2024-09-20T18:52:20.297 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T18:52:20.320 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T18:52:20.396 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744 branch: wip-guits-main-2024-09-20-1028 description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7912744' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-guits-main-2024-09-20-1028 ceph: conf: global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 workunit: branch: wip-guits-main-2024-09-20-1028 sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9945 sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-20-1028 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4D0UvkqY6Z0+7CAUsxS7cWGFyMBDAZnSNheGQ2/Ai4ZLJcn4M2TrsYBSpZj8TPig8/sDSdiqQXvatEIPzewsk= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNCE/ihSvlml7pt6yXNmAb91nxY0JPwq46l3QVnA6oH33nwmYzjkxh704bYztJhkJyt6mSaADnH8EuIygv50C1E= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_12:24:58 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T18:52:20.396 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa; will attempt to use it 2024-09-20T18:52:20.397 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_88deadb8082f4a33832e228ccefc59cedba158a5/qa/tasks 2024-09-20T18:52:20.397 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T18:52:20.398 INFO:teuthology.task.internal:Checking packages... 2024-09-20T18:52:20.419 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '88deadb8082f4a33832e228ccefc59cedba158a5' 2024-09-20T18:52:20.419 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T18:52:20.419 INFO:teuthology.packaging:ref: None 2024-09-20T18:52:20.419 INFO:teuthology.packaging:tag: None 2024-09-20T18:52:20.419 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-20-1028 2024-09-20T18:52:20.419 INFO:teuthology.packaging:sha1: 88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T18:52:20.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-guits-main-2024-09-20-1028 2024-09-20T18:52:20.562 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5099-g88deadb8-1jammy 2024-09-20T18:52:20.563 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T18:52:20.572 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T18:52:20.573 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T18:52:20.596 INFO:teuthology.task.internal:Saving configuration 2024-09-20T18:52:20.608 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T18:52:20.620 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T18:52:20.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 18:46:32.855245', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT3f5srMoSbfMZj58PKxIlFISGEdAL5KsxxhSC7k45H'} 2024-09-20T18:52:20.655 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 18:46:32.857243', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFr8MUHZLsLwvTgR7S5X/b3WJ63XcV6DEMnfuL76+q2n'} 2024-09-20T18:52:20.655 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T18:52:20.664 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T18:52:20.664 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-20T18:52:20.665 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T18:52:20.723 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f5eadc280>, signals=[15]) 2024-09-20T18:52:20.723 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T18:52:20.763 INFO:teuthology.task.internal:Opening connections... 2024-09-20T18:52:20.763 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T18:52:20.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:52:20.858 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-20T18:52:20.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:52:20.940 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T18:52:20.949 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-09-20T18:52:20.954 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-09-20T18:52:20.954 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-09-20T18:52:21.000 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-20T18:52:21.000 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2024-09-20T18:52:21.000 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="22.04" 2024-09-20T18:52:21.000 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-20T18:52:21.000 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=jammy 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-20T18:52:21.001 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=jammy 2024-09-20T18:52:21.002 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-09-20T18:52:21.024 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-20T18:52:21.029 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-20T18:52:21.029 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-20T18:52:21.081 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Ubuntu" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="22.04" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_CODENAME=jammy 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:ID=ubuntu 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE=debian 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-20T18:52:21.082 INFO:teuthology.orchestra.run.smithi162.stdout:UBUNTU_CODENAME=jammy 2024-09-20T18:52:21.083 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-20T18:52:21.103 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T18:52:21.117 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T18:52:21.163 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T18:52:21.164 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-09-20T18:52:21.166 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-20T18:52:21.171 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T18:52:21.350 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T18:52:21.350 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-09-20T18:52:21.353 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-20T18:52:21.368 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T18:52:21.450 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T18:52:21.502 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T18:52:21.502 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T18:52:21.502 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T18:52:21.502 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T18:52:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:52:21.507 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-09-20T18:52:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T18:52:21.509 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-20T18:52:21.554 INFO:teuthology.orchestra.run.smithi162.stdout:5.15.0-56-generic 2024-09-20T18:52:21.554 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.15.0-56-generic 2024-09-20T18:52:21.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-20T18:52:21.556 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-56-generic 2024-09-20T18:52:21.556 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.15.0-56-generic 2024-09-20T18:52:21.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-09-20T18:52:21.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-20T18:52:21.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-20T18:52:21.828 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-20T18:52:21.843 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T18:52:21.856 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-20T18:52:21.921 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-20T18:52:21.947 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T18:52:21.958 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-20T18:52:22.028 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-20T18:52:22.091 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-20T18:52:22.131 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-20T18:52:22.159 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-20T18:52:22.194 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-20T18:52:22.197 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-20T18:52:22.197 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-20T18:52:22.216 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-20T18:52:22.217 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-20T18:52:22.221 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-20T18:52:22.221 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-20T18:52:22.226 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-20T18:52:22.234 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-20T18:52:22.236 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-20T18:52:22.237 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-20T18:52:22.237 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-20T18:52:22.240 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-20T18:52:22.245 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-20T18:52:22.248 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-20T18:52:22.248 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-20T18:52:22.248 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-20T18:52:22.258 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-20T18:52:22.258 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-20T18:52:22.258 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-20T18:52:22.259 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-20T18:52:22.259 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-20T18:52:22.264 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-20T18:52:22.265 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-20T18:52:22.265 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-20T18:52:22.265 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-20T18:52:22.265 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-20T18:52:22.268 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-20T18:52:22.272 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-20T18:52:22.272 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-20T18:52:22.277 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-20T18:52:22.285 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-20T18:52:22.287 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-20T18:52:22.287 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-20T18:52:22.288 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-20T18:52:22.288 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-20T18:52:22.323 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-20T18:52:22.384 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-20T18:52:22.410 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-20T18:52:22.523 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-20T18:52:22.695 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-20T18:52:22.844 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-20T18:52:22.873 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-20T18:52:22.878 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-20T18:52:22.883 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-20T18:52:22.883 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-20T18:52:22.893 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-20T18:52:22.893 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-20T18:52:22.895 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-20T18:52:22.968 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-20T18:52:22.969 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-20T18:52:22.970 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-20T18:52:22.978 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-20T18:52:22.978 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-20T18:52:22.983 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-20T18:52:22.983 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-20T18:52:22.996 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-20T18:52:23.004 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-20T18:52:23.031 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-20T18:52:23.032 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-20T18:52:23.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-20T18:52:23.038 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-20T18:52:23.039 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-20T18:52:23.041 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-20T18:52:23.042 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-20T18:52:23.043 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-20T18:52:23.044 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-20T18:52:23.090 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-20T18:52:23.091 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-20T18:52:23.092 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-20T18:52:23.094 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-20T18:52:23.120 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-20T18:52:23.121 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-20T18:52:23.172 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-20T18:52:23.172 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-20T18:52:23.173 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-20T18:52:23.174 INFO:teuthology.orchestra.run.smithi162.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-20T18:52:23.174 INFO:teuthology.orchestra.run.smithi162.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-20T18:52:28.697 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 16.3 MB in 2s (7,163 kB/s) 2024-09-20T18:52:28.819 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 16.3 MB in 2s (7,226 kB/s) 2024-09-20T18:52:30.170 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T18:52:30.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-20T18:52:30.251 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T18:52:30.305 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T18:52:30.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-20T18:52:30.391 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T18:52:30.463 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-20T18:52:30.464 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-20T18:52:30.595 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-20T18:52:30.596 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-20T18:52:30.654 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T18:52:30.654 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T18:52:30.654 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T18:52:30.654 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T18:52:30.654 INFO:teuthology.orchestra.run.smithi162.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-20T18:52:30.655 INFO:teuthology.orchestra.run.smithi162.stdout: python2.7-minimal 2024-09-20T18:52:30.655 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T18:52:30.656 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-20T18:52:30.656 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-20T18:52:30.656 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-20T18:52:30.657 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-20T18:52:30.658 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-20T18:52:30.658 INFO:teuthology.orchestra.run.smithi162.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-20T18:52:30.658 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-20T18:52:30.658 INFO:teuthology.orchestra.run.smithi162.stdout: thermald 2024-09-20T18:52:30.711 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-20T18:52:30.712 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-20T18:52:30.712 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-20T18:52:30.712 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-20T18:52:30.713 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be upgraded: 2024-09-20T18:52:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-20T18:52:30.765 INFO:teuthology.orchestra.run.smithi162.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-20T18:52:30.765 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 113 MB of archives. 2024-09-20T18:52:30.765 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-20T18:52:30.765 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-20T18:52:30.777 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T18:52:30.777 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T18:52:30.777 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T18:52:30.777 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T18:52:30.777 INFO:teuthology.orchestra.run.smithi042.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-20T18:52:30.778 INFO:teuthology.orchestra.run.smithi042.stdout: python2.7-minimal 2024-09-20T18:52:30.778 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T18:52:30.779 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-09-20T18:52:30.779 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-20T18:52:30.780 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-20T18:52:30.780 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-20T18:52:30.781 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-09-20T18:52:30.781 INFO:teuthology.orchestra.run.smithi042.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-20T18:52:30.781 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-09-20T18:52:30.782 INFO:teuthology.orchestra.run.smithi042.stdout: thermald 2024-09-20T18:52:30.834 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-09-20T18:52:30.834 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-20T18:52:30.834 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-20T18:52:30.834 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-20T18:52:30.835 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2024-09-20T18:52:30.836 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-20T18:52:31.016 INFO:teuthology.orchestra.run.smithi042.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-20T18:52:31.017 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 113 MB of archives. 2024-09-20T18:52:31.017 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-20T18:52:31.017 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-20T18:52:31.201 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-20T18:52:31.311 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-20T18:52:32.143 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-20T18:52:32.143 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-20T18:52:32.144 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-20T18:52:32.264 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-20T18:52:32.286 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-20T18:52:32.423 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-20T18:52:32.668 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 113 MB in 2s (72.4 MB/s) 2024-09-20T18:52:32.783 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-20T18:52:32.839 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-20T18:52:34.794 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-20T18:52:34.798 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:34.886 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:35.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-20T18:52:35.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-20T18:52:35.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-20T18:52:35.397 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-20T18:52:35.480 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-20T18:52:35.896 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-09-20T18:52:36.096 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-20T18:52:36.961 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-20T18:52:36.982 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:37.074 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:37.689 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-20T18:52:37.709 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:37.809 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:38.171 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-20T18:52:38.175 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:38.360 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:40.761 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-20T18:52:40.783 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:41.373 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-20T18:52:41.394 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:41.431 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:44.521 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:44.648 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:44.956 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:45.078 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:45.371 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-20T18:52:45.393 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-20T18:52:45.438 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-20T18:52:48.126 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:48.250 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:48.609 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:48.736 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:49.056 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-20T18:52:49.079 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-20T18:52:49.114 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-20T18:52:51.355 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-20T18:52:51.391 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:51.430 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:54.025 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:54.125 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:54.449 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-20T18:52:54.571 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:54.698 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-20T18:52:54.817 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:55.009 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-20T18:52:55.047 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-20T18:52:55.110 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:56.124 INFO:teuthology.orchestra.run.smithi162.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:52:56.124 INFO:teuthology.orchestra.run.smithi162.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-20T18:52:56.124 INFO:teuthology.orchestra.run.smithi162.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:52:56.124 INFO:teuthology.orchestra.run.smithi162.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-20T18:52:56.294 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:58.163 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-20T18:52:58.288 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-20T18:52:58.414 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:58.945 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-20T18:52:59.070 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-20T18:52:59.429 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-20T18:52:59.533 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:59.659 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-20T18:52:59.780 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:59.800 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:52:59.866 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-20T18:52:59.866 INFO:teuthology.orchestra.run.smithi162.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:01.156 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:53:01.156 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-20T18:53:01.156 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:53:01.156 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:01.315 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:53:02.741 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-20T18:53:02.842 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-20T18:53:02.943 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:53:04.537 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-20T18:53:04.593 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-20T18:53:04.593 INFO:teuthology.orchestra.run.smithi042.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:14.281 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-20T18:53:14.281 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub' 2024-09-20T18:53:14.292 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-20T18:53:14.309 INFO:teuthology.orchestra.run.smithi162.stdout:Generating grub configuration file ... 2024-09-20T18:53:14.568 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:53:14.593 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:14.855 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:53:14.860 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-20T18:53:14.934 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-20T18:53:14.939 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-20T18:53:15.102 INFO:teuthology.orchestra.run.smithi162.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-20T18:53:15.102 INFO:teuthology.orchestra.run.smithi162.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-20T18:53:15.103 INFO:teuthology.orchestra.run.smithi162.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-20T18:53:15.147 INFO:teuthology.orchestra.run.smithi162.stdout:done 2024-09-20T18:53:15.650 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 21 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.15.0.122.122 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-20T18:53:15.674 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-20T18:53:15.675 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-20T18:53:15.675 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-20T18:53:15.675 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-20T18:53:15.675 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-20T18:53:15.675 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-20T18:53:15.675 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T18:53:15.675 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T18:53:15.675 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T18:53:15.675 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T18:53:15.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-20T18:53:15.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-20T18:53:16.141 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T18:53:16.141 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-20T18:53:16.205 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-20T18:53:16.287 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-20T18:53:17.596 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T18:53:17.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-20T18:53:17.668 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T18:53:17.859 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-20T18:53:17.859 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T18:53:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-20T18:53:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-20T18:53:18.026 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T18:53:18.064 INFO:teuthology.orchestra.run.smithi162.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-20T18:53:18.066 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 21 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.15.0.122.122 2024-09-20T18:53:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-20T18:53:18.125 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-20T18:53:18.125 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-20T18:53:18.125 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-20T18:53:18.125 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-20T18:53:18.125 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-20T18:53:18.125 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-09-20T18:53:18.169 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.jQv69hz1MB 2024-09-20T18:53:18.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jQv69hz1MB 2024-09-20T18:53:18.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.jQv69hz1MB 2024-09-20T18:53:18.355 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.jQv69hz1MB is 10KB 2024-09-20T18:53:18.368 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.jQv69hz1MB 2024-09-20T18:53:18.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-20T18:53:18.429 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T18:53:18.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-20T18:53:18.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-20T18:53:18.495 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-20T18:53:18.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo update-grub 2024-09-20T18:53:19.069 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-20T18:53:19.069 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub' 2024-09-20T18:53:19.079 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-20T18:53:19.107 INFO:teuthology.orchestra.run.smithi042.stdout:Generating grub configuration file ... 2024-09-20T18:53:19.398 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:53:19.418 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:19.716 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:53:19.722 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-20T18:53:19.735 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub' 2024-09-20T18:53:19.736 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-20T18:53:19.739 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-09-20T18:53:19.794 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-20T18:53:19.799 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-20T18:53:19.903 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:53:19.908 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:20.103 INFO:teuthology.orchestra.run.smithi042.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-20T18:53:20.103 INFO:teuthology.orchestra.run.smithi042.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-20T18:53:20.103 INFO:teuthology.orchestra.run.smithi042.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-20T18:53:20.125 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:53:20.130 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-20T18:53:20.152 INFO:teuthology.orchestra.run.smithi042.stdout:done 2024-09-20T18:53:20.167 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-20T18:53:20.171 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-20T18:53:20.290 INFO:teuthology.orchestra.run.smithi162.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-20T18:53:20.290 INFO:teuthology.orchestra.run.smithi162.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-20T18:53:20.290 INFO:teuthology.orchestra.run.smithi162.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-20T18:53:20.304 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-09-20T18:53:20.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-09-20T18:53:20.695 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-09-20T18:53:20.726 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-09-20T18:53:20.726 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-09-20T18:53:20.726 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.122.122 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-20T18:53:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-09-20T18:53:20.728 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-09-20T18:53:20.728 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-09-20T18:53:20.728 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-09-20T18:53:20.728 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-20T18:53:20.728 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T18:53:20.728 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T18:53:20.728 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T18:53:20.728 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T18:53:20.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-09-20T18:53:20.893 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-20T18:53:21.026 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-20T18:53:21.146 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T18:53:21.228 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-20T18:53:21.309 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-20T18:53:22.616 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T18:53:22.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-20T18:53:22.702 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T18:53:22.931 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-20T18:53:22.931 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-20T18:53:23.130 INFO:teuthology.orchestra.run.smithi042.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-20T18:53:23.130 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-20T18:53:23.131 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T18:53:23.176 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-20T18:53:23.178 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-09-20T18:53:23.192 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-09-20T18:53:23.192 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-09-20T18:53:23.192 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-09-20T18:53:23.192 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-09-20T18:53:23.192 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.122.122 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-09-20T18:53:23.193 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-09-20T18:53:23.193 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-09-20T18:53:23.240 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.k1PX4nQg8N 2024-09-20T18:53:23.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub/grub.cfg /tmp/tmp.k1PX4nQg8N 2024-09-20T18:53:23.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.k1PX4nQg8N 2024-09-20T18:53:23.409 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.k1PX4nQg8N is 10KB 2024-09-20T18:53:23.421 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.k1PX4nQg8N 2024-09-20T18:53:23.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-20T18:53:23.480 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T18:53:23.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-20T18:53:23.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-20T18:53:23.547 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-20T18:53:23.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo update-grub 2024-09-20T18:53:24.780 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub' 2024-09-20T18:53:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-20T18:53:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-09-20T18:53:24.962 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-20T18:53:24.967 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-20T18:53:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-20T18:53:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-20T18:53:25.300 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-20T18:53:25.306 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-20T18:53:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-20T18:53:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-20T18:53:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-20T18:53:25.478 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-09-20T18:53:25.480 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-09-20T18:53:50.339 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T18:53:50.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T18:53:50.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:53:55.488 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T18:53:55.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T18:53:55.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:54:08.879 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-20T18:54:13.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2024-09-20T18:54:17.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T18:54:17.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:54:20.943 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-20T18:54:23.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T18:54:23.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:54:32.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T18:54:32.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:55:23.055 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T18:55:32.958 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T18:55:35.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T18:55:35.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:55:35.329 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-09-20T18:55:35.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T18:55:35.895 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-20T18:55:35.895 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-09-20T18:55:35.943 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-122-generic 2024-09-20T18:55:35.944 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-20T18:55:35.944 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T18:55:35.944 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-20T18:55:36.945 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T18:55:36.945 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T18:55:37.011 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-09-20T18:55:37.045 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:47.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T18:55:47.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T18:55:48.545 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-20T18:55:49.100 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T18:55:49.101 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-20T18:55:49.101 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-20T18:55:49.148 INFO:teuthology.orchestra.run.smithi162.stdout:5.15.0-122-generic 2024-09-20T18:55:49.148 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-20T18:55:49.148 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T18:55:49.148 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-20T18:55:50.149 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T18:55:50.150 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T18:55:50.207 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-20T18:55:50.230 DEBUG:teuthology.parallel:result is None 2024-09-20T18:55:50.230 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T18:55:50.247 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T18:55:50.247 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T18:55:50.250 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T18:55:50.255 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T18:55:50.267 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T18:55:50.282 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T18:55:50.282 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T18:55:50.297 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T18:55:50.317 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T18:55:50.329 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T18:55:50.329 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T18:55:50.353 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T18:55:50.373 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:55:50.375 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:55:50.381 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:55:50.384 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T18:55:50.385 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T18:55:50.396 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T18:55:50.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T18:55:50.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T18:55:50.442 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T18:55:50.452 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T18:55:50.452 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T18:55:50.484 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T18:55:50.490 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:55:50.536 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:55:50.584 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T18:55:50.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T18:55:50.641 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T18:55:50.647 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T18:55:50.692 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T18:55:50.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T18:55:50.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-09-20T18:55:50.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-20T18:55:50.835 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T18:55:50.846 INFO:teuthology.task.internal:Starting timer... 2024-09-20T18:55:50.846 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T18:55:50.919 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T18:55:50.978 DEBUG:teuthology.task.selinux:Excluding smithi042: OS 'ubuntu' does not support SELinux 2024-09-20T18:55:50.978 DEBUG:teuthology.task.selinux:Excluding smithi162: OS 'ubuntu' does not support SELinux 2024-09-20T18:55:50.978 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T18:55:50.978 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-20T18:55:50.978 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T18:55:50.978 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T18:55:51.026 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T18:55:51.037 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T18:55:51.037 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T19:03:26.573 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-09-20T19:03:26.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T19:03:26.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:03:26.656 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-09-20T19:03:26.735 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-20T19:03:26.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T19:03:26.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:03:26.816 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-20T19:03:26.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-20T19:03:26.900 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T19:03:26.911 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T19:03:26.911 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:03:26.911 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:03:26.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:03:26.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Command line: ntpd -gq 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: ---------------------------------------------------- 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: ntp-4 is maintained by Network Time Foundation, 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: corporation. Support and training for ntp-4 are 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: available at https://www.nwtime.org/support 2024-09-20T19:03:26.942 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: ---------------------------------------------------- 2024-09-20T19:03:26.943 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: proto: precision = 0.062 usec (-24) 2024-09-20T19:03:26.943 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: basedate set to 2022-02-04 2024-09-20T19:03:26.943 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: gps base set to 2022-02-06 (week 2196) 2024-09-20T19:03:26.943 INFO:teuthology.orchestra.run.smithi042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-20T19:03:26.944 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-20T19:03:26.944 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: restrict ::: KOD does nothing without LIMITED. 2024-09-20T19:03:26.944 INFO:teuthology.orchestra.run.smithi042.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen and drop on 0 v6wildcard [::]:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen normally on 2 lo 127.0.0.1:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen normally on 3 ens1f0 172.21.15.42:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen normally on 4 lo [::1]:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1440%4]:123 2024-09-20T19:03:26.945 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:26 ntpd[18353]: Listening on routing socket on fd #22 for interface updates 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Command line: ntpd -gq 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: ---------------------------------------------------- 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: ntp-4 is maintained by Network Time Foundation, 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-20T19:03:26.969 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: corporation. Support and training for ntp-4 are 2024-09-20T19:03:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: available at https://www.nwtime.org/support 2024-09-20T19:03:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: ---------------------------------------------------- 2024-09-20T19:03:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: proto: precision = 0.063 usec (-24) 2024-09-20T19:03:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: basedate set to 2022-02-04 2024-09-20T19:03:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: gps base set to 2022-02-06 (week 2196) 2024-09-20T19:03:26.971 INFO:teuthology.orchestra.run.smithi162.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-20T19:03:26.971 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-20T19:03:26.971 INFO:teuthology.orchestra.run.smithi162.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: restrict ::: KOD does nothing without LIMITED. 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen and drop on 0 v6wildcard [::]:123 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen normally on 2 lo 127.0.0.1:123 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen normally on 3 enp3s0f1 172.21.15.162:123 2024-09-20T19:03:26.972 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen normally on 4 lo [::1]:123 2024-09-20T19:03:26.973 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f41%5]:123 2024-09-20T19:03:26.973 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:26 ntpd[18490]: Listening on routing socket on fd #22 for interface updates 2024-09-20T19:03:27.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:27.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:27 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:27.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:27 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:28.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:28.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:28.972 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:28.972 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:28.972 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:28.972 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:28 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:29.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:29 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:29 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:30 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:30 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:31 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:31.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:31.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:31 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:32.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:32.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:32.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:32.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:32.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:32.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:32 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:32.969 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:32 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stdout:20 Sep 19:03:33 ntpd[18353]: ntpd: time slew +0.001842 s 2024-09-20T19:03:33.943 INFO:teuthology.orchestra.run.smithi042.stdout:ntpd: time slew +0.001842s 2024-09-20T19:03:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-20T19:03:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:20 Sep 19:03:33 ntpd[18353]: can't open /var/log/ntpstats/loopstats.20240920: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240920: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-20T19:03:33.970 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240920: Permission denied 2024-09-20T19:03:33.971 INFO:teuthology.orchestra.run.smithi162.stdout:20 Sep 19:03:33 ntpd[18490]: ntpd: time slew -0.001231 s 2024-09-20T19:03:33.971 INFO:teuthology.orchestra.run.smithi162.stdout:ntpd: time slew -0.001231s 2024-09-20T19:03:33.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-20T19:03:33.971 INFO:teuthology.orchestra.run.smithi162.stderr:20 Sep 19:03:33 ntpd[18490]: can't open /var/log/ntpstats/loopstats.20240920: Permission denied 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:33.992 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-20T19:03:34.025 INFO:teuthology.run_tasks:Running task install... 2024-09-20T19:03:34.033 DEBUG:teuthology.task.install:project ceph 2024-09-20T19:03:34.033 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}} 2024-09-20T19:03:34.033 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'} 2024-09-20T19:03:34.033 INFO:teuthology.task.install:Using flavor: default 2024-09-20T19:03:34.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-20T19:03:34.039 INFO:teuthology.task.install:extra packages: [] 2024-09-20T19:03:34.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-key list | grep Ceph 2024-09-20T19:03:34.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-key list | grep Ceph 2024-09-20T19:03:34.098 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-20T19:03:34.119 INFO:teuthology.orchestra.run.smithi162.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-20T19:03:34.128 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-20T19:03:34.128 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph.com (release key) 2024-09-20T19:03:34.129 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-20T19:03:34.129 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:03:34.145 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-20T19:03:34.145 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph.com (release key) 2024-09-20T19:03:34.145 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-20T19:03:34.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:03:34.266 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default/ 2024-09-20T19:03:34.266 INFO:teuthology.task.install.deb:Package version is 19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:03:34.286 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default/ 2024-09-20T19:03:34.286 INFO:teuthology.task.install.deb:Package version is 19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:03:34.378 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:03:34.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-20T19:03:34.393 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:03:34.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-20T19:03:34.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-20T19:03:34.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-20T19:03:34.566 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T19:03:34.566 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-20T19:03:34.575 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-20T19:03:34.578 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-20T19:03:34.589 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-20T19:03:34.592 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-20T19:03:34.606 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-20T19:03:34.621 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy InRelease 2024-09-20T19:03:34.632 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy InRelease 2024-09-20T19:03:34.642 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-20T19:03:34.653 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-20T19:03:34.664 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-20T19:03:34.675 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-20T19:03:34.688 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-20T19:03:34.696 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-20T19:03:34.701 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-20T19:03:34.715 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-20T19:03:34.722 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-20T19:03:34.915 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 29.3 kB in 0s (90.1 kB/s) 2024-09-20T19:03:34.935 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 29.3 kB in 0s (82.1 kB/s) 2024-09-20T19:03:35.961 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T19:03:35.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5099-g88deadb8-1jammy cephadm=19.3.0-5099-g88deadb8-1jammy ceph-mds=19.3.0-5099-g88deadb8-1jammy ceph-mgr=19.3.0-5099-g88deadb8-1jammy ceph-common=19.3.0-5099-g88deadb8-1jammy ceph-fuse=19.3.0-5099-g88deadb8-1jammy ceph-test=19.3.0-5099-g88deadb8-1jammy ceph-volume=19.3.0-5099-g88deadb8-1jammy radosgw=19.3.0-5099-g88deadb8-1jammy python3-rados=19.3.0-5099-g88deadb8-1jammy python3-rgw=19.3.0-5099-g88deadb8-1jammy python3-cephfs=19.3.0-5099-g88deadb8-1jammy python3-rbd=19.3.0-5099-g88deadb8-1jammy libcephfs2=19.3.0-5099-g88deadb8-1jammy libcephfs-dev=19.3.0-5099-g88deadb8-1jammy librados2=19.3.0-5099-g88deadb8-1jammy librbd1=19.3.0-5099-g88deadb8-1jammy rbd-fuse=19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:03:36.005 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T19:03:36.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5099-g88deadb8-1jammy cephadm=19.3.0-5099-g88deadb8-1jammy ceph-mds=19.3.0-5099-g88deadb8-1jammy ceph-mgr=19.3.0-5099-g88deadb8-1jammy ceph-common=19.3.0-5099-g88deadb8-1jammy ceph-fuse=19.3.0-5099-g88deadb8-1jammy ceph-test=19.3.0-5099-g88deadb8-1jammy ceph-volume=19.3.0-5099-g88deadb8-1jammy radosgw=19.3.0-5099-g88deadb8-1jammy python3-rados=19.3.0-5099-g88deadb8-1jammy python3-rgw=19.3.0-5099-g88deadb8-1jammy python3-cephfs=19.3.0-5099-g88deadb8-1jammy python3-rbd=19.3.0-5099-g88deadb8-1jammy libcephfs2=19.3.0-5099-g88deadb8-1jammy libcephfs-dev=19.3.0-5099-g88deadb8-1jammy librados2=19.3.0-5099-g88deadb8-1jammy librbd1=19.3.0-5099-g88deadb8-1jammy rbd-fuse=19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:03:36.030 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-20T19:03:36.079 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-20T19:03:36.217 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-20T19:03:36.218 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-20T19:03:36.270 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-20T19:03:36.270 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-20T19:03:36.405 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T19:03:36.405 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T19:03:36.405 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T19:03:36.405 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T19:03:36.406 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-20T19:03:36.406 INFO:teuthology.orchestra.run.smithi162.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-20T19:03:36.406 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-20T19:03:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-20T19:03:36.408 INFO:teuthology.orchestra.run.smithi162.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-20T19:03:36.408 INFO:teuthology.orchestra.run.smithi162.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-20T19:03:36.408 INFO:teuthology.orchestra.run.smithi162.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-20T19:03:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet zip 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-20T19:03:36.410 INFO:teuthology.orchestra.run.smithi162.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-20T19:03:36.411 INFO:teuthology.orchestra.run.smithi162.stdout: btrfs-tools 2024-09-20T19:03:36.462 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-20T19:03:36.463 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-20T19:03:36.463 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-20T19:03:36.463 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-20T19:03:36.463 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-20T19:03:36.463 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-20T19:03:36.464 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-20T19:03:36.464 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-09-20T19:03:36.464 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-20T19:03:36.464 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-20T19:03:36.465 INFO:teuthology.orchestra.run.smithi042.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-20T19:03:36.465 INFO:teuthology.orchestra.run.smithi042.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-20T19:03:36.465 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-20T19:03:36.465 INFO:teuthology.orchestra.run.smithi042.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-20T19:03:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet zip 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-20T19:03:36.467 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-20T19:03:36.468 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-20T19:03:36.468 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-20T19:03:36.468 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-09-20T19:03:36.468 INFO:teuthology.orchestra.run.smithi042.stdout: btrfs-tools 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-20T19:03:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-20T19:03:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-20T19:03:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-20T19:03:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-20T19:03:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-20T19:03:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-20T19:03:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-20T19:03:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-20T19:03:36.573 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-09-20T19:03:36.573 INFO:teuthology.orchestra.run.smithi042.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-20T19:03:36.573 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-20T19:03:36.573 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-20T19:03:36.573 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-20T19:03:36.574 INFO:teuthology.orchestra.run.smithi042.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-20T19:03:36.574 INFO:teuthology.orchestra.run.smithi042.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-20T19:03:36.575 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-20T19:03:36.576 INFO:teuthology.orchestra.run.smithi042.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-20T19:03:36.634 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-20T19:03:36.635 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 244 MB of archives. 2024-09-20T19:03:36.635 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-20T19:03:36.635 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-20T19:03:36.658 INFO:teuthology.orchestra.run.smithi162.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-20T19:03:36.658 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 244 MB of archives. 2024-09-20T19:03:36.658 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-20T19:03:36.658 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5099-g88deadb8-1jammy [3,634 kB] 2024-09-20T19:03:36.703 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-20T19:03:36.715 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5099-g88deadb8-1jammy [3,634 kB] 2024-09-20T19:03:36.722 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-20T19:03:36.725 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-20T19:03:36.742 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-20T19:03:36.801 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-20T19:03:36.806 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-20T19:03:36.819 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-20T19:03:36.823 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-20T19:03:36.824 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-20T19:03:36.824 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-20T19:03:36.825 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-20T19:03:36.834 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-20T19:03:36.836 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-20T19:03:36.839 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-20T19:03:36.840 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-20T19:03:36.840 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-20T19:03:36.857 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-20T19:03:36.859 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-20T19:03:36.861 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-20T19:03:36.861 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-20T19:03:36.862 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-20T19:03:36.863 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-20T19:03:36.874 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-20T19:03:36.875 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-20T19:03:36.875 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-20T19:03:36.875 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-20T19:03:36.891 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-20T19:03:36.892 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-20T19:03:36.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-20T19:03:36.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-20T19:03:36.896 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5099-g88deadb8-1jammy [3,335 kB] 2024-09-20T19:03:36.896 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-20T19:03:36.896 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-20T19:03:36.908 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-20T19:03:36.909 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-20T19:03:36.909 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-20T19:03:36.910 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-20T19:03:36.925 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-20T19:03:36.926 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-20T19:03:36.929 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5099-g88deadb8-1jammy [3,335 kB] 2024-09-20T19:03:36.931 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-20T19:03:36.931 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-20T19:03:36.932 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-20T19:03:36.932 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-20T19:03:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-20T19:03:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-20T19:03:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-20T19:03:36.946 INFO:teuthology.orchestra.run.smithi042.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-20T19:03:36.959 INFO:teuthology.orchestra.run.smithi042.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-20T19:03:36.962 INFO:teuthology.orchestra.run.smithi042.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-20T19:03:36.962 INFO:teuthology.orchestra.run.smithi042.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-20T19:03:36.963 INFO:teuthology.orchestra.run.smithi042.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-20T19:03:36.966 INFO:teuthology.orchestra.run.smithi042.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-20T19:03:36.990 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5099-g88deadb8-1jammy [722 kB] 2024-09-20T19:03:37.014 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5099-g88deadb8-1jammy [356 kB] 2024-09-20T19:03:37.018 INFO:teuthology.orchestra.run.smithi042.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-20T19:03:37.019 INFO:teuthology.orchestra.run.smithi042.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-20T19:03:37.020 INFO:teuthology.orchestra.run.smithi042.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-20T19:03:37.021 INFO:teuthology.orchestra.run.smithi042.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-20T19:03:37.022 INFO:teuthology.orchestra.run.smithi042.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-20T19:03:37.022 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5099-g88deadb8-1jammy [32.8 kB] 2024-09-20T19:03:37.023 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5099-g88deadb8-1jammy [180 kB] 2024-09-20T19:03:37.027 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5099-g88deadb8-1jammy [72.9 kB] 2024-09-20T19:03:37.029 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5099-g88deadb8-1jammy [340 kB] 2024-09-20T19:03:37.032 INFO:teuthology.orchestra.run.smithi042.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5099-g88deadb8-1jammy [722 kB] 2024-09-20T19:03:37.039 INFO:teuthology.orchestra.run.smithi042.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-20T19:03:37.041 INFO:teuthology.orchestra.run.smithi042.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-20T19:03:37.041 INFO:teuthology.orchestra.run.smithi042.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-20T19:03:37.044 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5099-g88deadb8-1jammy [7,965 kB] 2024-09-20T19:03:37.056 INFO:teuthology.orchestra.run.smithi042.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-20T19:03:37.056 INFO:teuthology.orchestra.run.smithi042.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-20T19:03:37.056 INFO:teuthology.orchestra.run.smithi042.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-20T19:03:37.057 INFO:teuthology.orchestra.run.smithi042.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-20T19:03:37.057 INFO:teuthology.orchestra.run.smithi042.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-20T19:03:37.058 INFO:teuthology.orchestra.run.smithi042.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-20T19:03:37.058 INFO:teuthology.orchestra.run.smithi042.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5099-g88deadb8-1jammy [356 kB] 2024-09-20T19:03:37.065 INFO:teuthology.orchestra.run.smithi042.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5099-g88deadb8-1jammy [32.8 kB] 2024-09-20T19:03:37.067 INFO:teuthology.orchestra.run.smithi042.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5099-g88deadb8-1jammy [180 kB] 2024-09-20T19:03:37.070 INFO:teuthology.orchestra.run.smithi042.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5099-g88deadb8-1jammy [72.9 kB] 2024-09-20T19:03:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-20T19:03:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-20T19:03:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5099-g88deadb8-1jammy [340 kB] 2024-09-20T19:03:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-20T19:03:37.088 INFO:teuthology.orchestra.run.smithi042.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-20T19:03:37.090 INFO:teuthology.orchestra.run.smithi042.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-20T19:03:37.090 INFO:teuthology.orchestra.run.smithi042.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-20T19:03:37.092 INFO:teuthology.orchestra.run.smithi042.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5099-g88deadb8-1jammy [7,965 kB] 2024-09-20T19:03:37.093 INFO:teuthology.orchestra.run.smithi042.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-20T19:03:37.096 INFO:teuthology.orchestra.run.smithi042.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-20T19:03:37.096 INFO:teuthology.orchestra.run.smithi042.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-20T19:03:37.097 INFO:teuthology.orchestra.run.smithi042.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-20T19:03:37.105 INFO:teuthology.orchestra.run.smithi042.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-20T19:03:37.106 INFO:teuthology.orchestra.run.smithi042.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-20T19:03:37.109 INFO:teuthology.orchestra.run.smithi042.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-20T19:03:37.123 INFO:teuthology.orchestra.run.smithi042.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-20T19:03:37.123 INFO:teuthology.orchestra.run.smithi042.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-20T19:03:37.123 INFO:teuthology.orchestra.run.smithi042.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-20T19:03:37.125 INFO:teuthology.orchestra.run.smithi042.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-20T19:03:37.125 INFO:teuthology.orchestra.run.smithi042.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-20T19:03:37.126 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-20T19:03:37.131 INFO:teuthology.orchestra.run.smithi042.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-20T19:03:37.132 INFO:teuthology.orchestra.run.smithi042.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-20T19:03:37.138 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-20T19:03:37.140 INFO:teuthology.orchestra.run.smithi042.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-20T19:03:37.142 INFO:teuthology.orchestra.run.smithi042.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-20T19:03:37.142 INFO:teuthology.orchestra.run.smithi042.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-20T19:03:37.220 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-20T19:03:37.300 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5099-g88deadb8-1jammy [112 kB] 2024-09-20T19:03:37.303 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5099-g88deadb8-1jammy [473 kB] 2024-09-20T19:03:37.312 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5099-g88deadb8-1jammy [28.3 MB] 2024-09-20T19:03:37.366 INFO:teuthology.orchestra.run.smithi042.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5099-g88deadb8-1jammy [112 kB] 2024-09-20T19:03:37.369 INFO:teuthology.orchestra.run.smithi042.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5099-g88deadb8-1jammy [473 kB] 2024-09-20T19:03:37.380 INFO:teuthology.orchestra.run.smithi042.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5099-g88deadb8-1jammy [28.3 MB] 2024-09-20T19:03:37.449 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-20T19:03:37.460 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-20T19:03:37.490 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-20T19:03:37.499 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-20T19:03:37.501 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-20T19:03:37.502 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-20T19:03:37.503 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-20T19:03:37.521 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-20T19:03:37.525 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-20T19:03:37.532 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-20T19:03:37.611 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-20T19:03:37.612 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-20T19:03:37.616 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-20T19:03:37.620 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-20T19:03:37.624 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-20T19:03:37.625 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-20T19:03:37.626 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-20T19:03:37.628 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-20T19:03:37.628 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-20T19:03:37.695 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-20T19:03:37.712 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-20T19:03:37.712 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-20T19:03:37.713 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-20T19:03:37.713 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-20T19:03:37.719 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-20T19:03:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-20T19:03:37.777 INFO:teuthology.orchestra.run.smithi162.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-20T19:03:37.778 INFO:teuthology.orchestra.run.smithi162.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-20T19:03:37.780 INFO:teuthology.orchestra.run.smithi162.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-20T19:03:37.858 INFO:teuthology.orchestra.run.smithi162.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-20T19:03:37.859 INFO:teuthology.orchestra.run.smithi162.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-20T19:03:37.861 INFO:teuthology.orchestra.run.smithi162.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-20T19:03:37.862 INFO:teuthology.orchestra.run.smithi162.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-20T19:03:37.863 INFO:teuthology.orchestra.run.smithi162.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-20T19:03:37.874 INFO:teuthology.orchestra.run.smithi162.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-20T19:03:37.940 INFO:teuthology.orchestra.run.smithi162.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-20T19:03:37.941 INFO:teuthology.orchestra.run.smithi162.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-20T19:03:37.941 INFO:teuthology.orchestra.run.smithi162.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-20T19:03:37.944 INFO:teuthology.orchestra.run.smithi162.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-20T19:03:38.021 INFO:teuthology.orchestra.run.smithi162.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-20T19:03:38.023 INFO:teuthology.orchestra.run.smithi162.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-20T19:03:38.025 INFO:teuthology.orchestra.run.smithi162.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-20T19:03:38.027 INFO:teuthology.orchestra.run.smithi162.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-20T19:03:38.031 INFO:teuthology.orchestra.run.smithi162.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-20T19:03:38.032 INFO:teuthology.orchestra.run.smithi162.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-20T19:03:38.103 INFO:teuthology.orchestra.run.smithi162.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-20T19:03:38.107 INFO:teuthology.orchestra.run.smithi162.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-20T19:03:38.331 INFO:teuthology.orchestra.run.smithi162.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-20T19:03:38.403 INFO:teuthology.orchestra.run.smithi162.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-20T19:03:38.405 INFO:teuthology.orchestra.run.smithi162.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5099-g88deadb8-1jammy [5,212 kB] 2024-09-20T19:03:38.406 INFO:teuthology.orchestra.run.smithi162.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-20T19:03:38.407 INFO:teuthology.orchestra.run.smithi162.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-20T19:03:38.409 INFO:teuthology.orchestra.run.smithi162.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-20T19:03:38.434 INFO:teuthology.orchestra.run.smithi042.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5099-g88deadb8-1jammy [5,212 kB] 2024-09-20T19:03:38.454 INFO:teuthology.orchestra.run.smithi162.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-20T19:03:38.458 INFO:teuthology.orchestra.run.smithi162.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-20T19:03:38.458 INFO:teuthology.orchestra.run.smithi162.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-20T19:03:38.484 INFO:teuthology.orchestra.run.smithi162.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5099-g88deadb8-1jammy [252 kB] 2024-09-20T19:03:38.487 INFO:teuthology.orchestra.run.smithi162.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5099-g88deadb8-1jammy [124 kB] 2024-09-20T19:03:38.489 INFO:teuthology.orchestra.run.smithi162.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5099-g88deadb8-1jammy [1,561 kB] 2024-09-20T19:03:38.497 INFO:teuthology.orchestra.run.smithi162.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-20T19:03:38.497 INFO:teuthology.orchestra.run.smithi162.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-20T19:03:38.519 INFO:teuthology.orchestra.run.smithi162.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5099-g88deadb8-1jammy [6,310 kB] 2024-09-20T19:03:38.539 INFO:teuthology.orchestra.run.smithi042.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5099-g88deadb8-1jammy [252 kB] 2024-09-20T19:03:38.543 INFO:teuthology.orchestra.run.smithi042.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5099-g88deadb8-1jammy [124 kB] 2024-09-20T19:03:38.545 INFO:teuthology.orchestra.run.smithi042.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5099-g88deadb8-1jammy [1,561 kB] 2024-09-20T19:03:38.569 INFO:teuthology.orchestra.run.smithi162.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-20T19:03:38.570 INFO:teuthology.orchestra.run.smithi162.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-20T19:03:38.571 INFO:teuthology.orchestra.run.smithi162.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-20T19:03:38.572 INFO:teuthology.orchestra.run.smithi162.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-20T19:03:38.573 INFO:teuthology.orchestra.run.smithi162.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-20T19:03:38.574 INFO:teuthology.orchestra.run.smithi162.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-20T19:03:38.582 INFO:teuthology.orchestra.run.smithi162.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-20T19:03:38.617 INFO:teuthology.orchestra.run.smithi042.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5099-g88deadb8-1jammy [6,310 kB] 2024-09-20T19:03:38.650 INFO:teuthology.orchestra.run.smithi162.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-20T19:03:38.654 INFO:teuthology.orchestra.run.smithi162.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-20T19:03:38.655 INFO:teuthology.orchestra.run.smithi162.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-20T19:03:38.719 INFO:teuthology.orchestra.run.smithi162.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5099-g88deadb8-1jammy [23.1 MB] 2024-09-20T19:03:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-20T19:03:38.738 INFO:teuthology.orchestra.run.smithi162.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-20T19:03:38.740 INFO:teuthology.orchestra.run.smithi162.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-20T19:03:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-20T19:03:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-20T19:03:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-20T19:03:38.761 INFO:teuthology.orchestra.run.smithi162.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-20T19:03:38.814 INFO:teuthology.orchestra.run.smithi162.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-20T19:03:38.815 INFO:teuthology.orchestra.run.smithi162.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-20T19:03:38.816 INFO:teuthology.orchestra.run.smithi162.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-20T19:03:38.838 INFO:teuthology.orchestra.run.smithi042.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5099-g88deadb8-1jammy [23.1 MB] 2024-09-20T19:03:38.895 INFO:teuthology.orchestra.run.smithi162.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-20T19:03:38.897 INFO:teuthology.orchestra.run.smithi162.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-20T19:03:38.914 INFO:teuthology.orchestra.run.smithi162.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-20T19:03:38.915 INFO:teuthology.orchestra.run.smithi162.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-20T19:03:38.915 INFO:teuthology.orchestra.run.smithi162.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-20T19:03:38.920 INFO:teuthology.orchestra.run.smithi162.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-20T19:03:38.922 INFO:teuthology.orchestra.run.smithi162.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-20T19:03:39.496 INFO:teuthology.orchestra.run.smithi162.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5099-g88deadb8-1jammy [14.0 kB] 2024-09-20T19:03:39.497 INFO:teuthology.orchestra.run.smithi162.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5099-g88deadb8-1jammy [880 kB] 2024-09-20T19:03:39.512 INFO:teuthology.orchestra.run.smithi162.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5099-g88deadb8-1jammy [2,453 kB] 2024-09-20T19:03:39.553 INFO:teuthology.orchestra.run.smithi162.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5099-g88deadb8-1jammy [813 kB] 2024-09-20T19:03:39.571 INFO:teuthology.orchestra.run.smithi162.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5099-g88deadb8-1jammy [160 kB] 2024-09-20T19:03:39.573 INFO:teuthology.orchestra.run.smithi162.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5099-g88deadb8-1jammy [61.5 MB] 2024-09-20T19:03:39.726 INFO:teuthology.orchestra.run.smithi042.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5099-g88deadb8-1jammy [14.0 kB] 2024-09-20T19:03:39.727 INFO:teuthology.orchestra.run.smithi042.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5099-g88deadb8-1jammy [880 kB] 2024-09-20T19:03:39.745 INFO:teuthology.orchestra.run.smithi042.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5099-g88deadb8-1jammy [2,453 kB] 2024-09-20T19:03:39.788 INFO:teuthology.orchestra.run.smithi042.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5099-g88deadb8-1jammy [813 kB] 2024-09-20T19:03:39.799 INFO:teuthology.orchestra.run.smithi042.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5099-g88deadb8-1jammy [160 kB] 2024-09-20T19:03:39.801 INFO:teuthology.orchestra.run.smithi042.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5099-g88deadb8-1jammy [61.5 MB] 2024-09-20T19:03:41.762 INFO:teuthology.orchestra.run.smithi162.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5099-g88deadb8-1jammy [8,625 kB] 2024-09-20T19:03:41.898 INFO:teuthology.orchestra.run.smithi162.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5099-g88deadb8-1jammy [14.2 kB] 2024-09-20T19:03:41.898 INFO:teuthology.orchestra.run.smithi162.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5099-g88deadb8-1jammy [48.5 MB] 2024-09-20T19:03:42.037 INFO:teuthology.orchestra.run.smithi042.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5099-g88deadb8-1jammy [8,625 kB] 2024-09-20T19:03:42.241 INFO:teuthology.orchestra.run.smithi042.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5099-g88deadb8-1jammy [14.2 kB] 2024-09-20T19:03:42.242 INFO:teuthology.orchestra.run.smithi042.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5099-g88deadb8-1jammy [48.5 MB] 2024-09-20T19:03:43.472 INFO:teuthology.orchestra.run.smithi162.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5099-g88deadb8-1jammy [131 kB] 2024-09-20T19:03:43.475 INFO:teuthology.orchestra.run.smithi162.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5099-g88deadb8-1jammy [39.0 kB] 2024-09-20T19:03:43.476 INFO:teuthology.orchestra.run.smithi162.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5099-g88deadb8-1jammy [15.8 MB] 2024-09-20T19:03:43.980 INFO:teuthology.orchestra.run.smithi162.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5099-g88deadb8-1jammy [91.2 kB] 2024-09-20T19:03:44.009 INFO:teuthology.orchestra.run.smithi042.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5099-g88deadb8-1jammy [131 kB] 2024-09-20T19:03:44.011 INFO:teuthology.orchestra.run.smithi042.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5099-g88deadb8-1jammy [39.0 kB] 2024-09-20T19:03:44.013 INFO:teuthology.orchestra.run.smithi042.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5099-g88deadb8-1jammy [15.8 MB] 2024-09-20T19:03:44.316 INFO:teuthology.orchestra.run.smithi042.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5099-g88deadb8-1jammy [91.2 kB] 2024-09-20T19:03:44.554 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 244 MB in 7s (32.7 MB/s) 2024-09-20T19:03:44.727 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-20T19:03:44.826 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-20T19:03:44.832 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-20T19:03:44.897 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 244 MB in 8s (31.5 MB/s) 2024-09-20T19:03:44.917 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-20T19:03:45.030 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-20T19:03:45.115 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-20T19:03:45.121 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-20T19:03:45.196 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-20T19:03:45.329 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-20T19:03:45.355 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-20T19:03:45.404 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-20T19:03:45.541 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-20T19:03:45.568 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-20T19:03:45.608 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-20T19:03:45.799 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-20T19:03:45.825 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-20T19:03:45.874 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-20T19:03:45.936 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-20T19:03:45.963 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-20T19:03:46.003 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-20T19:03:46.403 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-20T19:03:46.429 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:46.448 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-20T19:03:46.475 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:46.477 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:46.523 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:46.898 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-20T19:03:46.918 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-20T19:03:46.924 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:46.945 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:46.964 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:46.985 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:47.309 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-20T19:03:47.330 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-20T19:03:47.338 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:47.356 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-20T19:03:47.384 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:47.396 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:03:47.725 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-20T19:03:47.752 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-20T19:03:47.783 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-20T19:03:47.787 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-20T19:03:47.814 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-20T19:03:47.862 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-20T19:03:48.111 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librados2. 2024-09-20T19:03:48.138 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../007-librados2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:48.177 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librados2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:48.265 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librados2. 2024-09-20T19:03:48.292 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../007-librados2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:48.340 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librados2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:48.664 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libnbd0. 2024-09-20T19:03:48.691 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-20T19:03:48.730 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-20T19:03:49.033 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librbd1. 2024-09-20T19:03:49.050 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../009-librbd1_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:49.077 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libnbd0. 2024-09-20T19:03:49.083 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librbd1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:49.103 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-20T19:03:49.151 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-20T19:03:49.455 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librbd1. 2024-09-20T19:03:49.481 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../009-librbd1_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:49.495 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs2. 2024-09-20T19:03:49.521 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librbd1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:49.522 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:49.561 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:49.847 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rados. 2024-09-20T19:03:49.874 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:49.914 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rados (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:50.016 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs2. 2024-09-20T19:03:50.043 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:50.091 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:50.418 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rados. 2024-09-20T19:03:50.434 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:50.485 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rados (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:50.500 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-20T19:03:50.527 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:03:50.567 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-argparse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:50.786 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cephfs. 2024-09-20T19:03:50.802 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:50.813 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-20T19:03:50.836 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cephfs (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:50.839 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:03:50.880 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-argparse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:51.133 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cephfs. 2024-09-20T19:03:51.147 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-20T19:03:51.159 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:51.174 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:03:51.199 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cephfs (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:51.214 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:51.559 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-20T19:03:51.569 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-20T19:03:51.585 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-20T19:03:51.594 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:03:51.625 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-20T19:03:51.643 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:51.912 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-prettytable. 2024-09-20T19:03:51.939 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-20T19:03:51.970 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-20T19:03:52.030 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-20T19:03:52.057 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-20T19:03:52.105 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-20T19:03:52.223 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rbd. 2024-09-20T19:03:52.239 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:52.281 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rbd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:52.433 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-prettytable. 2024-09-20T19:03:52.459 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-20T19:03:52.499 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-20T19:03:52.634 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-20T19:03:52.662 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-20T19:03:52.701 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-20T19:03:52.794 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rbd. 2024-09-20T19:03:52.821 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:52.861 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rbd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:52.987 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-20T19:03:53.013 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-20T19:03:53.053 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-20T19:03:53.222 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-20T19:03:53.249 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-20T19:03:53.289 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-20T19:03:53.323 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-20T19:03:53.340 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-20T19:03:53.373 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-20T19:03:53.575 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-20T19:03:53.590 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-20T19:03:53.633 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-20T19:03:53.702 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua5.1. 2024-09-20T19:03:53.729 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-20T19:03:53.768 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-20T19:03:53.970 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-20T19:03:53.985 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-20T19:03:54.020 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-20T19:03:54.105 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-any. 2024-09-20T19:03:54.131 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-20T19:03:54.172 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-20T19:03:54.340 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua5.1. 2024-09-20T19:03:54.367 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-20T19:03:54.406 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-20T19:03:54.407 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package zip. 2024-09-20T19:03:54.423 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-20T19:03:54.466 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking zip (3.0-12build2) ... 2024-09-20T19:03:54.726 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-any. 2024-09-20T19:03:54.742 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-20T19:03:54.784 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-20T19:03:54.811 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package unzip. 2024-09-20T19:03:54.838 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-20T19:03:54.880 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-20T19:03:55.029 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package zip. 2024-09-20T19:03:55.044 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-20T19:03:55.079 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking zip (3.0-12build2) ... 2024-09-20T19:03:55.242 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package luarocks. 2024-09-20T19:03:55.258 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-20T19:03:55.308 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-20T19:03:55.382 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package unzip. 2024-09-20T19:03:55.409 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-20T19:03:55.449 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-20T19:03:55.845 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package luarocks. 2024-09-20T19:03:55.871 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-20T19:03:55.875 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librgw2. 2024-09-20T19:03:55.902 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../026-librgw2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:55.911 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-20T19:03:55.941 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librgw2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:56.294 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librgw2. 2024-09-20T19:03:56.321 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../026-librgw2_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:56.361 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librgw2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:56.569 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rgw. 2024-09-20T19:03:56.585 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:56.627 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:56.972 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-20T19:03:56.997 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rgw. 2024-09-20T19:03:56.999 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-20T19:03:57.012 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:57.038 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-20T19:03:57.055 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:57.299 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libradosstriper1. 2024-09-20T19:03:57.326 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:57.366 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libradosstriper1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:57.408 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-20T19:03:57.435 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-20T19:03:57.475 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-20T19:03:57.786 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libradosstriper1. 2024-09-20T19:03:57.794 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-common. 2024-09-20T19:03:57.813 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:57.821 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:57.853 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libradosstriper1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:57.886 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:58.365 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-common. 2024-09-20T19:03:58.391 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:58.439 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:59.366 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-base. 2024-09-20T19:03:59.382 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:03:59.424 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-base (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:03:59.986 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-base. 2024-09-20T19:04:00.013 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:00.045 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-20T19:04:00.061 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-20T19:04:00.062 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-base (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:00.103 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-20T19:04:00.415 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cheroot. 2024-09-20T19:04:00.442 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-20T19:04:00.481 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-20T19:04:00.658 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-20T19:04:00.685 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-20T19:04:00.733 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-20T19:04:00.809 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-20T19:04:00.836 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-20T19:04:00.876 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-20T19:04:00.961 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cheroot. 2024-09-20T19:04:00.987 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-20T19:04:01.035 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-20T19:04:01.179 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-20T19:04:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-20T19:04:01.245 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-20T19:04:01.389 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-20T19:04:01.415 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-20T19:04:01.455 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-20T19:04:01.523 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-20T19:04:01.550 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-20T19:04:01.590 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-20T19:04:01.759 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-20T19:04:01.785 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-20T19:04:01.825 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-20T19:04:01.926 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempora. 2024-09-20T19:04:01.953 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-20T19:04:01.993 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-20T19:04:02.195 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-20T19:04:02.221 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-20T19:04:02.269 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-20T19:04:02.295 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-portend. 2024-09-20T19:04:02.323 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-20T19:04:02.362 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-20T19:04:02.598 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-tempora. 2024-09-20T19:04:02.624 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-20T19:04:02.632 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-20T19:04:02.648 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-20T19:04:02.664 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-20T19:04:02.681 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-20T19:04:02.968 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-portend. 2024-09-20T19:04:02.976 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-20T19:04:02.994 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-20T19:04:03.003 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-20T19:04:03.034 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-20T19:04:03.043 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-20T19:04:03.371 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-natsort. 2024-09-20T19:04:03.387 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-20T19:04:03.404 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-20T19:04:03.421 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-20T19:04:03.430 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-20T19:04:03.478 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-20T19:04:03.699 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-logutils. 2024-09-20T19:04:03.727 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-20T19:04:03.766 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-20T19:04:03.815 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-20T19:04:03.842 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-20T19:04:03.882 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-20T19:04:04.069 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-20T19:04:04.096 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-20T19:04:04.135 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-20T19:04:04.236 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-natsort. 2024-09-20T19:04:04.262 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-20T19:04:04.302 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-20T19:04:04.438 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-mako. 2024-09-20T19:04:04.454 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-20T19:04:04.496 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-20T19:04:04.647 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-logutils. 2024-09-20T19:04:04.674 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-20T19:04:04.714 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-20T19:04:04.875 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-20T19:04:04.902 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-20T19:04:04.941 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-20T19:04:05.042 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-20T19:04:05.068 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-20T19:04:05.108 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-20T19:04:05.186 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-20T19:04:05.202 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-20T19:04:05.244 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-20T19:04:05.370 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-mako. 2024-09-20T19:04:05.385 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-20T19:04:05.420 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-20T19:04:05.563 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webob. 2024-09-20T19:04:05.590 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-20T19:04:05.630 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-20T19:04:05.765 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-20T19:04:05.792 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-20T19:04:05.831 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-20T19:04:05.992 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-waitress. 2024-09-20T19:04:06.018 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-20T19:04:06.134 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-20T19:04:06.161 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-20T19:04:06.201 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-20T19:04:06.262 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-20T19:04:06.479 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webob. 2024-09-20T19:04:06.505 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-20T19:04:06.545 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-20T19:04:06.569 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempita. 2024-09-20T19:04:06.596 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-20T19:04:06.636 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-20T19:04:06.857 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-waitress. 2024-09-20T19:04:06.884 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-20T19:04:06.905 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-paste. 2024-09-20T19:04:06.918 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-20T19:04:06.932 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-20T19:04:06.972 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-20T19:04:07.202 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-tempita. 2024-09-20T19:04:07.229 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-20T19:04:07.268 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-20T19:04:07.309 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-20T19:04:07.324 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-20T19:04:07.367 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-20T19:04:07.580 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-paste. 2024-09-20T19:04:07.607 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-20T19:04:07.645 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-20T19:04:07.646 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-20T19:04:07.671 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-20T19:04:07.703 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-20T19:04:07.941 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-20T19:04:07.968 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-20T19:04:07.973 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webtest. 2024-09-20T19:04:08.000 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-20T19:04:08.008 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-20T19:04:08.039 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-20T19:04:08.344 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-20T19:04:08.351 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pecan. 2024-09-20T19:04:08.370 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-20T19:04:08.377 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-20T19:04:08.417 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-20T19:04:08.419 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-20T19:04:08.714 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webtest. 2024-09-20T19:04:08.740 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-20T19:04:08.772 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-certifi. 2024-09-20T19:04:08.780 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-20T19:04:08.799 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-20T19:04:08.838 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-20T19:04:09.042 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pecan. 2024-09-20T19:04:09.068 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-20T19:04:09.100 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-20T19:04:09.133 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-idna. 2024-09-20T19:04:09.149 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-20T19:04:09.191 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-20T19:04:09.519 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-urllib3. 2024-09-20T19:04:09.546 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-20T19:04:09.586 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-20T19:04:09.605 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-certifi. 2024-09-20T19:04:09.620 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-20T19:04:09.663 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-20T19:04:10.389 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests. 2024-09-20T19:04:10.417 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-20T19:04:10.798 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-20T19:04:10.876 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-idna. 2024-09-20T19:04:10.902 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-20T19:04:10.942 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-20T19:04:11.159 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-20T19:04:11.186 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-20T19:04:11.226 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-20T19:04:11.262 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-urllib3. 2024-09-20T19:04:11.288 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-20T19:04:11.328 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-20T19:04:11.529 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-20T19:04:11.556 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:11.631 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-requests. 2024-09-20T19:04:11.647 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-20T19:04:11.681 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-20T19:04:11.960 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-20T19:04:11.975 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-20T19:04:11.986 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-20T19:04:12.002 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:12.017 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-20T19:04:12.041 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:12.361 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-20T19:04:12.363 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-20T19:04:12.388 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-20T19:04:12.389 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:12.427 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-20T19:04:12.429 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:12.689 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-openssl. 2024-09-20T19:04:12.716 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-20T19:04:12.747 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-20T19:04:12.776 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-20T19:04:12.803 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:12.834 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:13.092 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr. 2024-09-20T19:04:13.119 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:13.137 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-20T19:04:13.158 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:13.164 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-20T19:04:13.203 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-20T19:04:13.507 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-openssl. 2024-09-20T19:04:13.533 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-20T19:04:13.561 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mon. 2024-09-20T19:04:13.573 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-20T19:04:13.589 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:13.628 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mon (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:13.885 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr. 2024-09-20T19:04:13.909 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:13.951 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:14.190 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-osd. 2024-09-20T19:04:14.217 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:14.248 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-osd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:14.339 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mon. 2024-09-20T19:04:14.365 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:14.405 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mon (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:15.025 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-osd. 2024-09-20T19:04:15.052 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:15.092 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-osd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:15.326 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph. 2024-09-20T19:04:15.342 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../067-ceph_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:15.384 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:15.704 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-fuse. 2024-09-20T19:04:15.731 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:15.770 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:16.157 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mds. 2024-09-20T19:04:16.184 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:16.223 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mds (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:16.228 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph. 2024-09-20T19:04:16.255 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../067-ceph_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:16.295 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:16.597 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-fuse. 2024-09-20T19:04:16.613 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:16.647 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:16.760 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package cephadm. 2024-09-20T19:04:16.787 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../070-cephadm_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:16.818 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:16.992 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mds. 2024-09-20T19:04:17.019 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:17.059 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mds (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:17.146 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-20T19:04:17.173 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-20T19:04:17.212 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-20T19:04:17.470 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package cephadm. 2024-09-20T19:04:17.497 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../070-cephadm_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:17.523 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-20T19:04:17.536 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:17.539 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-20T19:04:17.573 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:17.857 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-20T19:04:17.883 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-20T19:04:17.923 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-20T19:04:18.251 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-20T19:04:18.278 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-20T19:04:18.317 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:18.628 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-babel. 2024-09-20T19:04:18.656 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-20T19:04:18.694 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:19.006 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jinja2. 2024-09-20T19:04:19.033 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-20T19:04:19.323 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-babel. 2024-09-20T19:04:19.350 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-20T19:04:19.380 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:19.409 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-20T19:04:19.436 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:19.475 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:19.762 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-20T19:04:19.789 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-20T19:04:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-20T19:04:19.850 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jinja2. 2024-09-20T19:04:19.877 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-20T19:04:19.933 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-20T19:04:20.123 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-routes. 2024-09-20T19:04:20.140 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-20T19:04:20.190 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-20T19:04:20.262 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-20T19:04:20.288 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:20.328 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:20.526 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-20T19:04:20.554 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:20.592 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:20.649 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-20T19:04:20.675 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-20T19:04:20.715 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-20T19:04:21.001 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-routes. 2024-09-20T19:04:21.028 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-20T19:04:21.068 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-20T19:04:21.405 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-20T19:04:21.432 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:21.471 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:23.456 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-20T19:04:23.486 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-20T19:04:23.531 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-20T19:04:24.026 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-joblib. 2024-09-20T19:04:24.054 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-20T19:04:24.092 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-20T19:04:24.294 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-20T19:04:24.323 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-20T19:04:24.368 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-20T19:04:24.438 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-20T19:04:24.466 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-20T19:04:24.504 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-20T19:04:24.832 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn. 2024-09-20T19:04:24.860 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-20T19:04:24.891 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-joblib. 2024-09-20T19:04:24.918 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-20T19:04:24.957 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-20T19:04:24.984 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-20T19:04:25.369 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-20T19:04:25.397 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-20T19:04:25.436 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-20T19:04:25.713 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn. 2024-09-20T19:04:25.735 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-20T19:04:25.741 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-20T19:04:25.764 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:25.801 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:25.857 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-20T19:04:26.542 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-20T19:04:26.558 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:26.591 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:27.013 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cachetools. 2024-09-20T19:04:27.041 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-20T19:04:27.079 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-20T19:04:27.349 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-20T19:04:27.366 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-20T19:04:27.399 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-20T19:04:27.545 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cachetools. 2024-09-20T19:04:27.573 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-20T19:04:27.611 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-20T19:04:27.669 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-20T19:04:27.698 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-20T19:04:27.736 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-20T19:04:27.923 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-20T19:04:28.020 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rsa. 2024-09-20T19:04:28.021 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-20T19:04:28.021 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-20T19:04:28.042 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-20T19:04:28.080 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-20T19:04:28.309 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-20T19:04:28.337 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-20T19:04:28.375 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-20T19:04:28.401 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-google-auth. 2024-09-20T19:04:28.429 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-20T19:04:28.467 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-20T19:04:28.687 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rsa. 2024-09-20T19:04:28.715 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-20T19:04:28.753 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-20T19:04:28.829 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-20T19:04:28.857 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-20T19:04:28.895 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-20T19:04:29.032 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-google-auth. 2024-09-20T19:04:29.057 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-20T19:04:29.099 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-20T19:04:29.207 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-websocket. 2024-09-20T19:04:29.235 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-20T19:04:29.274 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-20T19:04:29.377 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-20T19:04:29.393 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-20T19:04:29.426 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-20T19:04:29.593 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-20T19:04:29.622 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-20T19:04:29.660 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-20T19:04:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-websocket. 2024-09-20T19:04:29.733 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-20T19:04:29.771 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-20T19:04:30.091 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-20T19:04:30.120 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-20T19:04:30.307 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-20T19:04:30.336 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:30.358 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-20T19:04:30.382 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:30.710 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-20T19:04:30.738 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-20T19:04:30.776 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-20T19:04:31.038 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-20T19:04:31.055 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:31.096 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:31.104 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-20T19:04:31.122 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-20T19:04:31.162 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-20T19:04:31.374 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-20T19:04:31.403 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-20T19:04:31.518 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package jq. 2024-09-20T19:04:31.543 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-20T19:04:31.566 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-20T19:04:31.573 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-20T19:04:31.793 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package socat. 2024-09-20T19:04:31.810 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-20T19:04:31.851 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-20T19:04:31.894 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-20T19:04:31.922 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-20T19:04:31.960 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-20T19:04:32.104 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package xmlstarlet. 2024-09-20T19:04:32.122 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-20T19:04:32.179 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package jq. 2024-09-20T19:04:32.187 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-20T19:04:32.196 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-20T19:04:32.229 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-20T19:04:32.474 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package socat. 2024-09-20T19:04:32.502 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-20T19:04:32.540 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-20T19:04:32.558 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-test. 2024-09-20T19:04:32.575 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:32.616 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-test (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:32.827 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package xmlstarlet. 2024-09-20T19:04:32.855 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-20T19:04:32.894 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-20T19:04:33.240 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-test. 2024-09-20T19:04:33.256 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:33.289 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-test (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:34.946 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-volume. 2024-09-20T19:04:34.974 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:35.012 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-volume (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:35.300 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-20T19:04:35.318 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:35.358 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs-dev (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-volume. 2024-09-20T19:04:35.572 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5099-g88deadb8-1jammy_all.deb ... 2024-09-20T19:04:35.610 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-volume (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:35.644 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-20T19:04:35.673 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-20T19:04:35.710 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-20T19:04:35.898 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-20T19:04:35.926 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:35.964 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs-dev (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:36.073 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-20T19:04:36.102 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-20T19:04:36.140 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-20T19:04:36.292 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-20T19:04:36.320 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-20T19:04:36.358 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-20T19:04:36.488 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package nvme-cli. 2024-09-20T19:04:36.516 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-20T19:04:36.546 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-20T19:04:36.705 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-20T19:04:36.733 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-20T19:04:36.771 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-20T19:04:36.900 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package pkg-config. 2024-09-20T19:04:36.917 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-20T19:04:36.959 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-20T19:04:37.083 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package nvme-cli. 2024-09-20T19:04:37.112 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-20T19:04:37.141 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-20T19:04:37.220 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-20T19:04:37.237 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-20T19:04:37.495 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-20T19:04:37.547 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package pkg-config. 2024-09-20T19:04:37.575 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-20T19:04:37.613 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-20T19:04:37.833 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-20T19:04:37.849 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-20T19:04:37.891 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-20T19:04:37.906 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-20T19:04:37.935 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-20T19:04:37.973 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-20T19:04:38.259 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastescript. 2024-09-20T19:04:38.286 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-20T19:04:38.288 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-20T19:04:38.314 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-20T19:04:38.325 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-20T19:04:38.344 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-20T19:04:38.613 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastescript. 2024-09-20T19:04:38.642 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-20T19:04:38.646 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pluggy. 2024-09-20T19:04:38.675 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-20T19:04:38.688 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-20T19:04:38.712 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-20T19:04:38.990 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-psutil. 2024-09-20T19:04:39.009 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pluggy. 2024-09-20T19:04:39.019 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-20T19:04:39.037 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-20T19:04:39.048 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-20T19:04:39.098 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-20T19:04:39.401 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-py. 2024-09-20T19:04:39.411 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-psutil. 2024-09-20T19:04:39.429 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-20T19:04:39.439 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-20T19:04:39.477 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-20T19:04:39.484 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-20T19:04:39.779 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pygments. 2024-09-20T19:04:39.796 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-20T19:04:39.823 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-py. 2024-09-20T19:04:39.837 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-20T19:04:39.851 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-20T19:04:39.897 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-20T19:04:40.201 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pygments. 2024-09-20T19:04:40.229 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-20T19:04:40.267 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-20T19:04:40.275 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-20T19:04:40.292 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-20T19:04:40.325 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-20T19:04:40.603 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-toml. 2024-09-20T19:04:40.631 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-20T19:04:40.669 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-20T19:04:40.713 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-20T19:04:40.741 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-20T19:04:40.779 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-20T19:04:40.972 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pytest. 2024-09-20T19:04:41.001 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-20T19:04:41.038 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-20T19:04:41.065 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-toml. 2024-09-20T19:04:41.094 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-20T19:04:41.123 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-20T19:04:41.367 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplejson. 2024-09-20T19:04:41.393 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pytest. 2024-09-20T19:04:41.395 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-20T19:04:41.422 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-20T19:04:41.433 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-20T19:04:41.459 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-20T19:04:41.695 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-20T19:04:41.723 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-20T19:04:41.761 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-20T19:04:41.796 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplejson. 2024-09-20T19:04:41.825 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-20T19:04:41.863 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-20T19:04:42.132 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-20T19:04:42.149 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-20T19:04:42.182 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-20T19:04:42.523 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package radosgw. 2024-09-20T19:04:42.552 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../117-radosgw_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:42.589 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking radosgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:42.870 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package radosgw. 2024-09-20T19:04:42.898 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../117-radosgw_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:42.961 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking radosgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:43.526 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package rbd-fuse. 2024-09-20T19:04:43.543 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:43.584 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking rbd-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:43.831 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package rbd-fuse. 2024-09-20T19:04:43.847 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5099-g88deadb8-1jammy_amd64.deb ... 2024-09-20T19:04:43.873 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-20T19:04:43.889 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking rbd-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:44.147 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-20T19:04:44.202 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-20T19:04:44.273 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-20T19:04:44.452 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-20T19:04:44.542 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-20T19:04:44.578 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-20T19:04:44.842 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-20T19:04:44.851 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-20T19:04:45.181 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-20T19:04:45.271 INFO:teuthology.orchestra.run.smithi042.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-20T19:04:45.342 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:45.602 INFO:teuthology.orchestra.run.smithi162.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-20T19:04:45.680 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:45.808 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user cephadm....done 2024-09-20T19:04:45.923 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-20T19:04:46.199 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-20T19:04:46.467 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-20T19:04:46.471 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user cephadm....done 2024-09-20T19:04:46.602 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-20T19:04:46.636 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-20T19:04:46.853 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-20T19:04:46.920 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-20T19:04:47.112 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-20T19:04:47.180 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-20T19:04:47.238 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-20T19:04:47.306 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-20T19:04:47.531 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-20T19:04:47.590 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-20T19:04:47.866 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-20T19:04:47.900 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-20T19:04:47.984 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-20T19:04:48.160 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-20T19:04:48.269 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-20T19:04:48.419 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-20T19:04:48.621 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-20T19:04:48.673 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-20T19:04:48.881 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-20T19:04:48.931 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-20T19:04:49.165 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-20T19:04:49.198 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-20T19:04:49.425 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-20T19:04:49.457 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-argparse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:49.676 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-20T19:04:49.716 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-20T19:04:49.944 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-20T19:04:49.976 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-20T19:04:50.085 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-20T19:04:50.203 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-argparse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:50.211 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-20T19:04:50.338 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-20T19:04:50.471 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-20T19:04:50.589 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-20T19:04:50.615 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-20T19:04:50.740 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-20T19:04:50.866 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-20T19:04:50.978 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-20T19:04:50.992 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-20T19:04:51.002 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-20T19:04:51.058 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-20T19:04:51.185 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-20T19:04:51.243 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-20T19:04:51.553 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:51.651 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-20T19:04:51.683 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-20T19:04:51.703 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-20T19:04:51.738 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-20T19:04:51.864 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-20T19:04:51.980 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-20T19:04:52.224 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-20T19:04:52.247 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-20T19:04:52.349 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-20T19:04:52.548 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up zip (3.0-12build2) ... 2024-09-20T19:04:52.617 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-20T19:04:52.675 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-20T19:04:52.885 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-20T19:04:53.186 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up zip (3.0-12build2) ... 2024-09-20T19:04:53.287 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-20T19:04:53.410 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-20T19:04:53.670 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-20T19:04:53.796 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-20T19:04:53.905 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-20T19:04:53.931 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-20T19:04:54.189 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-20T19:04:54.190 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-20T19:04:54.316 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-20T19:04:54.467 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-20T19:04:54.574 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-20T19:04:54.760 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-20T19:04:54.825 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-20T19:04:55.177 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-20T19:04:55.244 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-20T19:04:55.421 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-20T19:04:55.520 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-20T19:04:55.789 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-20T19:04:55.879 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-20T19:04:56.074 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-20T19:04:56.131 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-20T19:04:56.257 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:56.417 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-20T19:04:56.566 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-20T19:04:56.668 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-20T19:04:56.794 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:04:56.893 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-20T19:04:57.087 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-20T19:04:57.396 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-20T19:04:58.165 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-20T19:04:58.425 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:04:58.510 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-20T19:04:58.551 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-20T19:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:04:58.836 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-20T19:04:58.880 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-20T19:04:58.962 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-20T19:04:59.088 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-20T19:04:59.156 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-20T19:04:59.282 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-20T19:04:59.364 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-20T19:04:59.408 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-20T19:04:59.615 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-20T19:04:59.693 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-20T19:04:59.875 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:04:59.944 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-20T19:05:00.001 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-20T19:05:00.170 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:05:00.296 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-20T19:05:00.302 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-20T19:05:00.553 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-20T19:05:00.606 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-20T19:05:00.813 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-20T19:05:00.874 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-20T19:05:01.064 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-20T19:05:01.158 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-20T19:05:01.290 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-20T19:05:01.566 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-20T19:05:01.634 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-20T19:05:01.692 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-20T19:05:01.885 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-20T19:05:01.952 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-20T19:05:02.078 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-20T19:05:02.170 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-20T19:05:02.304 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-20T19:05:02.379 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-20T19:05:02.581 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-20T19:05:02.697 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-20T19:05:02.706 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-20T19:05:02.948 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-20T19:05:03.008 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-20T19:05:03.074 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-20T19:05:03.225 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-20T19:05:03.309 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-20T19:05:03.314 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-20T19:05:03.560 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-20T19:05:03.568 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-20T19:05:03.686 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-20T19:05:03.694 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-20T19:05:03.820 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-20T19:05:03.909 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-20T19:05:04.079 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-20T19:05:04.155 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-20T19:05:04.273 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-20T19:05:04.331 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-20T19:05:04.457 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-20T19:05:04.650 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-20T19:05:04.724 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:05:04.842 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-20T19:05:04.909 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-20T19:05:05.010 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-20T19:05:05.160 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-20T19:05:05.261 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-20T19:05:05.387 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-20T19:05:05.486 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-20T19:05:05.705 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-20T19:05:05.771 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-20T19:05:05.888 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-20T19:05:06.006 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-20T19:05:06.139 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-20T19:05:06.316 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-20T19:05:06.442 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-20T19:05:06.484 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-20T19:05:06.709 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-20T19:05:06.744 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-20T19:05:07.113 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-20T19:05:07.129 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-20T19:05:07.389 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-20T19:05:07.414 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-20T19:05:07.749 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-20T19:05:07.774 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-20T19:05:07.875 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librados2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.002 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.059 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-20T19:05:08.136 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-20T19:05:08.405 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-20T19:05:08.472 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.530 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librados2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.598 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libradosstriper1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.647 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:08.724 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-20T19:05:08.765 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-20T19:05:09.009 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librbd1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.101 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.135 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-modules-core (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.227 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libradosstriper1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.270 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.353 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-20T19:05:09.488 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-20T19:05:09.488 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-20T19:05:09.632 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librbd1 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.757 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-modules-core (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.925 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:09.963 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs-dev (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.089 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rados (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.136 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-20T19:05:10.136 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-20T19:05:10.215 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librgw2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.341 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-20T19:05:10.618 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs-dev (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.744 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rados (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.870 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librgw2 (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:10.995 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-20T19:05:11.620 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rbd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:11.746 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up rbd-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:11.855 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:11.956 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cephfs (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.057 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.149 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rbd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.267 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up rbd-fuse (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.368 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.469 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cephfs (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.586 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-common (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:12.600 INFO:teuthology.orchestra.run.smithi042.stdout:Adding group ceph....done 2024-09-20T19:05:12.993 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user ceph....done 2024-09-20T19:05:13.136 INFO:teuthology.orchestra.run.smithi042.stdout:Setting system user ceph properties....done 2024-09-20T19:05:13.138 INFO:teuthology.orchestra.run.smithi162.stdout:Adding group ceph....done 2024-09-20T19:05:13.148 INFO:teuthology.orchestra.run.smithi042.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-20T19:05:13.293 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-20T19:05:13.497 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user ceph....done 2024-09-20T19:05:13.639 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-20T19:05:13.716 INFO:teuthology.orchestra.run.smithi162.stdout:Setting system user ceph properties....done 2024-09-20T19:05:13.733 INFO:teuthology.orchestra.run.smithi162.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-20T19:05:13.867 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-20T19:05:14.091 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-test (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:14.167 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-20T19:05:14.192 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up radosgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:14.645 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-test (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:14.655 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:05:14.655 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:05:14.780 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up radosgw (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:15.120 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-base (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:15.231 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:05:15.232 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:05:15.504 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-20T19:05:15.690 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-base (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:16.032 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-20T19:05:16.114 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mds (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:16.373 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-20T19:05:16.373 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-20T19:05:16.475 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mds (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:16.697 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-20T19:05:16.697 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-20T19:05:16.832 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:17.043 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-20T19:05:17.043 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-20T19:05:17.143 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:17.376 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-20T19:05:17.376 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-20T19:05:17.483 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-osd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:17.756 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-20T19:05:17.756 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-20T19:05:17.828 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-osd (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.104 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-20T19:05:18.104 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-20T19:05:18.219 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.344 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.604 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.714 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.751 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mon (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:18.840 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mon (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.037 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-20T19:05:19.037 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-20T19:05:19.107 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-20T19:05:19.107 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-20T19:05:19.482 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.573 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-cephadm (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.625 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.699 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.751 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-dashboard (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.902 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-volume (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:19.975 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-dashboard (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:20.101 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-volume (19.3.0-5099-g88deadb8-1jammy) ... 2024-09-20T19:05:20.263 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-20T19:05:20.463 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-20T19:05:20.506 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-20T19:05:20.564 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-20T19:05:20.650 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-20T19:05:20.732 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-20T19:05:21.734 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-20T19:05:21.875 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-20T19:05:22.394 INFO:teuthology.orchestra.run.smithi162.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-20T19:05:22.399 DEBUG:teuthology.parallel:result is None 2024-09-20T19:05:22.544 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-20T19:05:22.548 DEBUG:teuthology.parallel:result is None 2024-09-20T19:05:22.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:05:22.689 DEBUG:teuthology.orchestra.run.smithi042:> dpkg-query -W -f '${Version}' ceph 2024-09-20T19:05:22.707 INFO:teuthology.orchestra.run.smithi042.stdout:19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:05:22.707 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:05:22.707 INFO:teuthology.task.install:The correct ceph version 19.3.0-5099-g88deadb8-1jammy is installed. 2024-09-20T19:05:22.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:05:22.847 DEBUG:teuthology.orchestra.run.smithi162:> dpkg-query -W -f '${Version}' ceph 2024-09-20T19:05:22.865 INFO:teuthology.orchestra.run.smithi162.stdout:19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:05:22.866 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5099-g88deadb8-1jammy 2024-09-20T19:05:22.866 INFO:teuthology.task.install:The correct ceph version 19.3.0-5099-g88deadb8-1jammy is installed. 2024-09-20T19:05:22.867 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-20T19:05:22.867 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:05:22.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T19:05:22.881 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:05:22.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T19:05:22.924 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-20T19:05:22.924 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:05:22.924 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T19:05:22.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T19:05:22.997 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:05:22.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T19:05:23.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T19:05:23.065 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-20T19:05:23.066 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:05:23.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T19:05:23.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T19:05:23.133 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:05:23.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T19:05:23.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T19:05:23.201 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-20T19:05:23.201 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:05:23.201 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T19:05:23.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T19:05:23.269 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:05:23.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T19:05:23.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T19:05:23.337 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T19:05:23.425 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5', 'cephadm_mode': 'root'} 2024-09-20T19:05:23.425 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:05:23.426 INFO:tasks.cephadm:Cluster fsid is 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:05:23.426 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T19:05:23.426 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.42:6789]', 'mon.c': '[v1:172.21.15.42:6790]', 'mon.b': '[v1:172.21.15.162:6789]'} 2024-09-20T19:05:23.426 INFO:tasks.cephadm:First mon is mon.a on smithi042 2024-09-20T19:05:23.426 INFO:tasks.cephadm:First mgr is y 2024-09-20T19:05:23.426 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T19:05:23.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo hostname $(hostname -s) 2024-09-20T19:05:23.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-09-20T19:05:23.454 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-20T19:05:23.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:05:23.591 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/83186/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5099-g88deadb8', 'node_name': '172.21.2.3+braggi03', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5099-g88deadb8-1jammy'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-09-20 12:13:16.402626', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-20-1028', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-09-20T19:05:23.725 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-guits-main-2024-09-20-1028, sha1 88deadb8082f4a33832e228ccefc59cedba158a5 from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:05:23.726 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-09-20T19:05:23.726 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-09-20T19:05:23.727 DEBUG:teuthology.orchestra.run.smithi042:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.197 INFO:teuthology.orchestra.run.smithi042.stdout:-rw-rw-r-- 1 ubuntu ubuntu 811803 Sep 20 19:05 /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.198 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-20-1028/88deadb8082f4a33832e228ccefc59cedba158a5/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.632 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-rw-r-- 1 ubuntu ubuntu 811803 Sep 20 19:05 /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.633 DEBUG:teuthology.orchestra.run.smithi042:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.641 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:05:24.661 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 on all hosts... 2024-09-20T19:05:24.661 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 pull 2024-09-20T19:05:24.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 pull 2024-09-20T19:05:24.874 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5... 2024-09-20T19:05:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5... 2024-09-20T19:06:45.598 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "ceph_version": "ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev)", 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "image_id": "441f9a3fe51528ef653b21f42e9d884db6124a0c7d482e865382bdfdb9ec979c", 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "repo_digests": [ 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8a1a6edbc97129e01d4344a6578add33aaab808082c84049bd2e48e1f97b80d2" 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:06:45.599 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:06:45.651 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev)", 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "441f9a3fe51528ef653b21f42e9d884db6124a0c7d482e865382bdfdb9ec979c", 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8a1a6edbc97129e01d4344a6578add33aaab808082c84049bd2e48e1f97b80d2" 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:06:45.652 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:06:45.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph 2024-09-20T19:06:45.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-09-20T19:06:45.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /etc/ceph 2024-09-20T19:06:45.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-09-20T19:06:45.785 INFO:tasks.cephadm:Writing seed config... 2024-09-20T19:06:45.786 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T19:06:45.786 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T19:06:45.786 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-20T19:06:45.786 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-20T19:06:45.787 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-20T19:06:45.788 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-20T19:06:45.788 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-20T19:06:45.788 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-20T19:06:45.788 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:06:45.788 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T19:06:45.820 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 49cb5aae-7783-11ef-baf5-efdc52797490 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T19:06:45.821 DEBUG:teuthology.orchestra.run.smithi042:mon.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T19:06:45.865 DEBUG:teuthology.orchestra.run.smithi042:mgr.y> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T19:06:45.909 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T19:06:45.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 -v bootstrap --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.42:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:06:46.141 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:06:46.141 INFO:teuthology.orchestra.run.smithi042.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5', '-v', 'bootstrap', '--fsid', '49cb5aae-7783-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.42:6789]', '--skip-admin-label'] 2024-09-20T19:06:46.142 INFO:teuthology.orchestra.run.smithi042.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T19:06:46.142 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying podman|docker is present... 2024-09-20T19:06:46.142 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying lvm2 is present... 2024-09-20T19:06:46.143 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying time synchronization is in place... 2024-09-20T19:06:46.148 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:06:46.148 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:06:46.154 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:06:46.154 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.158 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-20T19:06:46.158 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-20T19:06:46.162 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-20T19:06:46.162 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.166 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-20T19:06:46.166 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout masked 2024-09-20T19:06:46.170 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-20T19:06:46.170 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.174 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-20T19:06:46.174 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-20T19:06:46.178 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-20T19:06:46.178 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.183 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout enabled 2024-09-20T19:06:46.187 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout active 2024-09-20T19:06:46.187 INFO:teuthology.orchestra.run.smithi042.stdout:Unit ntp.service is enabled and running 2024-09-20T19:06:46.187 INFO:teuthology.orchestra.run.smithi042.stdout:Repeating the final host check... 2024-09-20T19:06:46.187 INFO:teuthology.orchestra.run.smithi042.stdout:docker (/usr/bin/docker) is present 2024-09-20T19:06:46.188 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl is present 2024-09-20T19:06:46.188 INFO:teuthology.orchestra.run.smithi042.stdout:lvcreate is present 2024-09-20T19:06:46.191 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:06:46.191 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:06:46.195 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:06:46.195 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.200 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-20T19:06:46.200 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-20T19:06:46.203 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-20T19:06:46.204 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.207 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-20T19:06:46.207 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout masked 2024-09-20T19:06:46.211 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-20T19:06:46.211 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.215 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-20T19:06:46.215 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-20T19:06:46.219 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-20T19:06:46.219 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-09-20T19:06:46.224 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout enabled 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout active 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Unit ntp.service is enabled and running 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Host looks OK 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Cluster fsid: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Acquiring lock 140062547879600 on /run/cephadm/49cb5aae-7783-11ef-baf5-efdc52797490.lock 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Lock 140062547879600 acquired on /run/cephadm/49cb5aae-7783-11ef-baf5-efdc52797490.lock 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying IP 172.21.15.42 port 6789 ... 2024-09-20T19:06:46.229 INFO:teuthology.orchestra.run.smithi042.stdout:Base mon IP(s) is [172.21.15.42:6789], mon addrv is [v1:172.21.15.42:6789] 2024-09-20T19:06:46.232 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 2024-09-20T19:06:46.232 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-09-20T19:06:46.232 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.42 2024-09-20T19:06:46.234 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T19:06:46.234 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-20T19:06:46.234 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1440/64 scope link 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:Mon IP `172.21.15.42` is in CIDR network `172.21.0.0/20` 2024-09-20T19:06:46.237 INFO:teuthology.orchestra.run.smithi042.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-20T19:06:46.238 INFO:teuthology.orchestra.run.smithi042.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T19:06:46.238 INFO:teuthology.orchestra.run.smithi042.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5... 2024-09-20T19:06:46.497 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/docker: stdout 88deadb8082f4a33832e228ccefc59cedba158a5: Pulling from ceph-ci/ceph 2024-09-20T19:06:46.497 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/docker: stdout Digest: sha256:8a1a6edbc97129e01d4344a6578add33aaab808082c84049bd2e48e1f97b80d2 2024-09-20T19:06:46.497 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:06:46.497 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 2024-09-20T19:06:48.370 INFO:teuthology.orchestra.run.smithi042.stdout:ceph: stdout ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev) 2024-09-20T19:06:48.370 INFO:teuthology.orchestra.run.smithi042.stdout:Ceph version: ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev) 2024-09-20T19:06:48.370 INFO:teuthology.orchestra.run.smithi042.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T19:06:49.892 INFO:teuthology.orchestra.run.smithi042.stdout:stat: stdout 167 167 2024-09-20T19:06:49.892 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial keys... 2024-09-20T19:06:51.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQDKx+1mA17FMhAAjrV/ukI3M1xpZN1MRDSRiA== 2024-09-20T19:06:53.112 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQDMx+1mKlDNHRAAsM5uUwbtuBNUTWtgy69TjQ== 2024-09-20T19:06:54.635 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQDOx+1m6DSlARAAl/mnfEYoWvqHOgsc3DryQg== 2024-09-20T19:06:54.636 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial monmap... 2024-09-20T19:06:56.308 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool for a [v1:172.21.15.42:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = quincy 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: set fsid to 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:06:56.309 INFO:teuthology.orchestra.run.smithi042.stdout:Creating mon... 2024-09-20T19:06:57.823 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.286+0000 7f73fbabdc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:06:57.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.286+0000 7f73fbabdc80 1 imported monmap: 2024-09-20T19:06:57.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-09-20T19:06:57.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:06:57.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-09-20T19:06:55.589722+0000 2024-09-20T19:06:57.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.286+0000 7f73fbabdc80 0 /usr/bin/ceph-mon: set fsid to 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: RocksDB version: 7.9.2 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Git sha 0 2024-09-20T19:06:57.825 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Compile date 2024-09-20 10:48:39 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: DB SUMMARY 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: DB Session ID: NUELERGC5T49H2LW7JD6 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.827 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.error_if_exists: 0 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.create_if_missing: 1 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.paranoid_checks: 1 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:06:57.828 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.env: 0x5654e56af120 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.info_log: 0x5654e6426bc0 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.statistics: (nil) 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.use_fsync: 0 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_log_file_size: 0 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:06:57.830 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_fallocate: 1 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.use_direct_reads: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:06:57.831 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:06:57.833 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.db_log_dir: 2024-09-20T19:06:57.833 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.wal_dir: 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.write_buffer_manager: 0x5654e641fcc0 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:06:57.834 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.unordered_write: 0 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:06:57.835 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.row_cache: None 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.wal_filter: None 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.two_write_queues: 0 2024-09-20T19:06:57.838 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.wal_compression: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.atomic_flush: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.log_readahead_size: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:06:57.839 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_background_jobs: 2 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_background_compactions: -1 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_subcompactions: 1 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:06:57.840 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_open_files: -1 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:06:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_background_flushes: -1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Compression algorithms supported: 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kZSTD supported: 0 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kXpressCompression supported: 0 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kBZip2Compression supported: 0 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kLZ4Compression supported: 1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kZlibCompression supported: 1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: kSnappyCompression supported: 1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-09-20T19:06:57.844 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.merge_operator: 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_filter: None 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:06:57.845 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:06:57.848 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5654e6426820) 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x5654e640f610 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-09-20T19:06:57.849 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-09-20T19:06:57.850 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-09-20T19:06:57.851 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression: NoCompression 2024-09-20T19:06:57.855 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.num_levels: 7 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:06:57.856 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:06:57.857 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:06:57.858 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:06:57.858 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:06:57.867 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:06:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.inplace_update_support: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.bloom_locality: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.max_successive_merges: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:06:57.869 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.ttl: 2592000 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enable_blob_files: false 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.min_blob_size: 0 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:06:57.870 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 03e6b5f0-3ddf-4060-aaa7-65762c646672 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.290+0000 7f73fbabdc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.871 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73fbabdc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5654e643ee00 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73fbabdc80 4 rocksdb: DB pointer 0x5654e6520000 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73f3248640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73f3248640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-20T19:06:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.875 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x5654e640f610#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:06:57.877 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr 2024-09-20T19:06:57.877 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73fbabdc80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-09-20T19:06:57.877 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73fbabdc80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-09-20T19:06:57.877 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-20T19:06:57.294+0000 7f73fbabdc80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-09-20T19:06:57.877 INFO:teuthology.orchestra.run.smithi042.stdout:create mon.a on 2024-09-20T19:06:58.192 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-09-20T19:06:58.385 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-09-20T19:06:58.609 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target -> /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target. 2024-09-20T19:06:58.609 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target -> /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target. 2024-09-20T19:06:58.881 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a 2024-09-20T19:06:58.881 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to reset failed state of unit ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T19:06:59.081 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target.wants/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service -> /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service. 2024-09-20T19:06:59.091 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld does not appear to be present 2024-09-20T19:06:59.091 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:06:59.091 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mon to start... 2024-09-20T19:06:59.091 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mon... 2024-09-20T19:07:01.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:00 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.848363+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:07:01.694 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T19:07:01.694 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout id: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:01.694 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T19:07:01.694 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:01.695 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout services: 2024-09-20T19:07:01.695 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.231188s) 2024-09-20T19:07:01.695 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T19:07:01.695 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T19:07:01.695 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout data: 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T19:07:01.696 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:01.697 INFO:teuthology.orchestra.run.smithi042.stdout:mon is available 2024-09-20T19:07:01.697 INFO:teuthology.orchestra.run.smithi042.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851627+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851627+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851794+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851794+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851804+0000 mon.a (mon.0) 4 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851804+0000 mon.a (mon.0) 4 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851812+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:02.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851812+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851818+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851818+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851826+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851826+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851833+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851833+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851839+0000 mon.a (mon.0) 9 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.851839+0000 mon.a (mon.0) 9 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.853828+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.853828+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.856284+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.856284+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.856799+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-09-20T19:07:02.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: cluster 2024-09-20T19:07:00.856799+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-09-20T19:07:02.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: audit 2024-09-20T19:07:01.082765+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/808951570' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-20T19:07:02.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:01 smithi042 bash[22225]: audit 2024-09-20T19:07:01.082765+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/808951570' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-20T19:07:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:02 smithi042 bash[22225]: audit 2024-09-20T19:07:02.907284+0000 mon.a (mon.0) 14 : audit [INF] from='client.? v1:172.21.15.42:0/4293891205' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-20T19:07:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:02 smithi042 bash[22225]: audit 2024-09-20T19:07:02.907284+0000 mon.a (mon.0) 14 : audit [INF] from='client.? v1:172.21.15.42:0/4293891205' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-20T19:07:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:02 smithi042 bash[22225]: audit 2024-09-20T19:07:02.911496+0000 mon.a (mon.0) 15 : audit [INF] from='client.? v1:172.21.15.42:0/4293891205' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T19:07:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:02 smithi042 bash[22225]: audit 2024-09-20T19:07:02.911496+0000 mon.a (mon.0) 15 : audit [INF] from='client.? v1:172.21.15.42:0/4293891205' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T19:07:03.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:03.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:07:03.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout fsid = 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:03.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-20T19:07:03.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.42:6789] 2024-09-20T19:07:03.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:07:03.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:07:03.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:07:03.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:07:03.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:03.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:07:03.414 INFO:teuthology.orchestra.run.smithi042.stdout:Generating new minimal ceph.conf... 2024-09-20T19:07:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:04 smithi042 bash[22225]: audit 2024-09-20T19:07:04.656998+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? v1:172.21.15.42:0/1425964318' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:07:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:04 smithi042 bash[22225]: audit 2024-09-20T19:07:04.656998+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? v1:172.21.15.42:0/1425964318' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout:Restarting the monitor... 2024-09-20T19:07:05.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:05 smithi042 systemd[1]: Stopping Ceph mon.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:07:05.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:05 smithi042 bash[22225]: debug 2024-09-20T19:07:05.414+0000 7eff3e444640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-20T19:07:05.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:05 smithi042 bash[22225]: debug 2024-09-20T19:07:05.414+0000 7eff3e444640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T19:07:06.112 INFO:teuthology.orchestra.run.smithi042.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T19:07:06.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:05 smithi042 bash[22606]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:07:06.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:06 smithi042 bash[22668]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:07:06.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:06 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T19:07:06.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:06 smithi042 systemd[1]: Stopped Ceph mon.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:07:06.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:06 smithi042 systemd[1]: Started Ceph mon.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:07:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.706+0000 7fc242f16c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:07:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.706+0000 7fc242f16c80 0 ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev), process ceph-mon, pid 7 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.706+0000 7fc242f16c80 0 pidfile_write: ignore empty --pid-file 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 0 load: jerasure load: lrc 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: RocksDB version: 7.9.2 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Git sha 0 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Compile date 2024-09-20 10:48:39 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: DB SUMMARY 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: DB Session ID: EU36CJFCEAUC7NXNB0FB 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: CURRENT file: CURRENT 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91562 ; 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.error_if_exists: 0 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.create_if_missing: 0 2024-09-20T19:07:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.paranoid_checks: 1 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.env: 0x55b036f73120 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.info_log: 0x55b037ee16a0 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.statistics: (nil) 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.use_fsync: 0 2024-09-20T19:07:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_log_file_size: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_fallocate: 1 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.use_direct_reads: 0 2024-09-20T19:07:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.db_log_dir: 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.wal_dir: 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.write_buffer_manager: 0x55b037ee5b80 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:07:07.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.unordered_write: 0 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.row_cache: None 2024-09-20T19:07:07.937 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.wal_filter: None 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.two_write_queues: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.wal_compression: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.atomic_flush: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.log_readahead_size: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:07:07.938 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_background_jobs: 2 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_background_compactions: -1 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_subcompactions: 1 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:07:07.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_open_files: -1 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_background_flushes: -1 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Compression algorithms supported: 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kZSTD supported: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kXpressCompression supported: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kBZip2Compression supported: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kLZ4Compression supported: 1 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kZlibCompression supported: 1 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:07:07.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: kSnappyCompression supported: 1 2024-09-20T19:07:07.941 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:07:07.941 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:07:07.941 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.merge_operator: 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_filter: None 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:07:07.942 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b037ee12c0) 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: cache_index_and_filter_blocks: 1 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: pin_top_level_index_and_filter: 1 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: index_type: 0 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: data_block_index_type: 0 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: index_shortening: 1 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: checksum: 4 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: no_block_cache: 0 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_cache: 0x55b037ed7610 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_cache_name: BinnedLRUCache 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_cache_options: 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: capacity : 536870912 2024-09-20T19:07:07.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: num_shard_bits : 4 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: strict_capacity_limit : 0 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: high_pri_pool_ratio: 0.000 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_cache_compressed: (nil) 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: persistent_cache: (nil) 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_size: 4096 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_size_deviation: 10 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_restart_interval: 16 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: index_block_restart_interval: 1 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: metadata_block_size: 4096 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: partition_filters: 0 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: use_delta_encoding: 1 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: filter_policy: bloomfilter 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: whole_key_filtering: 1 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: verify_compression: 0 2024-09-20T19:07:07.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: read_amp_bytes_per_bit: 0 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: format_version: 5 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: enable_index_compression: 1 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: block_align: 0 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: max_auto_readahead_size: 262144 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: prepopulate_block_cache: 0 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: initial_auto_readahead_size: 8192 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: num_file_reads_for_auto_readahead: 2 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression: NoCompression 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.num_levels: 7 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:07:07.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:07:07.946 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:07:07.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:07:07.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.inplace_update_support: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.bloom_locality: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.max_successive_merges: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:07:07.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.ttl: 2592000 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enable_blob_files: false 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.min_blob_size: 0 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:07:07.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 03e6b5f0-3ddf-4060-aaa7-65762c646672 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859227713910, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.710+0000 7fc242f16c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859227715090, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 281, "table_properties": {"data_size": 85767, "index_size": 243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11476, "raw_average_key_size": 47, "raw_value_size": 79379, "raw_average_value_size": 330, "num_data_blocks": 10, "num_entries": 240, "num_filter_entries": 240, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726859227, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "03e6b5f0-3ddf-4060-aaa7-65762c646672", "db_session_id": "EU36CJFCEAUC7NXNB0FB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859227715230, "job": 1, "event": "recovery_finished"} 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b037f04e00 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 4 rocksdb: DB pointer 0x55b038008000 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc238cde640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:07:07.951 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc238cde640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: ** DB Stats ** 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: ** Compaction Stats [default] ** 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: L0 2/0 87.46 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Sum 2/0 87.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: ** Compaction Stats [default] ** 2024-09-20T19:07:07.952 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Cumulative compaction: 0.00 GB write, 20.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Interval compaction: 0.00 GB write, 20.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Block cache BinnedLRUCache@0x55b037ed7610#7 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:07:07.953 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.714+0000 7fc242f16c80 0 starting mon.a rank 0 at public addrs v1:172.21.15.42:6789/0 at bind addrs v1:172.21.15.42:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 1 mon.a@-1(???) e1 preinit fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).mds e1 new map 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).mds e1 print_map 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: e1 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: btime 2024-09-20T19:07:00:852215+0000 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: legacy client fscid: -1 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: No filesystems configured 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:07:07.954 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:07:07.955 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:07 smithi042 bash[22706]: debug 2024-09-20T19:07:07.718+0000 7fc242f16c80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T19:07:08.572 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T19:07:08.590 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:07:08.590 INFO:teuthology.orchestra.run.smithi042.stdout:Creating mgr... 2024-09-20T19:07:08.590 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T19:07:08.590 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722831+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722831+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722898+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722898+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722909+0000 mon.a (mon.0) 3 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722909+0000 mon.a (mon.0) 3 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722921+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722921+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722934+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:08.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722934+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722945+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722945+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722955+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722955+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722966+0000 mon.a (mon.0) 8 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.722966+0000 mon.a (mon.0) 8 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723436+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723436+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723475+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723475+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723797+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: cluster 2024-09-20T19:07:07.723797+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: audit 2024-09-20T19:07:07.987183+0000 mon.a (mon.0) 12 : audit [INF] from='client.? v1:172.21.15.42:0/594930061' entity='client.admin' 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 bash[22706]: audit 2024-09-20T19:07:07.987183+0000 mon.a (mon.0) 12 : audit [INF] from='client.? v1:172.21.15.42:0/594930061' entity='client.admin' 2024-09-20T19:07:08.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:08 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:07:08.908 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y 2024-09-20T19:07:08.908 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to reset failed state of unit ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T19:07:09.108 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490.target.wants/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service -> /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service. 2024-09-20T19:07:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld does not appear to be present 2024-09-20T19:07:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:07:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld does not appear to be present 2024-09-20T19:07:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-20T19:07:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr to start... 2024-09-20T19:07:09.118 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr... 2024-09-20T19:07:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:09 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:07:09.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:09 smithi042 systemd[1]: Started Ceph mgr.y for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:07:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:10 smithi042 bash[22706]: audit 2024-09-20T19:07:10.924502+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/593515372' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:10 smithi042 bash[22706]: audit 2024-09-20T19:07:10.924502+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/593515372' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:07:11.591 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.592 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:07:11.593 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T19:07:00:852215+0000", 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:07:11.594 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:07:00.852957+0000", 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:07:11.595 INFO:teuthology.orchestra.run.smithi042.stdout:mgr not available, waiting (1/15)... 2024-09-20T19:07:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:14 smithi042 bash[22706]: audit 2024-09-20T19:07:14.835613+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3613734420' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:14 smithi042 bash[22706]: audit 2024-09-20T19:07:14.835613+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3613734420' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:15.410 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:07:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:07:15.412 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T19:07:00:852215+0000", 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:07:15.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:07:00.852957+0000", 2024-09-20T19:07:15.414 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:15.415 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:15.415 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:07:15.415 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:07:15.415 INFO:teuthology.orchestra.run.smithi042.stdout:mgr not available, waiting (2/15)... 2024-09-20T19:07:17.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.948995+0000 mon.a (mon.0) 15 : cluster [INF] Activating manager daemon y 2024-09-20T19:07:17.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.948995+0000 mon.a (mon.0) 15 : cluster [INF] Activating manager daemon y 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.956246+0000 mon.a (mon.0) 16 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00761132s) 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.956246+0000 mon.a (mon.0) 16 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00761132s) 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.957953+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.957953+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958133+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958133+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958264+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958264+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958390+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958390+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958508+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958508+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958623+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958623+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958749+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.958749+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.959370+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.959370+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.984006+0000 mon.a (mon.0) 25 : cluster [INF] Manager daemon y is now available 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: cluster 2024-09-20T19:07:16.984006+0000 mon.a (mon.0) 25 : cluster [INF] Manager daemon y is now available 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.996603+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.996603+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.998845+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:07:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:17 smithi042 bash[22706]: audit 2024-09-20T19:07:16.998845+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:07:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.001467+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.001467+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.006573+0000 mon.a (mon.0) 29 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.006573+0000 mon.a (mon.0) 29 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.008408+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: audit 2024-09-20T19:07:17.008408+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.14100 v1:172.21.15.42:0/3020069796' entity='mgr.y' 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: cluster 2024-09-20T19:07:17.964616+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e3: y(active, since 1.01598s) 2024-09-20T19:07:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:18 smithi042 bash[22706]: cluster 2024-09-20T19:07:17.964616+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e3: y(active, since 1.01598s) 2024-09-20T19:07:19.816 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:19.816 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:07:19.816 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-09-20T19:07:19.818 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:07:19.820 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:07:19.821 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:07:19.822 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T19:07:00:852215+0000", 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:07:19.824 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:07:00.852957+0000", 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:07:19.826 INFO:teuthology.orchestra.run.smithi042.stdout:mgr is available 2024-09-20T19:07:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:20 smithi042 bash[22706]: cluster 2024-09-20T19:07:19.011103+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-20T19:07:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:20 smithi042 bash[22706]: cluster 2024-09-20T19:07:19.011103+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-20T19:07:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:20 smithi042 bash[22706]: audit 2024-09-20T19:07:19.179374+0000 mon.a (mon.0) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/1466521777' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:20 smithi042 bash[22706]: audit 2024-09-20T19:07:19.179374+0000 mon.a (mon.0) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/1466521777' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-20T19:07:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:21 smithi042 bash[22706]: audit 2024-09-20T19:07:21.288598+0000 mon.a (mon.0) 34 : audit [INF] from='client.? v1:172.21.15.42:0/1224170994' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-20T19:07:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:21 smithi042 bash[22706]: audit 2024-09-20T19:07:21.288598+0000 mon.a (mon.0) 34 : audit [INF] from='client.? v1:172.21.15.42:0/1224170994' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-20T19:07:21.882 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout fsid = 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.42:6789] 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:07:21.883 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:07:21.884 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:07:21.884 INFO:teuthology.orchestra.run.smithi042.stdout:Enabling cephadm module... 2024-09-20T19:07:23.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:23 smithi042 bash[22706]: audit 2024-09-20T19:07:23.367678+0000 mon.a (mon.0) 35 : audit [INF] from='client.? v1:172.21.15.42:0/917049267' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-20T19:07:23.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:23 smithi042 bash[22706]: audit 2024-09-20T19:07:23.367678+0000 mon.a (mon.0) 35 : audit [INF] from='client.? v1:172.21.15.42:0/917049267' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-20T19:07:23.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:23 smithi042 bash[22958]: ignoring --setuser ceph since I am not root 2024-09-20T19:07:23.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:23 smithi042 bash[22958]: ignoring --setgroup ceph since I am not root 2024-09-20T19:07:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:24 smithi042 bash[22706]: audit 2024-09-20T19:07:23.419593+0000 mon.a (mon.0) 36 : audit [INF] from='client.? v1:172.21.15.42:0/917049267' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T19:07:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:24 smithi042 bash[22706]: audit 2024-09-20T19:07:23.419593+0000 mon.a (mon.0) 36 : audit [INF] from='client.? v1:172.21.15.42:0/917049267' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T19:07:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:24 smithi042 bash[22706]: cluster 2024-09-20T19:07:23.425627+0000 mon.a (mon.0) 37 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-09-20T19:07:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:24 smithi042 bash[22706]: cluster 2024-09-20T19:07:23.425627+0000 mon.a (mon.0) 37 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-09-20T19:07:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:25 smithi042 bash[22706]: audit 2024-09-20T19:07:25.470749+0000 mon.a (mon.0) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/2447440492' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-20T19:07:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:25 smithi042 bash[22706]: audit 2024-09-20T19:07:25.470749+0000 mon.a (mon.0) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/2447440492' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for the mgr to restart... 2024-09-20T19:07:26.011 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr epoch 5... 2024-09-20T19:07:30.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.716169+0000 mon.a (mon.0) 39 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:07:30.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.716169+0000 mon.a (mon.0) 39 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:07:30.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.716910+0000 mon.a (mon.0) 40 : cluster [INF] Activating manager daemon y 2024-09-20T19:07:30.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.716910+0000 mon.a (mon.0) 40 : cluster [INF] Activating manager daemon y 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.724781+0000 mon.a (mon.0) 41 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.724781+0000 mon.a (mon.0) 41 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.725039+0000 mon.a (mon.0) 42 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00850011s) 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.725039+0000 mon.a (mon.0) 42 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00850011s) 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.726458+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.726458+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.728308+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.728308+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729327+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729327+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:07:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729433+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729433+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:07:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729544+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: audit 2024-09-20T19:07:29.729544+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:07:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.752390+0000 mon.a (mon.0) 48 : cluster [INF] Manager daemon y is now available 2024-09-20T19:07:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:29 smithi042 bash[22706]: cluster 2024-09-20T19:07:29.752390+0000 mon.a (mon.0) 48 : cluster [INF] Manager daemon y is now available 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:mgr epoch 5 is available 2024-09-20T19:07:31.318 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying orchestrator module is enabled... 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: cluster 2024-09-20T19:07:30.729571+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.01303s) 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: cluster 2024-09-20T19:07:30.729571+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.01303s) 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.840084+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.840084+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.842205+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.842205+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: cephadm 2024-09-20T19:07:30.842585+0000 mgr.y (mgr.14118) 3 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: cephadm 2024-09-20T19:07:30.842585+0000 mgr.y (mgr.14118) 3 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.844438+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.844438+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.846712+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.846712+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.849946+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.849946+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.852858+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.852858+0000 mon.a (mon.0) 55 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.862993+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.862993+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:07:32.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.865947+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:07:32.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:31 smithi042 bash[22706]: audit 2024-09-20T19:07:30.865947+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: debug 2024-09-20T19:07:32.449+0000 7f2692726640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:07:32] ENGINE Error in HTTPServer.serve 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: Traceback (most recent call last): 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: self._connections.run(self.expiration_interval) 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: self._run(expiration_interval) 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:07:32.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: s = self.context.wrap_socket( 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: return self.sslsocket_class._create( 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: self.do_handshake() 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: self._sslobj.do_handshake() 2024-09-20T19:07:32.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22958]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:07:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.222817+0000 mgr.y (mgr.14118) 4 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Bus STARTING 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.222817+0000 mgr.y (mgr.14118) 4 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Bus STARTING 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.324942+0000 mgr.y (mgr.14118) 5 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.324942+0000 mgr.y (mgr.14118) 5 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.448841+0000 mgr.y (mgr.14118) 6 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.448841+0000 mgr.y (mgr.14118) 6 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.449054+0000 mgr.y (mgr.14118) 7 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Bus STARTED 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.449054+0000 mgr.y (mgr.14118) 7 : cephadm [INF] [20/Sep/2024:19:07:32] ENGINE Bus STARTED 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: audit 2024-09-20T19:07:32.449974+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: audit 2024-09-20T19:07:32.449974+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.453309+0000 mgr.y (mgr.14118) 8 : cephadm [ERR] [20/Sep/2024:19:07:32] ENGINE Error in HTTPServer.serve 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:07:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:07:33.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: cephadm 2024-09-20T19:07:32.453309+0000 mgr.y (mgr.14118) 8 : cephadm [ERR] [20/Sep/2024:19:07:32] ENGINE Error in HTTPServer.serve 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:07:33.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:07:33.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:07:33.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:07:33.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:07:33.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:07:33.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:32 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:07:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:33 smithi042 bash[22706]: cluster 2024-09-20T19:07:32.730697+0000 mon.a (mon.0) 59 : cluster [DBG] mgrmap e8: y(active, since 3s) 2024-09-20T19:07:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:33 smithi042 bash[22706]: cluster 2024-09-20T19:07:32.730697+0000 mon.a (mon.0) 59 : cluster [DBG] mgrmap e8: y(active, since 3s) 2024-09-20T19:07:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:33 smithi042 bash[22706]: audit 2024-09-20T19:07:32.948391+0000 mon.a (mon.0) 60 : audit [INF] from='client.? v1:172.21.15.42:0/1832315415' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-20T19:07:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:33 smithi042 bash[22706]: audit 2024-09-20T19:07:32.948391+0000 mon.a (mon.0) 60 : audit [INF] from='client.? v1:172.21.15.42:0/1832315415' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-20T19:07:34.318 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-20T19:07:34.318 INFO:teuthology.orchestra.run.smithi042.stdout:Setting orchestrator backend to cephadm... 2024-09-20T19:07:35.121 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:34 smithi042 bash[22706]: audit 2024-09-20T19:07:33.733144+0000 mon.a (mon.0) 61 : audit [INF] from='client.? v1:172.21.15.42:0/1832315415' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-20T19:07:35.122 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:34 smithi042 bash[22706]: audit 2024-09-20T19:07:33.733144+0000 mon.a (mon.0) 61 : audit [INF] from='client.? v1:172.21.15.42:0/1832315415' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-20T19:07:35.122 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:34 smithi042 bash[22706]: cluster 2024-09-20T19:07:33.739042+0000 mon.a (mon.0) 62 : cluster [DBG] mgrmap e9: y(active, since 4s) 2024-09-20T19:07:35.122 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:34 smithi042 bash[22706]: cluster 2024-09-20T19:07:33.739042+0000 mon.a (mon.0) 62 : cluster [DBG] mgrmap e9: y(active, since 4s) 2024-09-20T19:07:37.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.699921+0000 mgr.y (mgr.14118) 9 : audit [DBG] from='client.14132 v1:172.21.15.42:0/2941743058' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:37.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.699921+0000 mgr.y (mgr.14118) 9 : audit [DBG] from='client.14132 v1:172.21.15.42:0/2941743058' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:37.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.707368+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:37.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.707368+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:37.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.719647+0000 mon.a (mon.0) 64 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:37.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:36 smithi042 bash[22706]: audit 2024-09-20T19:07:35.719647+0000 mon.a (mon.0) 64 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:38.239 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T19:07:38.239 INFO:teuthology.orchestra.run.smithi042.stdout:Generating ssh key... 2024-09-20T19:07:38.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:38 smithi042 bash[22706]: audit 2024-09-20T19:07:37.624693+0000 mgr.y (mgr.14118) 10 : audit [DBG] from='client.14134 v1:172.21.15.42:0/2629766265' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:38.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:38 smithi042 bash[22706]: audit 2024-09-20T19:07:37.624693+0000 mgr.y (mgr.14118) 10 : audit [DBG] from='client.14134 v1:172.21.15.42:0/2629766265' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:40.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: Generating public/private rsa key pair. 2024-09-20T19:07:40.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: Your identification has been saved in /tmp/tmpgwkj45e5/key 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: Your public key has been saved in /tmp/tmpgwkj45e5/key.pub 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: The key fingerprint is: 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: SHA256:FzeY7QrslfA3q+NgNdzkdKpDqj8F9MNMmNmGs56HODc ceph-49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: The key's randomart image is: 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: +---[RSA 3072]----+ 2024-09-20T19:07:40.184 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | * | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | B ++ | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | ..O+ * . | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | .o+=X + | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | oS+X.B | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | o.EBo= o | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | o=++ . | 2024-09-20T19:07:40.185 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | o...o | 2024-09-20T19:07:40.186 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: | ...oo. | 2024-09-20T19:07:40.186 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:07:39 smithi042 bash[22958]: +----[SHA256]-----+ 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.693575+0000 mgr.y (mgr.14118) 11 : audit [DBG] from='client.14136 v1:172.21.15.42:0/908514081' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.693575+0000 mgr.y (mgr.14118) 11 : audit [DBG] from='client.14136 v1:172.21.15.42:0/908514081' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: cephadm 2024-09-20T19:07:39.693962+0000 mgr.y (mgr.14118) 12 : cephadm [INF] Generating ssh key... 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: cephadm 2024-09-20T19:07:39.693962+0000 mgr.y (mgr.14118) 12 : cephadm [INF] Generating ssh key... 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.871445+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.871445+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.875760+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:40 smithi042 bash[22706]: audit 2024-09-20T19:07:39.875760+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:42.487 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqS1CrxupPiP5+HSpH4JR/ZgGnsoOiPeE2WqdXFtn3szc81AhWI9dDMP6EfEbsN69jx2bY5mkOYP3fEHgK7XPvo8d3x7XTdtK69wEx7fLwUUho91W8P3Rc7me2TyM46QX3bVSJoAQ9rxWyWKlRvnKcHmloK5BhDeMdTtMeNshNYNofotNz3zdPnFgoVOl2DC4dlqAwRic8hM6itBxcq3M0JjtOhRlF7qzXNGUo+VtPIfwr3gSU2rMrtPcQXrTrEenKJUptMqmGxAMhFJGBxxe9h3vTGtvbpZGNixZZj7aS7QMYEoysRbBztldC6KhxbrJXP8zNK7HxMOsRdMU0sZePVICaRdz0VhcZKqSVcQ+WtpZNZv13qYZjzoBHRXaEcEYwzRJpozunrpwPbswAsyAdmqo88cylUknniLR+jWGpuvCUcEvGfDXomuzi+7+I5lKTuWKVU5S0OwzyIXM01ahV1jnzPknVJfVpsGbqNy6oVXdJwtEAQ9ptsT4bYGqV8Qc= ceph-49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:07:42.487 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:07:42.487 INFO:teuthology.orchestra.run.smithi042.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T19:07:42.487 INFO:teuthology.orchestra.run.smithi042.stdout:Adding host smithi042... 2024-09-20T19:07:42.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:42 smithi042 bash[22706]: audit 2024-09-20T19:07:41.895919+0000 mgr.y (mgr.14118) 13 : audit [DBG] from='client.14138 v1:172.21.15.42:0/1704377443' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:42.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:42 smithi042 bash[22706]: audit 2024-09-20T19:07:41.895919+0000 mgr.y (mgr.14118) 13 : audit [DBG] from='client.14138 v1:172.21.15.42:0/1704377443' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:44.599 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:44 smithi042 bash[22706]: audit 2024-09-20T19:07:44.128654+0000 mgr.y (mgr.14118) 14 : audit [DBG] from='client.14140 v1:172.21.15.42:0/4264933304' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi042", "addr": "172.21.15.42", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:44.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:44 smithi042 bash[22706]: audit 2024-09-20T19:07:44.128654+0000 mgr.y (mgr.14118) 14 : audit [DBG] from='client.14140 v1:172.21.15.42:0/4264933304' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi042", "addr": "172.21.15.42", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:45.919 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:45 smithi042 bash[22706]: cephadm 2024-09-20T19:07:45.401257+0000 mgr.y (mgr.14118) 15 : cephadm [INF] Deploying cephadm binary to smithi042 2024-09-20T19:07:45.920 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:45 smithi042 bash[22706]: cephadm 2024-09-20T19:07:45.401257+0000 mgr.y (mgr.14118) 15 : cephadm [INF] Deploying cephadm binary to smithi042 2024-09-20T19:07:49.909 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Added host 'smithi042' with addr '172.21.15.42' 2024-09-20T19:07:49.910 INFO:teuthology.orchestra.run.smithi042.stdout:Deploying unmanaged mon service... 2024-09-20T19:07:50.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: audit 2024-09-20T19:07:49.042184+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:50.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: audit 2024-09-20T19:07:49.042184+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:50.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: cephadm 2024-09-20T19:07:49.042553+0000 mgr.y (mgr.14118) 16 : cephadm [INF] Added host smithi042 2024-09-20T19:07:50.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: cephadm 2024-09-20T19:07:49.042553+0000 mgr.y (mgr.14118) 16 : cephadm [INF] Added host smithi042 2024-09-20T19:07:50.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: audit 2024-09-20T19:07:49.042936+0000 mon.a (mon.0) 68 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:50.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:50 smithi042 bash[22706]: audit 2024-09-20T19:07:49.042936+0000 mon.a (mon.0) 68 : audit [DBG] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:07:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:51 smithi042 bash[22706]: cluster 2024-09-20T19:07:49.731013+0000 mgr.y (mgr.14118) 17 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:51.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:51 smithi042 bash[22706]: cluster 2024-09-20T19:07:49.731013+0000 mgr.y (mgr.14118) 17 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:52.908 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T19:07:52.908 INFO:teuthology.orchestra.run.smithi042.stdout:Deploying unmanaged mgr service... 2024-09-20T19:07:53.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: cluster 2024-09-20T19:07:51.731323+0000 mgr.y (mgr.14118) 18 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:53.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: cluster 2024-09-20T19:07:51.731323+0000 mgr.y (mgr.14118) 18 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:53.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.084431+0000 mgr.y (mgr.14118) 19 : audit [DBG] from='client.14142 v1:172.21.15.42:0/336541098' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:53.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.084431+0000 mgr.y (mgr.14118) 19 : audit [DBG] from='client.14142 v1:172.21.15.42:0/336541098' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:53.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: cephadm 2024-09-20T19:07:52.087342+0000 mgr.y (mgr.14118) 20 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-20T19:07:53.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: cephadm 2024-09-20T19:07:52.087342+0000 mgr.y (mgr.14118) 20 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-20T19:07:53.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.089963+0000 mon.a (mon.0) 69 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:53.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.089963+0000 mon.a (mon.0) 69 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:53.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.805928+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:53.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:53 smithi042 bash[22706]: audit 2024-09-20T19:07:52.805928+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:54.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:54 smithi042 bash[22706]: audit 2024-09-20T19:07:53.390674+0000 mon.a (mon.0) 71 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:54.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:54 smithi042 bash[22706]: audit 2024-09-20T19:07:53.390674+0000 mon.a (mon.0) 71 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:55.217 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T19:07:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:55 smithi042 bash[22706]: cluster 2024-09-20T19:07:53.731693+0000 mgr.y (mgr.14118) 21 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:55 smithi042 bash[22706]: cluster 2024-09-20T19:07:53.731693+0000 mgr.y (mgr.14118) 21 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:55 smithi042 bash[22706]: audit 2024-09-20T19:07:54.643082+0000 mon.a (mon.0) 72 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:55 smithi042 bash[22706]: audit 2024-09-20T19:07:54.643082+0000 mon.a (mon.0) 72 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:56.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:56 smithi042 bash[22706]: audit 2024-09-20T19:07:54.638548+0000 mgr.y (mgr.14118) 22 : audit [DBG] from='client.14144 v1:172.21.15.42:0/1510717508' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:56.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:56 smithi042 bash[22706]: audit 2024-09-20T19:07:54.638548+0000 mgr.y (mgr.14118) 22 : audit [DBG] from='client.14144 v1:172.21.15.42:0/1510717508' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:07:56.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:56 smithi042 bash[22706]: cephadm 2024-09-20T19:07:54.639930+0000 mgr.y (mgr.14118) 23 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-20T19:07:56.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:56 smithi042 bash[22706]: cephadm 2024-09-20T19:07:54.639930+0000 mgr.y (mgr.14118) 23 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-20T19:07:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:57 smithi042 bash[22706]: cluster 2024-09-20T19:07:55.732038+0000 mgr.y (mgr.14118) 24 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:57 smithi042 bash[22706]: cluster 2024-09-20T19:07:55.732038+0000 mgr.y (mgr.14118) 24 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:57 smithi042 bash[22706]: audit 2024-09-20T19:07:56.738452+0000 mon.a (mon.0) 73 : audit [INF] from='client.? v1:172.21.15.42:0/528819155' entity='client.admin' 2024-09-20T19:07:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:57 smithi042 bash[22706]: audit 2024-09-20T19:07:56.738452+0000 mon.a (mon.0) 73 : audit [INF] from='client.? v1:172.21.15.42:0/528819155' entity='client.admin' 2024-09-20T19:07:59.505 INFO:teuthology.orchestra.run.smithi042.stdout:Enabling the dashboard module... 2024-09-20T19:07:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: cluster 2024-09-20T19:07:57.732467+0000 mgr.y (mgr.14118) 25 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:59.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: cluster 2024-09-20T19:07:57.732467+0000 mgr.y (mgr.14118) 25 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:07:59.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: audit 2024-09-20T19:07:58.922062+0000 mon.a (mon.0) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2506894747' entity='client.admin' 2024-09-20T19:07:59.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: audit 2024-09-20T19:07:58.922062+0000 mon.a (mon.0) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2506894747' entity='client.admin' 2024-09-20T19:07:59.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: audit 2024-09-20T19:07:59.077677+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:07:59.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:07:59 smithi042 bash[22706]: audit 2024-09-20T19:07:59.077677+0000 mon.a (mon.0) 75 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:08:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: cluster 2024-09-20T19:07:59.732825+0000 mgr.y (mgr.14118) 26 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: cluster 2024-09-20T19:07:59.732825+0000 mgr.y (mgr.14118) 26 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: audit 2024-09-20T19:08:00.179954+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:08:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: audit 2024-09-20T19:08:00.179954+0000 mon.a (mon.0) 76 : audit [INF] from='mgr.14118 v1:172.21.15.42:0/2483966124' entity='mgr.y' 2024-09-20T19:08:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: audit 2024-09-20T19:08:01.112151+0000 mon.a (mon.0) 77 : audit [INF] from='client.? v1:172.21.15.42:0/3213208721' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-20T19:08:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22706]: audit 2024-09-20T19:08:01.112151+0000 mon.a (mon.0) 77 : audit [INF] from='client.? v1:172.21.15.42:0/3213208721' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-20T19:08:01.932 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22958]: ignoring --setuser ceph since I am not root 2024-09-20T19:08:01.932 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:01 smithi042 bash[22958]: ignoring --setgroup ceph since I am not root 2024-09-20T19:08:02.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:02 smithi042 bash[22706]: audit 2024-09-20T19:08:01.403458+0000 mon.a (mon.0) 78 : audit [INF] from='client.? v1:172.21.15.42:0/3213208721' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T19:08:02.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:02 smithi042 bash[22706]: audit 2024-09-20T19:08:01.403458+0000 mon.a (mon.0) 78 : audit [INF] from='client.? v1:172.21.15.42:0/3213208721' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T19:08:02.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:02 smithi042 bash[22706]: cluster 2024-09-20T19:08:01.409534+0000 mon.a (mon.0) 79 : cluster [DBG] mgrmap e10: y(active, since 31s) 2024-09-20T19:08:02.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:02 smithi042 bash[22706]: cluster 2024-09-20T19:08:01.409534+0000 mon.a (mon.0) 79 : cluster [DBG] mgrmap e10: y(active, since 31s) 2024-09-20T19:08:04.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:03 smithi042 bash[22706]: audit 2024-09-20T19:08:03.773973+0000 mon.a (mon.0) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/501135678' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-20T19:08:04.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:03 smithi042 bash[22706]: audit 2024-09-20T19:08:03.773973+0000 mon.a (mon.0) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/501135678' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 10, 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:08:04.370 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:08:04.371 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for the mgr to restart... 2024-09-20T19:08:04.371 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr epoch 10... 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.715678+0000 mon.a (mon.0) 81 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.715678+0000 mon.a (mon.0) 81 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.716440+0000 mon.a (mon.0) 82 : cluster [INF] Activating manager daemon y 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.716440+0000 mon.a (mon.0) 82 : cluster [INF] Activating manager daemon y 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.720852+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.720852+0000 mon.a (mon.0) 83 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.720996+0000 mon.a (mon.0) 84 : cluster [DBG] mgrmap e11: y(active, starting, since 0.00494478s) 2024-09-20T19:08:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.720996+0000 mon.a (mon.0) 84 : cluster [DBG] mgrmap e11: y(active, starting, since 0.00494478s) 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.725490+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.725490+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.727342+0000 mon.a (mon.0) 86 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.727342+0000 mon.a (mon.0) 86 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.728501+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.728501+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.728880+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.728880+0000 mon.a (mon.0) 88 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.729455+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: audit 2024-09-20T19:08:07.729455+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.756801+0000 mon.a (mon.0) 90 : cluster [INF] Manager daemon y is now available 2024-09-20T19:08:08.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:07 smithi042 bash[22706]: cluster 2024-09-20T19:08:07.756801+0000 mon.a (mon.0) 90 : cluster [INF] Manager daemon y is now available 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.172195+0000 mon.a (mon.0) 91 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.172195+0000 mon.a (mon.0) 91 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.187444+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.187444+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.189666+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:08:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: audit 2024-09-20T19:08:08.189666+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:08:09.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: cluster 2024-09-20T19:08:08.729650+0000 mon.a (mon.0) 94 : cluster [DBG] mgrmap e12: y(active, since 1.01359s) 2024-09-20T19:08:09.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:08 smithi042 bash[22706]: cluster 2024-09-20T19:08:08.729650+0000 mon.a (mon.0) 94 : cluster [DBG] mgrmap e12: y(active, since 1.01359s) 2024-09-20T19:08:09.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:08:09.427 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 12, 2024-09-20T19:08:09.427 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:08:09.427 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:08:09.427 INFO:teuthology.orchestra.run.smithi042.stdout:mgr epoch 10 is available 2024-09-20T19:08:09.427 INFO:teuthology.orchestra.run.smithi042.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-20T19:08:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:09 smithi042 bash[22706]: audit 2024-09-20T19:08:08.725834+0000 mgr.y (mgr.14152) 1 : audit [DBG] from='client.14156 v1:172.21.15.42:0/4207865151' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:08:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:09 smithi042 bash[22706]: audit 2024-09-20T19:08:08.725834+0000 mgr.y (mgr.14152) 1 : audit [DBG] from='client.14156 v1:172.21.15.42:0/4207865151' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:08:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:09 smithi042 bash[22706]: audit 2024-09-20T19:08:08.740554+0000 mgr.y (mgr.14152) 2 : audit [DBG] from='client.14156 v1:172.21.15.42:0/4207865151' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:08:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:09 smithi042 bash[22706]: audit 2024-09-20T19:08:08.740554+0000 mgr.y (mgr.14152) 2 : audit [DBG] from='client.14156 v1:172.21.15.42:0/4207865151' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: debug 2024-09-20T19:08:10.300+0000 7f5874195640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:08:10] ENGINE Error in HTTPServer.serve 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: Traceback (most recent call last): 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: self._connections.run(self.expiration_interval) 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: self._run(expiration_interval) 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:08:10.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: s = self.context.wrap_socket( 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: return self.sslsocket_class._create( 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: self.do_handshake() 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: self._sslobj.do_handshake() 2024-09-20T19:08:10.684 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22958]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cluster 2024-09-20T19:08:09.777401+0000 mon.a (mon.0) 95 : cluster [DBG] mgrmap e13: y(active, since 2s) 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cluster 2024-09-20T19:08:09.777401+0000 mon.a (mon.0) 95 : cluster [DBG] mgrmap e13: y(active, since 2s) 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.077703+0000 mgr.y (mgr.14152) 3 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Bus STARTING 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.077703+0000 mgr.y (mgr.14152) 3 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Bus STARTING 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.179514+0000 mgr.y (mgr.14152) 4 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.179514+0000 mgr.y (mgr.14152) 4 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.303048+0000 mgr.y (mgr.14152) 5 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.303048+0000 mgr.y (mgr.14152) 5 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:08:11.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.303267+0000 mgr.y (mgr.14152) 6 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Bus STARTED 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.303267+0000 mgr.y (mgr.14152) 6 : cephadm [INF] [20/Sep/2024:19:08:10] ENGINE Bus STARTED 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.305467+0000 mgr.y (mgr.14152) 7 : cephadm [ERR] [20/Sep/2024:19:08:10] ENGINE Error in HTTPServer.serve 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:08:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: cephadm 2024-09-20T19:08:10.305467+0000 mgr.y (mgr.14152) 7 : cephadm [ERR] [20/Sep/2024:19:08:10] ENGINE Error in HTTPServer.serve 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:08:11.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:08:11.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:10 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:08:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:11 smithi042 bash[22706]: audit 2024-09-20T19:08:10.995940+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14164 v1:172.21.15.42:0/4084212117' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:11 smithi042 bash[22706]: audit 2024-09-20T19:08:10.995940+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14164 v1:172.21.15.42:0/4084212117' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:13.434 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUSXLagy+fVhq+GnBVb5ojC/xZDOMwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMDE5MDgxMloXDTM0\nMDkyMTE5MDgxMlowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjQyMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAuqvQ2x7O3bkB6kLdCZWSZWcu4m8Mjwa6LvqW\n4yPCaDJjnBKwDWrARpsVYuaLpalv9QRDdOx05KI0e0gu1DCxfXAYuBNmoh/Pf3e+\nN5amg6CI8GHKhLZ57US2OaiZkCNDTf3UmEsH7HlKQ52XusvsbFRbw8ebvdPs+Njy\np++ULurldfl97yLEMa6M8lmd3Wfh1O1iVM7cimZTNt28X9r/jRDLGrAxuKvYA6NS\n6KxZ4ogMn+K1PqURFBC+iybkhr4uo86dJpK3g1pfU2R4vGe1xGDc5BckcEFpC2pG\nNSKgl3CgTpAs3pLj6cfUVZRym/jwlxQf8Rp8Fim9TnO+0OeBoXMlTH4HVXygOCj4\nTc3dVG6eina896s5gwUB4azUVk0wYkd7suGZiaPGPnV8rN//motCtVKc+1c4SU85\nvC0fySrdO8yWN7pZnWpTLhZ6MiYPEqScpZC3hROtX8yybtf08JmcExO/0IP5HXQ6\nN52bw3OsSXJJ2V0btYc1mZiD0C8mR+2w92OJ610By7/3vwN7IgNLSvpiQrjEkjIn\n3GmIm1Dd3hTarlhg0/mh9b4nvlriDVl8sqhmFMNMuvJsu0fESK2It+ySmoCdJwhO\n/bc28gEB8UVXss6pXqXnJR4jyDRewEqitJPtI4Rmq2ph+3Av8TgrHX2QZDnp3284\ntBYIeBECAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDQyghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPKjAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQCF\nqgxahOYhmI/sLnbJAetPjktbaoKrzFEzRxPCsxTlN6dCRnup2ib22vucMzSg25Xc\n92KwloKwNtWQIVdbpgsaAZlGoEaJA3fsLIyu2YjGdJuUACtsoBri5g9saYkrbmPo\nQt+yyb4WLYgkX+7z36Zcwi0KcTAun4YhJ/NJDZOji2qmGk/ODhaViUewgOOLJfQh\nReKZnLD1ybyRFDMFTXv8iJKEINx3H5YAujhgmwOM6hQLlb4eQvfSTK9S7dXKqdcr\nOfYarQ/9wWN+XkvlwJhKtBTBMxjyS2CnY7MCTAQD3onZneNUdhxWauJunYhpLW17\n5Lv9xCoIo+tROfYibGx7MV3Nwk0gYwZPj3YWAZg/F0R3dsBVYHkEvobp/2puRQz7\nebskxB/ek5FyZAp9e4USX6tLyJz75yC0VLt8G/e5Bjv5gHXst2gEjvan6C4WyxOX\nXu27oEwCBFduf07xtkZXubXbVWQ69sZgNf2dVs70h++1LMr3LEkFT0syVENrTVC8\nMHyq9Yf3LctNOMMQsK+lK31Q0szw/t5e38eJT88SFJX/K1y8YZm9iheNi7xd/+g/\nLOT51iAQVtBZgmqBpt6JMBHdDnUuH5tc62QDL2BQ9egfGLY7N+Mp5u189I4+lSyR\nE6cS93ts4euCEUzX3xr1Qlt1vnec2kEiYbefkMhNTw==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAuqvQ2x7O3bkB6kLdCZWSZWcu4m8Mjwa6LvqW4yPCaDJjnBKw\nDWrARpsVYuaLpalv9QRDdOx05KI0e0gu1DCxfXAYuBNmoh/Pf3e+N5amg6CI8GHK\nhLZ57US2OaiZkCNDTf3UmEsH7HlKQ52XusvsbFRbw8ebvdPs+Njyp++ULurldfl9\n7yLEMa6M8lmd3Wfh1O1iVM7cimZTNt28X9r/jRDLGrAxuKvYA6NS6KxZ4ogMn+K1\nPqURFBC+iybkhr4uo86dJpK3g1pfU2R4vGe1xGDc5BckcEFpC2pGNSKgl3CgTpAs\n3pLj6cfUVZRym/jwlxQf8Rp8Fim9TnO+0OeBoXMlTH4HVXygOCj4Tc3dVG6eina8\n96s5gwUB4azUVk0wYkd7suGZiaPGPnV8rN//motCtVKc+1c4SU85vC0fySrdO8yW\nN7pZnWpTLhZ6MiYPEqScpZC3hROtX8yybtf08JmcExO/0IP5HXQ6N52bw3OsSXJJ\n2V0btYc1mZiD0C8mR+2w92OJ610By7/3vwN7IgNLSvpiQrjEkjIn3GmIm1Dd3hTa\nrlhg0/mh9b4nvlriDVl8sqhmFMNMuvJsu0fESK2It+ySmoCdJwhO/bc28gEB8UVX\nss6pXqXnJR4jyDRewEqitJPtI4Rmq2ph+3Av8TgrHX2QZDnp3284tBYIeBECAwEA\nAQKCAgAnJzaNXy7gjpd6PUADbJ3iKvxIfiZSMYUHqYPQbIWwOTFvYPQQPF3L8WdQ\n8S5FNv7Gk3hao+X4tpTvbrpzwyrXNpXYtikGxElPYRxCqJSQczGFyd0ibOux+oGG\nInjjJKJPT3gx4TruPJp5CCr1ASS7rA37SHjAHh1MLKTo2HVqSdaLIiNp7w7bpNLn\nWiikcc+KS6geRa04UjIOBgAm0tZbDjSril1DcLfEO5N+Pfg0bXjIaFK/qvukC8cL\nhMLWCDncHdAUcR2QrXnlEePJEK+tWuu546i+6tlPjaDgMD0ixnq5P8BJhuHK/SDq\nOlK3qEU+SvhQJ8/zDhooxFo3kvc7O3bhDPxoQKV+L0gJoVBAjaoCicX69/crfTD0\nM70NIPgNh1YZLqeaqPi359TmmcW2CShYbOfy/27IyuiFg96dvFkqkeFayWN97JpT\nPFtU6x8mJDxEjQMUdItZGOjHKfez13TAZb8gtfXBB66zKuuUbtRmdxdsFu2dAf4D\nqLbQZ8GsVQo31PQUpNKlDLaPuFJdg3SwRIeOJsebWqNxILNbryMuX04xshEmFAO0\nywHLIIez3pV7IPxPtWoISyTXmtFt/6nJ4Mj9psomcb/C1UKcfHIrQ0WrBtOqyLPN\nytV9D/7nawSJ+dWBBpPwNYqGbDNDyvLP3myU/BHFYkllCXRf4QKCAQEA90kxMVj+\nTbFfsjY6C6I1U/KkvlDaF+2jELBvX5HGSpDg3UWv4dYSbPG/A4+uiSmvn2cU/SFP\nrxCtx/9omonL5v9DwTlr1R+wJDpvDok0Hw36G+QG58DkD3dVcvQLwW/p5875cvcM\nefQ+11BYwresuQz0sb/ihMm03K93kbdEFhUAv7fHRIK6K5TQl6w2EFUWpeX2iPjx\nDkDvT5p6/gEUjDkIJ4Gxw/6VupB3fBHWVCIPOl7lC6Q7RYu8jjerIgCZ7ZYq5AmF\nDK7WIoOid3dP7tpGjrAbzb5mKFCkNCGULcduhoME1ACt5JOymS04DHyPIrQLv3RK\niD2nsALUo2YWJQKCAQEAwT/Oy9phhLfQ42dwEcGDCjwm6DNpVlH0zlrhnut2YkB6\niRiKHi4K7QSHxzTycd8L+ww0VLstxdhETQ97+oDNkhtMRy7SLSU67rXIjbkF89uj\njVULGZsV1KZsAgdj5L3B1xMXH4uY7IK1nUD51D8+pn5OUVuuZIY5cC0BwcVphZq3\n82kZfYafUPM47NsT3YsscZR9bfNhtp/BYseQpZa3VEpL/d/zrDTMgk209gQP/JDn\nnnB/T14zTKEwpzsoXVFkm4YO8N4JWgK0WXr3VUAkAbd71GXXzfNlUexZYghQTlRv\nMV7gYzmeoce+8du0s1dO5ccGGTVA1CZJebrbofyIfQKCAQEAh2rUfpKeiItWvXAG\nr3I/w93/q5z5Idtnze7gvrREo3bdPS85VvH1GyXzlY6sZL6oRyGbE9eXHGaE1B4g\n+Wf2emRVhU3dqFN3EQ/S22H1YgA0c+uWgHlYKsQNziqo+WBA9kmYvwVdFrfTZfPA\n97i/5IKcyW4sLM2MtsF8OBblOTZ9N3/7pp/0NYOEC1sqLB18jpL1RCMXVtfmdXBC\nZDDRQMzGk2PppkwXUJhmZlIaTe5s6mknAw4Hy1jvg+4mUDOGF96BVGmGK+DesP3n\nDYcQBdwQgXkdcLkQlok6srWkwMES3KSJE1Pf5mmfh1SE+gmIFNgGCu0zF5Kz7qZO\nLcDs5QKCAQAa5SeBxY3m67I9w+qb/gOfzAlaLbWOVu+sPIfEF18tg23KYMjwNFY+\n8G95MQxA/trrAKLT3A0dyOCQnQQTNBPoLrZp/cEwmUrfaBtWZhbG/p1/kQNvd98M\n7nB/w8mKT0KIDVcdg88vs21VSnK6sGamw1NhDVJzGR8rVog1EKWD+6l4TftHNhI8\nMO2OQVs9bR4tXI4sXEAQK6G9fqrJ8PTo3UJAPgfqwppn1CrIRn53Fy+1JFEDmcen\n9ZMQ4zHV8vr+jiTMLwq8JT42ad+8VKXuOzwXrmjfd+10/mDePOv4StYC2AKArMdA\nGs2fWYWHpDIN1qopnfi8zetR7sicfcn9AoIBAQDI5MlMFj1aiF1OIuqvgtugKc2F\nsGuCDiRJy2rcz6TLPwiIs8Viz/8lm4nINLB3bL6zVgf9MUQW7dhE92s67lwHpLwy\nzPdyhStBLDW4k2H/zMnZCSstEByKWe5wemqx7q16trBbz9QkZ38WJx7D9f8Uf9EP\nLsPKrlvIPa2vGj9a4qnQAwpWDTIZg2OJp9mvUE3NwMNHYiKM39KVMlDHTh2HY4E9\nz03A30Yl2aOlR/FeDQSbytT+D7N8ZMCy5qUC7/vdaweBrEdfX3IbfHQ4oOv5KzPw\nYv19n0qkKwCNZ9ygqcgBKTIYngV4WjafPbObK+BEQnLQpU2lHD7UHMQp63yn\n-----END RSA PRIVATE KEY-----\n"} 2024-09-20T19:08:15.567 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:14.948175+0000 mgr.y (mgr.14152) 9 : audit [DBG] from='client.14166 v1:172.21.15.42:0/2439384700' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:14.948175+0000 mgr.y (mgr.14152) 9 : audit [DBG] from='client.14166 v1:172.21.15.42:0/2439384700' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:14.951049+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:14.951049+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:15.114803+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:16.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:15 smithi042 bash[22706]: audit 2024-09-20T19:08:15.114803+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:17 smithi042 bash[22706]: audit 2024-09-20T19:08:16.223619+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:17 smithi042 bash[22706]: audit 2024-09-20T19:08:16.223619+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:17 smithi042 bash[22706]: audit 2024-09-20T19:08:17.142843+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:17 smithi042 bash[22706]: audit 2024-09-20T19:08:17.142843+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:17.817 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-20T19:08:17.817 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial admin user... 2024-09-20T19:08:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:18 smithi042 bash[22706]: audit 2024-09-20T19:08:17.139654+0000 mgr.y (mgr.14152) 10 : audit [DBG] from='client.14168 v1:172.21.15.42:0/3858376844' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:18 smithi042 bash[22706]: audit 2024-09-20T19:08:17.139654+0000 mgr.y (mgr.14152) 10 : audit [DBG] from='client.14168 v1:172.21.15.42:0/3858376844' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:19.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:19 smithi042 bash[22706]: audit 2024-09-20T19:08:19.382768+0000 mgr.y (mgr.14152) 11 : audit [DBG] from='client.14170 v1:172.21.15.42:0/3936019995' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:19.911 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:19 smithi042 bash[22706]: audit 2024-09-20T19:08:19.382768+0000 mgr.y (mgr.14152) 11 : audit [DBG] from='client.14170 v1:172.21.15.42:0/3936019995' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:20.316 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$NDOFVnY2QPg2wajt1fSB3.bSyfiJNuOhnlY1VouQCPos6VzchAgES", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726859299, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T19:08:20.316 INFO:teuthology.orchestra.run.smithi042.stdout:Fetching dashboard port number... 2024-09-20T19:08:21.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:20 smithi042 bash[22706]: audit 2024-09-20T19:08:19.685861+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:21.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:20 smithi042 bash[22706]: audit 2024-09-20T19:08:19.685861+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:21.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:21 smithi042 bash[22706]: cluster 2024-09-20T19:08:20.691236+0000 mon.a (mon.0) 101 : cluster [DBG] mgrmap e14: y(active, since 12s) 2024-09-20T19:08:21.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:21 smithi042 bash[22706]: cluster 2024-09-20T19:08:20.691236+0000 mon.a (mon.0) 101 : cluster [DBG] mgrmap e14: y(active, since 12s) 2024-09-20T19:08:22.792 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T19:08:22.792 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld does not appear to be present 2024-09-20T19:08:22.792 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T19:08:22.794 INFO:teuthology.orchestra.run.smithi042.stdout:Ceph Dashboard is now available at: 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout: URL: https://smithi042.front.sepia.ceph.com:8443/ 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout: User: admin 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout: Password: g291owldc6 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:22.795 INFO:teuthology.orchestra.run.smithi042.stdout:Saving cluster configuration to /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config directory 2024-09-20T19:08:23.015 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:22 smithi042 bash[22706]: audit 2024-09-20T19:08:21.754028+0000 mon.a (mon.0) 102 : audit [DBG] from='client.? v1:172.21.15.42:0/874323524' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-20T19:08:23.015 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:22 smithi042 bash[22706]: audit 2024-09-20T19:08:21.754028+0000 mon.a (mon.0) 102 : audit [DBG] from='client.? v1:172.21.15.42:0/874323524' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.170 INFO:teuthology.orchestra.run.smithi042.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: ceph telemetry on 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout:For more information see: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:26.171 INFO:teuthology.orchestra.run.smithi042.stdout:Bootstrap complete. 2024-09-20T19:08:26.222 INFO:tasks.cephadm:Fetching config... 2024-09-20T19:08:26.223 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:08:26.223 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T19:08:26.228 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T19:08:26.228 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:08:26.228 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T19:08:26.277 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T19:08:26.278 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:08:26.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T19:08:26.333 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T19:08:26.333 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:08:26.333 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T19:08:26.381 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T19:08:26.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqS1CrxupPiP5+HSpH4JR/ZgGnsoOiPeE2WqdXFtn3szc81AhWI9dDMP6EfEbsN69jx2bY5mkOYP3fEHgK7XPvo8d3x7XTdtK69wEx7fLwUUho91W8P3Rc7me2TyM46QX3bVSJoAQ9rxWyWKlRvnKcHmloK5BhDeMdTtMeNshNYNofotNz3zdPnFgoVOl2DC4dlqAwRic8hM6itBxcq3M0JjtOhRlF7qzXNGUo+VtPIfwr3gSU2rMrtPcQXrTrEenKJUptMqmGxAMhFJGBxxe9h3vTGtvbpZGNixZZj7aS7QMYEoysRbBztldC6KhxbrJXP8zNK7HxMOsRdMU0sZePVICaRdz0VhcZKqSVcQ+WtpZNZv13qYZjzoBHRXaEcEYwzRJpozunrpwPbswAsyAdmqo88cylUknniLR+jWGpuvCUcEvGfDXomuzi+7+I5lKTuWKVU5S0OwzyIXM01ahV1jnzPknVJfVpsGbqNy6oVXdJwtEAQ9ptsT4bYGqV8Qc= ceph-49cb5aae-7783-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:08:26.447 INFO:teuthology.orchestra.run.smithi042.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqS1CrxupPiP5+HSpH4JR/ZgGnsoOiPeE2WqdXFtn3szc81AhWI9dDMP6EfEbsN69jx2bY5mkOYP3fEHgK7XPvo8d3x7XTdtK69wEx7fLwUUho91W8P3Rc7me2TyM46QX3bVSJoAQ9rxWyWKlRvnKcHmloK5BhDeMdTtMeNshNYNofotNz3zdPnFgoVOl2DC4dlqAwRic8hM6itBxcq3M0JjtOhRlF7qzXNGUo+VtPIfwr3gSU2rMrtPcQXrTrEenKJUptMqmGxAMhFJGBxxe9h3vTGtvbpZGNixZZj7aS7QMYEoysRbBztldC6KhxbrJXP8zNK7HxMOsRdMU0sZePVICaRdz0VhcZKqSVcQ+WtpZNZv13qYZjzoBHRXaEcEYwzRJpozunrpwPbswAsyAdmqo88cylUknniLR+jWGpuvCUcEvGfDXomuzi+7+I5lKTuWKVU5S0OwzyIXM01ahV1jnzPknVJfVpsGbqNy6oVXdJwtEAQ9ptsT4bYGqV8Qc= ceph-49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:08:26.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqS1CrxupPiP5+HSpH4JR/ZgGnsoOiPeE2WqdXFtn3szc81AhWI9dDMP6EfEbsN69jx2bY5mkOYP3fEHgK7XPvo8d3x7XTdtK69wEx7fLwUUho91W8P3Rc7me2TyM46QX3bVSJoAQ9rxWyWKlRvnKcHmloK5BhDeMdTtMeNshNYNofotNz3zdPnFgoVOl2DC4dlqAwRic8hM6itBxcq3M0JjtOhRlF7qzXNGUo+VtPIfwr3gSU2rMrtPcQXrTrEenKJUptMqmGxAMhFJGBxxe9h3vTGtvbpZGNixZZj7aS7QMYEoysRbBztldC6KhxbrJXP8zNK7HxMOsRdMU0sZePVICaRdz0VhcZKqSVcQ+WtpZNZv13qYZjzoBHRXaEcEYwzRJpozunrpwPbswAsyAdmqo88cylUknniLR+jWGpuvCUcEvGfDXomuzi+7+I5lKTuWKVU5S0OwzyIXM01ahV1jnzPknVJfVpsGbqNy6oVXdJwtEAQ9ptsT4bYGqV8Qc= ceph-49cb5aae-7783-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:08:26.481 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqS1CrxupPiP5+HSpH4JR/ZgGnsoOiPeE2WqdXFtn3szc81AhWI9dDMP6EfEbsN69jx2bY5mkOYP3fEHgK7XPvo8d3x7XTdtK69wEx7fLwUUho91W8P3Rc7me2TyM46QX3bVSJoAQ9rxWyWKlRvnKcHmloK5BhDeMdTtMeNshNYNofotNz3zdPnFgoVOl2DC4dlqAwRic8hM6itBxcq3M0JjtOhRlF7qzXNGUo+VtPIfwr3gSU2rMrtPcQXrTrEenKJUptMqmGxAMhFJGBxxe9h3vTGtvbpZGNixZZj7aS7QMYEoysRbBztldC6KhxbrJXP8zNK7HxMOsRdMU0sZePVICaRdz0VhcZKqSVcQ+WtpZNZv13qYZjzoBHRXaEcEYwzRJpozunrpwPbswAsyAdmqo88cylUknniLR+jWGpuvCUcEvGfDXomuzi+7+I5lKTuWKVU5S0OwzyIXM01ahV1jnzPknVJfVpsGbqNy6oVXdJwtEAQ9ptsT4bYGqV8Qc= ceph-49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:08:26.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T19:08:26.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:26 smithi042 bash[22706]: audit 2024-09-20T19:08:25.466286+0000 mon.a (mon.0) 103 : audit [INF] from='client.? v1:172.21.15.42:0/2043197379' entity='client.admin' 2024-09-20T19:08:26.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:26 smithi042 bash[22706]: audit 2024-09-20T19:08:25.466286+0000 mon.a (mon.0) 103 : audit [INF] from='client.? v1:172.21.15.42:0/2043197379' entity='client.admin' 2024-09-20T19:08:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:28 smithi042 bash[22706]: cluster 2024-09-20T19:08:27.729942+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:28 smithi042 bash[22706]: cluster 2024-09-20T19:08:27.729942+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:30.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:30.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:30 smithi042 bash[22706]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:31.584 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:31 smithi042 bash[22706]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:31.584 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:31 smithi042 bash[22706]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:08:33.582 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:33 smithi042 bash[22706]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:33.582 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:33 smithi042 bash[22706]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:34.579 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T19:08:34.579 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:35.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:34 smithi042 bash[22706]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:36 smithi042 bash[22706]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:36 smithi042 bash[22706]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:39.315 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:08:39.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:38 smithi042 bash[22706]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:39.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:38 smithi042 bash[22706]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:08:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:08:41.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:08:41.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:40 smithi042 bash[22706]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:08:41.511 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-09-20T19:08:41.511 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:08:41.511 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-09-20T19:08:41.517 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:08:41.518 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:08:41.563 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-09-20T19:08:41.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch host add smithi162 2024-09-20T19:08:42.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:42.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:42.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:42.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:42 smithi042 bash[22706]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:43.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:43 smithi042 bash[22706]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:43.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:43 smithi042 bash[22706]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:45.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:45 smithi042 bash[22706]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:45.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:45 smithi042 bash[22706]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:46.299 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:08:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:47 smithi042 bash[22706]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:47 smithi042 bash[22706]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:48 smithi042 bash[22706]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:48 smithi042 bash[22706]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:48 smithi042 bash[22706]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:48 smithi042 bash[22706]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:08:50.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:49 smithi042 bash[22706]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:08:50.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:49 smithi042 bash[22706]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:08:51.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:50 smithi042 bash[22706]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:51.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:50 smithi042 bash[22706]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:52.761 INFO:teuthology.orchestra.run.smithi042.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-09-20T19:08:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:52 smithi042 bash[22706]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:52 smithi042 bash[22706]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:53.412 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T19:08:54.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:54.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:54.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:08:54.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:08:54.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:54.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:08:54.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:54.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:53 smithi042 bash[22706]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:55 smithi042 bash[22706]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:55 smithi042 bash[22706]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:55 smithi042 bash[22706]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:55 smithi042 bash[22706]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:57 smithi042 bash[22706]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:57 smithi042 bash[22706]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:57 smithi042 bash[22706]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:57 smithi042 bash[22706]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:08:58.147 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:08:59.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:59 smithi042 bash[22706]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:59.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:08:59 smithi042 bash[22706]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:08:59.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:08:59.541 INFO:teuthology.orchestra.run.smithi042.stdout:[{"addr": "172.21.15.42", "hostname": "smithi042", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-09-20T19:09:00.255 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T19:09:00.256 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T19:09:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:01 smithi042 bash[22706]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:01 smithi042 bash[22706]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:01 smithi042 bash[22706]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:01 smithi042 bash[22706]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:02.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:02.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:02 smithi042 bash[22706]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:04.995 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:09:05.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:04 smithi042 bash[22706]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:05.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:04 smithi042 bash[22706]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:06.802 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-09-20T19:09:07.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:06 smithi042 bash[22706]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:07.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:06 smithi042 bash[22706]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:07.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:06 smithi042 bash[22706]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:07.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:06 smithi042 bash[22706]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:07.493 INFO:tasks.cephadm:Adding mon.a on smithi042 2024-09-20T19:09:07.494 INFO:tasks.cephadm:Adding mon.c on smithi042 2024-09-20T19:09:07.494 INFO:tasks.cephadm:Adding mon.b on smithi162 2024-09-20T19:09:07.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b' 2024-09-20T19:09:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:07 smithi042 bash[22706]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:07 smithi042 bash[22706]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:07 smithi042 bash[22706]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:08.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:07 smithi042 bash[22706]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:08 smithi042 bash[22706]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:09.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:08 smithi042 bash[22706]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:10.055 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-09-20T19:09:10.621 DEBUG:teuthology.orchestra.run.smithi042:mon.c> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T19:09:10.624 DEBUG:teuthology.orchestra.run.smithi162:mon.b> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T19:09:10.626 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:09:10.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:09:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:11.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:11.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:11.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:11.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:11.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:11 smithi042 bash[22706]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:11.886 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:13.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:13 smithi042 bash[22706]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:13.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:13 smithi042 bash[22706]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:13.894 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:09:13.894 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","modified":"2024-09-20T19:06:55.589722Z","created":"2024-09-20T19:06:55.589722Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:09:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-09-20T19:09:14.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:14 smithi042 bash[22706]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:14.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:14 smithi042 bash[22706]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:15.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:15 smithi042 bash[22706]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:15.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:15 smithi042 bash[22706]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:15.745 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:09:15.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:09:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:15 smithi162 systemd[1]: Started Ceph mon.b for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:09:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:09:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:17 smithi042 bash[22706]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.656+0000 7fa09899dc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.656+0000 7fa09899dc80 0 ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev), process ceph-mon, pid 7 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.656+0000 7fa09899dc80 0 pidfile_write: ignore empty --pid-file 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.656+0000 7fa09899dc80 0 load: jerasure load: lrc 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: RocksDB version: 7.9.2 2024-09-20T19:09:17.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Git sha 0 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Compile date 2024-09-20 10:48:39 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: DB SUMMARY 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: DB Session ID: AM4EWP9MML0MXLDA8UXD 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: CURRENT file: CURRENT 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.error_if_exists: 0 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.create_if_missing: 0 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.paranoid_checks: 1 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:09:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.env: 0x564e839db120 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.info_log: 0x564e85bcb680 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.statistics: (nil) 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.use_fsync: 0 2024-09-20T19:09:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_log_file_size: 0 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_fallocate: 1 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:09:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.use_direct_reads: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.db_log_dir: 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.wal_dir: 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:09:17.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.write_buffer_manager: 0x564e85bcfb80 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.unordered_write: 0 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:09:17.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.row_cache: None 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.wal_filter: None 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.two_write_queues: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.wal_compression: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.atomic_flush: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.log_readahead_size: 0 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:09:17.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_background_jobs: 2 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_background_compactions: -1 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_subcompactions: 1 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:09:17.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_open_files: -1 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_background_flushes: -1 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Compression algorithms supported: 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kZSTD supported: 0 2024-09-20T19:09:17.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kXpressCompression supported: 0 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kBZip2Compression supported: 0 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kLZ4Compression supported: 1 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kZlibCompression supported: 1 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: kSnappyCompression supported: 1 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.merge_operator: 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_filter: None 2024-09-20T19:09:17.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564e85bcb2a0) 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cache_index_and_filter_blocks: 1 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: pin_top_level_index_and_filter: 1 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: index_type: 0 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: data_block_index_type: 0 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: index_shortening: 1 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: checksum: 4 2024-09-20T19:09:17.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: no_block_cache: 0 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_cache: 0x564e85bc1610 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_cache_name: BinnedLRUCache 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_cache_options: 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: capacity : 536870912 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: num_shard_bits : 4 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: strict_capacity_limit : 0 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: high_pri_pool_ratio: 0.000 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_cache_compressed: (nil) 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: persistent_cache: (nil) 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_size: 4096 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_size_deviation: 10 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_restart_interval: 16 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: index_block_restart_interval: 1 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: metadata_block_size: 4096 2024-09-20T19:09:17.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: partition_filters: 0 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: use_delta_encoding: 1 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: filter_policy: bloomfilter 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: whole_key_filtering: 1 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: verify_compression: 0 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: read_amp_bytes_per_bit: 0 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: format_version: 5 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: enable_index_compression: 1 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: block_align: 0 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: max_auto_readahead_size: 262144 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: prepopulate_block_cache: 0 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: initial_auto_readahead_size: 8192 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: num_file_reads_for_auto_readahead: 2 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:09:17.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression: NoCompression 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.num_levels: 7 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:09:17.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:09:17.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:09:17.794 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:09:17.795 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:09:17.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.inplace_update_support: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.bloom_locality: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.max_successive_merges: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.ttl: 2592000 2024-09-20T19:09:17.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enable_blob_files: false 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.min_blob_size: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T19:09:17.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7d001ed3-afab-4b65-8252-7aa378ab7ae3 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357662997, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357663431, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726859357, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7d001ed3-afab-4b65-8252-7aa378ab7ae3", "db_session_id": "AM4EWP9MML0MXLDA8UXD", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357663555, "job": 1, "event": "recovery_finished"} 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.660+0000 7fa09899dc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564e85beee00 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 4 rocksdb: DB pointer 0x564e85cf2000 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa08e765640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:09:17.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa08e765640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: ** DB Stats ** 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: ** Compaction Stats [default] ** 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:17.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: ** Compaction Stats [default] ** 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:09:17.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Block cache BinnedLRUCache@0x564e85bc1610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 0 using public_addrv v1:172.21.15.162:6789/0 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 0 starting mon.b rank -1 at public addrs v1:172.21.15.162:6789/0 at bind addrs v1:172.21.15.162:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.664+0000 7fa09899dc80 1 mon.b@-1(???) e0 preinit fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).mds e1 new map 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).mds e1 print_map 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: e1 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: btime 2024-09-20T19:07:00:852215+0000 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: legacy client fscid: -1 2024-09-20T19:09:17.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: No filesystems configured 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa08cf62640 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.690003) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa095f74640 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa095f74640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357690103, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1173, "num_deletes": 24, "total_data_size": 2577679, "memory_usage": 2671952, "flush_reason": "Manual Compaction"} 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa095f74640 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:09:17.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.688+0000 7fa09176b640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:09:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.812 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:17.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa09176b640 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-20T19:09:17.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357698450, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2292858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1178, "table_properties": {"data_size": 2286989, "index_size": 2621, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 25540, "raw_average_key_size": 29, "raw_value_size": 2269134, "raw_average_value_size": 2632, "num_data_blocks": 123, "num_entries": 862, "num_filter_entries": 862, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726859357, "oldest_key_time": 1726859357, "file_creation_time": 1726859357, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7d001ed3-afab-4b65-8252-7aa378ab7ae3", "db_session_id": "AM4EWP9MML0MXLDA8UXD", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 8473 microseconds, and 5052 cpu microseconds. 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.698484) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2292858 bytes OK 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.698503) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.698632) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.698645) EVENT_LOG_v1 {"time_micros": 1726859357698641, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.698658) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2569504, prev total WAL file size 2569645, number of live WAL files 2. 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa095f74640 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa096775640 4 rocksdb: (Original Log Time 2024/09/20-19:09:17.699015) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003232' seq:0, type:0; will stop at (end) 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa096775640 4 rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa096775640 4 rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(2239KB) 8(1613B)] 2024-09-20T19:09:17.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:17 smithi162 bash[24557]: debug 2024-09-20T19:09:17.696+0000 7fa096775640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859357699079, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2294471, "oldest_snapshot_seqno": -1} 2024-09-20T19:09:20.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.401 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.683 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:20 smithi042 systemd[1]: Started Ceph mon.c for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:09:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.763+0000 7fca2e533c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.763+0000 7fca2e533c80 0 ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev), process ceph-mon, pid 7 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.763+0000 7fca2e533c80 0 pidfile_write: ignore empty --pid-file 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 0 load: jerasure load: lrc 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: RocksDB version: 7.9.2 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Git sha 0 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Compile date 2024-09-20 10:48:39 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: DB SUMMARY 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: DB Session ID: PDMD4GX8DGPFX80IH8RV 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: CURRENT file: CURRENT 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:09:22.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.error_if_exists: 0 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.create_if_missing: 0 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.paranoid_checks: 1 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.env: 0x560677beb120 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.info_log: 0x560678c01680 2024-09-20T19:09:22.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.statistics: (nil) 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.use_fsync: 0 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_log_file_size: 0 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:09:22.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_fallocate: 1 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.use_direct_reads: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.db_log_dir: 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.wal_dir: 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:09:22.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.write_buffer_manager: 0x560678c05b80 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:09:22.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.unordered_write: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.row_cache: None 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.wal_filter: None 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.two_write_queues: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.wal_compression: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.atomic_flush: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:09:22.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.log_readahead_size: 0 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_background_jobs: 2 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_background_compactions: -1 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_subcompactions: 1 2024-09-20T19:09:22.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_open_files: -1 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_background_flushes: -1 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Compression algorithms supported: 2024-09-20T19:09:22.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kZSTD supported: 0 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kXpressCompression supported: 0 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kBZip2Compression supported: 0 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kLZ4Compression supported: 1 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kZlibCompression supported: 1 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: kSnappyCompression supported: 1 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-20T19:09:22.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:09:22.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:09:22.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.merge_operator: 2024-09-20T19:09:22.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_filter: None 2024-09-20T19:09:22.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560678c012a0) 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cache_index_and_filter_blocks: 1 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: pin_top_level_index_and_filter: 1 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: index_type: 0 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: data_block_index_type: 0 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: index_shortening: 1 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: checksum: 4 2024-09-20T19:09:22.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: no_block_cache: 0 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_cache: 0x560678bf7610 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_cache_name: BinnedLRUCache 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_cache_options: 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: capacity : 536870912 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: num_shard_bits : 4 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: strict_capacity_limit : 0 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: high_pri_pool_ratio: 0.000 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_cache_compressed: (nil) 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: persistent_cache: (nil) 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_size: 4096 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_size_deviation: 10 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_restart_interval: 16 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: index_block_restart_interval: 1 2024-09-20T19:09:22.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: metadata_block_size: 4096 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: partition_filters: 0 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: use_delta_encoding: 1 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: filter_policy: bloomfilter 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: whole_key_filtering: 1 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: verify_compression: 0 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: read_amp_bytes_per_bit: 0 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: format_version: 5 2024-09-20T19:09:22.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: enable_index_compression: 1 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: block_align: 0 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: max_auto_readahead_size: 262144 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: prepopulate_block_cache: 0 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: initial_auto_readahead_size: 8192 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: num_file_reads_for_auto_readahead: 2 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression: NoCompression 2024-09-20T19:09:22.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.num_levels: 7 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:09:22.199 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:09:22.200 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:09:22.201 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:09:22.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.inplace_update_support: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.bloom_locality: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.max_successive_merges: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:09:22.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.ttl: 2592000 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enable_blob_files: false 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.min_blob_size: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:09:22.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e750f61f-2a2a-46c7-aab1-1b34dd45d01d 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361773914, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.767+0000 7fca2e533c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361774382, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726859361, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e750f61f-2a2a-46c7-aab1-1b34dd45d01d", "db_session_id": "PDMD4GX8DGPFX80IH8RV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361774521, "job": 1, "event": "recovery_finished"} 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560678c24e00 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 4 rocksdb: DB pointer 0x560678d28000 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca242fb640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca242fb640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: ** DB Stats ** 2024-09-20T19:09:22.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: ** Compaction Stats [default] ** 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:22.206 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: ** Compaction Stats [default] ** 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Cumulative compaction: 0.00 GB write, 0.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:09:22.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Interval compaction: 0.00 GB write, 0.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Block cache BinnedLRUCache@0x560678bf7610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 0 using public_addrv v1:172.21.15.42:6790/0 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.771+0000 7fca2e533c80 0 starting mon.c rank -1 at public addrs v1:172.21.15.42:6790/0 at bind addrs v1:172.21.15.42:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.775+0000 7fca2e533c80 1 mon.c@-1(???) e0 preinit fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).mds e1 new map 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).mds e1 print_map 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: e1 2024-09-20T19:09:22.208 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: btime 2024-09-20T19:07:00:852215+0000 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: legacy client fscid: -1 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: No filesystems configured 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca22af8640 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.795328) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca2bb0a640 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca2bb0a640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361795428, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1177, "num_deletes": 24, "total_data_size": 2578412, "memory_usage": 2672864, "flush_reason": "Manual Compaction"} 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca2bb0a640 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T19:09:22.209 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.791+0000 7fca27301640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.285934+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.289833+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.291405+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.298307+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.312631+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:29.318021+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:29.730366+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:31.730870+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:33.731362+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.940989+0000 mon.a (mon.0) 110 : audit [INF] from='client.? v1:172.21.15.42:0/3167087663' entity='client.admin' 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.955559+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:33.960693+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:35.731825+0000 mgr.y (mgr.14152) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:37.732251+0000 mgr.y (mgr.14152) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:39.732674+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.795159+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.796477+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.798937+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.799622+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:40.791466+0000 mgr.y (mgr.14152) 19 : audit [DBG] from='client.14178 v1:172.21.15.42:0/3352485956' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.800829+0000 mgr.y (mgr.14152) 20 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.865933+0000 mgr.y (mgr.14152) 21 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.916851+0000 mgr.y (mgr.14152) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:40.968861+0000 mgr.y (mgr.14152) 23 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.023916+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.213 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.026564+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:41.029598+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:41.733084+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:43.733491+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:45.733941+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:47.734318+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:47.889786+0000 mgr.y (mgr.14152) 28 : audit [DBG] from='client.14180 v1:172.21.15.42:0/65180960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:09:22.214 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:49.162949+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying cephadm binary to smithi162 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:49.734702+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:51.735125+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:52.756608+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:08:52.757494+0000 mgr.y (mgr.14152) 32 : cephadm [INF] Added host smithi162 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:52.758241+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:53.267695+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:53.735551+0000 mgr.y (mgr.14152) 33 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:54.797835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:55.735956+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:56.033958+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:57.736361+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:08:59.536536+0000 mgr.y (mgr.14152) 36 : audit [DBG] from='client.14182 v1:172.21.15.42:0/2224605266' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:08:59.736750+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.217 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.621659+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.626801+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.632730+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.637887+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.639646+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.641750+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:01.643411+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:22.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.645849+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:01.737132+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.760292+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.860348+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:01.951521+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.046307+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.051648+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:02.057829+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:03.737543+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:05.737919+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:06.425417+0000 mon.a (mon.0) 135 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:06.797979+0000 mon.a (mon.0) 136 : audit [INF] from='client.? v1:172.21.15.42:0/1093209346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:06.800760+0000 mon.a (mon.0) 137 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:07.738368+0000 mgr.y (mgr.14152) 45 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:09.738798+0000 mgr.y (mgr.14152) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.039883+0000 mgr.y (mgr.14152) 47 : audit [DBG] from='client.14186 v1:172.21.15.162:0/4042576933' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:10.045130+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Saving service mon spec with placement smithi042:[v1:172.21.15.42:6789]=a;smithi042:[v1:172.21.15.42:6790]=c;smithi162:[v1:172.21.15.162:6789]=b;count:3 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.050403+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.051930+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.055283+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.056928+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.064519+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:22.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.068249+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:10.069995+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:10.071935+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Deploying daemon mon.b on smithi162 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:11.739206+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:13.891131+0000 mon.a (mon.0) 145 : audit [DBG] from='client.? v1:172.21.15.162:0/1167950424' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:13.739606+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cluster 2024-09-20T19:09:15.739996+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.039493+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.044790+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.050659+0000 mon.a (mon.0) 148 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.051949+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: audit 2024-09-20T19:09:16.053459+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: cephadm 2024-09-20T19:09:16.055286+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.c on smithi042 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca27301640 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-20T19:09:22.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361801743, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2293517, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1182, "table_properties": {"data_size": 2287641, "index_size": 2628, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 25573, "raw_average_key_size": 29, "raw_value_size": 2269756, "raw_average_value_size": 2627, "num_data_blocks": 123, "num_entries": 864, "num_filter_entries": 864, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726859361, "oldest_key_time": 1726859361, "file_creation_time": 1726859361, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e750f61f-2a2a-46c7-aab1-1b34dd45d01d", "db_session_id": "PDMD4GX8DGPFX80IH8RV", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 6427 microseconds, and 5217 cpu microseconds. 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.801776) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2293517 bytes OK 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.801804) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.801955) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.801968) EVENT_LOG_v1 {"time_micros": 1726859361801964, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.801980) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.795+0000 7fca2bb0a640 4 rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2570209, prev total WAL file size 2570350, number of live WAL files 2. 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.799+0000 7fca2bb0a640 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.799+0000 7fca2c30b640 4 rocksdb: (Original Log Time 2024/09/20-19:09:21.802388) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003232' seq:0, type:0; will stop at (end) 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.799+0000 7fca2c30b640 4 rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.799+0000 7fca2c30b640 4 rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(2239KB) 8(1613B)] 2024-09-20T19:09:22.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:21 smithi042 bash[32075]: debug 2024-09-20T19:09:21.799+0000 7fca2c30b640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726859361802444, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2295130, "oldest_snapshot_seqno": -1} 2024-09-20T19:09:23.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:09:23.154 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","modified":"2024-09-20T19:09:17.752473Z","created":"2024-09-20T19:06:55.589722Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T19:09:23.154 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:23.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:23.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:22 smithi162 bash[24557]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:23.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:23.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:23.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:22 smithi042 bash[22706]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:24.944 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:09:24.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:09:25.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:09:24 smithi042 bash[22958]: debug 2024-09-20T19:09:24.747+0000 7f58ad387640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-20T19:09:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:29.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:29.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:29.187 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.188 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:28 smithi042 bash[22706]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:29.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:29.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:29.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:29.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:29.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:28 smithi162 bash[24557]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:30.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:30.186 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:29 smithi042 bash[22706]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:30.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:30.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:30.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:30.312 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:29 smithi162 bash[24557]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:30.332 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-09-20T19:09:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:09:30.332 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","modified":"2024-09-20T19:09:23.810145Z","created":"2024-09-20T19:06:55.589722Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T19:09:30.961 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T19:09:30.961 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:17.740322+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:17.755386+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:17.755532+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:17.755754+0000 mon.a (mon.0) 154 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:18.749826+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:19.740821+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:19.750324+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:19.756250+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:20.749972+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:21.741204+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:21.749814+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:21.807560+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.750281+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.797319+0000 mon.a (mon.0) 161 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803455+0000 mon.a (mon.0) 162 : cluster [DBG] monmap epoch 2 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803485+0000 mon.a (mon.0) 163 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803499+0000 mon.a (mon.0) 164 : cluster [DBG] last_changed 2024-09-20T19:09:17.752473+0000 2024-09-20T19:09:31.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803514+0000 mon.a (mon.0) 165 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803527+0000 mon.a (mon.0) 166 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803544+0000 mon.a (mon.0) 167 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803569+0000 mon.a (mon.0) 168 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.803587+0000 mon.a (mon.0) 169 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:31.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.822990+0000 mon.a (mon.0) 170 : cluster [DBG] fsmap 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823039+0000 mon.a (mon.0) 171 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823322+0000 mon.a (mon.0) 172 : cluster [DBG] mgrmap e14: y(active, since 75s) 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:31.187 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:22.823551+0000 mon.a (mon.0) 173 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.829956+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.830955+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.837127+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.861342+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:22.868092+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.741586+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:31.188 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.816295+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.816705+0000 mon.a (mon.0) 185 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:23.818740+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.835644+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:23.835929+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:24.807629+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:25.741970+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:25.807824+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.189 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:26.807977+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:27.742318+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:27.808050+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.808164+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.835627+0000 mon.a (mon.0) 193 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841639+0000 mon.a (mon.0) 194 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.190 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841678+0000 mon.a (mon.0) 195 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841712+0000 mon.a (mon.0) 196 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841740+0000 mon.a (mon.0) 197 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841773+0000 mon.a (mon.0) 198 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841806+0000 mon.a (mon.0) 199 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841836+0000 mon.a (mon.0) 200 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841864+0000 mon.a (mon.0) 201 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.841897+0000 mon.a (mon.0) 202 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843038+0000 mon.a (mon.0) 203 : cluster [DBG] fsmap 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843105+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843410+0000 mon.a (mon.0) 205 : cluster [DBG] mgrmap e14: y(active, since 81s) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.843707+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861171+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861228+0000 mon.a (mon.0) 208 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: cluster 2024-09-20T19:09:28.861277+0000 mon.a (mon.0) 209 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.42:6790/0 is down (out of quorum) 2024-09-20T19:09:31.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.865801+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.871123+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.876780+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:28.882519+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.534772+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:31.193 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.536128+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.720998+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.723621+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.727309+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.734507+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.739208+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.194 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.764034+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.770151+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.777688+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.785139+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.786050+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.787118+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.788118+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:31.195 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.196 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:30 smithi042 bash[32075]: audit 2024-09-20T19:09:29.808207+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:31.818 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.818 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.818 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:31.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[32075]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:31.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:31.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:31.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:31.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:31.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:31.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:31 smithi042 bash[22706]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:25.811860+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.854032+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.854838+0000 mon.a (mon.0) 230 : cluster [INF] mon.a calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.856142+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.881227+0000 mon.a (mon.0) 231 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887421+0000 mon.a (mon.0) 232 : cluster [DBG] monmap epoch 3 2024-09-20T19:09:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887453+0000 mon.a (mon.0) 233 : cluster [DBG] fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887480+0000 mon.a (mon.0) 234 : cluster [DBG] last_changed 2024-09-20T19:09:23.810145+0000 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887509+0000 mon.a (mon.0) 235 : cluster [DBG] created 2024-09-20T19:06:55.589722+0000 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887541+0000 mon.a (mon.0) 236 : cluster [DBG] min_mon_release 19 (squid) 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887570+0000 mon.a (mon.0) 237 : cluster [DBG] election_strategy: 1 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887598+0000 mon.a (mon.0) 238 : cluster [DBG] 0: v1:172.21.15.42:6789/0 mon.a 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:32.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887628+0000 mon.a (mon.0) 239 : cluster [DBG] 1: v1:172.21.15.162:6789/0 mon.b 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.887658+0000 mon.a (mon.0) 240 : cluster [DBG] 2: v1:172.21.15.42:6790/0 mon.c 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.888790+0000 mon.a (mon.0) 241 : cluster [DBG] fsmap 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.888860+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889172+0000 mon.a (mon.0) 243 : cluster [DBG] mgrmap e14: y(active, since 83s) 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:32.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889455+0000 mon.a (mon.0) 244 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T19:09:32.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:32.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.889492+0000 mon.a (mon.0) 245 : cluster [INF] Cluster is now healthy 2024-09-20T19:09:32.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:32.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:31 smithi162 bash[24557]: cluster 2024-09-20T19:09:30.895437+0000 mon.a (mon.0) 246 : cluster [INF] overall HEALTH_OK 2024-09-20T19:09:33.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:33.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:32 smithi162 bash[24557]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[22706]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cluster 2024-09-20T19:09:31.743182+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:31.808681+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.316946+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.322189+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cephadm 2024-09-20T19:09:32.323569+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.324071+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.325542+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: audit 2024-09-20T19:09:32.326970+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:33.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:32 smithi042 bash[32075]: cephadm 2024-09-20T19:09:32.328691+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.c on smithi042 2024-09-20T19:09:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:34 smithi162 bash[24557]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[22706]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: cluster 2024-09-20T19:09:33.743623+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.680310+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.685333+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.686629+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.687542+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:34 smithi042 bash[32075]: audit 2024-09-20T19:09:34.688294+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:09:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:35 smithi162 bash[24557]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:35 smithi162 bash[24557]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:35 smithi162 bash[24557]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:35 smithi162 bash[24557]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[22706]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[22706]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[22706]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[22706]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[32075]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[32075]: cephadm 2024-09-20T19:09:34.686287+0000 mgr.y (mgr.14152) 71 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[32075]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:36.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:35 smithi042 bash[32075]: cephadm 2024-09-20T19:09:34.689214+0000 mgr.y (mgr.14152) 72 : cephadm [INF] Reconfiguring daemon mon.b on smithi162 2024-09-20T19:09:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:36 smithi162 bash[24557]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:36 smithi162 bash[24557]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.111 INFO:teuthology.orchestra.run.smithi042.stdout:# minimal ceph.conf for 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:37.111 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-09-20T19:09:37.111 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:09:37.111 INFO:teuthology.orchestra.run.smithi042.stdout: mon_host = 172.21.15.42:6789/0 172.21.15.162:6789/0 v1:172.21.15.42:6790/0 2024-09-20T19:09:37.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:36 smithi042 bash[32075]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:36 smithi042 bash[32075]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.123 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:36 smithi042 bash[22706]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.123 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:36 smithi042 bash[22706]: cluster 2024-09-20T19:09:35.744061+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:37.807 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T19:09:37.807 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:09:37.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:09:37.822 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:09:37.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:37.877 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:09:37.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:09:37.892 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:09:37.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:09:37.945 INFO:tasks.cephadm:Adding mgr.y on smithi042 2024-09-20T19:09:37.945 INFO:tasks.cephadm:Adding mgr.x on smithi162 2024-09-20T19:09:37.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi042=y;smithi162=x' 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[22706]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:37 smithi042 bash[32075]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.855680+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.862897+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.864575+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.866387+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.867263+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:36.874536+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:38.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:37 smithi162 bash[24557]: audit 2024-09-20T19:09:37.107713+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.42:0/2910430960' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:38 smithi042 bash[32075]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:38 smithi042 bash[32075]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:39.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:38 smithi042 bash[22706]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:39.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:38 smithi042 bash[22706]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:39.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:38 smithi162 bash[24557]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:39.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:38 smithi162 bash[24557]: cluster 2024-09-20T19:09:37.744515+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:40 smithi042 bash[22706]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:40 smithi042 bash[22706]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:40 smithi042 bash[32075]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:40 smithi042 bash[32075]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:40 smithi162 bash[24557]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:40 smithi162 bash[24557]: cluster 2024-09-20T19:09:39.745056+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:41.703 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:09:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:42 smithi162 bash[24557]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:42 smithi162 bash[24557]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.070 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-09-20T19:09:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:42 smithi042 bash[22706]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:42 smithi042 bash[22706]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:42 smithi042 bash[32075]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:42 smithi042 bash[32075]: cluster 2024-09-20T19:09:41.745526+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:43.668 DEBUG:teuthology.orchestra.run.smithi162:mgr.x> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T19:09:43.670 DEBUG:tasks.cephadm:set 0 configs 2024-09-20T19:09:43.670 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T19:09:43.671 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:09:43.671 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:09:43.678 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:09:43.678 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-09-20T19:09:43.726 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:09:43.726 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:43.726 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-09-20T19:09:43.726 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:43.727 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-20 18:59:50.392493737 +0000 2024-09-20T19:09:43.727 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-20 18:59:50.136496023 +0000 2024-09-20T19:09:43.727 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-20 18:59:50.136496023 +0000 2024-09-20T19:09:43.727 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-20T19:09:43.727 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:09:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-20T19:09:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-20T19:09:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000336724 s, 1.5 MB/s 2024-09-20T19:09:43.782 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:09:43.831 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-09-20T19:09:43.877 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-20 18:59:50.872489448 +0000 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-20 18:59:50.616491736 +0000 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-20 18:59:50.616491736 +0000 2024-09-20T19:09:43.878 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-20T19:09:43.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:09:43.932 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-20T19:09:43.933 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-20T19:09:43.933 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000209731 s, 2.4 MB/s 2024-09-20T19:09:43.934 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:09:43.983 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-09-20T19:09:44.029 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-20 18:59:51.356485125 +0000 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-20 18:59:51.092487483 +0000 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-20 18:59:51.092487483 +0000 2024-09-20T19:09:44.030 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-20T19:09:44.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:09:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-20T19:09:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-20T19:09:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000252672 s, 2.0 MB/s 2024-09-20T19:09:44.088 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:09:44.140 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-09-20T19:09:44.185 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:09:44.185 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.185 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-20T19:09:44.185 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.186 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-20 18:59:51.812481053 +0000 2024-09-20T19:09:44.186 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-20 18:59:51.576483160 +0000 2024-09-20T19:09:44.186 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-20 18:59:51.576483160 +0000 2024-09-20T19:09:44.186 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-20T19:09:44.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:09:44.244 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-20T19:09:44.244 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-20T19:09:44.244 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.00029323 s, 1.7 MB/s 2024-09-20T19:09:44.245 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:09:44.295 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:09:44.295 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:09:44.301 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:09:44.301 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-20 18:59:47.787616653 +0000 2024-09-20T19:09:44.347 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-20 18:59:47.559618689 +0000 2024-09-20T19:09:44.348 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-20 18:59:47.559618689 +0000 2024-09-20T19:09:44.348 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-20T19:09:44.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:09:44.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:44 smithi162 bash[24557]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.403 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-20T19:09:44.403 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-20T19:09:44.403 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000379735 s, 1.3 MB/s 2024-09-20T19:09:44.403 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:09:44.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.056496+0000 mgr.y (mgr.14152) 77 : audit [DBG] from='client.24112 v1:172.21.15.162:0/2150480937' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi042=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: cephadm 2024-09-20T19:09:43.059627+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Saving service mgr spec with placement smithi042=y;smithi162=x;count:2 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.066196+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.067935+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.071643+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.073387+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.082568+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.085655+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[22706]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.088070+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.094897+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:44.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: audit 2024-09-20T19:09:43.096638+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:44.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:44 smithi042 bash[32075]: cephadm 2024-09-20T19:09:43.098447+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Deploying daemon mgr.x on smithi162 2024-09-20T19:09:44.452 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-20 18:59:48.003614724 +0000 2024-09-20T19:09:44.499 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-20 18:59:47.999614759 +0000 2024-09-20T19:09:44.500 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-20 18:59:47.999614759 +0000 2024-09-20T19:09:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-20T19:09:44.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:09:44.552 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-20T19:09:44.553 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-20T19:09:44.553 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000371012 s, 1.4 MB/s 2024-09-20T19:09:44.554 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:09:44.604 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-09-20T19:09:44.650 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:09:44.650 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.650 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-09-20T19:09:44.651 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.651 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-20 18:59:48.743608113 +0000 2024-09-20T19:09:44.651 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-20 18:59:48.503610258 +0000 2024-09-20T19:09:44.651 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-20 18:59:48.503610258 +0000 2024-09-20T19:09:44.651 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-20T19:09:44.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:09:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-20T19:09:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-20T19:09:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000368074 s, 1.4 MB/s 2024-09-20T19:09:44.710 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:09:44.759 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-20 18:59:48.971606077 +0000 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-20 18:59:48.967606113 +0000 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-20 18:59:48.967606113 +0000 2024-09-20T19:09:44.806 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-20T19:09:44.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:09:44.858 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-20T19:09:44.859 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-20T19:09:44.859 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000264856 s, 1.9 MB/s 2024-09-20T19:09:44.860 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:09:44.908 INFO:tasks.cephadm:Deploying osd.0 on smithi042 with /dev/vg_nvme/lv_4... 2024-09-20T19:09:44.908 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:09:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:45 smithi042 bash[22706]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:45 smithi042 bash[22706]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:45 smithi042 bash[32075]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:45 smithi042 bash[32075]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:45 smithi162 bash[24557]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:45 smithi162 bash[24557]: cluster 2024-09-20T19:09:43.745943+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:45.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:45 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:45.895 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:09:45 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:45.895 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:09:45 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:45.896 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:09:45 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:46.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:46 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:46.220 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:09:46 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:09:46.220 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:09:46 smithi162 systemd[1]: Started Ceph mgr.x for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:47 smithi162 bash[24557]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[22706]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: cluster 2024-09-20T19:09:45.746381+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.162957+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.169725+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.179340+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.186350+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:47.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:47.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:47 smithi042 bash[32075]: audit 2024-09-20T19:09:46.218501+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:49 smithi162 bash[24557]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:49 smithi162 bash[24557]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:49 smithi162 bash[24557]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:49 smithi162 bash[24557]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:09:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[32075]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[32075]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[32075]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:49.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[32075]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:49.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[22706]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[22706]: cluster 2024-09-20T19:09:47.746726+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:49.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[22706]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:49.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:49 smithi042 bash[22706]: audit 2024-09-20T19:09:48.200157+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:51 smithi162 bash[24557]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:51.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:51 smithi162 bash[24557]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:51.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:51 smithi042 bash[22706]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:51.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:51 smithi042 bash[22706]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:51 smithi042 bash[32075]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:51 smithi042 bash[32075]: cluster 2024-09-20T19:09:49.747192+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:52.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[22706]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.750 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.750 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:52 smithi042 bash[32075]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.436906+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.444110+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.445679+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.447355+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.458286+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.493635+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.495420+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:52.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:52 smithi162 bash[24557]: audit 2024-09-20T19:09:51.496763+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:53.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:53 smithi162 bash[24557]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:53.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:53.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[32075]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cephadm 2024-09-20T19:09:51.492945+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cephadm 2024-09-20T19:09:51.498495+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Reconfiguring daemon mgr.y on smithi042 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cluster 2024-09-20T19:09:51.747483+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.055059+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.057361+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.057998+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:09:53.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.059121+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:09:53.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:53.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:53 smithi042 bash[22706]: audit 2024-09-20T19:09:53.059497+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1741301687' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:09:54.258 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:09:54.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi042:vg_nvme/lv_4 2024-09-20T19:09:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[32075]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[32075]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[22706]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:54 smithi042 bash[22706]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:54.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:54 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:54 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.472485+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e15: y(active, since 105s), standbys: x 2024-09-20T19:09:54.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:54 smithi162 bash[24557]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:54.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:54 smithi162 bash[24557]: audit 2024-09-20T19:09:53.473158+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:09:55.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:55 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:55.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:55 smithi042 bash[22706]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:55.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:55 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:55.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:55 smithi042 bash[32075]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:55 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:55 smithi162 bash[24557]: cluster 2024-09-20T19:09:53.747750+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[32075]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:57 smithi042 bash[22706]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: cluster 2024-09-20T19:09:55.748141+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.125626+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.133541+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.136684+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.838826+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.840589+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:57.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:57 smithi162 bash[24557]: audit 2024-09-20T19:09:56.852734+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:09:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:09:59.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:59 smithi042 bash[22706]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:59.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:09:59 smithi042 bash[22706]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:59.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:59 smithi042 bash[32075]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:59.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:09:59 smithi042 bash[32075]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:59.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:59 smithi162 bash[24557]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:09:59.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:09:59 smithi162 bash[24557]: cluster 2024-09-20T19:09:57.748609+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:00 smithi042 bash[32075]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:00 smithi042 bash[32075]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:00.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:00 smithi042 bash[22706]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:00.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:00 smithi042 bash[22706]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:00.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:00 smithi162 bash[24557]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:00.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:00 smithi162 bash[24557]: cluster 2024-09-20T19:10:00.000188+0000 mon.a (mon.0) 296 : cluster [INF] overall HEALTH_OK 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[32075]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:01.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:01 smithi042 bash[22706]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: cluster 2024-09-20T19:09:59.749045+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.498870+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.505416+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:01.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:01 smithi162 bash[24557]: audit 2024-09-20T19:10:00.506555+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:02 smithi042 bash[22706]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:02 smithi042 bash[22706]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:02 smithi042 bash[32075]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:02 smithi042 bash[32075]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:02.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:02 smithi162 bash[24557]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:02.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:02 smithi162 bash[24557]: audit 2024-09-20T19:10:00.494708+0000 mgr.y (mgr.14152) 91 : audit [DBG] from='client.14217 v1:172.21.15.42:0/3644999426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:10:03.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:03 smithi042 bash[22706]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:03.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:03 smithi042 bash[22706]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:03.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:03 smithi042 bash[32075]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:03.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:03 smithi042 bash[32075]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:03.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:03 smithi162 bash[24557]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:03.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:03 smithi162 bash[24557]: cluster 2024-09-20T19:10:01.749435+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:05 smithi042 bash[22706]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:05 smithi042 bash[22706]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:05 smithi042 bash[32075]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:05 smithi042 bash[32075]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:05 smithi162 bash[24557]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:05.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:05 smithi162 bash[24557]: cluster 2024-09-20T19:10:03.749861+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:07 smithi042 bash[22706]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:07 smithi042 bash[22706]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:07 smithi042 bash[32075]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:07 smithi042 bash[32075]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:07 smithi162 bash[24557]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:07.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:07 smithi162 bash[24557]: cluster 2024-09-20T19:10:05.750374+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:09 smithi042 bash[22706]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:09 smithi042 bash[22706]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:09 smithi042 bash[32075]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:09 smithi042 bash[32075]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:09 smithi162 bash[24557]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:09.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:09 smithi162 bash[24557]: cluster 2024-09-20T19:10:07.750908+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[22706]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:10.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:10 smithi042 bash[32075]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.214197+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.42:0/1208856745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.214790+0000 mon.a (mon.0) 300 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"} : dispatch 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.217398+0000 mon.a (mon.0) 301 : audit [INF] from='client.24125 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b"}]': finished 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: cluster 2024-09-20T19:10:09.221960+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:10:10.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.222277+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:10.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:10.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:10 smithi162 bash[24557]: audit 2024-09-20T19:10:09.894564+0000 mon.a (mon.0) 304 : audit [DBG] from='client.? v1:172.21.15.42:0/2482948126' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:10:11.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:11 smithi162 bash[24557]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:11.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:11 smithi162 bash[24557]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:11 smithi042 bash[22706]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:11 smithi042 bash[22706]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:11.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:11 smithi042 bash[32075]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:11.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:11 smithi042 bash[32075]: cluster 2024-09-20T19:10:09.751400+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:13 smithi042 bash[32075]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:13 smithi042 bash[32075]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:13 smithi042 bash[22706]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:13 smithi042 bash[22706]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:13 smithi162 bash[24557]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:13.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:13 smithi162 bash[24557]: cluster 2024-09-20T19:10:11.751962+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:15 smithi162 bash[24557]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:15 smithi162 bash[24557]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:15 smithi042 bash[32075]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:15 smithi042 bash[32075]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:15 smithi042 bash[22706]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:15.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:15 smithi042 bash[22706]: cluster 2024-09-20T19:10:13.752423+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:17 smithi162 bash[24557]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:17 smithi162 bash[24557]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:17 smithi042 bash[22706]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:17 smithi042 bash[22706]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:17 smithi042 bash[32075]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:17.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:17 smithi042 bash[32075]: cluster 2024-09-20T19:10:15.752854+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:19 smithi042 bash[22706]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:19 smithi042 bash[22706]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:19 smithi042 bash[32075]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:19 smithi042 bash[32075]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:19 smithi162 bash[24557]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:19.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:19 smithi162 bash[24557]: cluster 2024-09-20T19:10:17.753286+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:21 smithi162 bash[24557]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:21 smithi162 bash[24557]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:21 smithi042 bash[32075]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:21 smithi042 bash[32075]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:21 smithi042 bash[22706]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:21.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:21 smithi042 bash[22706]: cluster 2024-09-20T19:10:19.753729+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[22706]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:23.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:23 smithi042 bash[32075]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:23.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: cluster 2024-09-20T19:10:21.754212+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:23.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: audit 2024-09-20T19:10:22.938317+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:10:23.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:23.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:23 smithi162 bash[24557]: audit 2024-09-20T19:10:22.940046+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:24.551 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:24 smithi042 bash[32075]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:24.551 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:24 smithi042 bash[32075]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:24.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:24 smithi042 bash[22706]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:24.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:24 smithi042 bash[22706]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:24.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:24 smithi162 bash[24557]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:24.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:24 smithi162 bash[24557]: cephadm 2024-09-20T19:10:22.941451+0000 mgr.y (mgr.14152) 103 : cephadm [INF] Deploying daemon osd.0 on smithi042 2024-09-20T19:10:25.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:25 smithi162 bash[24557]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:25.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:25 smithi162 bash[24557]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:25 smithi042 bash[22706]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:25 smithi042 bash[22706]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:25 smithi042 bash[32075]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:25.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:25 smithi042 bash[32075]: cluster 2024-09-20T19:10:23.754584+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:27 smithi162 bash[24557]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:27 smithi162 bash[24557]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:27 smithi042 bash[22706]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:27 smithi042 bash[22706]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:27 smithi042 bash[32075]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:27 smithi042 bash[32075]: cluster 2024-09-20T19:10:25.755016+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:27.884 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:10:27 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:28 smithi162 bash[24557]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[22706]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.919690+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.923688+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:28 smithi042 bash[32075]: audit 2024-09-20T19:10:27.928381+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:29.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:29 smithi042 bash[22706]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:29.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:29 smithi042 bash[22706]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:29 smithi042 bash[32075]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:29 smithi042 bash[32075]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:29.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:29 smithi162 bash[24557]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:29 smithi162 bash[24557]: cluster 2024-09-20T19:10:27.755330+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:31 smithi162 bash[24557]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:31 smithi162 bash[24557]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:31 smithi042 bash[22706]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:31 smithi042 bash[22706]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.709 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:31 smithi042 bash[32075]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:31.709 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:31 smithi042 bash[32075]: cluster 2024-09-20T19:10:29.755723+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[22706]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:33 smithi042 bash[32075]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: cluster 2024-09-20T19:10:31.756184+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: audit 2024-09-20T19:10:33.249906+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:33.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:33 smithi162 bash[24557]: audit 2024-09-20T19:10:33.254232+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[22706]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:34 smithi042 bash[32075]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.877301+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.878601+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:34 smithi162 bash[24557]: audit 2024-09-20T19:10:33.886684+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:35 smithi042 bash[22706]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:35 smithi042 bash[22706]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:35 smithi042 bash[32075]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:35 smithi042 bash[32075]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:35.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:35 smithi162 bash[24557]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:35 smithi162 bash[24557]: cluster 2024-09-20T19:10:33.756522+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:37 smithi162 bash[24557]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:37 smithi162 bash[24557]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:37 smithi042 bash[32075]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:37 smithi042 bash[32075]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:37 smithi042 bash[22706]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:37.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:37 smithi042 bash[22706]: cluster 2024-09-20T19:10:35.756905+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:39 smithi162 bash[24557]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:39 smithi162 bash[24557]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:39 smithi042 bash[22706]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:39 smithi042 bash[22706]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:39 smithi042 bash[32075]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:39.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:39 smithi042 bash[32075]: cluster 2024-09-20T19:10:37.757364+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:40.998 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 0 on host 'smithi042' 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:41 smithi162 bash[24557]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[32075]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: cluster 2024-09-20T19:10:39.757691+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.976993+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:10:41.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.983922+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:41 smithi042 bash[22706]: audit 2024-09-20T19:10:40.991272+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:41.720 DEBUG:teuthology.orchestra.run.smithi042:osd.0> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T19:10:41.722 INFO:tasks.cephadm:Deploying osd.1 on smithi042 with /dev/vg_nvme/lv_3... 2024-09-20T19:10:41.723 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:42 smithi162 bash[24557]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:42 smithi162 bash[24557]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:42 smithi162 bash[24557]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:42 smithi162 bash[24557]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[22706]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[22706]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[22706]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[22706]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[32075]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[32075]: audit 2024-09-20T19:10:41.344077+0000 mon.c (mon.2) 5 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[32075]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:42.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:42 smithi042 bash[32075]: audit 2024-09-20T19:10:41.344411+0000 mon.a (mon.0) 318 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:10:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[32075]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[22706]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: cluster 2024-09-20T19:10:41.758114+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.304928+0000 mon.a (mon.0) 319 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.309556+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.310302+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:43.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.310560+0000 mon.c (mon.2) 6 : audit [INF] from='osd.0 v1:172.21.15.42:6801/2265043130' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:43.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:43 smithi162 bash[24557]: audit 2024-09-20T19:10:42.311259+0000 mon.a (mon.0) 322 : audit [INF] from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:10:44.182 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[35151]: debug 2024-09-20T19:10:43.686+0000 7f917b630640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T19:10:44.182 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:10:43 smithi042 bash[35151]: debug 2024-09-20T19:10:43.690+0000 7f917344f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:10:44.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[22706]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:44.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:44 smithi042 bash[32075]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.306770+0000 mon.a (mon.0) 323 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: cluster 2024-09-20T19:10:43.311807+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.312716+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.324028+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:44.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:44 smithi162 bash[24557]: audit 2024-09-20T19:10:43.698655+0000 mon.a (mon.0) 327 : audit [INF] from='osd.0 ' entity='osd.0' 2024-09-20T19:10:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[22706]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:45 smithi042 bash[32075]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.375511+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:42.375605+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:43.758504+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: audit 2024-09-20T19:10:44.324579+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:44.702201+0000 mon.a (mon.0) 329 : cluster [INF] osd.0 v1:172.21.15.42:6801/2265043130 boot 2024-09-20T19:10:45.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: cluster 2024-09-20T19:10:44.702255+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:10:45.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:45.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:45 smithi162 bash[24557]: audit 2024-09-20T19:10:44.702870+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:10:46.034 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:10:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:46 smithi162 bash[24557]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.107 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:46 smithi162 bash[24557]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.107 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:46 smithi162 bash[24557]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:47.107 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:46 smithi162 bash[24557]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[22706]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[22706]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[22706]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[22706]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[32075]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[32075]: cluster 2024-09-20T19:10:45.711523+0000 mon.a (mon.0) 332 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:10:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[32075]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:46 smithi042 bash[32075]: cluster 2024-09-20T19:10:45.759094+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:48 smithi042 bash[22706]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:48 smithi042 bash[22706]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:48 smithi042 bash[32075]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:48 smithi042 bash[32075]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:48 smithi162 bash[24557]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:48 smithi162 bash[24557]: cluster 2024-09-20T19:10:47.759475+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.113 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:50 smithi042 bash[22706]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.113 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:50 smithi042 bash[22706]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.113 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:50 smithi042 bash[32075]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.114 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:50 smithi042 bash[32075]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:50 smithi162 bash[24557]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:51.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:50 smithi162 bash[24557]: cluster 2024-09-20T19:10:49.759935+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[22706]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:52 smithi042 bash[32075]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cluster 2024-09-20T19:10:51.760533+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cephadm 2024-09-20T19:10:51.812384+0000 mgr.y (mgr.14152) 119 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.819239+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.831782+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.833606+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: cephadm 2024-09-20T19:10:51.834334+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 16245M 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.839994+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.842614+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.843421+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:53.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:52 smithi162 bash[24557]: audit 2024-09-20T19:10:51.849133+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:10:54.430 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:10:54.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi042:vg_nvme/lv_3 2024-09-20T19:10:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:54 smithi042 bash[22706]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:54 smithi042 bash[22706]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:55.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:54 smithi042 bash[32075]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:55.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:54 smithi042 bash[32075]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:55.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:54 smithi162 bash[24557]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:55.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:54 smithi162 bash[24557]: cluster 2024-09-20T19:10:53.760881+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:56 smithi042 bash[22706]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:56 smithi042 bash[22706]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:56 smithi042 bash[32075]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:56 smithi042 bash[32075]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:56 smithi162 bash[24557]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:57.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:56 smithi162 bash[24557]: cluster 2024-09-20T19:10:55.761323+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:58 smithi042 bash[22706]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:10:58 smithi042 bash[22706]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:58 smithi042 bash[32075]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:10:58 smithi042 bash[32075]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.233 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:10:59.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:58 smithi162 bash[24557]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:10:59.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:10:58 smithi162 bash[24557]: cluster 2024-09-20T19:10:57.761802+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[22706]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:01.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:00 smithi042 bash[32075]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:01.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: cluster 2024-09-20T19:10:59.762312+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.771835+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.777217+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:01.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:00 smithi162 bash[24557]: audit 2024-09-20T19:11:00.778285+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:02.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:01 smithi042 bash[22706]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:02.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:01 smithi042 bash[22706]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:01 smithi042 bash[32075]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:02.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:01 smithi042 bash[32075]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:02.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:01 smithi162 bash[24557]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:02.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:01 smithi162 bash[24557]: audit 2024-09-20T19:11:00.769156+0000 mgr.y (mgr.14152) 125 : audit [DBG] from='client.14244 v1:172.21.15.42:0/2587037964' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:11:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:02 smithi042 bash[22706]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:03.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:02 smithi042 bash[22706]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:03.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:02 smithi042 bash[32075]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:03.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:02 smithi042 bash[32075]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:03.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:02 smithi162 bash[24557]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:03.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:02 smithi162 bash[24557]: cluster 2024-09-20T19:11:01.762797+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:04 smithi042 bash[22706]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:04 smithi042 bash[22706]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:04 smithi042 bash[32075]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:04 smithi042 bash[32075]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:04 smithi162 bash[24557]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:05.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:04 smithi162 bash[24557]: cluster 2024-09-20T19:11:03.763246+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.156 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:06 smithi042 bash[22706]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.156 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:06 smithi042 bash[22706]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.156 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:06 smithi042 bash[32075]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.156 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:06 smithi042 bash[32075]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:06 smithi162 bash[24557]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:07.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:06 smithi162 bash[24557]: cluster 2024-09-20T19:11:05.763727+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:08 smithi042 bash[32075]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:08 smithi042 bash[32075]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:08 smithi042 bash[22706]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:08 smithi042 bash[22706]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:08 smithi162 bash[24557]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:09.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:08 smithi162 bash[24557]: cluster 2024-09-20T19:11:07.764137+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[22706]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:10.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:09 smithi042 bash[32075]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:10.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.499427+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.42:0/869343151' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.499914+0000 mon.a (mon.0) 343 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"} : dispatch 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.502620+0000 mon.a (mon.0) 344 : audit [INF] from='client.24152 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d"}]': finished 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: cluster 2024-09-20T19:11:09.507633+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:10.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:09 smithi162 bash[24557]: audit 2024-09-20T19:11:09.507950+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[22706]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[22706]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[22706]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[22706]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:11.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[32075]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[32075]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[32075]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:11.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:10 smithi042 bash[32075]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:11.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:10 smithi162 bash[24557]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:10 smithi162 bash[24557]: cluster 2024-09-20T19:11:09.764709+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:11.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:10 smithi162 bash[24557]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:11.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:10 smithi162 bash[24557]: audit 2024-09-20T19:11:10.179376+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.42:0/224353499' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:11:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:12 smithi042 bash[32075]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:12 smithi042 bash[32075]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:12 smithi042 bash[22706]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:12 smithi042 bash[22706]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:13.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:12 smithi162 bash[24557]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:13.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:12 smithi162 bash[24557]: cluster 2024-09-20T19:11:11.765154+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:14 smithi042 bash[22706]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:14 smithi042 bash[22706]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:14 smithi042 bash[32075]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:14 smithi042 bash[32075]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:14 smithi162 bash[24557]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:15.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:14 smithi162 bash[24557]: cluster 2024-09-20T19:11:13.765617+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:16 smithi042 bash[22706]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:16 smithi042 bash[22706]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:16 smithi042 bash[32075]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:16 smithi042 bash[32075]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:16 smithi162 bash[24557]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:17.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:16 smithi162 bash[24557]: cluster 2024-09-20T19:11:15.766120+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:18 smithi042 bash[22706]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:18 smithi042 bash[22706]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:18 smithi042 bash[32075]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:18 smithi042 bash[32075]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:18 smithi162 bash[24557]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:19.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:18 smithi162 bash[24557]: cluster 2024-09-20T19:11:17.766529+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:20.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:20 smithi042 bash[22706]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:20.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:20 smithi042 bash[22706]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:20 smithi042 bash[32075]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:20 smithi042 bash[32075]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:21.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:20 smithi162 bash[24557]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:21.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:20 smithi162 bash[24557]: cluster 2024-09-20T19:11:19.766968+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:22 smithi042 bash[22706]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:22 smithi042 bash[22706]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:22 smithi042 bash[32075]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:22 smithi042 bash[32075]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:22 smithi162 bash[24557]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:23.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:22 smithi162 bash[24557]: cluster 2024-09-20T19:11:21.767435+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[22706]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:24.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:23 smithi042 bash[32075]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:24.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: audit 2024-09-20T19:11:23.210112+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:11:24.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: audit 2024-09-20T19:11:23.211511+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:24.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:24.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:23 smithi162 bash[24557]: cephadm 2024-09-20T19:11:23.212803+0000 mgr.y (mgr.14152) 137 : cephadm [INF] Deploying daemon osd.1 on smithi042 2024-09-20T19:11:25.166 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:24 smithi042 bash[22706]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:25.166 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:24 smithi042 bash[22706]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:25.166 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:24 smithi042 bash[32075]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:25.166 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:24 smithi042 bash[32075]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:25.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:24 smithi162 bash[24557]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:25.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:24 smithi162 bash[24557]: cluster 2024-09-20T19:11:23.767814+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.129 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:26 smithi042 bash[22706]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.130 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:26 smithi042 bash[22706]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.130 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:26 smithi042 bash[32075]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.130 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:26 smithi042 bash[32075]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:26 smithi162 bash[24557]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:27.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:26 smithi162 bash[24557]: cluster 2024-09-20T19:11:25.768239+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:28.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.179 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.432 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:28.432 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:11:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:11:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[22706]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:28 smithi042 bash[32075]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: cluster 2024-09-20T19:11:27.768675+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:29.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.447679+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:29.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.451670+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:29.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:28 smithi162 bash[24557]: audit 2024-09-20T19:11:28.457261+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:31.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:30 smithi042 bash[32075]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:31.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:30 smithi042 bash[32075]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:31.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:30 smithi042 bash[22706]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:31.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:30 smithi042 bash[22706]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:31.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:30 smithi162 bash[24557]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:31.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:30 smithi162 bash[24557]: cluster 2024-09-20T19:11:29.768998+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:32 smithi042 bash[22706]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:32 smithi042 bash[22706]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:32 smithi042 bash[32075]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:32 smithi042 bash[32075]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:32 smithi162 bash[24557]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:33.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:32 smithi162 bash[24557]: cluster 2024-09-20T19:11:31.769444+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[22706]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:34 smithi042 bash[32075]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: cluster 2024-09-20T19:11:33.769810+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.813470+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.817596+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.818403+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.819100+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:35.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:34 smithi162 bash[24557]: audit 2024-09-20T19:11:33.823823+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:36 smithi042 bash[22706]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:36 smithi042 bash[22706]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:36 smithi042 bash[32075]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:36 smithi042 bash[32075]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:37.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:36 smithi162 bash[24557]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:37.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:36 smithi162 bash[24557]: cluster 2024-09-20T19:11:35.770299+0000 mgr.y (mgr.14152) 144 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:38 smithi042 bash[22706]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:38 smithi042 bash[22706]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:38 smithi042 bash[32075]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:38 smithi042 bash[32075]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:39.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:38 smithi162 bash[24557]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:39.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:38 smithi162 bash[24557]: cluster 2024-09-20T19:11:37.770803+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:40 smithi042 bash[22706]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:40 smithi042 bash[22706]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:40 smithi042 bash[32075]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:40 smithi042 bash[32075]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:40 smithi162 bash[24557]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:40 smithi162 bash[24557]: cluster 2024-09-20T19:11:39.771144+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:41.780 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 1 on host 'smithi042' 2024-09-20T19:11:42.101 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[22706]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:41 smithi042 bash[32075]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.759809+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:11:42.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.766916+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:41 smithi162 bash[24557]: audit 2024-09-20T19:11:41.774718+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:42.549 DEBUG:teuthology.orchestra.run.smithi042:osd.1> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T19:11:42.551 INFO:tasks.cephadm:Deploying osd.2 on smithi042 with /dev/vg_nvme/lv_2... 2024-09-20T19:11:42.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:11:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[22706]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:42 smithi042 bash[32075]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: cluster 2024-09-20T19:11:41.771476+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: audit 2024-09-20T19:11:42.102790+0000 mon.c (mon.2) 9 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:42 smithi162 bash[24557]: audit 2024-09-20T19:11:42.103068+0000 mon.a (mon.0) 360 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:11:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[22706]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[32075]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.184 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[40690]: debug 2024-09-20T19:11:43.846+0000 7f207623a640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T19:11:44.185 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:11:43 smithi042 bash[40690]: debug 2024-09-20T19:11:43.850+0000 7f2072061640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:11:44.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.832349+0000 mon.a (mon.0) 361 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:11:44.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: cluster 2024-09-20T19:11:42.834881+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.835132+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.836300+0000 mon.c (mon.2) 10 : audit [INF] from='osd.1 v1:172.21.15.42:6805/3513258758' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:43 smithi162 bash[24557]: audit 2024-09-20T19:11:42.836640+0000 mon.a (mon.0) 364 : audit [INF] from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:11:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[22706]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:44 smithi042 bash[32075]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.771895+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.834467+0000 mon.a (mon.0) 365 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.840139+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.841114+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:45.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:44 smithi162 bash[24557]: audit 2024-09-20T19:11:43.852480+0000 mon.a (mon.0) 368 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[22706]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:45 smithi042 bash[32075]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.139854+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:43.139952+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:44.851703+0000 mon.a (mon.0) 369 : cluster [INF] osd.1 v1:172.21.15.42:6805/3513258758 boot 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: cluster 2024-09-20T19:11:44.851801+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:11:46.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:46.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:45 smithi162 bash[24557]: audit 2024-09-20T19:11:44.852580+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[22706]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[22706]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[22706]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[22706]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[32075]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[32075]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[32075]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:46 smithi042 bash[32075]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:46 smithi162 bash[24557]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:46 smithi162 bash[24557]: cluster 2024-09-20T19:11:45.772366+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:11:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:46 smithi162 bash[24557]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:46 smithi162 bash[24557]: cluster 2024-09-20T19:11:45.869366+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:11:47.359 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:11:49.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:48 smithi042 bash[22706]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:49.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:48 smithi042 bash[22706]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:49.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:48 smithi042 bash[32075]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:49.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:48 smithi042 bash[32075]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:48 smithi162 bash[24557]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:48 smithi162 bash[24557]: cluster 2024-09-20T19:11:47.772834+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[22706]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:51.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:50 smithi042 bash[32075]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:51.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.756905+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:11:51.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.757003+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:11:51.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.773285+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:51.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:51.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:50 smithi162 bash[24557]: cluster 2024-09-20T19:11:49.880152+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:11:53.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:52 smithi042 bash[22706]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:53.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:52 smithi042 bash[22706]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:52 smithi042 bash[32075]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:52 smithi042 bash[32075]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:53.256 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:11:53.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi042:vg_nvme/lv_2 2024-09-20T19:11:53.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:52 smithi162 bash[24557]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:53.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:52 smithi162 bash[24557]: cluster 2024-09-20T19:11:51.773606+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:54 smithi042 bash[22706]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:54 smithi042 bash[22706]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:54 smithi042 bash[32075]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:54 smithi042 bash[32075]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:54 smithi162 bash[24557]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:55.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:54 smithi162 bash[24557]: cluster 2024-09-20T19:11:53.774010+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[22706]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:57 smithi042 bash[32075]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cluster 2024-09-20T19:11:55.774524+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cephadm 2024-09-20T19:11:56.313484+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.322198+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.330718+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.333413+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.335316+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: cephadm 2024-09-20T19:11:56.336496+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 8122M 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.343014+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.347525+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.349078+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:57.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:57 smithi162 bash[24557]: audit 2024-09-20T19:11:56.361211+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:11:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:11:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:59 smithi042 bash[22706]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:11:59 smithi042 bash[22706]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:59.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:59 smithi042 bash[32075]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:59.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:11:59 smithi042 bash[32075]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:59 smithi162 bash[24557]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:11:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:11:59 smithi162 bash[24557]: cluster 2024-09-20T19:11:57.774995+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[22706]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:01.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:01 smithi042 bash[32075]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: cluster 2024-09-20T19:11:59.775488+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.840936+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.846311+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:12:01.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:01.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:01 smithi162 bash[24557]: audit 2024-09-20T19:12:00.847670+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:02.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:02 smithi042 bash[22706]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:02.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:02 smithi042 bash[22706]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:02.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:02 smithi042 bash[32075]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:02.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:02 smithi042 bash[32075]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:02.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:02 smithi162 bash[24557]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:02.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:02 smithi162 bash[24557]: audit 2024-09-20T19:12:00.834638+0000 mgr.y (mgr.14152) 159 : audit [DBG] from='client.24173 v1:172.21.15.42:0/2817390716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:12:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:03 smithi042 bash[22706]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:03 smithi042 bash[22706]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:03.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:03 smithi042 bash[32075]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:03.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:03 smithi042 bash[32075]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:03.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:03 smithi162 bash[24557]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:03.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:03 smithi162 bash[24557]: cluster 2024-09-20T19:12:01.775996+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:05 smithi042 bash[22706]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:05 smithi042 bash[22706]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:05 smithi042 bash[32075]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:05 smithi042 bash[32075]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:05 smithi162 bash[24557]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:05.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:05 smithi162 bash[24557]: cluster 2024-09-20T19:12:03.776494+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:07 smithi042 bash[32075]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:07 smithi042 bash[32075]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:07 smithi042 bash[22706]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:07 smithi042 bash[22706]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:07 smithi162 bash[24557]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:07.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:07 smithi162 bash[24557]: cluster 2024-09-20T19:12:05.777002+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:09 smithi042 bash[32075]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:09 smithi042 bash[32075]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:09 smithi042 bash[22706]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:09 smithi042 bash[22706]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:09 smithi162 bash[24557]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:09.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:09 smithi162 bash[24557]: cluster 2024-09-20T19:12:07.777507+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:10.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[22706]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:10.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:10 smithi042 bash[32075]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.698129+0000 mon.a (mon.0) 385 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"} : dispatch 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.700904+0000 mon.a (mon.0) 386 : audit [INF] from='client.? v1:172.21.15.42:0/2724638345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d"}]': finished 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: cluster 2024-09-20T19:12:09.704476+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:10.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:10 smithi162 bash[24557]: audit 2024-09-20T19:12:09.704832+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[22706]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[22706]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[22706]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[22706]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[32075]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[32075]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[32075]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:11.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:11 smithi042 bash[32075]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:11.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:11 smithi162 bash[24557]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:11 smithi162 bash[24557]: cluster 2024-09-20T19:12:09.777991+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:11.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:11 smithi162 bash[24557]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:11.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:11 smithi162 bash[24557]: audit 2024-09-20T19:12:10.365931+0000 mon.a (mon.0) 389 : audit [DBG] from='client.? v1:172.21.15.42:0/3181464577' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:12:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:13 smithi042 bash[32075]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:13 smithi042 bash[32075]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:13 smithi042 bash[22706]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:13 smithi042 bash[22706]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:13 smithi162 bash[24557]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:13 smithi162 bash[24557]: cluster 2024-09-20T19:12:11.778449+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:15 smithi042 bash[22706]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:15 smithi042 bash[22706]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:15 smithi042 bash[32075]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:15 smithi042 bash[32075]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:15 smithi162 bash[24557]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:15.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:15 smithi162 bash[24557]: cluster 2024-09-20T19:12:13.778870+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:17 smithi042 bash[22706]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:17 smithi042 bash[22706]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:17 smithi042 bash[32075]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:17 smithi042 bash[32075]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:17 smithi162 bash[24557]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:17.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:17 smithi162 bash[24557]: cluster 2024-09-20T19:12:15.779446+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:19 smithi042 bash[22706]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:19 smithi042 bash[22706]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:19 smithi042 bash[32075]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:19 smithi042 bash[32075]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:19 smithi162 bash[24557]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:19.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:19 smithi162 bash[24557]: cluster 2024-09-20T19:12:17.779913+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:21 smithi042 bash[22706]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:21 smithi042 bash[22706]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:21 smithi042 bash[32075]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:21 smithi042 bash[32075]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:21 smithi162 bash[24557]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:21 smithi162 bash[24557]: cluster 2024-09-20T19:12:19.780426+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:23 smithi042 bash[32075]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:23 smithi042 bash[32075]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:23 smithi042 bash[22706]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:23 smithi042 bash[22706]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:23 smithi162 bash[24557]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:23 smithi162 bash[24557]: cluster 2024-09-20T19:12:21.780986+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[22706]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[22706]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[22706]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[22706]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[32075]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[32075]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[32075]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:24 smithi042 bash[32075]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:24 smithi162 bash[24557]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:24 smithi162 bash[24557]: audit 2024-09-20T19:12:23.941419+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:12:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:24 smithi162 bash[24557]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:24 smithi162 bash[24557]: audit 2024-09-20T19:12:23.942992+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[32075]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[32075]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[32075]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[32075]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[22706]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[22706]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[22706]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:25 smithi042 bash[22706]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:25.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:25 smithi162 bash[24557]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:25 smithi162 bash[24557]: cluster 2024-09-20T19:12:23.781465+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:25.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:25 smithi162 bash[24557]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:25.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:25 smithi162 bash[24557]: cephadm 2024-09-20T19:12:23.944138+0000 mgr.y (mgr.14152) 172 : cephadm [INF] Deploying daemon osd.2 on smithi042 2024-09-20T19:12:27.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:27 smithi042 bash[32075]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:27.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:27 smithi042 bash[32075]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:27.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:27 smithi042 bash[22706]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:27.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:27 smithi042 bash[22706]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:27.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:27 smithi162 bash[24557]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:27.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:27 smithi162 bash[24557]: cluster 2024-09-20T19:12:25.781860+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:28.997 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:28.997 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:28.998 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:12:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:28.998 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:12:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:28.998 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:12:28 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.354 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:12:29 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.355 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.355 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:12:29 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.355 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:12:29 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:12:29.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[32075]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:29 smithi042 bash[22706]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: cluster 2024-09-20T19:12:27.782416+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.264976+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.269145+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:29 smithi162 bash[24557]: audit 2024-09-20T19:12:29.274015+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:31.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:31 smithi042 bash[22706]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:31.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:31 smithi042 bash[22706]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:31 smithi042 bash[32075]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:31 smithi042 bash[32075]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:31 smithi162 bash[24557]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:31 smithi162 bash[24557]: cluster 2024-09-20T19:12:29.782904+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:33 smithi042 bash[22706]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:33 smithi042 bash[22706]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:33 smithi042 bash[32075]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:33 smithi042 bash[32075]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:33 smithi162 bash[24557]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:33 smithi162 bash[24557]: cluster 2024-09-20T19:12:31.783391+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.577 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[32075]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:35 smithi042 bash[22706]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: cluster 2024-09-20T19:12:33.783715+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.649790+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.654328+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.655175+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:35.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.655845+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:35.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:35.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:35 smithi162 bash[24557]: audit 2024-09-20T19:12:34.660944+0000 mon.a (mon.0) 399 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:37 smithi042 bash[22706]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:37 smithi042 bash[22706]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:37.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:37 smithi042 bash[32075]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:37.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:37 smithi042 bash[32075]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:37 smithi162 bash[24557]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:37 smithi162 bash[24557]: cluster 2024-09-20T19:12:35.784181+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:39 smithi042 bash[22706]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:39 smithi042 bash[22706]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:39 smithi042 bash[32075]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:39 smithi042 bash[32075]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:39 smithi162 bash[24557]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:39.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:39 smithi162 bash[24557]: cluster 2024-09-20T19:12:37.784705+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:41 smithi042 bash[22706]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:41 smithi042 bash[22706]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:41 smithi042 bash[32075]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:41 smithi042 bash[32075]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:41 smithi162 bash[24557]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:41 smithi162 bash[24557]: cluster 2024-09-20T19:12:39.785172+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:42.670 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 2 on host 'smithi042' 2024-09-20T19:12:43.304 DEBUG:teuthology.orchestra.run.smithi042:osd.2> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T19:12:43.306 INFO:tasks.cephadm:Deploying osd.3 on smithi042 with /dev/vg_nvme/lv_1... 2024-09-20T19:12:43.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[32075]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:43 smithi042 bash[22706]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:43.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: cluster 2024-09-20T19:12:41.785681+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.649162+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.654785+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:42.662810+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:43.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:43 smithi162 bash[24557]: audit 2024-09-20T19:12:43.229298+0000 mon.a (mon.0) 403 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:12:44.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[22706]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:44 smithi042 bash[32075]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:45.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:45.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.660841+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: cluster 2024-09-20T19:12:43.665447+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.665917+0000 mon.a (mon.0) 406 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: audit 2024-09-20T19:12:43.666450+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:44 smithi162 bash[24557]: cluster 2024-09-20T19:12:43.786077+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:45.432 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[46409]: debug 2024-09-20T19:12:45.065+0000 7f06fea8a640 -1 osd.2 0 waiting for initial osdmap 2024-09-20T19:12:45.432 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[46409]: debug 2024-09-20T19:12:45.069+0000 7f06f68a9640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[22706]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:45.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:45.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:45 smithi042 bash[32075]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:46.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:46.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.669132+0000 mon.a (mon.0) 408 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.676655+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.677765+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:44.685067+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:45 smithi162 bash[24557]: audit 2024-09-20T19:12:45.071498+0000 mon.a (mon.0) 412 : audit [INF] from='osd.2 v1:172.21.15.42:6809/2620956901' entity='osd.2' 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:46 smithi162 bash[24557]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[22706]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.262163+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:44.262299+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:45.683524+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:45.786594+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:46.076084+0000 mon.a (mon.0) 414 : cluster [INF] osd.2 v1:172.21.15.42:6809/2620956901 boot 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: cluster 2024-09-20T19:12:46.076127+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:12:47.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:46.076335+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:12:47.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:47.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:46 smithi042 bash[32075]: audit 2024-09-20T19:12:46.304587+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.104 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[32075]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.341 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.342 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.342 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.342 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.342 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.342 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:48 smithi042 bash[22706]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: audit 2024-09-20T19:12:47.074236+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:48.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: cluster 2024-09-20T19:12:47.079895+0000 mon.a (mon.0) 419 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:12:48.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:48.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:48 smithi162 bash[24557]: audit 2024-09-20T19:12:47.081575+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[32075]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.439 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.439 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.439 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:49 smithi042 bash[22706]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:47.787219+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v150: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:48.074006+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.083336+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: cluster 2024-09-20T19:12:48.090170+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.206483+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.292759+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.292924+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.293114+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.293197+0000 mon.a (mon.0) 428 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.333728+0000 mon.a (mon.0) 429 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336573+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336835+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.336971+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.339368+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.395586+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.437393+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.439999+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.440260+0000 mon.c (mon.2) 11 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.440531+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.441026+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:49.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:49 smithi162 bash[24557]: audit 2024-09-20T19:12:48.491487+0000 mon.c (mon.2) 12 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:12:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:50 smithi042 bash[22706]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:50 smithi042 bash[22706]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:50 smithi042 bash[32075]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:50 smithi042 bash[32075]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:50 smithi162 bash[24557]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:50 smithi162 bash[24557]: cluster 2024-09-20T19:12:49.107706+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[22706]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:51.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:51 smithi042 bash[32075]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:51.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:49.787753+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v153: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.101933+0000 mon.a (mon.0) 438 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.101960+0000 mon.a (mon.0) 439 : cluster [INF] Cluster is now healthy 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:51.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:51 smithi162 bash[24557]: cluster 2024-09-20T19:12:50.116539+0000 mon.a (mon.0) 440 : cluster [DBG] mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:12:53.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:53 smithi042 bash[22706]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:53.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:53 smithi042 bash[22706]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:53.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:53 smithi042 bash[32075]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:53.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:53 smithi042 bash[32075]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:53.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:53 smithi162 bash[24557]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:53.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:53 smithi162 bash[24557]: cluster 2024-09-20T19:12:51.788424+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[22706]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:55 smithi042 bash[32075]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cluster 2024-09-20T19:12:53.789043+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cephadm 2024-09-20T19:12:54.316725+0000 mgr.y (mgr.14152) 188 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.324973+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.335099+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.337392+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.338665+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.339845+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: cephadm 2024-09-20T19:12:54.340782+0000 mgr.y (mgr.14152) 189 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 5415M 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.347555+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.352712+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.354050+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:55.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:55 smithi162 bash[24557]: audit 2024-09-20T19:12:54.364951+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:12:57.087 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:12:57.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi042:vg_nvme/lv_1 2024-09-20T19:12:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:57 smithi042 bash[32075]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:57 smithi042 bash[32075]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:57 smithi042 bash[22706]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:57 smithi042 bash[22706]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:57.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:57 smithi162 bash[24557]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:57.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:57 smithi162 bash[24557]: cluster 2024-09-20T19:12:55.789421+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:59 smithi042 bash[22706]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:12:59 smithi042 bash[22706]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:59 smithi042 bash[32075]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:12:59 smithi042 bash[32075]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:59 smithi162 bash[24557]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:12:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:12:59 smithi162 bash[24557]: cluster 2024-09-20T19:12:57.789952+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:01 smithi042 bash[22706]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:01 smithi042 bash[22706]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:01 smithi042 bash[32075]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:01 smithi042 bash[32075]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:01 smithi162 bash[24557]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:01 smithi162 bash[24557]: cluster 2024-09-20T19:12:59.790466+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:03 smithi042 bash[22706]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:03 smithi042 bash[22706]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:03 smithi042 bash[32075]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:03 smithi042 bash[32075]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:03.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:03 smithi162 bash[24557]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:03.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:03 smithi162 bash[24557]: cluster 2024-09-20T19:13:01.791018+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[22706]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:04.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:04 smithi042 bash[32075]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.562688+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.568115+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:04.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:04 smithi162 bash[24557]: audit 2024-09-20T19:13:03.569231+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[22706]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[22706]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[22706]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[22706]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[32075]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[32075]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[32075]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:05.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:05 smithi042 bash[32075]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:05.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:05 smithi162 bash[24557]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:05 smithi162 bash[24557]: audit 2024-09-20T19:13:03.556790+0000 mgr.y (mgr.14152) 194 : audit [DBG] from='client.24209 v1:172.21.15.42:0/3657894928' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:13:05.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:05 smithi162 bash[24557]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:05.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:05 smithi162 bash[24557]: cluster 2024-09-20T19:13:03.791411+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:07 smithi042 bash[22706]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:07 smithi042 bash[22706]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:07 smithi042 bash[32075]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:07 smithi042 bash[32075]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:07 smithi162 bash[24557]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:07 smithi162 bash[24557]: cluster 2024-09-20T19:13:05.791951+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:09 smithi042 bash[22706]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:09 smithi042 bash[22706]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:09 smithi042 bash[32075]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:09 smithi042 bash[32075]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:09 smithi162 bash[24557]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:09.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:09 smithi162 bash[24557]: cluster 2024-09-20T19:13:07.792446+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:11 smithi042 bash[22706]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:11 smithi042 bash[22706]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:11 smithi042 bash[32075]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:11 smithi042 bash[32075]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:11 smithi162 bash[24557]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:11 smithi162 bash[24557]: cluster 2024-09-20T19:13:09.792991+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[22706]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:13 smithi042 bash[32075]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:11.793430+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.598284+0000 mon.a (mon.0) 453 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"} : dispatch 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.600831+0000 mon.a (mon.0) 454 : audit [INF] from='client.? v1:172.21.15.42:0/2140589347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e"}]': finished 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.603785+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:12.604284+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.973039+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: cluster 2024-09-20T19:13:12.973155+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:13.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:13 smithi162 bash[24557]: audit 2024-09-20T19:13:13.270501+0000 mon.a (mon.0) 457 : audit [DBG] from='client.? v1:172.21.15.42:0/2835641220' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:13:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[22706]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:14 smithi042 bash[32075]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: cluster 2024-09-20T19:13:13.608296+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: audit 2024-09-20T19:13:13.608523+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:14 smithi162 bash[24557]: cluster 2024-09-20T19:13:13.793924+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.155 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:16 smithi042 bash[22706]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.155 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:16 smithi042 bash[22706]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.155 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:16 smithi042 bash[32075]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.155 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:16 smithi042 bash[32075]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:16 smithi162 bash[24557]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:17.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:16 smithi162 bash[24557]: cluster 2024-09-20T19:13:15.794474+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:18.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[32075]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:18.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:17 smithi042 bash[22706]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.567858+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.568012+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: cluster 2024-09-20T19:13:16.861539+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:18.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:17 smithi162 bash[24557]: audit 2024-09-20T19:13:16.861754+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:18 smithi042 bash[32075]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:18 smithi042 bash[32075]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:18 smithi042 bash[22706]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:18 smithi042 bash[22706]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:19.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:18 smithi162 bash[24557]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:19.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:18 smithi162 bash[24557]: cluster 2024-09-20T19:13:17.794984+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:20 smithi042 bash[32075]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:20 smithi042 bash[32075]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:20 smithi042 bash[22706]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:20 smithi042 bash[22706]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:20 smithi162 bash[24557]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:21.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:20 smithi162 bash[24557]: cluster 2024-09-20T19:13:19.795517+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:22 smithi042 bash[32075]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:22 smithi042 bash[32075]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:22 smithi042 bash[22706]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:22 smithi042 bash[22706]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:22 smithi162 bash[24557]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:23.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:22 smithi162 bash[24557]: cluster 2024-09-20T19:13:21.796071+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:24 smithi042 bash[22706]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:24 smithi042 bash[22706]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:24 smithi042 bash[32075]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:24 smithi042 bash[32075]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:24 smithi162 bash[24557]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:25.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:24 smithi162 bash[24557]: cluster 2024-09-20T19:13:23.796584+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:26 smithi042 bash[22706]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:26 smithi042 bash[22706]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:26 smithi042 bash[32075]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:26 smithi042 bash[32075]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:26 smithi162 bash[24557]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:27.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:26 smithi162 bash[24557]: cluster 2024-09-20T19:13:25.797128+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:28.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[32075]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:28.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:27 smithi042 bash[22706]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:28.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: audit 2024-09-20T19:13:27.022909+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:13:28.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: audit 2024-09-20T19:13:27.024335+0000 mon.a (mon.0) 463 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:28.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:28.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:27 smithi162 bash[24557]: cephadm 2024-09-20T19:13:27.025458+0000 mgr.y (mgr.14152) 207 : cephadm [INF] Deploying daemon osd.3 on smithi042 2024-09-20T19:13:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:28 smithi042 bash[22706]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:28 smithi042 bash[22706]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:29.146 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:28 smithi042 bash[32075]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:29.146 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:28 smithi042 bash[32075]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:29.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:28 smithi162 bash[24557]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:29.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:28 smithi162 bash[24557]: cluster 2024-09-20T19:13:27.797692+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.065 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:30 smithi042 bash[32075]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.065 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:30 smithi042 bash[32075]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:30 smithi042 bash[22706]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:30 smithi042 bash[22706]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:30 smithi162 bash[24557]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:31.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:30 smithi162 bash[24557]: cluster 2024-09-20T19:13:29.798288+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:32.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.199 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.199 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.199 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.200 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.541 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.541 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.542 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.542 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:32.542 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:13:32 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:13:33.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[32075]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:32 smithi042 bash[22706]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: cluster 2024-09-20T19:13:31.798954+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.464730+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.468915+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:33.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:32 smithi162 bash[24557]: audit 2024-09-20T19:13:32.473688+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:34 smithi042 bash[22706]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:34 smithi042 bash[22706]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:35.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:34 smithi042 bash[32075]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:35.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:34 smithi042 bash[32075]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:35.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:34 smithi162 bash[24557]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:35.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:34 smithi162 bash[24557]: cluster 2024-09-20T19:13:33.799523+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:36 smithi042 bash[22706]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:36 smithi042 bash[22706]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:36 smithi042 bash[32075]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:36 smithi042 bash[32075]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:36 smithi162 bash[24557]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:37.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:36 smithi162 bash[24557]: cluster 2024-09-20T19:13:35.800047+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[22706]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.231 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.232 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.232 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.232 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:38 smithi042 bash[32075]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: cluster 2024-09-20T19:13:37.800402+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.893320+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.899338+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.900146+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.900835+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:38 smithi162 bash[24557]: audit 2024-09-20T19:13:37.905385+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:40 smithi042 bash[22706]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:40 smithi042 bash[22706]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:40 smithi042 bash[32075]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:40 smithi042 bash[32075]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:40 smithi162 bash[24557]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:40 smithi162 bash[24557]: cluster 2024-09-20T19:13:39.800966+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:42 smithi042 bash[22706]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:42 smithi042 bash[22706]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:42 smithi042 bash[32075]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:42 smithi042 bash[32075]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:42 smithi162 bash[24557]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:42 smithi162 bash[24557]: cluster 2024-09-20T19:13:41.801478+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:44 smithi042 bash[32075]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:44 smithi042 bash[32075]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:44 smithi042 bash[22706]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:44 smithi042 bash[22706]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:44 smithi162 bash[24557]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:44 smithi162 bash[24557]: cluster 2024-09-20T19:13:43.801970+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:46.070 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 3 on host 'smithi042' 2024-09-20T19:13:46.788 DEBUG:teuthology.orchestra.run.smithi042:osd.3> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T19:13:46.790 INFO:tasks.cephadm:Deploying osd.4 on smithi162 with /dev/vg_nvme/lv_4... 2024-09-20T19:13:46.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:13:47.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:47.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:46 smithi162 bash[24557]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[32075]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: cluster 2024-09-20T19:13:45.802507+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.048300+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.056039+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.064895+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:46 smithi042 bash[22706]: audit 2024-09-20T19:13:46.135122+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[22706]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:48.433 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[51953]: debug 2024-09-20T19:13:48.077+0000 7f3ae328b640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T19:13:48.433 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[51953]: debug 2024-09-20T19:13:48.085+0000 7f3adf0b2640 -1 osd.3 27 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:48 smithi042 bash[32075]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:48.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.062902+0000 mon.a (mon.0) 476 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.067991+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.068423+0000 mon.a (mon.0) 478 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:48.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:48 smithi162 bash[24557]: audit 2024-09-20T19:13:47.068911+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[22706]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:49 smithi042 bash[32075]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.552629+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.552735+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.803081+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.065164+0000 mon.a (mon.0) 480 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1507333597' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: cluster 2024-09-20T19:13:48.071260+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.072532+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:49.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:49 smithi162 bash[24557]: audit 2024-09-20T19:13:48.081977+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[22706]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:50 smithi042 bash[32075]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.149753+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:47.149881+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.082836+0000 mon.a (mon.0) 484 : cluster [INF] osd.3 v1:172.21.15.42:6813/1507333597 boot 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.082902+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:50.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:50 smithi162 bash[24557]: audit 2024-09-20T19:13:49.083595+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:13:51.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[22706]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[22706]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[22706]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[32075]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[32075]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:51.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:51 smithi042 bash[32075]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:51.522 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:13:51.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:51 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:51 smithi162 bash[24557]: cluster 2024-09-20T19:13:49.803548+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:13:51.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:51 smithi162 bash[24557]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:51.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:51 smithi162 bash[24557]: cluster 2024-09-20T19:13:50.100457+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:13:53.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:53 smithi162 bash[24557]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:53.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:53 smithi162 bash[24557]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:53 smithi042 bash[32075]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:53 smithi042 bash[32075]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:53.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:53 smithi042 bash[22706]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:53.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:53 smithi042 bash[22706]: cluster 2024-09-20T19:13:51.804053+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:55 smithi162 bash[24557]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:55 smithi162 bash[24557]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:55 smithi042 bash[32075]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:55 smithi042 bash[32075]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:55 smithi042 bash[22706]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:55 smithi042 bash[22706]: cluster 2024-09-20T19:13:53.804603+0000 mgr.y (mgr.14152) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:13:55.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-09-20T19:13:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[22706]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[22706]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[22706]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[22706]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[32075]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[32075]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[32075]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:57 smithi042 bash[32075]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:57 smithi162 bash[24557]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:57 smithi162 bash[24557]: cluster 2024-09-20T19:13:55.805166+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:57 smithi162 bash[24557]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:57.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:57 smithi162 bash[24557]: audit 2024-09-20T19:13:57.098361+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[32075]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:58 smithi042 bash[22706]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: cephadm 2024-09-20T19:13:57.089692+0000 mgr.y (mgr.14152) 223 : cephadm [INF] Detected new or changed devices on smithi042 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.109108+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.117132+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.118908+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.120323+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.121780+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: cephadm 2024-09-20T19:13:57.122952+0000 mgr.y (mgr.14152) 224 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 4061M 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.128766+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.133066+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.134469+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:58.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:58 smithi162 bash[24557]: audit 2024-09-20T19:13:57.142370+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:13:59.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:59 smithi042 bash[22706]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:59.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:13:59 smithi042 bash[22706]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:59.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:59 smithi042 bash[32075]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:59.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:13:59 smithi042 bash[32075]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:59.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:59 smithi162 bash[24557]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:13:59.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:13:59 smithi162 bash[24557]: cluster 2024-09-20T19:13:57.805720+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:14:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:01 smithi042 bash[22706]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:01 smithi042 bash[22706]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:01 smithi042 bash[32075]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:01 smithi042 bash[32075]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:01.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:01 smithi162 bash[24557]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:01.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:01 smithi162 bash[24557]: cluster 2024-09-20T19:13:59.806265+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[22706]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:02.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:02 smithi042 bash[32075]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:02.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.970359+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:02.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.976416+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:02.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:02.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:02 smithi162 bash[24557]: audit 2024-09-20T19:14:01.978004+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[22706]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[22706]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[22706]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[22706]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[32075]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[32075]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[32075]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:03.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:03 smithi042 bash[32075]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:03.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:03 smithi162 bash[24557]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:03 smithi162 bash[24557]: cluster 2024-09-20T19:14:01.806801+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:03.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:03 smithi162 bash[24557]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:03.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:03 smithi162 bash[24557]: audit 2024-09-20T19:14:01.963408+0000 mgr.y (mgr.14152) 228 : audit [DBG] from='client.24235 v1:172.21.15.162:0/2366421658' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:05 smithi042 bash[22706]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:05.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:05 smithi042 bash[22706]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:05 smithi042 bash[32075]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:05 smithi042 bash[32075]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:05.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:05 smithi162 bash[24557]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:05.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:05 smithi162 bash[24557]: cluster 2024-09-20T19:14:03.807321+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:07 smithi042 bash[22706]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:07 smithi042 bash[22706]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:07 smithi042 bash[32075]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:07 smithi042 bash[32075]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:07 smithi162 bash[24557]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:07.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:07 smithi162 bash[24557]: cluster 2024-09-20T19:14:05.807843+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:09 smithi162 bash[24557]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:09 smithi162 bash[24557]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:09 smithi042 bash[22706]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:09 smithi042 bash[22706]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:09 smithi042 bash[32075]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:09 smithi042 bash[32075]: cluster 2024-09-20T19:14:07.808390+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:10.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:10 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:10.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:10 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:10 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.572273+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:14:10.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:10 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:10.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:10 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.575961+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[32075]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:11.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:11 smithi042 bash[22706]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:11.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: cluster 2024-09-20T19:14:09.808925+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.563210+0000 mon.a (mon.0) 501 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.565254+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.162:0/2644735903' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"} : dispatch 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.566447+0000 mon.a (mon.0) 502 : audit [INF] from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c5738d-14a9-415f-9249-eac0455635ad"}]': finished 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: cluster 2024-09-20T19:14:10.572312+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:11.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:11 smithi162 bash[24557]: audit 2024-09-20T19:14:10.572802+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:12.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:12 smithi162 bash[24557]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:12.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:12 smithi162 bash[24557]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:12 smithi042 bash[32075]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:12 smithi042 bash[32075]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:12 smithi042 bash[22706]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:12.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:12 smithi042 bash[22706]: audit 2024-09-20T19:14:11.228986+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.162:0/4034290333' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:14:13.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:13 smithi162 bash[24557]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:13.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:13 smithi162 bash[24557]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:13 smithi042 bash[22706]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:13 smithi042 bash[22706]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:13 smithi042 bash[32075]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:13 smithi042 bash[32075]: cluster 2024-09-20T19:14:11.809462+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:14.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:14 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:14 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:14 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:14.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:14 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.931261+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:14 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.931370+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:15.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:15 smithi162 bash[24557]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[22706]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: cluster 2024-09-20T19:14:13.809981+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: cluster 2024-09-20T19:14:14.197495+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:15 smithi042 bash[32075]: audit 2024-09-20T19:14:14.198297+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:17.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:17 smithi162 bash[24557]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:17.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:17 smithi162 bash[24557]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:17 smithi042 bash[22706]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:17 smithi042 bash[22706]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:17 smithi042 bash[32075]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:17 smithi042 bash[32075]: cluster 2024-09-20T19:14:15.810547+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:19 smithi162 bash[24557]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:19 smithi162 bash[24557]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:19 smithi042 bash[22706]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:19 smithi042 bash[22706]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:19 smithi042 bash[32075]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:19.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:19 smithi042 bash[32075]: cluster 2024-09-20T19:14:17.811112+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:21 smithi162 bash[24557]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:21 smithi162 bash[24557]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:21 smithi042 bash[22706]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:21 smithi042 bash[22706]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:21 smithi042 bash[32075]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:21 smithi042 bash[32075]: cluster 2024-09-20T19:14:19.811649+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:23 smithi162 bash[24557]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:23 smithi162 bash[24557]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:23 smithi042 bash[22706]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:23 smithi042 bash[22706]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:23 smithi042 bash[32075]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:23.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:23 smithi042 bash[32075]: cluster 2024-09-20T19:14:21.812327+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:24 smithi162 bash[24557]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:24 smithi162 bash[24557]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:24 smithi162 bash[24557]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:24 smithi162 bash[24557]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[22706]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[22706]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[22706]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[22706]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[32075]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[32075]: audit 2024-09-20T19:14:23.778112+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:14:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[32075]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:24 smithi042 bash[32075]: audit 2024-09-20T19:14:23.779967+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:25.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:25 smithi162 bash[24557]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:25 smithi162 bash[24557]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:25 smithi162 bash[24557]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:25.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:25 smithi162 bash[24557]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[22706]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[22706]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[22706]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[22706]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[32075]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[32075]: cephadm 2024-09-20T19:14:23.781510+0000 mgr.y (mgr.14152) 239 : cephadm [INF] Deploying daemon osd.4 on smithi162 2024-09-20T19:14:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[32075]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:25 smithi042 bash[32075]: cluster 2024-09-20T19:14:23.812877+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:27 smithi162 bash[24557]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:27 smithi162 bash[24557]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:27 smithi042 bash[32075]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:27 smithi042 bash[32075]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:27 smithi042 bash[22706]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:27.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:27 smithi042 bash[22706]: cluster 2024-09-20T19:14:25.813527+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:28.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:27 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:14:28.056 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:14:27 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:14:28.556 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:14:28 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:14:28.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:28 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:29 smithi162 bash[24557]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[22706]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: cluster 2024-09-20T19:14:27.814257+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:29.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.286641+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:29.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.295865+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:29.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:29 smithi042 bash[32075]: audit 2024-09-20T19:14:28.306142+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:31.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:31 smithi162 bash[24557]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:31.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:31 smithi162 bash[24557]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:31.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:31 smithi042 bash[22706]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:31.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:31 smithi042 bash[22706]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:31 smithi042 bash[32075]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:31 smithi042 bash[32075]: cluster 2024-09-20T19:14:29.814805+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:32.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:32 smithi162 bash[24557]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[22706]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:32.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:30.468482+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:30.468570+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: cluster 2024-09-20T19:14:31.250387+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:32.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:32 smithi042 bash[32075]: audit 2024-09-20T19:14:31.250629+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:33.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:33 smithi162 bash[24557]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:33.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:33 smithi162 bash[24557]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:33 smithi042 bash[22706]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:33 smithi042 bash[22706]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:33 smithi042 bash[32075]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:33 smithi042 bash[32075]: cluster 2024-09-20T19:14:31.815312+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:34 smithi162 bash[24557]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[22706]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:33.634936+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:33.642994+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: cluster 2024-09-20T19:14:33.815819+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.301903+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.303528+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:34 smithi042 bash[32075]: audit 2024-09-20T19:14:34.313734+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:36 smithi042 bash[22706]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:36 smithi042 bash[22706]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:36 smithi042 bash[32075]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:36 smithi042 bash[32075]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:37.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:36 smithi162 bash[24557]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:37.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:36 smithi162 bash[24557]: cluster 2024-09-20T19:14:35.816328+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:38 smithi042 bash[22706]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:38 smithi042 bash[22706]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:38 smithi042 bash[32075]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:38 smithi042 bash[32075]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:38 smithi162 bash[24557]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:38 smithi162 bash[24557]: cluster 2024-09-20T19:14:37.816961+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:40.863 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 4 on host 'smithi162' 2024-09-20T19:14:41.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.161 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.161 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.161 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:40 smithi162 bash[24557]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[22706]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: cluster 2024-09-20T19:14:39.817514+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.840018+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.847762+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:40 smithi042 bash[32075]: audit 2024-09-20T19:14:40.855783+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:41.584 DEBUG:teuthology.orchestra.run.smithi162:osd.4> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T19:14:41.586 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-09-20T19:14:41.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:14:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[22706]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[22706]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[22706]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[22706]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[32075]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[32075]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[32075]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:41 smithi042 bash[32075]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:41 smithi162 bash[24557]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:41 smithi162 bash[24557]: audit 2024-09-20T19:14:40.970265+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:41 smithi162 bash[24557]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:42.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:41 smithi162 bash[24557]: audit 2024-09-20T19:14:40.972804+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:14:43.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[32075]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:42 smithi042 bash[22706]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.818090+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.888149+0000 mon.a (mon.0) 523 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.894147+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:14:43.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.895053+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:43.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.895766+0000 mon.a (mon.0) 526 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:42 smithi162 bash[24557]: audit 2024-09-20T19:14:41.896304+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.162:6800/2898101120' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:14:43.806 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[28094]: debug 2024-09-20T19:14:43.407+0000 7f43ca4c9640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T19:14:43.806 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[28094]: debug 2024-09-20T19:14:43.411+0000 7f43c1ad4640 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:14:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[22706]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:43 smithi042 bash[32075]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.890567+0000 mon.a (mon.0) 527 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: cluster 2024-09-20T19:14:42.898643+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.899842+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:42.908949+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:44.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:44.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:43 smithi162 bash[24557]: audit 2024-09-20T19:14:43.416944+0000 mon.a (mon.0) 531 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[22706]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:44 smithi042 bash[32075]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.956986+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:41.957100+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.818615+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.900105+0000 mon.a (mon.0) 532 : cluster [INF] osd.4 v1:172.21.15.162:6800/2898101120 boot 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:43.900209+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: audit 2024-09-20T19:14:43.900533+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:44 smithi162 bash[24557]: cluster 2024-09-20T19:14:44.901147+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:14:45.892 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[22706]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[22706]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[22706]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[22706]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[32075]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[32075]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[32075]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:46 smithi042 bash[32075]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:47.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:46 smithi162 bash[24557]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:46 smithi162 bash[24557]: cluster 2024-09-20T19:14:45.819177+0000 mgr.y (mgr.14152) 251 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:47.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:46 smithi162 bash[24557]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:47.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:46 smithi162 bash[24557]: cluster 2024-09-20T19:14:45.904800+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-20T19:14:49.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:48 smithi042 bash[22706]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:49.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:48 smithi042 bash[22706]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:48 smithi042 bash[32075]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:48 smithi042 bash[32075]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:49.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:48 smithi162 bash[24557]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:48 smithi162 bash[24557]: cluster 2024-09-20T19:14:47.819722+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v227: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:50.842 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:14:50.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-09-20T19:14:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:50 smithi042 bash[22706]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:50 smithi042 bash[22706]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:51.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:50 smithi042 bash[32075]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:51.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:50 smithi042 bash[32075]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:51.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:50 smithi162 bash[24557]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:51.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:50 smithi162 bash[24557]: cluster 2024-09-20T19:14:49.820267+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v228: 1 pgs: 1 peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:14:52.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[22706]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:51 smithi042 bash[32075]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.776539+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:14:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.776631+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:14:52.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:52.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:51 smithi162 bash[24557]: cluster 2024-09-20T19:14:50.920012+0000 mon.a (mon.0) 537 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-20T19:14:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:52 smithi042 bash[22706]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:53.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:52 smithi042 bash[22706]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:52 smithi042 bash[32075]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:52 smithi042 bash[32075]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:53.215 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:52 smithi162 bash[24557]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:53.215 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:52 smithi162 bash[24557]: cluster 2024-09-20T19:14:51.820850+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T19:14:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:54.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[32075]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:54.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:54 smithi042 bash[22706]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:55.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:55.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cephadm 2024-09-20T19:14:53.641105+0000 mgr.y (mgr.14152) 255 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:14:55.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.650262+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.658411+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.660955+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cephadm 2024-09-20T19:14:53.662249+0000 mgr.y (mgr.14152) 256 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 17265M 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.668559+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.673159+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.674916+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: audit 2024-09-20T19:14:53.685181+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:14:55.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:54 smithi162 bash[24557]: cluster 2024-09-20T19:14:53.821492+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:14:56.099 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:14:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:56 smithi042 bash[22706]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:56 smithi042 bash[22706]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:56 smithi042 bash[32075]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:56 smithi042 bash[32075]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:57.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:56 smithi162 bash[24557]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:57.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:56 smithi162 bash[24557]: cluster 2024-09-20T19:14:55.822234+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:58.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[22706]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:57 smithi042 bash[32075]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.425124+0000 mgr.y (mgr.14152) 259 : audit [DBG] from='client.24262 v1:172.21.15.162:0/3514533334' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.431734+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.438006+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:14:58.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:58.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:57 smithi162 bash[24557]: audit 2024-09-20T19:14:57.439703+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:14:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:58 smithi042 bash[22706]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:14:58 smithi042 bash[22706]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:59.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:58 smithi042 bash[32075]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:59.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:14:58 smithi042 bash[32075]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:59.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:58 smithi162 bash[24557]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:14:59.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:14:58 smithi162 bash[24557]: cluster 2024-09-20T19:14:57.822927+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.146 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:00 smithi162 bash[24557]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.146 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:00 smithi162 bash[24557]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:00 smithi042 bash[22706]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:00 smithi042 bash[22706]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:00 smithi042 bash[32075]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:01.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:00 smithi042 bash[32075]: cluster 2024-09-20T19:14:59.823473+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:03.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:02 smithi042 bash[32075]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:03.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:02 smithi042 bash[32075]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:03.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:02 smithi042 bash[22706]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:03.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:02 smithi042 bash[22706]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:03.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:02 smithi162 bash[24557]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:03.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:02 smithi162 bash[24557]: cluster 2024-09-20T19:15:01.824027+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:04 smithi162 bash[24557]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:04 smithi162 bash[24557]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:04 smithi042 bash[22706]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:04 smithi042 bash[22706]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:04 smithi042 bash[32075]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:05.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:04 smithi042 bash[32075]: cluster 2024-09-20T19:15:03.824557+0000 mgr.y (mgr.14152) 263 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[32075]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:07.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:06 smithi042 bash[22706]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.633972+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.634061+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.825192+0000 mgr.y (mgr.14152) 264 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.915492+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.949545+0000 mon.a (mon.0) 549 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.951996+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.162:0/1265551414' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"} : dispatch 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.952790+0000 mon.a (mon.0) 550 : audit [INF] from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27"}]': finished 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: cluster 2024-09-20T19:15:05.960396+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:05.961025+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:06 smithi162 bash[24557]: audit 2024-09-20T19:15:06.622585+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.162:0/2598466413' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:15:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:08 smithi042 bash[22706]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:08 smithi042 bash[22706]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:09.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:08 smithi042 bash[32075]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:09.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:08 smithi042 bash[32075]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:09.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:08 smithi162 bash[24557]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:09.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:08 smithi162 bash[24557]: cluster 2024-09-20T19:15:07.825768+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:10 smithi042 bash[22706]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:10 smithi042 bash[22706]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:10 smithi042 bash[32075]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:10 smithi042 bash[32075]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:10 smithi162 bash[24557]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:11.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:10 smithi162 bash[24557]: cluster 2024-09-20T19:15:09.826302+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:12 smithi042 bash[22706]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:12 smithi042 bash[22706]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:12 smithi042 bash[32075]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:12 smithi042 bash[32075]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:12 smithi162 bash[24557]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:13.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:12 smithi162 bash[24557]: cluster 2024-09-20T19:15:11.826873+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:14 smithi042 bash[22706]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:14 smithi042 bash[22706]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:14 smithi042 bash[32075]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:14 smithi042 bash[32075]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:14 smithi162 bash[24557]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:14 smithi162 bash[24557]: cluster 2024-09-20T19:15:13.827470+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:16 smithi162 bash[24557]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:16 smithi162 bash[24557]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:16 smithi042 bash[22706]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:16 smithi042 bash[22706]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:16 smithi042 bash[32075]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:16 smithi042 bash[32075]: cluster 2024-09-20T19:15:15.828143+0000 mgr.y (mgr.14152) 269 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:18 smithi162 bash[24557]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:18 smithi162 bash[24557]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:18 smithi042 bash[22706]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:18 smithi042 bash[22706]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:18 smithi042 bash[32075]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:18 smithi042 bash[32075]: cluster 2024-09-20T19:15:17.828830+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:20.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:20.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:19 smithi162 bash[24557]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:20.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[22706]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: audit 2024-09-20T19:15:19.397230+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: audit 2024-09-20T19:15:19.400226+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:20.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:19 smithi042 bash[32075]: cephadm 2024-09-20T19:15:19.401757+0000 mgr.y (mgr.14152) 271 : cephadm [INF] Deploying daemon osd.5 on smithi162 2024-09-20T19:15:21.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:21.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:20 smithi162 bash[24557]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[22706]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.655300+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.658487+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:21.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:20 smithi042 bash[32075]: cluster 2024-09-20T19:15:19.829487+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:22 smithi162 bash[24557]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:22 smithi162 bash[24557]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:22 smithi042 bash[22706]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:22 smithi042 bash[22706]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:22 smithi042 bash[32075]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:22 smithi042 bash[32075]: cluster 2024-09-20T19:15:21.830100+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:23.933 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:15:23 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:23.934 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:15:23 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:23.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:23 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:24.283 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:24.284 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:15:24 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:24.284 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:15:24 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:15:25.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:24 smithi162 bash[24557]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[22706]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: cluster 2024-09-20T19:15:23.830628+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:25.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.214661+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:25.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.224213+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:25.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:24 smithi042 bash[32075]: audit 2024-09-20T19:15:24.232908+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:27.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:26 smithi162 bash[24557]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:27.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:26 smithi162 bash[24557]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:27.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:26 smithi042 bash[22706]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:27.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:26 smithi042 bash[22706]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:26 smithi042 bash[32075]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:26 smithi042 bash[32075]: cluster 2024-09-20T19:15:25.831163+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:28 smithi162 bash[24557]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.278 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:28 smithi162 bash[24557]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:28 smithi042 bash[22706]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:28 smithi042 bash[22706]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:28 smithi042 bash[32075]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:29.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:28 smithi042 bash[32075]: cluster 2024-09-20T19:15:27.831730+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[22706]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:30.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:30 smithi042 bash[32075]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.563936+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.572592+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.574611+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.576454+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: audit 2024-09-20T19:15:29.589112+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:31.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:31.007 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:30 smithi162 bash[24557]: cluster 2024-09-20T19:15:29.832280+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[22706]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:31 smithi042 bash[32075]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:32.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:32.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.156489+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:32.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:32.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.156607+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:32.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:32.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: cluster 2024-09-20T19:15:30.595436+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-20T19:15:32.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:32.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:31 smithi162 bash[24557]: audit 2024-09-20T19:15:30.595692+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:32 smithi042 bash[22706]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:32 smithi042 bash[22706]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:32 smithi042 bash[32075]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:32 smithi042 bash[32075]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:33.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:32 smithi162 bash[24557]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:33.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:32 smithi162 bash[24557]: cluster 2024-09-20T19:15:31.832840+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:34 smithi042 bash[22706]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:34 smithi042 bash[22706]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:34 smithi042 bash[32075]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:34 smithi042 bash[32075]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:34 smithi162 bash[24557]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:35.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:34 smithi162 bash[24557]: cluster 2024-09-20T19:15:33.833366+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.026 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-09-20T19:15:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:36 smithi042 bash[22706]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:36 smithi042 bash[22706]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:36 smithi042 bash[32075]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:36 smithi042 bash[32075]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.264 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:36 smithi162 bash[24557]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.265 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:36 smithi162 bash[24557]: cluster 2024-09-20T19:15:35.833899+0000 mgr.y (mgr.14152) 280 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:37.650 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T19:15:37.651 INFO:tasks.cephadm:Deploying osd.6 on smithi162 with /dev/vg_nvme/lv_2... 2024-09-20T19:15:37.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:15:38.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:37 smithi162 bash[24557]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[22706]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.001490+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.009464+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.018281+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:38.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.662583+0000 mon.a (mon.0) 568 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:38.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:37 smithi042 bash[32075]: audit 2024-09-20T19:15:37.665528+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T19:15:39.305 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[33802]: debug 2024-09-20T19:15:39.038+0000 7fedc658d640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T19:15:39.305 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[33802]: debug 2024-09-20T19:15:39.050+0000 7fedc23b4640 -1 osd.5 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:39 smithi162 bash[24557]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[22706]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: cluster 2024-09-20T19:15:37.834476+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.021982+0000 mon.a (mon.0) 569 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.027527+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.028076+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.028930+0000 mon.a (mon.0) 572 : audit [INF] from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:39.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:39 smithi042 bash[32075]: audit 2024-09-20T19:15:38.031445+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.162:6804/4269525575' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:15:40.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:40.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:40 smithi162 bash[24557]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:40.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[22706]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.024405+0000 mon.a (mon.0) 573 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: cluster 2024-09-20T19:15:39.032221+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.033391+0000 mon.a (mon.0) 575 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: audit 2024-09-20T19:15:39.043008+0000 mon.a (mon.0) 576 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:40.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:40 smithi042 bash[32075]: cluster 2024-09-20T19:15:40.033251+0000 mon.a (mon.0) 577 : cluster [INF] osd.5 v1:172.21.15.162:6804/4269525575 boot 2024-09-20T19:15:41.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:41.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:41 smithi162 bash[24557]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[32075]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.618555+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:38.618684+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:39.835011+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 152 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:40.033363+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: audit 2024-09-20T19:15:40.033614+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:41.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:41 smithi042 bash[22706]: cluster 2024-09-20T19:15:41.035173+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:15:42.423 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:15:43.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:43 smithi162 bash[24557]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:43 smithi162 bash[24557]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:43 smithi162 bash[24557]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:43.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:43 smithi162 bash[24557]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[32075]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[32075]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[32075]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[32075]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[22706]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[22706]: cluster 2024-09-20T19:15:41.835625+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v262: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[22706]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:43 smithi042 bash[22706]: cluster 2024-09-20T19:15:42.037815+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:15:45.239 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.239 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.239 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.239 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.240 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.240 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.240 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:45.240 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:45 smithi162 bash[24557]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[22706]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.469597+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.469691+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:43.836193+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v264: 1 pgs: 1 remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:45 smithi042 bash[32075]: cluster 2024-09-20T19:15:44.050622+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-20T19:15:47.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:47 smithi162 bash[24557]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:47 smithi162 bash[24557]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:47 smithi042 bash[22706]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:47 smithi042 bash[22706]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:47 smithi042 bash[32075]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:47 smithi042 bash[32075]: cluster 2024-09-20T19:15:45.836745+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:15:47.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-09-20T19:15:49.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:49 smithi162 bash[24557]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:49.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:49 smithi162 bash[24557]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:49 smithi042 bash[22706]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:49 smithi042 bash[22706]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:49 smithi042 bash[32075]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:49 smithi042 bash[32075]: cluster 2024-09-20T19:15:47.837285+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[32075]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:51 smithi042 bash[22706]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cluster 2024-09-20T19:15:49.837833+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cephadm 2024-09-20T19:15:50.401881+0000 mgr.y (mgr.14152) 288 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.410986+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.420185+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.423089+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.424728+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: cephadm 2024-09-20T19:15:50.425924+0000 mgr.y (mgr.14152) 289 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8632M 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.432599+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.437153+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.438737+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:51 smithi162 bash[24557]: audit 2024-09-20T19:15:50.448735+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:15:52.860 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:15:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:53 smithi042 bash[22706]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:53 smithi042 bash[22706]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:53.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:53 smithi042 bash[32075]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:53.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:53 smithi042 bash[32075]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:53 smithi162 bash[24557]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:53 smithi162 bash[24557]: cluster 2024-09-20T19:15:51.838419+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[22706]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:54 smithi042 bash[32075]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.336332+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.342234+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:54 smithi162 bash[24557]: audit 2024-09-20T19:15:54.343708+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[22706]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[22706]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[22706]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[22706]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[32075]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[32075]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[32075]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:55 smithi042 bash[32075]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:55 smithi162 bash[24557]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:55 smithi162 bash[24557]: cluster 2024-09-20T19:15:53.838946+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T19:15:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:55 smithi162 bash[24557]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:55 smithi162 bash[24557]: audit 2024-09-20T19:15:54.329779+0000 mgr.y (mgr.14152) 292 : audit [DBG] from='client.24289 v1:172.21.15.162:0/787829442' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:15:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:57 smithi042 bash[22706]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:57 smithi042 bash[22706]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:57.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:57 smithi042 bash[32075]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:57.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:57 smithi042 bash[32075]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:57.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:57 smithi162 bash[24557]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:57.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:57 smithi162 bash[24557]: cluster 2024-09-20T19:15:55.839507+0000 mgr.y (mgr.14152) 293 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T19:15:59.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:59 smithi162 bash[24557]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:59.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:15:59 smithi162 bash[24557]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:59.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:59 smithi042 bash[22706]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:59.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:15:59 smithi042 bash[22706]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:59 smithi042 bash[32075]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:15:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:15:59 smithi042 bash[32075]: cluster 2024-09-20T19:15:57.840046+0000 mgr.y (mgr.14152) 294 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:01.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:15:59.840594+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.316916+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.317044+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:00.449306+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:03.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:03 smithi162 bash[24557]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[32075]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: cluster 2024-09-20T19:16:01.841118+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.873125+0000 mon.a (mon.0) 595 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.875991+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.162:0/3715493793' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"} : dispatch 2024-09-20T19:16:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.876661+0000 mon.a (mon.0) 596 : audit [INF] from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a"}]': finished 2024-09-20T19:16:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: cluster 2024-09-20T19:16:02.882727+0000 mon.a (mon.0) 597 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-20T19:16:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:03 smithi042 bash[22706]: audit 2024-09-20T19:16:02.883422+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:04.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:04 smithi162 bash[24557]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:04.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:04 smithi162 bash[24557]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:04.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:04 smithi042 bash[22706]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:04.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:04 smithi042 bash[22706]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:04 smithi042 bash[32075]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:04 smithi042 bash[32075]: audit 2024-09-20T19:16:03.567861+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.162:0/1725332902' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:16:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:05 smithi162 bash[24557]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:05 smithi162 bash[24557]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:05 smithi042 bash[22706]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:05 smithi042 bash[22706]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:05 smithi042 bash[32075]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:05 smithi042 bash[32075]: cluster 2024-09-20T19:16:03.841636+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:07 smithi162 bash[24557]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:07 smithi162 bash[24557]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:07 smithi042 bash[22706]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:07 smithi042 bash[22706]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:07 smithi042 bash[32075]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:07 smithi042 bash[32075]: cluster 2024-09-20T19:16:05.842244+0000 mgr.y (mgr.14152) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:09 smithi162 bash[24557]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:09 smithi162 bash[24557]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:09 smithi042 bash[22706]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:09 smithi042 bash[22706]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:09 smithi042 bash[32075]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:09 smithi042 bash[32075]: cluster 2024-09-20T19:16:07.842852+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:11 smithi162 bash[24557]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:11 smithi162 bash[24557]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:11 smithi042 bash[22706]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:11 smithi042 bash[22706]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:11 smithi042 bash[32075]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:11 smithi042 bash[32075]: cluster 2024-09-20T19:16:09.843451+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:13 smithi162 bash[24557]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:13 smithi162 bash[24557]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:13 smithi042 bash[32075]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:13 smithi042 bash[32075]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:13 smithi042 bash[22706]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:13 smithi042 bash[22706]: cluster 2024-09-20T19:16:11.844115+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:15 smithi162 bash[24557]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:15 smithi162 bash[24557]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:15 smithi042 bash[22706]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:15 smithi042 bash[22706]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:15 smithi042 bash[32075]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:15 smithi042 bash[32075]: cluster 2024-09-20T19:16:13.844703+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:17.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:17 smithi162 bash[24557]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[22706]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: cluster 2024-09-20T19:16:15.845278+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: audit 2024-09-20T19:16:16.704954+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:16:17.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:17.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:17 smithi042 bash[32075]: audit 2024-09-20T19:16:16.706631+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:18.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:18 smithi162 bash[24557]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:18.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:18 smithi162 bash[24557]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:18.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:18 smithi042 bash[22706]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:18 smithi042 bash[22706]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:18 smithi042 bash[32075]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:18 smithi042 bash[32075]: cephadm 2024-09-20T19:16:16.708149+0000 mgr.y (mgr.14152) 304 : cephadm [INF] Deploying daemon osd.6 on smithi162 2024-09-20T19:16:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:19 smithi162 bash[24557]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:19 smithi162 bash[24557]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:19 smithi042 bash[22706]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:19 smithi042 bash[22706]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:19 smithi042 bash[32075]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:19 smithi042 bash[32075]: cluster 2024-09-20T19:16:17.845925+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:20 smithi162 bash[24557]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[22706]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.479125+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.479233+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.504630+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:20 smithi042 bash[32075]: audit 2024-09-20T19:16:19.504915+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:21.554 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:21.555 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:21.555 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:21 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:21 smithi162 bash[24557]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:21 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:21 smithi042 bash[22706]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:21.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:21 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:21.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:21 smithi042 bash[32075]: cluster 2024-09-20T19:16:19.846527+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:22.054 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:22.055 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:22.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:22.055 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:16:21 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:22 smithi162 bash[24557]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[22706]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.798150+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.805740+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:22 smithi042 bash[32075]: audit 2024-09-20T19:16:21.814522+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:23 smithi162 bash[24557]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:23.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:23 smithi162 bash[24557]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:23.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:23 smithi042 bash[22706]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:23.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:23 smithi042 bash[22706]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:23.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:23 smithi042 bash[32075]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:23.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:23 smithi042 bash[32075]: cluster 2024-09-20T19:16:21.847107+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:25 smithi162 bash[24557]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:25 smithi162 bash[24557]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:25 smithi042 bash[22706]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:25 smithi042 bash[22706]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:25 smithi042 bash[32075]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:25 smithi042 bash[32075]: cluster 2024-09-20T19:16:23.847653+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:26.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:26 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:26 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:26 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:26.793 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:26 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:26.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.648336+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:26.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:26 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.652254+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:27 smithi162 bash[24557]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[22706]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: cluster 2024-09-20T19:16:25.848233+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.167031+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.175523+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.177447+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.179212+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:27.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:27.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:27 smithi042 bash[32075]: audit 2024-09-20T19:16:27.189554+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:29 smithi162 bash[24557]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:29 smithi162 bash[24557]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:29.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:29 smithi042 bash[22706]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:29 smithi042 bash[22706]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:29 smithi042 bash[32075]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:29 smithi042 bash[32075]: cluster 2024-09-20T19:16:27.848789+0000 mgr.y (mgr.14152) 310 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:30.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:30 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:30 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:30 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:30 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.177938+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:30 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.178031+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:31 smithi162 bash[24557]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[32075]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: cluster 2024-09-20T19:16:29.849281+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: cluster 2024-09-20T19:16:30.546718+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-20T19:16:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:31.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:31 smithi042 bash[22706]: audit 2024-09-20T19:16:30.546969+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:33 smithi162 bash[24557]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:33.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:33 smithi162 bash[24557]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:33 smithi042 bash[22706]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:33 smithi042 bash[22706]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:33.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:33 smithi042 bash[32075]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:33.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:33 smithi042 bash[32075]: cluster 2024-09-20T19:16:31.849823+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:34.855 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 6 on host 'smithi162' 2024-09-20T19:16:35.479 DEBUG:teuthology.orchestra.run.smithi162:osd.6> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T19:16:35.482 INFO:tasks.cephadm:Deploying osd.7 on smithi162 with /dev/vg_nvme/lv_1... 2024-09-20T19:16:35.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:35 smithi162 bash[24557]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[22706]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: cluster 2024-09-20T19:16:33.850375+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.830519+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.837915+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:34.846606+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:35.147201+0000 mon.a (mon.0) 616 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:35.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:35 smithi042 bash[32075]: audit 2024-09-20T19:16:35.150578+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[22706]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:36 smithi042 bash[32075]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.845090+0000 mon.a (mon.0) 617 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: cluster 2024-09-20T19:16:35.850866+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: cluster 2024-09-20T19:16:35.852887+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.853469+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.854264+0000 mon.a (mon.0) 620 : audit [INF] from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[24557]: audit 2024-09-20T19:16:35.855158+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.162:6808/1785075253' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:16:37.306 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[39432]: debug 2024-09-20T19:16:36.866+0000 7faea9592640 -1 osd.6 0 waiting for initial osdmap 2024-09-20T19:16:37.306 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:16:36 smithi162 bash[39432]: debug 2024-09-20T19:16:36.878+0000 7faea53b9640 -1 osd.6 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[22706]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:37 smithi042 bash[32075]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.853642+0000 mon.a (mon.0) 621 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.863426+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.864151+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:38.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:37 smithi162 bash[24557]: audit 2024-09-20T19:16:36.868843+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[22706]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:38 smithi042 bash[32075]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.129475+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:36.129610+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.851514+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.874524+0000 mon.a (mon.0) 625 : cluster [INF] osd.6 v1:172.21.15.162:6808/1785075253 boot 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: cluster 2024-09-20T19:16:37.874632+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:38 smithi162 bash[24557]: audit 2024-09-20T19:16:37.875862+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:16:39.914 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:16:40.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:39 smithi042 bash[22706]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:39 smithi042 bash[22706]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:40.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:39 smithi042 bash[32075]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:40.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:39 smithi042 bash[32075]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:40.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:39 smithi162 bash[24557]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:40.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:39 smithi162 bash[24557]: cluster 2024-09-20T19:16:38.882312+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-20T19:16:41.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[22706]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[22706]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[22706]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[22706]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[32075]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[32075]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[32075]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:40 smithi042 bash[32075]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:41.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:40 smithi162 bash[24557]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:40 smithi162 bash[24557]: cluster 2024-09-20T19:16:39.852189+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:41.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:40 smithi162 bash[24557]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:41.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:40 smithi162 bash[24557]: cluster 2024-09-20T19:16:39.892373+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-20T19:16:43.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:42 smithi042 bash[22706]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:42 smithi042 bash[22706]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:43.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:42 smithi042 bash[32075]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:43.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:42 smithi042 bash[32075]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:43.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:42 smithi162 bash[24557]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:43.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:42 smithi162 bash[24557]: cluster 2024-09-20T19:16:41.852917+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.055 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-20T19:16:45.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-09-20T19:16:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:44 smithi042 bash[22706]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:44 smithi042 bash[22706]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:44 smithi042 bash[32075]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:44 smithi042 bash[32075]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:44 smithi162 bash[24557]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:45.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:44 smithi162 bash[24557]: cluster 2024-09-20T19:16:43.853630+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:46 smithi042 bash[22706]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:46 smithi042 bash[22706]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:46 smithi042 bash[32075]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:46 smithi042 bash[32075]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.266 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:46 smithi162 bash[24557]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:47.266 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:46 smithi162 bash[24557]: cluster 2024-09-20T19:16:45.854356+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:48 smithi162 bash[24557]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cluster 2024-09-20T19:16:47.855043+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cephadm 2024-09-20T19:16:47.971685+0000 mgr.y (mgr.14152) 321 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.980552+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.989510+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.992389+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.993874+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:47.995360+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: cephadm 2024-09-20T19:16:47.996531+0000 mgr.y (mgr.14152) 322 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5755M 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.003030+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.007438+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[32075]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.009123+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:16:49.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:49.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:48 smithi042 bash[22706]: audit 2024-09-20T19:16:48.019703+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:16:50.444 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:16:51.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:50 smithi162 bash[24557]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:51.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:50 smithi162 bash[24557]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:50 smithi042 bash[22706]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:50 smithi042 bash[22706]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:50 smithi042 bash[32075]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:50 smithi042 bash[32075]: cluster 2024-09-20T19:16:49.855656+0000 mgr.y (mgr.14152) 323 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:52.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:51 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:51 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:51 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:52.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:52 smithi162 bash[24557]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[22706]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.912140+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.912259+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: cluster 2024-09-20T19:16:50.987660+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.861423+0000 mon.a (mon.0) 640 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.867383+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-20T19:16:52.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:52.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:51 smithi042 bash[32075]: audit 2024-09-20T19:16:51.869033+0000 mon.a (mon.0) 642 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:16:53.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:53 smithi162 bash[24557]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:53 smithi162 bash[24557]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:53 smithi162 bash[24557]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:53.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:53 smithi162 bash[24557]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[22706]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[22706]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[22706]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[22706]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[32075]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[32075]: audit 2024-09-20T19:16:51.854630+0000 mgr.y (mgr.14152) 324 : audit [DBG] from='client.24316 v1:172.21.15.162:0/1560904226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[32075]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:53 smithi042 bash[32075]: cluster 2024-09-20T19:16:51.856213+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:55 smithi162 bash[24557]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:55 smithi162 bash[24557]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:55 smithi042 bash[22706]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:55 smithi042 bash[22706]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:55 smithi042 bash[32075]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:55 smithi042 bash[32075]: cluster 2024-09-20T19:16:53.856764+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:57.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:57 smithi162 bash[24557]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[32075]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.655994+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.656081+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:55.857341+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:57 smithi042 bash[22706]: cluster 2024-09-20T19:16:56.022438+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-20T19:16:59.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:59 smithi162 bash[24557]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:59.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:16:59 smithi162 bash[24557]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:59 smithi042 bash[22706]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:16:59 smithi042 bash[22706]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:59 smithi042 bash[32075]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:16:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:16:59 smithi042 bash[32075]: cluster 2024-09-20T19:16:57.857913+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:01.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:01 smithi162 bash[24557]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[22706]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: cluster 2024-09-20T19:16:59.858497+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.520731+0000 mon.a (mon.0) 644 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.523510+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.162:0/660820396' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.524497+0000 mon.a (mon.0) 645 : audit [INF] from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e"}]': finished 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.530353+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:01.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:01 smithi042 bash[32075]: audit 2024-09-20T19:17:00.530871+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[22706]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:02 smithi042 bash[32075]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:02.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.248054+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:02.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: cluster 2024-09-20T19:17:00.248171+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:02.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:02.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:02 smithi162 bash[24557]: audit 2024-09-20T19:17:01.190367+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.162:0/2456099352' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-20T19:17:03.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:03 smithi162 bash[24557]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:03.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:03 smithi162 bash[24557]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:03 smithi042 bash[22706]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:03 smithi042 bash[22706]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:03.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:03 smithi042 bash[32075]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:03.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:03 smithi042 bash[32075]: cluster 2024-09-20T19:17:01.859094+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:05 smithi162 bash[24557]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:05 smithi162 bash[24557]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:05 smithi042 bash[22706]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:05 smithi042 bash[22706]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:05 smithi042 bash[32075]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:05.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:05 smithi042 bash[32075]: cluster 2024-09-20T19:17:03.859655+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:07 smithi162 bash[24557]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:07 smithi162 bash[24557]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:07 smithi042 bash[32075]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:07 smithi042 bash[32075]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:07 smithi042 bash[22706]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:07 smithi042 bash[22706]: cluster 2024-09-20T19:17:05.860205+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:09 smithi162 bash[24557]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:09 smithi162 bash[24557]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:09 smithi042 bash[32075]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:09 smithi042 bash[32075]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:09 smithi042 bash[22706]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:09 smithi042 bash[22706]: cluster 2024-09-20T19:17:07.860832+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:11 smithi162 bash[24557]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:11 smithi162 bash[24557]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:11 smithi042 bash[22706]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:11 smithi042 bash[22706]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:11 smithi042 bash[32075]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:11.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:11 smithi042 bash[32075]: cluster 2024-09-20T19:17:09.861440+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:13 smithi162 bash[24557]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:13 smithi162 bash[24557]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:13 smithi042 bash[32075]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:13 smithi042 bash[32075]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:13 smithi042 bash[22706]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:13 smithi042 bash[22706]: cluster 2024-09-20T19:17:11.862017+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:14.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:14 smithi162 bash[24557]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:14 smithi162 bash[24557]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:14 smithi162 bash[24557]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:14.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:14 smithi162 bash[24557]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[22706]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[22706]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[22706]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[22706]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[32075]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[32075]: audit 2024-09-20T19:17:13.744771+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[32075]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:14.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:14 smithi042 bash[32075]: audit 2024-09-20T19:17:13.746510+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:15.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:15 smithi162 bash[24557]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:15 smithi162 bash[24557]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:15 smithi162 bash[24557]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:15.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:15 smithi162 bash[24557]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[32075]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[32075]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[32075]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[32075]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[22706]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[22706]: cephadm 2024-09-20T19:17:13.748060+0000 mgr.y (mgr.14152) 336 : cephadm [INF] Deploying daemon osd.7 on smithi162 2024-09-20T19:17:15.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[22706]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:15.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:15 smithi042 bash[22706]: cluster 2024-09-20T19:17:13.862591+0000 mgr.y (mgr.14152) 337 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:17 smithi162 bash[24557]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:17 smithi162 bash[24557]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:17 smithi042 bash[32075]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:17 smithi042 bash[32075]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:17 smithi042 bash[22706]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:17 smithi042 bash[22706]: cluster 2024-09-20T19:17:15.863172+0000 mgr.y (mgr.14152) 338 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:18.164 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.165 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.165 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.165 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.165 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.165 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.165 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 bash[24557]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[22706]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.041436+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.041520+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.075901+0000 mon.a (mon.0) 650 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:18 smithi042 bash[32075]: audit 2024-09-20T19:17:17.076490+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:18.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.502 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.503 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.503 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:18.503 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:17:18 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:17:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[22706]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:19 smithi042 bash[32075]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: cluster 2024-09-20T19:17:17.863731+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.428055+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:19.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.435620+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:19.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:19 smithi162 bash[24557]: audit 2024-09-20T19:17:18.444476+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:21 smithi042 bash[32075]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:21 smithi042 bash[32075]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:21 smithi042 bash[22706]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:21 smithi042 bash[22706]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:21 smithi162 bash[24557]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:21 smithi162 bash[24557]: cluster 2024-09-20T19:17:19.864295+0000 mgr.y (mgr.14152) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.404 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:23 smithi162 bash[24557]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.404 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:23 smithi162 bash[24557]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:23 smithi042 bash[32075]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:23 smithi042 bash[32075]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:23 smithi042 bash[22706]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:23 smithi042 bash[22706]: cluster 2024-09-20T19:17:21.864879+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:24 smithi162 bash[24557]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[32075]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.238402+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.238500+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:23.829599+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:23.838817+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:23.865439+0000 mgr.y (mgr.14152) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: cluster 2024-09-20T19:17:24.103614+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.103936+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.521926+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.523364+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:25.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:24 smithi042 bash[22706]: audit 2024-09-20T19:17:24.533483+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:27.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:26 smithi042 bash[32075]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:27.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:26 smithi042 bash[32075]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:27.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:26 smithi042 bash[22706]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:26 smithi042 bash[22706]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:27.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:26 smithi162 bash[24557]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:27.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:26 smithi162 bash[24557]: cluster 2024-09-20T19:17:25.866009+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:28 smithi042 bash[32075]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:28 smithi042 bash[32075]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:28 smithi042 bash[22706]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:28 smithi042 bash[22706]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:28 smithi162 bash[24557]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:29.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:28 smithi162 bash[24557]: cluster 2024-09-20T19:17:27.866572+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:30.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:29 smithi162 bash[24557]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:29 smithi162 bash[24557]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:29 smithi162 bash[24557]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:30.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:29 smithi162 bash[24557]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:30.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[22706]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[22706]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[22706]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:30.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[22706]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[32075]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[32075]: cluster 2024-09-20T19:17:28.199012+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:17:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[32075]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:29 smithi042 bash[32075]: cluster 2024-09-20T19:17:28.202122+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:17:31.145 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 7 on host 'smithi162' 2024-09-20T19:17:31.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:30 smithi162 bash[24557]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.201 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:30 smithi162 bash[24557]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:30 smithi042 bash[22706]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:30 smithi042 bash[22706]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:30 smithi042 bash[32075]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:30 smithi042 bash[32075]: cluster 2024-09-20T19:17:29.867140+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:31.809 DEBUG:teuthology.orchestra.run.smithi162:osd.7> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T19:17:31.812 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T19:17:31.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T19:17:32.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[32075]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:31 smithi042 bash[22706]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.120127+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:17:32.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.130010+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.139318+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:32.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.167244+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:32.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:31 smithi162 bash[24557]: audit 2024-09-20T19:17:31.170882+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[22706]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:33 smithi042 bash[32075]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.554 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[45241]: debug 2024-09-20T19:17:33.161+0000 7f85d9ec5640 -1 osd.7 0 waiting for initial osdmap 2024-09-20T19:17:33.554 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[45241]: debug 2024-09-20T19:17:33.177+0000 7f85d5cec640 -1 osd.7 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:17:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: cluster 2024-09-20T19:17:31.867647+0000 mgr.y (mgr.14152) 346 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.143479+0000 mon.a (mon.0) 666 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.149088+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e62: 8 total, 7 up, 8 in 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.149668+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.150323+0000 mon.a (mon.0) 669 : audit [INF] from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:33 smithi162 bash[24557]: audit 2024-09-20T19:17:32.153159+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.162:6812/1845916231' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:17:34.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[22706]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:34 smithi042 bash[32075]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.146129+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: cluster 2024-09-20T19:17:33.152982+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.154190+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:34.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:34 smithi162 bash[24557]: audit 2024-09-20T19:17:33.163419+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[22706]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:35 smithi042 bash[32075]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.137927+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:32.138030+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:33.868181+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:34.156353+0000 mon.a (mon.0) 674 : cluster [INF] osd.7 v1:172.21.15.162:6812/1845916231 boot 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:34.156512+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: audit 2024-09-20T19:17:34.156944+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:35.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:35 smithi162 bash[24557]: cluster 2024-09-20T19:17:35.158531+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:17:36.597 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:17:37.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:37 smithi042 bash[22706]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:37.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:37 smithi042 bash[22706]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:37.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:37 smithi042 bash[32075]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:37.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:37 smithi042 bash[32075]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:37.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:37 smithi162 bash[24557]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:37.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:37 smithi162 bash[24557]: cluster 2024-09-20T19:17:35.868749+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:38.346 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:17:38.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:38 smithi042 bash[22706]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:38.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:38 smithi042 bash[22706]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:38.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:38 smithi042 bash[32075]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:38.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:38 smithi042 bash[32075]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:38 smithi162 bash[24557]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:38.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:38 smithi162 bash[24557]: cluster 2024-09-20T19:17:37.184399+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:17:39.051 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":66,"num_osds":8,"num_up_osds":8,"osd_up_since":1726859854,"num_in_osds":8,"osd_in_since":1726859820,"num_remapped_pgs":0} 2024-09-20T19:17:39.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:17:39.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:39 smithi162 bash[24557]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:39 smithi162 bash[24557]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:39 smithi162 bash[24557]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:39.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:39 smithi162 bash[24557]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[32075]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[32075]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[32075]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[32075]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:39.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[22706]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[22706]: cluster 2024-09-20T19:17:37.869322+0000 mgr.y (mgr.14152) 349 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[22706]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:39.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:39 smithi042 bash[22706]: audit 2024-09-20T19:17:38.345404+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.42:0/2583411347' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-20T19:17:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:41 smithi162 bash[24557]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: cluster 2024-09-20T19:17:39.870092+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.668025+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.677277+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.680294+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.682007+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.683624+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.685175+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.692927+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.697474+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[22706]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.699092+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:41.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:41 smithi042 bash[32075]: audit 2024-09-20T19:17:40.709045+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:17:42.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:42 smithi162 bash[24557]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:42 smithi162 bash[24557]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:42 smithi162 bash[24557]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:42.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:42 smithi162 bash[24557]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:42.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[22706]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[22706]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[22706]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:42.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[22706]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[32075]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[32075]: cephadm 2024-09-20T19:17:40.659093+0000 mgr.y (mgr.14152) 351 : cephadm [INF] Detected new or changed devices on smithi162 2024-09-20T19:17:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[32075]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:42 smithi042 bash[32075]: cephadm 2024-09-20T19:17:40.686422+0000 mgr.y (mgr.14152) 352 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4316M 2024-09-20T19:17:43.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:43 smithi162 bash[24557]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:43 smithi162 bash[24557]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:43 smithi042 bash[22706]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:43 smithi042 bash[22706]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:43 smithi042 bash[32075]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:43 smithi042 bash[32075]: cluster 2024-09-20T19:17:41.870599+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:17:45.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:45 smithi162 bash[24557]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:45.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:45 smithi162 bash[24557]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:45.636 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:17:45.637 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":66,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:17:37.169064+0000","last_up_change":"2024-09-20T19:17:34.145990+0000","last_in_change":"2024-09-20T19:17:00.522021+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":2265043130}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":2265043130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":2265043130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":2265043130}]},"public_addr":"172.21.15.42:6801/2265043130","cluster_addr":"172.21.15.42:6802/2265043130","heartbeat_back_addr":"172.21.15.42:6804/2265043130","heartbeat_front_addr":"172.21.15.42:6803/2265043130","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1507333597}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1507333597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1507333597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1507333597}]},"public_addr":"172.21.15.42:6813/1507333597","cluster_addr":"172.21.15.42:6814/1507333597","heartbeat_back_addr":"172.21.15.42:6816/1507333597","heartbeat_front_addr":"172.21.15.42:6815/1507333597","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":2898101120}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2898101120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2898101120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2898101120}]},"public_addr":"172.21.15.162:6800/2898101120","cluster_addr":"172.21.15.162:6801/2898101120","heartbeat_back_addr":"172.21.15.162:6803/2898101120","heartbeat_front_addr":"172.21.15.162:6802/2898101120","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:16:55.656086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:17:00.248178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:16:30.178035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:17:23.238504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:17:17.041525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:16:50.912265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:16:36.129616+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:17:32.138034+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:17:45.648 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:45 smithi042 bash[32075]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:45.648 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:45 smithi042 bash[32075]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:45.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:45 smithi042 bash[22706]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:45.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:45 smithi042 bash[22706]: cluster 2024-09-20T19:17:43.871064+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:46.418 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T19:12:46.311971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T19:17:46.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T19:17:46.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:46 smithi162 bash[24557]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:46.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:46 smithi162 bash[24557]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:46.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:46 smithi042 bash[22706]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:46.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:46 smithi042 bash[22706]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:46.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:46 smithi042 bash[32075]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:46.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:46 smithi042 bash[32075]: audit 2024-09-20T19:17:45.634575+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.42:0/3789061469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:17:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:47 smithi162 bash[24557]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:47.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:47 smithi162 bash[24557]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:47.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:47 smithi042 bash[22706]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:47.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:47 smithi042 bash[22706]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:47.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:47 smithi042 bash[32075]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:47.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:47 smithi042 bash[32075]: cluster 2024-09-20T19:17:45.871718+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:48.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:48 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:48 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:48 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:48.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:48 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:48.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.981704+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:48.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:48.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:48 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.981811+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:49.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:49 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:49 smithi162 bash[24557]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:49 smithi162 bash[24557]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:49.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:49 smithi162 bash[24557]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[22706]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[22706]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[22706]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[32075]: cluster 2024-09-20T19:17:47.872320+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:49.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[32075]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:49.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:49 smithi042 bash[32075]: cluster 2024-09-20T19:17:48.232044+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:17:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:17:51.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:51 smithi162 bash[24557]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:51.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:51 smithi162 bash[24557]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:51.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:51 smithi042 bash[32075]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:51.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:51 smithi042 bash[32075]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:51 smithi042 bash[22706]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:51 smithi042 bash[22706]: cluster 2024-09-20T19:17:49.873004+0000 mgr.y (mgr.14152) 357 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:52.831 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 1 2024-09-20T19:17:53.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[32075]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[32075]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[32075]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:53.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[32075]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:53.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[22706]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[22706]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[22706]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:53.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:53 smithi042 bash[22706]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:53.538 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi042 2024-09-20T19:17:53.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi042=foo.a' 2024-09-20T19:17:53.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:53 smithi162 bash[24557]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:53 smithi162 bash[24557]: cluster 2024-09-20T19:17:51.873609+0000 mgr.y (mgr.14152) 358 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:53.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:53 smithi162 bash[24557]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:53.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:53 smithi162 bash[24557]: audit 2024-09-20T19:17:52.830199+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.42:0/534384336' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-20T19:17:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:55 smithi162 bash[24557]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:55 smithi162 bash[24557]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:55.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:55 smithi042 bash[22706]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:55.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:55 smithi042 bash[22706]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:55 smithi042 bash[32075]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:55 smithi042 bash[32075]: cluster 2024-09-20T19:17:53.874207+0000 mgr.y (mgr.14152) 359 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:57 smithi162 bash[24557]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:57 smithi162 bash[24557]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:57 smithi042 bash[22706]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:57 smithi042 bash[22706]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:57 smithi042 bash[32075]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:57 smithi042 bash[32075]: cluster 2024-09-20T19:17:55.874818+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:58.300 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:17:58.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:58.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:58 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.048730+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.048836+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:58.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:58 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.247943+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:17:59.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:59 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:59.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:17:59 smithi162 bash[24557]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:59.653 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled rgw.foo.a update... 2024-09-20T19:17:59.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:59 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:59.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:17:59 smithi042 bash[32075]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:59.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:59 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:17:59.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:17:59 smithi042 bash[22706]: cluster 2024-09-20T19:17:57.875379+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:00.384 DEBUG:teuthology.orchestra.run.smithi042:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@rgw.foo.a.service 2024-09-20T19:18:00.386 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi162 2024-09-20T19:18:00.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-20T19:18:00.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[22706]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:00.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:00 smithi042 bash[32075]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.639790+0000 mgr.y (mgr.14152) 362 : audit [DBG] from='client.24358 v1:172.21.15.162:0/2764188833' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi042=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: cephadm 2024-09-20T19:17:59.643505+0000 mgr.y (mgr.14152) 363 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.650793+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:17:59.652663+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: cluster 2024-09-20T19:17:59.875807+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:18:00.215977+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:00 smithi162 bash[24557]: audit 2024-09-20T19:18:00.220692+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[22706]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:01.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:01 smithi042 bash[32075]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.852069+0000 mon.a (mon.0) 696 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.853596+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.861596+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.867231+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.870047+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.878434+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: audit 2024-09-20T19:18:00.881401+0000 mon.a (mon.0) 702 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:01 smithi162 bash[24557]: cephadm 2024-09-20T19:18:00.883033+0000 mgr.y (mgr.14152) 365 : cephadm [INF] Deploying daemon rgw.foo.a on smithi042 2024-09-20T19:18:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:02 smithi042 bash[22706]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:02 smithi042 bash[22706]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:02 smithi042 bash[32075]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:02 smithi042 bash[32075]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:03.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:02 smithi162 bash[24557]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:03.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:02 smithi162 bash[24557]: cluster 2024-09-20T19:18:01.876363+0000 mgr.y (mgr.14152) 366 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:03.907 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.907 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.907 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.907 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.908 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.908 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:03.908 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:03 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.160 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.160 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.161 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.161 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.161 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.161 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:04.162 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:18:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:05.197 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:05 smithi162 bash[24557]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[32075]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: cluster 2024-09-20T19:18:03.876899+0000 mgr.y (mgr.14152) 367 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.213342+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.218097+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.223070+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: cephadm 2024-09-20T19:18:04.223816+0000 mgr.y (mgr.14152) 368 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi042=foo.a;count:1 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.227806+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.232311+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:05.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:05.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:05 smithi042 bash[22706]: audit 2024-09-20T19:18:04.251834+0000 mon.a (mon.0) 708 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:07.244 INFO:teuthology.orchestra.run.smithi162.stderr:pool 'datapool' created 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:07 smithi162 bash[24557]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[22706]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: cluster 2024-09-20T19:18:05.877519+0000 mgr.y (mgr.14152) 369 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: cluster 2024-09-20T19:18:06.229053+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.239828+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.42:0/701504455' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.240299+0000 mon.a (mon.0) 710 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.722063+0000 mon.a (mon.0) 711 : audit [INF] from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:07.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:07 smithi042 bash[32075]: audit 2024-09-20T19:18:06.725696+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.162:0/1163361944' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-20T19:18:08.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- rbd pool init datapool 2024-09-20T19:18:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:08.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:08 smithi162 bash[24557]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[22706]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: audit 2024-09-20T19:18:07.236535+0000 mon.a (mon.0) 712 : audit [INF] from='client.24350 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: audit 2024-09-20T19:18:07.236770+0000 mon.a (mon.0) 713 : audit [INF] from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:08.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:08 smithi042 bash[32075]: cluster 2024-09-20T19:18:07.245219+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:18:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:09 smithi162 bash[24557]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[22706]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:07.878173+0000 mgr.y (mgr.14152) 370 : cluster [DBG] pgmap v360: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:09.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:08.236548+0000 mon.a (mon.0) 715 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: cluster 2024-09-20T19:18:08.257773+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: audit 2024-09-20T19:18:08.269183+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:09.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:09 smithi042 bash[32075]: audit 2024-09-20T19:18:08.273871+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-20T19:18:10.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:10 smithi162 bash[24557]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:10 smithi162 bash[24557]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:10 smithi162 bash[24557]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:10.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:10 smithi162 bash[24557]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[22706]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[22706]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[22706]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[22706]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[32075]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[32075]: audit 2024-09-20T19:18:09.267462+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[32075]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:10 smithi042 bash[32075]: cluster 2024-09-20T19:18:09.275663+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:18:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[22706]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:11 smithi042 bash[32075]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: cluster 2024-09-20T19:18:09.878893+0000 mgr.y (mgr.14152) 371 : cluster [DBG] pgmap v363: 68 pgs: 13 creating+peering, 42 unknown, 13 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: cluster 2024-09-20T19:18:10.287537+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.290853+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.660164+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.666977+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.667926+0000 mon.a (mon.0) 725 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:11 smithi162 bash[24557]: audit 2024-09-20T19:18:10.668626+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[22706]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:12 smithi042 bash[32075]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: cephadm 2024-09-20T19:18:10.673917+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: audit 2024-09-20T19:18:11.279957+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T19:18:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:12 smithi162 bash[24557]: cluster 2024-09-20T19:18:11.289013+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:18:12.870 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:13.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[22706]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:13 smithi042 bash[32075]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: cluster 2024-09-20T19:18:11.879765+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v366: 100 pgs: 3 creating+activating, 16 creating+peering, 23 unknown, 58 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: cluster 2024-09-20T19:18:12.322496+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.334892+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.337405+0000 mon.a (mon.0) 730 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:13 smithi162 bash[24557]: audit 2024-09-20T19:18:12.338605+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-20T19:18:14.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:14 smithi162 bash[24557]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[22706]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.315563+0000 mon.a (mon.0) 732 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.315804+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: cluster 2024-09-20T19:18:13.324208+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.342620+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.349150+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.42:0/3025752371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:13.352143+0000 mon.a (mon.0) 736 : audit [INF] from='client.24392 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-20T19:18:14.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:14.232075+0000 mon.a (mon.0) 737 : audit [INF] from='client.24418 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:14.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:14 smithi042 bash[32075]: audit 2024-09-20T19:18:14.236026+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.162:0/2294914623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[32075]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:15 smithi042 bash[22706]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: cluster 2024-09-20T19:18:13.880577+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v369: 132 pgs: 3 creating+activating, 16 creating+peering, 49 unknown, 64 active+clean; 578 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336292+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.42:0/1056266775' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336413+0000 mon.a (mon.0) 739 : audit [INF] from='client.24392 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.336587+0000 mon.a (mon.0) 740 : audit [INF] from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: cluster 2024-09-20T19:18:14.347422+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.630750+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.635806+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.644361+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.688101+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.689826+0000 mon.a (mon.0) 746 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:14.690546+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:15.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:15 smithi162 bash[24557]: audit 2024-09-20T19:18:15.001356+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:16.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[22706]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[22706]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[22706]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:16.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[22706]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:16.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[32075]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[32075]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[32075]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:16.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:16 smithi042 bash[32075]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:16 smithi162 bash[24557]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:16 smithi162 bash[24557]: cephadm 2024-09-20T19:18:14.694800+0000 mgr.y (mgr.14152) 375 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-20T19:18:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:16 smithi162 bash[24557]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:16 smithi162 bash[24557]: cluster 2024-09-20T19:18:15.346177+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:18:17.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.162 --placement '1;smithi162=iscsi.a' 2024-09-20T19:18:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:17 smithi162 bash[24557]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[22706]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:15.881547+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v372: 132 pgs: 4 creating+peering, 5 unknown, 123 active+clean; 580 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 117 op/s 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.350243+0000 mon.a (mon.0) 750 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.350290+0000 mon.a (mon.0) 751 : cluster [INF] Cluster is now healthy 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.370538+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.402873+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:17.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:17 smithi042 bash[32075]: cluster 2024-09-20T19:18:16.402954+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:18 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:18.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:18 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:18 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.380000+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:18:18.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:18 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:18.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:18 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.903981+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:18:19.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:19 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:19.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:19 smithi042 bash[22706]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:19.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:19 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:19.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:19 smithi042 bash[32075]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:19.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:19 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:19.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:19 smithi162 bash[24557]: cluster 2024-09-20T19:18:17.882321+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 581 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-09-20T19:18:20.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[22706]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:20 smithi042 bash[32075]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:20.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:18.878655+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:20.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:18.878755+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:20.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:20.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:20 smithi162 bash[24557]: cluster 2024-09-20T19:18:19.397229+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:18:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:21 smithi042 bash[22706]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:21 smithi042 bash[22706]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:21 smithi042 bash[32075]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:21 smithi042 bash[32075]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:21 smithi162 bash[24557]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:21 smithi162 bash[24557]: cluster 2024-09-20T19:18:19.883030+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.2 KiB/s wr, 169 op/s 2024-09-20T19:18:21.972 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:23.430 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled iscsi.datapool update... 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[22706]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[22706]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[22706]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[22706]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[32075]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[32075]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[32075]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:23.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:23 smithi042 bash[32075]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:23 smithi162 bash[24557]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:23 smithi162 bash[24557]: cluster 2024-09-20T19:18:21.884287+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-20T19:18:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:23 smithi162 bash[24557]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:23 smithi162 bash[24557]: cluster 2024-09-20T19:18:22.903066+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:18:24.086 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-20T19:18:24.086 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:18:24.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-20T19:18:24.102 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:18:24.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-20T19:18:24.113 DEBUG:teuthology.orchestra.run.smithi162:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@iscsi.iscsi.a.service 2024-09-20T19:18:24.157 INFO:tasks.cephadm:Adding prometheus.a on smithi162 2024-09-20T19:18:24.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi162=a' 2024-09-20T19:18:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:24.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:24 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.415109+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24466 v1:172.21.15.162:0/1835751435' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.162", "placement": "1;smithi162=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cephadm 2024-09-20T19:18:23.421374+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi162=iscsi.a;count:1 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.428306+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.430291+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.434525+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.436235+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.446644+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.453161+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.456264+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: audit 2024-09-20T19:18:23.479785+0000 mon.a (mon.0) 764 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cephadm 2024-09-20T19:18:23.481816+0000 mgr.y (mgr.14152) 382 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi162 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.562718+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:24.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:24.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:24 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.562825+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:25.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:25 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:25 smithi162 bash[24557]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:25 smithi162 bash[24557]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:25.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:25 smithi162 bash[24557]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[22706]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[22706]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[22706]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[32075]: cluster 2024-09-20T19:18:23.885144+0000 mgr.y (mgr.14152) 383 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 68 op/s 2024-09-20T19:18:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[32075]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:25 smithi042 bash[32075]: cluster 2024-09-20T19:18:24.470405+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:18:26.550 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.551 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.552 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.552 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.552 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.803 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.804 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.804 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.804 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.804 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:26.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:26 smithi162 systemd[1]: Started Ceph iscsi.iscsi.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:18:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:27 smithi162 bash[24557]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:27.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[32075]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: cluster 2024-09-20T19:18:25.886280+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 315 B/s wr, 0 op/s 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.852596+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.861699+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.870036+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.885788+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:27.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:27 smithi042 bash[22706]: audit 2024-09-20T19:18:26.905397+0000 mon.a (mon.0) 770 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:28.716 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.717 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[24557]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[22706]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: cephadm 2024-09-20T19:18:26.871618+0000 mgr.y (mgr.14152) 385 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: cluster 2024-09-20T19:18:27.909948+0000 mon.a (mon.0) 771 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:28 smithi042 bash[32075]: audit 2024-09-20T19:18:28.279140+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:28.936 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:29.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[48936]: debug Started the configuration object watcher 2024-09-20T19:18:29.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[48936]: debug Checking for config object changes every 1s 2024-09-20T19:18:29.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:28 smithi162 bash[48936]: debug Processing osd blocklist entries for this node 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: debug Reading the configuration object to update local LIO configuration 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: debug Configuration does not have an entry for this host(smithi162.front.sepia.ceph.com) - nothing to define to LIO 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: * Environment: production 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: Use a production WSGI server instead. 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: * Debug mode: off 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: debug * Running on all addresses. 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: * Running on all addresses. 2024-09-20T19:18:29.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T19:18:29.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: debug * Running on http://[fe80::ec4:7aff:fe88:7f41]:5000/ (Press CTRL+C to quit) 2024-09-20T19:18:29.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[48936]: * Running on http://[fe80::ec4:7aff:fe88:7f41]:5000/ (Press CTRL+C to quit) 2024-09-20T19:18:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[24557]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[24557]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[24557]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:29 smithi162 bash[24557]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[32075]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[32075]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[32075]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[32075]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[22706]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[22706]: cluster 2024-09-20T19:18:27.887197+0000 mgr.y (mgr.14152) 386 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:18:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[22706]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:29 smithi042 bash[22706]: audit 2024-09-20T19:18:29.094157+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.162:0/2890432247' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-20T19:18:30.629 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled prometheus update... 2024-09-20T19:18:31.355 DEBUG:teuthology.orchestra.run.smithi162:prometheus.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T19:18:31.357 INFO:tasks.cephadm:Adding node-exporter.a on smithi042 2024-09-20T19:18:31.358 INFO:tasks.cephadm:Adding node-exporter.b on smithi162 2024-09-20T19:18:31.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi042=a;smithi162=b' 2024-09-20T19:18:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:31 smithi162 bash[24557]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:31 smithi162 bash[24557]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:31 smithi162 bash[24557]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:31 smithi162 bash[24557]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[32075]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[32075]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[32075]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[32075]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[22706]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[22706]: cluster 2024-09-20T19:18:29.888232+0000 mgr.y (mgr.14152) 387 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[22706]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:31.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:31 smithi042 bash[22706]: audit 2024-09-20T19:18:30.627937+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:32 smithi162 bash[24557]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[22706]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:30.617580+0000 mgr.y (mgr.14152) 388 : audit [DBG] from='client.24502 v1:172.21.15.162:0/2167703861' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: cephadm 2024-09-20T19:18:30.620743+0000 mgr.y (mgr.14152) 389 : cephadm [INF] Saving service prometheus spec with placement smithi162=a;count:1 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:32.164720+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:32 smithi042 bash[32075]: audit 2024-09-20T19:18:32.174411+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:33 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:33.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:33 smithi162 bash[24557]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:33.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[22706]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:31.889385+0000 mgr.y (mgr.14152) 390 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 113 B/s wr, 1 op/s 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.029351+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.029418+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.501039+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.826888+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.828534+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:18:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: audit 2024-09-20T19:18:32.836682+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:33 smithi042 bash[32075]: cluster 2024-09-20T19:18:32.921350+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:18:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:34.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:34 smithi162 bash[24557]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[22706]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cephadm 2024-09-20T19:18:33.111353+0000 mgr.y (mgr.14152) 391 : cephadm [INF] Deploying daemon prometheus.a on smithi162 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cluster 2024-09-20T19:18:34.044394+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:34 smithi042 bash[32075]: cluster 2024-09-20T19:18:34.047484+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:18:35.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:35 smithi162 bash[24557]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:35.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:35 smithi162 bash[24557]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:35 smithi042 bash[22706]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:35 smithi042 bash[22706]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:35.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:35 smithi042 bash[32075]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:35.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:35 smithi042 bash[32075]: cluster 2024-09-20T19:18:33.890119+0000 mgr.y (mgr.14152) 392 : cluster [DBG] pgmap v389: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:37 smithi162 bash[24557]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:37 smithi162 bash[24557]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:37 smithi042 bash[32075]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:37 smithi042 bash[32075]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:37 smithi042 bash[22706]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:37 smithi042 bash[22706]: cluster 2024-09-20T19:18:35.891548+0000 mgr.y (mgr.14152) 393 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 127 B/s wr, 1 op/s 2024-09-20T19:18:39.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:38 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:18:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:39 smithi162 bash[24557]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:39 smithi162 bash[24557]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:39.751 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled node-exporter update... 2024-09-20T19:18:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:39 smithi042 bash[22706]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:39 smithi042 bash[22706]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:39.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:39 smithi042 bash[32075]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:39.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:39 smithi042 bash[32075]: cluster 2024-09-20T19:18:37.892516+0000 mgr.y (mgr.14152) 394 : cluster [DBG] pgmap v391: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:40.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:40 smithi162 bash[24557]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:40.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[22706]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: audit 2024-09-20T19:18:38.816695+0000 mgr.y (mgr.14152) 395 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.290930+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.291022+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.527301+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:40.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:40 smithi042 bash[32075]: audit 2024-09-20T19:18:39.749196+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:41.641 DEBUG:teuthology.orchestra.run.smithi042:node-exporter.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T19:18:41.644 DEBUG:teuthology.orchestra.run.smithi162:node-exporter.b> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T19:18:41.646 INFO:tasks.cephadm:Adding alertmanager.a on smithi042 2024-09-20T19:18:41.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi042=a' 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:41 smithi162 bash[24557]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[22706]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: audit 2024-09-20T19:18:39.738936+0000 mgr.y (mgr.14152) 396 : audit [DBG] from='client.24508 v1:172.21.15.162:0/2696429572' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi042=a;smithi162=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: cephadm 2024-09-20T19:18:39.742202+0000 mgr.y (mgr.14152) 397 : cephadm [INF] Saving service node-exporter spec with placement smithi042=a;smithi162=b;count:2 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:41 smithi042 bash[32075]: cluster 2024-09-20T19:18:39.893297+0000 mgr.y (mgr.14152) 398 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-20T19:18:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:43 smithi162 bash[24557]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:43 smithi162 bash[24557]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:43 smithi162 bash[24557]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:43 smithi162 bash[24557]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[22706]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[22706]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[22706]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[22706]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[32075]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[32075]: cluster 2024-09-20T19:18:41.894514+0000 mgr.y (mgr.14152) 399 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:18:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[32075]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:43 smithi042 bash[32075]: cluster 2024-09-20T19:18:42.927947+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:18:45.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:45 smithi162 bash[24557]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:45.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:45 smithi162 bash[24557]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:45 smithi042 bash[32075]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:45 smithi042 bash[32075]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:45 smithi042 bash[22706]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:45 smithi042 bash[22706]: cluster 2024-09-20T19:18:43.895209+0000 mgr.y (mgr.14152) 400 : cluster [DBG] pgmap v396: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:47 smithi162 bash[24557]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:47 smithi162 bash[24557]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:47 smithi042 bash[32075]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:47 smithi042 bash[32075]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:47.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:47 smithi042 bash[22706]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:47.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:47 smithi042 bash[22706]: cluster 2024-09-20T19:18:45.896464+0000 mgr.y (mgr.14152) 401 : cluster [DBG] pgmap v397: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:18:48.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.121 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.121 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.121 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.121 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.122 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.122 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.401 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.401 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.402 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.402 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.402 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.402 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.403 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.403 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.403 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:18:48.488 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled alertmanager update... 2024-09-20T19:18:48.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:48 smithi162 systemd[1]: Started Ceph prometheus.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:18:49.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:48 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:18:49.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[24557]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.737 DEBUG:teuthology.orchestra.run.smithi042:alertmanager.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T19:18:49.740 INFO:tasks.cephadm:Adding grafana.a on smithi162 2024-09-20T19:18:49.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi162=a' 2024-09-20T19:18:49.748 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22958]: ignoring --setuser ceph since I am not root 2024-09-20T19:18:49.748 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22958]: ignoring --setgroup ceph since I am not root 2024-09-20T19:18:49.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[32075]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: cluster 2024-09-20T19:18:47.897395+0000 mgr.y (mgr.14152) 402 : cluster [DBG] pgmap v398: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.453125+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.463404+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.471425+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.474417+0000 mgr.y (mgr.14152) 403 : audit [DBG] from='client.24514 v1:172.21.15.162:0/2465088480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi042=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.477393+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: cephadm 2024-09-20T19:18:48.479969+0000 mgr.y (mgr.14152) 404 : cephadm [INF] Saving service alertmanager spec with placement smithi042=a;count:1 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:49 smithi042 bash[22706]: audit 2024-09-20T19:18:48.486936+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' 2024-09-20T19:18:49.802 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[25406]: ignoring --setuser ceph since I am not root 2024-09-20T19:18:49.803 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[25406]: ignoring --setgroup ceph since I am not root 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.978Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.978Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.978Z caller=main.go:623 level=info host_details="(Linux 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 smithi162 (none))" 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.978Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.978Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.980Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.980Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.983Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.983Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:18:49.988 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.986Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.986Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.167µs 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.986Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.986Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.986Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.508µs wal_replay_duration=477.323µs wbl_replay_duration=120ns total_replay_duration=523.891µs 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.988Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.988Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.989Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.991Z caller=refresh.go:71 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.991Z caller=refresh.go:71 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.991Z caller=refresh.go:71 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.991Z caller=refresh.go:71 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:18:49.989 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:49 smithi162 bash[49971]: ts=2024-09-20T19:18:49.991Z caller=refresh.go:71 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:18:50.303 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[49971]: ts=2024-09-20T19:18:50.011Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.882003ms db_storage=928ns remote_storage=1.281µs web_handler=447ns query_engine=796ns scrape=332.219µs scrape_sd=144.982µs notify=1.067µs notify_sd=1.276µs rules=21.946222ms tracing=6.618µs 2024-09-20T19:18:50.303 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[49971]: ts=2024-09-20T19:18:50.011Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-20T19:18:50.303 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[49971]: ts=2024-09-20T19:18:50.011Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:18:50.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[24557]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[24557]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[24557]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:50 smithi162 bash[24557]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[32075]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[32075]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[32075]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[32075]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[22706]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[22706]: audit 2024-09-20T19:18:49.471973+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 v1:172.21.15.42:0/3654362393' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[22706]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:50.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:50 smithi042 bash[22706]: cluster 2024-09-20T19:18:49.490046+0000 mon.a (mon.0) 790 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:53 smithi162 bash[24557]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[32075]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:53.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.220212+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.220304+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.279959+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:53.752 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:53 smithi042 bash[22706]: cluster 2024-09-20T19:18:52.928618+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:18:54.514 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:55.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[32075]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:55 smithi042 bash[22706]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[25406]: [20/Sep/2024:19:18:55] ENGINE Bus STARTING 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[25406]: CherryPy Checker: 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[25406]: The Application mounted at '' has an empty config. 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[25406]: [20/Sep/2024:19:18:55] ENGINE Serving on http://:::9283 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[25406]: [20/Sep/2024:19:18:55] ENGINE Bus STARTED 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.599290+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-20T19:18:56.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.599624+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.601521+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.602349+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.605483+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:56.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:55 smithi162 bash[24557]: audit 2024-09-20T19:18:55.606051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.? v1:172.21.15.162:0/1999412793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-20T19:18:56.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.892 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.893 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[32075]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:56.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22706]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:56.899 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled grafana update... 2024-09-20T19:18:56.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.664735+0000 mon.a (mon.0) 795 : cluster [DBG] mgrmap e19: y(active, since 10m), standbys: x 2024-09-20T19:18:56.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.853414+0000 mon.a (mon.0) 796 : cluster [INF] Active manager daemon y restarted 2024-09-20T19:18:56.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.854051+0000 mon.a (mon.0) 797 : cluster [INF] Activating manager daemon y 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.861738+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.862188+0000 mon.a (mon.0) 799 : cluster [DBG] mgrmap e20: y(active, starting, since 0.00834804s), standbys: x 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.870690+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.870881+0000 mon.a (mon.0) 801 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.871063+0000 mon.a (mon.0) 802 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872472+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872730+0000 mon.a (mon.0) 804 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:18:56.913 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.872954+0000 mon.a (mon.0) 805 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873255+0000 mon.a (mon.0) 806 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873463+0000 mon.a (mon.0) 807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873675+0000 mon.a (mon.0) 808 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:18:56.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.873903+0000 mon.a (mon.0) 809 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874111+0000 mon.a (mon.0) 810 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874346+0000 mon.a (mon.0) 811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874569+0000 mon.a (mon.0) 812 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874802+0000 mon.a (mon.0) 813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.874945+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:55.875343+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: cluster 2024-09-20T19:18:55.903946+0000 mon.a (mon.0) 816 : cluster [INF] Manager daemon y is now available 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.359733+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.380275+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.385125+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.385467+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.393797+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:56.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:56 smithi162 bash[24557]: audit 2024-09-20T19:18:56.415685+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:18:57.179 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22958]: [20/Sep/2024:19:18:56] ENGINE Bus STARTING 2024-09-20T19:18:57.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22958]: CherryPy Checker: 2024-09-20T19:18:57.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:56 smithi042 bash[22958]: The Application mounted at '' has an empty config. 2024-09-20T19:18:57.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22958]: [20/Sep/2024:19:18:57] ENGINE Serving on http://:::9283 2024-09-20T19:18:57.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22958]: [20/Sep/2024:19:18:57] ENGINE Bus STARTED 2024-09-20T19:18:57.686 DEBUG:teuthology.orchestra.run.smithi162:grafana.a> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T19:18:57.688 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T19:18:57.688 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:57 smithi162 bash[24557]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[32075]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cluster 2024-09-20T19:18:56.879471+0000 mon.a (mon.0) 823 : cluster [DBG] mgrmap e21: y(active, since 1.0256s), standbys: x 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cephadm 2024-09-20T19:18:56.887312+0000 mgr.y (mgr.24491) 3 : cephadm [INF] Saving service grafana spec with placement smithi162=a;count:1 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: cluster 2024-09-20T19:18:56.889572+0000 mgr.y (mgr.24491) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:56.892345+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:57.259853+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:58.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:57 smithi042 bash[22706]: audit 2024-09-20T19:18:57.263990+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:18:59.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[32075]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22706]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:18:59.181 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: debug 2024-09-20T19:18:58.675+0000 7f16b92a2640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: Traceback (most recent call last): 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: self._connections.run(self.expiration_interval) 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: self._run(expiration_interval) 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: s = self.context.wrap_socket( 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: return self.sslsocket_class._create( 2024-09-20T19:18:59.182 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:18:59.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: self.do_handshake() 2024-09-20T19:18:59.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:18:59.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: self._sslobj.do_handshake() 2024-09-20T19:18:59.183 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:18:58 smithi042 bash[22958]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:18:59.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:57.874591+0000 mgr.y (mgr.24491) 5 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:57.927530+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.453360+0000 mgr.y (mgr.24491) 6 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTING 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:18:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:58 smithi162 bash[24557]: cluster 2024-09-20T19:18:58.899240+0000 mon.a (mon.0) 828 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-20T19:19:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: Traceback (most recent call last): 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._run(expiration_interval) 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: s = self.context.wrap_socket( 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: return self.sslsocket_class._create( 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self.do_handshake() 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._sslobj.do_handshake() 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: Traceback (most recent call last): 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._run(expiration_interval) 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: s = self.context.wrap_socket( 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: return self.sslsocket_class._create( 2024-09-20T19:19:00.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self.do_handshake() 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: self._sslobj.do_handshake() 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:00.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:18:59 smithi162 bash[24557]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: Traceback (most recent call last): 2024-09-20T19:19:00.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._run(expiration_interval) 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: s = self.context.wrap_socket( 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: return self.sslsocket_class._create( 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self.do_handshake() 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._sslobj.do_handshake() 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: Traceback (most recent call last): 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._run(expiration_interval) 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.555071+0000 mgr.y (mgr.24491) 7 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on http://172.21.15.42:8765 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.678838+0000 mgr.y (mgr.24491) 8 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Serving on https://172.21.15.42:7150 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.679031+0000 mgr.y (mgr.24491) 9 : cephadm [INF] [20/Sep/2024:19:18:58] ENGINE Bus STARTED 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: cephadm 2024-09-20T19:18:58.680946+0000 mgr.y (mgr.24491) 10 : cephadm [ERR] [20/Sep/2024:19:18:58] ENGINE Error in HTTPServer.serve 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: Traceback (most recent call last): 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._connections.run(self.expiration_interval) 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._run(expiration_interval) 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:19:00.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: s = self.context.wrap_socket( 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: return self.sslsocket_class._create( 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self.do_handshake() 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: self._sslobj.do_handshake() 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[32075]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:19:00.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: s = self.context.wrap_socket( 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: return self.sslsocket_class._create( 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self.do_handshake() 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: self._sslobj.do_handshake() 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:00.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:18:59 smithi042 bash[22706]: audit 2024-09-20T19:18:58.836369+0000 mgr.y (mgr.24491) 11 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:00 smithi162 bash[24557]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:00 smithi162 bash[24557]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:00 smithi162 bash[24557]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:00 smithi162 bash[24557]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[32075]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[32075]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[32075]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[32075]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[22706]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[22706]: cluster 2024-09-20T19:18:59.875281+0000 mgr.y (mgr.24491) 12 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[22706]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:00 smithi042 bash[22706]: cluster 2024-09-20T19:18:59.944055+0000 mon.a (mon.0) 829 : cluster [DBG] mgrmap e23: y(active, since 4s), standbys: x 2024-09-20T19:19:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[32075]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:03 smithi042 bash[22706]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: cluster 2024-09-20T19:19:01.875710+0000 mgr.y (mgr.24491) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.104354+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.108970+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.860918+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:03 smithi162 bash[24557]: audit 2024-09-20T19:19:02.869568+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:03.668 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-09-20T19:19:03.668 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQCnyu1my45iJxAALtXK2hKG3uB8ThTrIGUz7w== 2024-09-20T19:19:04.504 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-20T19:19:04.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T19:19:04.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.517 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:04.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[32075]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:04.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:04 smithi042 bash[22706]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:04.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.261755+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.268149+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.269489+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.270166+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.270748+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.271409+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.271865+0000 mgr.y (mgr.24491) 14 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3805M 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.276876+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.659649+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.42:0/459905461' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.660490+0000 mon.a (mon.0) 841 : audit [INF] from='client.24530 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.663355+0000 mon.a (mon.0) 842 : audit [INF] from='client.24530 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.914742+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.926310+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.933191+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.935399+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.937060+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.938667+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.948478+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.954094+0000 mon.a (mon.0) 850 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:04.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:04 smithi162 bash[24557]: audit 2024-09-20T19:19:03.955539+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:05.180 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:04 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[32075]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:05.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:05 smithi042 bash[22706]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cluster 2024-09-20T19:19:03.877054+0000 mgr.y (mgr.24491) 15 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.939857+0000 mgr.y (mgr.24491) 16 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3804M 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.957869+0000 mgr.y (mgr.24491) 17 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:03.958332+0000 mgr.y (mgr.24491) 18 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.092812+0000 mgr.y (mgr.24491) 19 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.093177+0000 mgr.y (mgr.24491) 20 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.191168+0000 mgr.y (mgr.24491) 21 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.191497+0000 mgr.y (mgr.24491) 22 : cephadm [INF] Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.286837+0000 mgr.y (mgr.24491) 23 : cephadm [INF] Updating smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.287002+0000 mgr.y (mgr.24491) 24 : cephadm [INF] Updating smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.386753+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.391172+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.396528+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.402179+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: audit 2024-09-20T19:19:04.408979+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:05 smithi162 bash[24557]: cephadm 2024-09-20T19:19:04.423352+0000 mgr.y (mgr.24491) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi042 2024-09-20T19:19:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:07 smithi042 bash[22706]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:07 smithi042 bash[22706]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:07 smithi042 bash[32075]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:07 smithi042 bash[32075]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:07.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:07 smithi162 bash[24557]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:07.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:07 smithi162 bash[24557]: cluster 2024-09-20T19:19:05.877973+0000 mgr.y (mgr.24491) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:08 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:08 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:08 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.168407+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:19:08.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:08 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:08.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:08 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.170750+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:19:09.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:08 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:09.314 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:09.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:09.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:09 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:09.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:07.879516+0000 mgr.y (mgr.24491) 27 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:19:09.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.061830+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:19:09.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:09.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:09 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.064204+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:10 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: audit 2024-09-20T19:19:08.847451+0000 mgr.y (mgr.24491) 28 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.450750+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:19:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:10 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.454422+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:19:10.816 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-09-20T19:19:10.816 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCuyu1mbao6MBAAhh1QOpm8JoWGM91BFB8TNw== 2024-09-20T19:19:11.470 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-20T19:19:11.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T19:19:11.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T19:19:11.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[22706]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:11 smithi042 bash[32075]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:09.880277+0000 mgr.y (mgr.24491) 29 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.240126+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.244050+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.470841+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: cluster 2024-09-20T19:19:10.473494+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.808943+0000 mon.a (mon.0) 857 : audit [INF] from='client.24562 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.811572+0000 mon.a (mon.0) 858 : audit [INF] from='client.24562 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:11 smithi162 bash[24557]: audit 2024-09-20T19:19:10.813247+0000 mon.b (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.162:0/4236446857' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:12 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.239813+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.241534+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: audit 2024-09-20T19:19:11.422491+0000 mon.a (mon.0) 859 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.979557+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:19:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:12 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.981730+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:13.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:13 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:11.881113+0000 mgr.y (mgr.24491) 30 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:12.202744+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:12.206172+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.017292+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.019577+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:19:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.241107+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-20T19:19:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:13 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.243624+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-20T19:19:14.447 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.447 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.447 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.447 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.448 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.448 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.448 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.448 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.809 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.810 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.810 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.810 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: Started Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:14.810 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.811 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:14.811 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:19:14 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[32075]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:14 smithi042 bash[22706]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.210526+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:19:15.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.215045+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.215351+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.217706+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.335267+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.335353+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.882596+0000 mgr.y (mgr.24491) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: cluster 2024-09-20T19:19:13.939882+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:19:15.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.733313+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.738849+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.742894+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:14 smithi162 bash[24557]: audit 2024-09-20T19:19:14.747639+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:15.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.835 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.835 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.835 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.836 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.836 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.836 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.837 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.837 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:15.837 INFO:journalctl@ceph.rgw.foo.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.154 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.155 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.155 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.155 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.156 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:15 smithi042 bash[22706]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.156 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:15 smithi042 bash[22706]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.156 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.156 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:15 smithi042 bash[32075]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.156 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:15 smithi042 bash[32075]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.156 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: Started Ceph node-exporter.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:16.156 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:15 smithi042 bash[60549]: Unable to find image 'quay.io/prometheus/node-exporter:v1.7.0' locally 2024-09-20T19:19:16.157 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.157 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:19:15 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:16.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:15 smithi162 bash[24557]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:15 smithi162 bash[24557]: cephadm 2024-09-20T19:19:14.750366+0000 mgr.y (mgr.24491) 32 : cephadm [INF] Deploying daemon node-exporter.a on smithi042 2024-09-20T19:19:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.219Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.220Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.221Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.42 port=9094 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.222Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.250Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.250Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.252Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-20T19:19:16.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60443]: ts=2024-09-20T19:19:16.252Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T19:19:17.193 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.193 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.194 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:16 smithi162 bash[24557]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:17.430 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[60549]: v1.7.0: Pulling from prometheus/node-exporter 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[22706]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.264418+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.268059+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.484321+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.487535+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cluster 2024-09-20T19:19:15.883024+0000 mgr.y (mgr.24491) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.988615+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.993261+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:15.997903+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: cephadm 2024-09-20T19:19:15.998694+0000 mgr.y (mgr.24491) 34 : cephadm [INF] Deploying daemon node-exporter.b on smithi162 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:16 smithi042 bash[32075]: audit 2024-09-20T19:19:16.383200+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:17.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:17 smithi162 systemd[1]: Started Ceph node-exporter.b for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:17.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[50824]: Unable to find image 'quay.io/prometheus/node-exporter:v1.7.0' locally 2024-09-20T19:19:17.929 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 2abcce694348: Pulling fs layer 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 455fd88e5221: Pulling fs layer 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 324153f2810a: Pulling fs layer 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 455fd88e5221: Download complete 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 2abcce694348: Download complete 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 324153f2810a: Verifying Checksum 2024-09-20T19:19:17.930 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[60549]: 324153f2810a: Download complete 2024-09-20T19:19:18.052 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[50824]: v1.7.0: Pulling from prometheus/node-exporter 2024-09-20T19:19:18.053 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 2abcce694348: Pulling fs layer 2024-09-20T19:19:18.053 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 455fd88e5221: Pulling fs layer 2024-09-20T19:19:18.053 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 324153f2810a: Pulling fs layer 2024-09-20T19:19:18.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:17 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.355 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:19:18.369 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:18 smithi042 bash[60549]: 2abcce694348: Pull complete 2024-09-20T19:19:18.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.213457+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.214937+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.238856+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:16.240963+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.111429+0000 osd.7 (osd.7) 7 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.113279+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.119240+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.120467+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.244837+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.254654+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.263145+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.270818+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.375 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: audit 2024-09-20T19:19:17.287022+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:18.375 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.375 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cephadm 2024-09-20T19:19:17.290607+0000 mgr.y (mgr.24491) 35 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:19:18.375 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.375 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:17 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.936893+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:19:18.375 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:18 smithi042 bash[60443]: ts=2024-09-20T19:19:18.222Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00020695s 2024-09-20T19:19:18.552 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 2abcce694348: Download complete 2024-09-20T19:19:18.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 455fd88e5221: Verifying Checksum 2024-09-20T19:19:18.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 455fd88e5221: Download complete 2024-09-20T19:19:18.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 324153f2810a: Verifying Checksum 2024-09-20T19:19:18.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 324153f2810a: Download complete 2024-09-20T19:19:18.553 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 2abcce694348: Pull complete 2024-09-20T19:19:19.179 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:18 smithi042 bash[60549]: 455fd88e5221: Pull complete 2024-09-20T19:19:19.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE Bus STOPPING 2024-09-20T19:19:19.286 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[50824]: 455fd88e5221: Pull complete 2024-09-20T19:19:19.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:18 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:19.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.542 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.542 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.552 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[50824]: 324153f2810a: Pull complete 2024-09-20T19:19:19.552 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[50824]: Digest: sha256:4cb2b9019f1757be8482419002cb7afe028fdba35d47958829e4cfeaf6246d80 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.696571+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.698423+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:17.884075+0000 mgr.y (mgr.24491) 36 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.138385+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.140767+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:18.355220+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.42:0/2921668522' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.024452+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.031800+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.035220+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: audit 2024-09-20T19:19:19.042273+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.059650+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:19:19.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.061317+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:19:19.848 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:19 smithi162 bash[50824]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.7.0 2024-09-20T19:19:19.929 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:19:19.930 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE Bus STOPPED 2024-09-20T19:19:19.930 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE Bus STARTING 2024-09-20T19:19:19.930 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE Serving on http://:::9283 2024-09-20T19:19:19.930 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:19 smithi042 bash[22958]: [20/Sep/2024:19:19:19] ENGINE Bus STARTED 2024-09-20T19:19:19.970 INFO:teuthology.orchestra.run.smithi042.stdout:[{"version":25,"timestamp":"2024-09-20T19:19:03.940418+0000","name":"","changes":[{"name":"osd/host:smithi162/osd_memory_target","previous_value":"4526056448","new_value":"3989185536"}]}] 2024-09-20T19:19:19.970 INFO:tasks.ceph_manager:config epoch is 25 2024-09-20T19:19:19.970 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T19:19:19.970 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T19:19:19.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T19:19:20.295 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[60549]: 324153f2810a: Pull complete 2024-09-20T19:19:20.296 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:19 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:20 smithi162 bash[24557]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:20.555 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[60549]: Digest: sha256:4cb2b9019f1757be8482419002cb7afe028fdba35d47958829e4cfeaf6246d80 2024-09-20T19:19:20.555 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[60549]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.7.0 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[32075]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.739359+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:18.741485+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:18.856399+0000 mgr.y (mgr.24491) 37 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.036238+0000 mgr.y (mgr.24491) 38 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cephadm 2024-09-20T19:19:19.076459+0000 mgr.y (mgr.24491) 39 : cephadm [INF] Deploying daemon grafana.a on smithi162 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.438661+0000 osd.1 (osd.1) 23 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.441331+0000 osd.1 (osd.1) 24 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.537786+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:19:20.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:20.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:20 smithi042 bash[22706]: audit 2024-09-20T19:19:19.539681+0000 mgr.y (mgr.24491) 40 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:19:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.382Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.382Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.383Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.383Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.383Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.383Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T19:19:21.554 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T19:19:21.555 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T19:19:21.556 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.384Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.385Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-20T19:19:21.557 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:21 smithi162 bash[50824]: ts=2024-09-20T19:19:21.385Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T19:19:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.749468+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.752202+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:19.884734+0000 mgr.y (mgr.24491) 41 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.177587+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.179885+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.205143+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.e scrub starts 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:21 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.207422+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.e scrub ok 2024-09-20T19:19:22.400 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.182Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-20T19:19:22.400 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.182Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-20T19:19:22.400 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.182Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T19:19:22.400 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.182Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.183Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.183Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T19:19:22.401 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T19:19:22.402 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T19:19:22.403 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-20T19:19:22.404 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[60549]: ts=2024-09-20T19:19:22.184Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T19:19:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[32075]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[32075]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[22706]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:22 smithi042 bash[22706]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:22.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:22 smithi162 bash[24557]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:22.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:22 smithi162 bash[24557]: audit 2024-09-20T19:19:21.392468+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[32075]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:23 smithi042 bash[22706]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:23.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:21.885608+0000 mgr.y (mgr.24491) 42 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:22.196434+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:19:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:23 smithi162 bash[24557]: cluster 2024-09-20T19:19:22.198442+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:24 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:24.768 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:24.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:24 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:24 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.107009+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:19:24.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:24 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:24.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:24 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.109058+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:25 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:25.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.734513+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:19:25.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.739617+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:23.886913+0000 mgr.y (mgr.24491) 43 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.378512+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.380890+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.166727+0000 osd.2 (osd.2) 15 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:25.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:25 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.169297+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:19:26.423 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[60443]: ts=2024-09-20T19:19:26.224Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001702053s 2024-09-20T19:19:26.560 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:26.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:26 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.691024+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:24.694632+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.407623+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.409791+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:19:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.161671+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:19:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:26 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.163166+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:19:27.300 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"flags":0,"active_gid":24491,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6800","nonce":4089929432}]},"active_addr":"172.21.15.42:6800/4089929432","active_change":"2024-09-20T19:18:55.853830+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24497,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.42:8443/","prometheus":"http://172.21.15.42:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":91,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":949767146}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":1781317076}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":577469704}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":1294768516}]}]} 2024-09-20T19:19:27.304 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T19:19:27.304 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T19:19:27.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:19:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:27.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:27 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:27.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:25.887623+0000 mgr.y (mgr.24491) 44 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.191632+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.192954+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.198400+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.e scrub starts 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:26.200987+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.e scrub ok 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: audit 2024-09-20T19:19:26.462726+0000 mon.a (mon.0) 881 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: audit 2024-09-20T19:19:26.550399+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.42:0/540285114' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-20T19:19:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.032527+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-09-20T19:19:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:27.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:27 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.036015+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-09-20T19:19:28.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.110967+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.113201+0000 osd.5 (osd.5) 18 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.176134+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:19:28.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.179387+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.377954+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.378062+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.430954+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.486373+0000 osd.1 (osd.1) 29 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.488635+0000 osd.1 (osd.1) 30 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.721828+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.727284+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.943897+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.166796+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.170297+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.172927+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:28.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:28 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.174146+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:19:29.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:28 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:29 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:27.888336+0000 mgr.y (mgr.24491) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.083794+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.085996+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.184498+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:19:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:28.186501+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:19:29.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.012620+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:19:29.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:29.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:29 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.014359+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:19:30.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:30.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:30 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:30.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: audit 2024-09-20T19:19:28.864342+0000 mgr.y (mgr.24491) 46 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.199780+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:19:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:30 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.203397+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:19:31.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:31 smithi162 bash[24557]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[22706]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:31.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:29.889052+0000 mgr.y (mgr.24491) 47 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:30.512179+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:19:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:31.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:31 smithi042 bash[32075]: cluster 2024-09-20T19:19:30.514664+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:19:32.120 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:32 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:32.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.463866+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.466238+0000 osd.1 (osd.1) 34 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.791391+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:32.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:32 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.793601+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:33.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:33 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:33.846 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:19:33.846 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":96,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:19:27.934466+0000","last_up_change":"2024-09-20T19:17:34.145990+0000","last_in_change":"2024-09-20T19:17:00.522021+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-20T19:18:05.791948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-20T19:18:06.729842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T19:18:07.353972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T19:18:09.310321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T19:18:11.438339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":2265043130}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":2265043130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":2265043130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":2265043130}]},"public_addr":"172.21.15.42:6801/2265043130","cluster_addr":"172.21.15.42:6802/2265043130","heartbeat_back_addr":"172.21.15.42:6804/2265043130","heartbeat_front_addr":"172.21.15.42:6803/2265043130","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1507333597}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1507333597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1507333597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1507333597}]},"public_addr":"172.21.15.42:6813/1507333597","cluster_addr":"172.21.15.42:6814/1507333597","heartbeat_back_addr":"172.21.15.42:6816/1507333597","heartbeat_front_addr":"172.21.15.42:6815/1507333597","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":2898101120}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2898101120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2898101120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2898101120}]},"public_addr":"172.21.15.162:6800/2898101120","cluster_addr":"172.21.15.162:6801/2898101120","heartbeat_back_addr":"172.21.15.162:6803/2898101120","heartbeat_front_addr":"172.21.15.162:6802/2898101120","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:23.562832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:16.402958+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:19:27.378089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:52.220308+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:32.029420+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:18.878761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:19:13.335358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:39.291027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/1743761075":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/3764180564":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3982292550":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/2784207196":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3505349280":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/368716713":"2024-09-21T19:18:55.853462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:31.890044+0000 mgr.y (mgr.24491) 48 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.081967+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:19:33.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.084310+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.221308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:32.223522+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.033664+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:33.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:33 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.036047+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:19:34.598 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T19:19:34.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:34.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:34 smithi162 bash[24557]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[22706]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.054856+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.057559+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.239532+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.242638+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.258865+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.262893+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-09-20T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:34.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:34 smithi042 bash[32075]: audit 2024-09-20T19:19:33.844046+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.42:0/4060406607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:35 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:33.890647+0000 mgr.y (mgr.24491) 49 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.011287+0000 osd.5 (osd.5) 25 : cluster [DBG] 6.d scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.013779+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.426891+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.429117+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.998677+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:35.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:35 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.001654+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:19:36.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:36.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:36 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.963837+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:34.966129+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.387698+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.d scrub starts 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.390051+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.d scrub ok 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.167213+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:36 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.169193+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:19:37.232 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.233 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.233 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.233 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.234 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.235 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.235 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.235 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.494 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.494 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.494 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.495 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.495 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.495 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.495 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:19:37.496 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:37 smithi162 systemd[1]: Started Ceph grafana.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:37 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.891287+0000 mgr.y (mgr.24491) 50 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.979036+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:35.981160+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.376248+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.378273+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.176039+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:37 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.178714+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:19:38.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.777866307Z level=info msg="Starting Grafana" version=10.4.8 commit=8c384b71bde142a25ddc01ed5a183ca572b2b7f7 branch=v10.4.x compiled=2024-09-20T19:19:38Z 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778301293Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778318711Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778325912Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778332326Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778338164Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778344294Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778354931Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778361607Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T19:19:38.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778368104Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778373961Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778379708Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778389947Z level=info msg=Target target=[all] 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778405858Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778412399Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778427622Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778433645Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.77843958Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=settings t=2024-09-20T19:19:38.778445225Z level=info msg="App mode production" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=sqlstore t=2024-09-20T19:19:38.779246879Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=sqlstore t=2024-09-20T19:19:38.779321281Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.780712985Z level=info msg="Starting DB migrations" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.781747671Z level=info msg="Executing migration" id="create migration_log table" 2024-09-20T19:19:38.805 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.782861325Z level=info msg="Migration successfully executed" id="create migration_log table" duration=1.112332ms 2024-09-20T19:19:38.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.941436+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.a scrub starts 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:36.943521+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.a scrub ok 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.546803+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.557141+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.565782+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.573285+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: audit 2024-09-20T19:19:37.605484+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.804709+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:38 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.807538+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:19:39.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:39.086 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.838233717Z level=info msg="Executing migration" id="create user table" 2024-09-20T19:19:39.086 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.839393925Z level=info msg="Migration successfully executed" id="create user table" duration=1.160391ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.863570884Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.86502663Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=1.454993ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.905614349Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.90696066Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=1.345518ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.947671798Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.949107851Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=1.43539ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.972969712Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:38 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:38.974359536Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=1.389234ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.01491166Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.019635412Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=4.721858ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.057085772Z level=info msg="Executing migration" id="create user table v2" 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.058736883Z level=info msg="Migration successfully executed" id="create user table v2" duration=1.650244ms 2024-09-20T19:19:39.087 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.073959079Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-20T19:19:39.088 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.075371239Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=1.411283ms 2024-09-20T19:19:39.088 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.090891966Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.092419761Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=1.527013ms 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.107917231Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.108791647Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=877.755µs 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.124693383Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.125934125Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=1.240085ms 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.141699476Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.143907698Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=2.207109ms 2024-09-20T19:19:39.349 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.166954114Z level=info msg="Executing migration" id="Update user table charset" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.167047427Z level=info msg="Migration successfully executed" id="Update user table charset" duration=95.433µs 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.19216534Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.194531321Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=2.360824ms 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.234263282Z level=info msg="Executing migration" id="Add missing user data" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.234851693Z level=info msg="Migration successfully executed" id="Add missing user data" duration=589.954µs 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.251286475Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.253844257Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=2.554332ms 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.276565953Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.278053735Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=1.483329ms 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.293466466Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.295719688Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=2.254022ms 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.336428873Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-20T19:19:39.350 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.353168632Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=16.741722ms 2024-09-20T19:19:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:39.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:39.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[24557]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.377541855Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.380516691Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=2.967775ms 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.402846303Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.403308531Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=462.549µs 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.444872056Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.446298772Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=1.428116ms 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.470209019Z level=info msg="Executing migration" id="update login field with orgid to allow for multiple service accounts with same name across orgs" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.470957549Z level=info msg="Migration successfully executed" id="update login field with orgid to allow for multiple service accounts with same name across orgs" duration=747.156µs 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.495400435Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.497080876Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=1.679351ms 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.529470433Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.530913616Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=1.442137ms 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.571553199Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T19:19:39.611 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.573155077Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=1.598905ms 2024-09-20T19:19:39.612 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.613720307Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-20T19:19:39.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.615528017Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=1.806553ms 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.638607425Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.63928202Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=674.358µs 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.680752057Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.680862653Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=76.86µs 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.72277263Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.724323042Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=1.550616ms 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.748081222Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.749605778Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=1.524249ms 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.77332354Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.774672278Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=1.348404ms 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.798594288Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.800011678Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=1.417239ms 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.823837646Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T19:19:39.879 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.829880529Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=6.041763ms 2024-09-20T19:19:39.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.857470079Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-20T19:19:39.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.859199385Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=1.725597ms 2024-09-20T19:19:39.880 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.8826817Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[22706]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:37.892715+0000 mgr.y (mgr.24491) 51 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.208638+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.e scrub starts 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.210702+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.355809+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.355903+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:39 smithi042 bash[32075]: cluster 2024-09-20T19:19:38.584246+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:19:40.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.884218236Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=1.536265ms 2024-09-20T19:19:40.147 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.92475537Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.926315858Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=1.557439ms 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.96664642Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:39 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:39.968177053Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=1.529973ms 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.00867204Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.010245965Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=1.573692ms 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.050908354Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.051820442Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=913.568µs 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.075961677Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.077228546Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=1.265545ms 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.101336095Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.102193717Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=856.982µs 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.12656472Z level=info msg="Executing migration" id="create star table" 2024-09-20T19:19:40.148 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.127922251Z level=info msg="Migration successfully executed" id="create star table" duration=1.356867ms 2024-09-20T19:19:40.149 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.151870748Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.153468273Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=1.596318ms 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.185633712Z level=info msg="Executing migration" id="create org table v1" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.187319993Z level=info msg="Migration successfully executed" id="create org table v1" duration=1.68693ms 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.210905447Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.212374993Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=1.46903ms 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.252938994Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.25441827Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=1.475736ms 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.278128509Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.279619865Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=1.494039ms 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.303385327Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T19:19:40.426 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.304958809Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=1.572722ms 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.328700308Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.330221171Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=1.520833ms 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.353961763Z level=info msg="Executing migration" id="Update org table charset" 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.354038796Z level=info msg="Migration successfully executed" id="Update org table charset" duration=78.472µs 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.387803364Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.387852093Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=50.729µs 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.429762071Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-20T19:19:40.427 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.430134394Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=370.707µs 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.455079195Z level=info msg="Executing migration" id="create dashboard table" 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.456622764Z level=info msg="Migration successfully executed" id="create dashboard table" duration=1.543059ms 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.497180767Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.498785499Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=1.617068ms 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.522415547Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.524088701Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=1.674594ms 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.547681371Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.549097825Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=1.415631ms 2024-09-20T19:19:40.687 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.58985058Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.591766388Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=1.907006ms 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.615088604Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.616764828Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=1.672985ms 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.640349816Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.650084401Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=9.735509ms 2024-09-20T19:19:40.688 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.690575912Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-20T19:19:40.835 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.835 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.835 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.835 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: audit 2024-09-20T19:19:38.872287+0000 mgr.y (mgr.24491) 52 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.224067+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.226151+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:19:40.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.896288+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:19:40.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:40 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.901158+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:19:40.939 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.691577094Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=1.002636ms 2024-09-20T19:19:40.939 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.72439738Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.725394243Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=999.281µs 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.741229514Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.741966348Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=736.563µs 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.758206664Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.758623792Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=417.109µs 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.775013162Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.775881216Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=867.866µs 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.792077951Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.79219171Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=112.141µs 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.809000435Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.811418248Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=2.417091ms 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.834271754Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-20T19:19:40.940 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.83667782Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=2.404664ms 2024-09-20T19:19:40.941 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.859705445Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-20T19:19:40.941 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.861957777Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=2.255132ms 2024-09-20T19:19:40.941 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.901681952Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-20T19:19:40.941 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.903448742Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=1.772407ms 2024-09-20T19:19:40.941 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.943794795Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-20T19:19:41.175 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:19:41.175 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":97,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:19:38.569848+0000","last_up_change":"2024-09-20T19:17:34.145990+0000","last_in_change":"2024-09-20T19:17:00.522021+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-20T19:18:05.791948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-20T19:18:06.729842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T19:18:07.353972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T19:18:09.310321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T19:18:11.438339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":2265043130}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":2265043130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":2265043130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":2265043130}]},"public_addr":"172.21.15.42:6801/2265043130","cluster_addr":"172.21.15.42:6802/2265043130","heartbeat_back_addr":"172.21.15.42:6804/2265043130","heartbeat_front_addr":"172.21.15.42:6803/2265043130","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1507333597}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1507333597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1507333597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1507333597}]},"public_addr":"172.21.15.42:6813/1507333597","cluster_addr":"172.21.15.42:6814/1507333597","heartbeat_back_addr":"172.21.15.42:6816/1507333597","heartbeat_front_addr":"172.21.15.42:6815/1507333597","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":2898101120}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2898101120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2898101120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2898101120}]},"public_addr":"172.21.15.162:6800/2898101120","cluster_addr":"172.21.15.162:6801/2898101120","heartbeat_back_addr":"172.21.15.162:6803/2898101120","heartbeat_front_addr":"172.21.15.162:6802/2898101120","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:23.562832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:19:38.355910+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:19:27.378089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:52.220308+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:32.029420+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:18.878761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:19:13.335358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:18:39.291027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/1743761075":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/3764180564":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3982292550":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/2784207196":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3505349280":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/368716713":"2024-09-21T19:18:55.853462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.94746651Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=3.668942ms 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.969143572Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:40 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:40.97070051Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=1.556172ms 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.019504349Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.021115854Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=1.611002ms 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.044858287Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.044940019Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=80.673µs 2024-09-20T19:19:41.209 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.086834971Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.08691713Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=80.399µs 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.112191961Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.115867187Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=3.674916ms 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.162759968Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.166437667Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=3.677236ms 2024-09-20T19:19:41.210 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.213187229Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-20T19:19:41.478 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.217079592Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=3.89268ms 2024-09-20T19:19:41.478 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.238513403Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-20T19:19:41.478 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.242194888Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=3.682519ms 2024-09-20T19:19:41.478 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.263761154Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.264241703Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=482.395µs 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.289064822Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.290674537Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=1.608908ms 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.314339167Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.315823947Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=1.480586ms 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.339545056Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.339593252Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=50.176µs 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.381569223Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.38328543Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=1.715307ms 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.423743164Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.425355126Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=1.613435ms 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.449106859Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T19:19:41.479 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.458607083Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=9.495148ms 2024-09-20T19:19:41.748 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.482648072Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-20T19:19:41.748 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.484188678Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=1.539932ms 2024-09-20T19:19:41.748 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.524705581Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T19:19:41.748 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.526343676Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=1.636572ms 2024-09-20T19:19:41.748 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.549996737Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.551595035Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=1.597435ms 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.592228664Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.59312712Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=901.795µs 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.61742017Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.618711761Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=1.294418ms 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.659560816Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.663614034Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=4.056665ms 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.701609633Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.703346006Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=1.736657ms 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.726928263Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.72732931Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=403.016µs 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.752170585Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-20T19:19:41.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.752589091Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=419.799µs 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[24557]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[22706]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:39.893451+0000 mgr.y (mgr.24491) 53 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:41.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.027490+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.029659+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.458700+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.461527+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.648994+0000 osd.5 (osd.5) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.649146+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.869991+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:19:41.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.871442+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.900725+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:40.905365+0000 osd.6 (osd.6) 28 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.042289+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.044418+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.139828+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.141963+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.172041+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.42:0/984786230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:19:41.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.405264+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:41.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:41 smithi042 bash[32075]: audit 2024-09-20T19:19:41.488065+0000 mon.a (mon.0) 892 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:41.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T19:19:41.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T19:19:41.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T19:19:41.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T19:19:41.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T19:19:41.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T19:19:41.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T19:19:41.940 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T19:19:42.008 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.77745752Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-20T19:19:42.008 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.779032139Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=1.576479ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.802675242Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.806702064Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=4.023169ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.827987123Z level=info msg="Executing migration" id="create data_source table" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.830126456Z level=info msg="Migration successfully executed" id="create data_source table" duration=2.13739ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.853267044Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.855012194Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=1.746613ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.895139313Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.896956526Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=1.820906ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.937402457Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.939055678Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=1.653214ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.971242015Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:41 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:41.97277792Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=1.535935ms 2024-09-20T19:19:42.009 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.013173809Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-20T19:19:42.261 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.024299584Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=11.124872ms 2024-09-20T19:19:42.261 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.063674689Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-20T19:19:42.261 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.065576161Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=1.900975ms 2024-09-20T19:19:42.261 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.088979771Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-20T19:19:42.261 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.090674958Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=1.698671ms 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.114212959Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.115905497Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=1.688904ms 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.139504108Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.140724686Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=1.220778ms 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.181375705Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.187830478Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=6.451569ms 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.214957853Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.217752809Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=2.795859ms 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.240191036Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.240245307Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=55.506µs 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.265349786Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-20T19:19:42.262 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.26568986Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=338.902µs 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.29075379Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.293510908Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=2.755874ms 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.316032585Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.316412571Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=380.04µs 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.341225112Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.34145961Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=234.274µs 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.366479676Z level=info msg="Executing migration" id="Add uid column" 2024-09-20T19:19:42.531 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.369145832Z level=info msg="Migration successfully executed" id="Add uid column" duration=2.665552ms 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.408703133Z level=info msg="Executing migration" id="Update uid value" 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.409148776Z level=info msg="Migration successfully executed" id="Update uid value" duration=444.462µs 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.433834623Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.434875038Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=1.039908ms 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.459166913Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.460175306Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=1.007819ms 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.48455477Z level=info msg="Executing migration" id="create api_key table" 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.486475495Z level=info msg="Migration successfully executed" id="create api_key table" duration=1.920218ms 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.509858207Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.511443166Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=1.586249ms 2024-09-20T19:19:42.532 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.535108102Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:42.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[24557]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.536909366Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=1.800547ms 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.56042845Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.562199733Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=1.773123ms 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.602488773Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.604127745Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=1.640391ms 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.619407377Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.620958499Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=1.549893ms 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.66132516Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.662860334Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=1.53459ms 2024-09-20T19:19:42.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.686649353Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-20T19:19:42.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.699700846Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=13.050304ms 2024-09-20T19:19:42.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.745198732Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-20T19:19:42.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.746306461Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=1.108881ms 2024-09-20T19:19:42.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.770394382Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-20T19:19:42.804 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.771473796Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=1.08346ms 2024-09-20T19:19:42.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[22706]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.575958+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.869701+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:19:42.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.871842+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:42 smithi042 bash[32075]: cluster 2024-09-20T19:19:41.894897+0000 mgr.y (mgr.24491) 54 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:43.068 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.820995803Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-20T19:19:43.068 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.82178774Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=806.95µs 2024-09-20T19:19:43.068 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.863091178Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.863884726Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=803.07µs 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.905157122Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.905587866Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=434.173µs 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.947315152Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.94796846Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=654.786µs 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.989051372Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:42 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:42.989085412Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=37.044µs 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.031191874Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.033134882Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.944145ms 2024-09-20T19:19:43.069 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.073271673Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-20T19:19:43.354 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.076308319Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=3.04033ms 2024-09-20T19:19:43.354 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.11553028Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-20T19:19:43.354 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.115833201Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=305.513µs 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.140656309Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.142505365Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.850551ms 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.182679347Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.18457195Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.894015ms 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.224699767Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.225502948Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=803.979µs 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.250272352Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.250995731Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=725.346µs 2024-09-20T19:19:43.355 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.359308746Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-20T19:19:43.624 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.360170837Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=865.703µs 2024-09-20T19:19:43.624 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.384640561Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T19:19:43.624 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.385459768Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=822.019µs 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.426719251Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.427557869Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=839.687µs 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.468817963Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.469631148Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=814.839µs 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.494162798Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.494274528Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=111.33µs 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.519442163Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.519483312Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=42.532µs 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.544673813Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.546605063Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.930981ms 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.570071369Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.573131998Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=3.061776ms 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.595285282Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T19:19:43.625 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.595385302Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=102.484µs 2024-09-20T19:19:43.626 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.629050855Z level=info msg="Executing migration" id="create quota table v1" 2024-09-20T19:19:43.877 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.629804969Z level=info msg="Migration successfully executed" id="create quota table v1" duration=754.843µs 2024-09-20T19:19:43.877 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.663010883Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.664154869Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=1.145643ms 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.69677399Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.696805188Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=34.308µs 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.738641677Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.739329109Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=686.452µs 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.763802992Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.764451537Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=648.023µs 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.8227018Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.828660627Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=5.958314ms 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.856349289Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-20T19:19:43.878 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.856430655Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=83.742µs 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.881574579Z level=info msg="Executing migration" id="create session table" 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.883545997Z level=info msg="Migration successfully executed" id="create session table" duration=1.967775ms 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.906965549Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.90717467Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=208.105µs 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.949069465Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-20T19:19:44.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.949297743Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=229.701µs 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.974369424Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:43.975861393Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=1.491519ms 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.016383008Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.01789363Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=1.509922ms 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.058404265Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.058510497Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=108.616µs 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.083749866Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.083803199Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=55.183µs 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.12569543Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.131327777Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=5.63184ms 2024-09-20T19:19:44.156 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.159363713Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:43 smithi162 bash[24557]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[22706]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:42.182145+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:42.187729+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:42.919868+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:42.923939+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.029735+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b scrub starts 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.031759+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b scrub ok 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.039167+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.040453+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.622374+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.632858+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.635036+0000 mon.a (mon.0) 898 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:19:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.636606+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:19:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:43 smithi042 bash[32075]: audit 2024-09-20T19:19:43.646745+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.165627928Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=6.259964ms 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.209729018Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.20994347Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=217.268µs 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.235103229Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.235316127Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=214.631µs 2024-09-20T19:19:44.441 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.277147237Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.278819264Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=1.671624ms 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.310894385Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.310945045Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=53.102µs 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.327894501Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.3337121Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=5.816882ms 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.35316517Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.353497147Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=332.173µs 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.370104197Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.375909496Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=5.799033ms 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.395351216Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.401126884Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=5.773649ms 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.445432614Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-20T19:19:44.442 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.445577536Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=148.885µs 2024-09-20T19:19:44.719 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.47095327Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-20T19:19:44.719 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.472798472Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=1.844303ms 2024-09-20T19:19:44.719 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.49646026Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-20T19:19:44.719 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.498244923Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=1.784183ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.521720885Z level=info msg="Executing migration" id="create alert table v1" 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.523847882Z level=info msg="Migration successfully executed" id="create alert table v1" duration=2.126686ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.546903344Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.548711047Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=1.806693ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.572356944Z level=info msg="Executing migration" id="add index alert state" 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.574075614Z level=info msg="Migration successfully executed" id="add index alert state" duration=1.717804ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.614255929Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.615919874Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=1.663505ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.639587324Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.641020017Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=1.431616ms 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.681565078Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T19:19:44.720 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.683329015Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=1.763224ms 2024-09-20T19:19:44.721 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.723531346Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T19:19:44.980 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.725452105Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=1.921938ms 2024-09-20T19:19:44.980 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.748804452Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T19:19:44.980 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.765707025Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=16.901813ms 2024-09-20T19:19:44.980 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.790749229Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-20T19:19:44.980 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.792226269Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=1.480043ms 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.83272696Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.83449094Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=1.76463ms 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.858119761Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.858766722Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=648.487µs 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.883290368Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.884464543Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=1.175375ms 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.908626085Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.91025169Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=1.626725ms 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.950811521Z level=info msg="Executing migration" id="Add column is_default" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.957358363Z level=info msg="Migration successfully executed" id="Add column is_default" duration=6.547635ms 2024-09-20T19:19:44.981 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.984499937Z level=info msg="Executing migration" id="Add column frequency" 2024-09-20T19:19:44.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:44.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:44.983 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:44.983 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:44.983 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:44.983 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:44.983 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:45.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:45.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.144388+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.c scrub starts 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.146454+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.203618+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.205137+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cephadm 2024-09-20T19:19:43.694727+0000 mgr.y (mgr.24491) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cephadm 2024-09-20T19:19:43.703236+0000 mgr.y (mgr.24491) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi042 2024-09-20T19:19:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:43.895991+0000 mgr.y (mgr.24491) 57 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.041372+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:19:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:45.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:44 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.045892+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:19:45.249 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:44 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:44.991748992Z level=info msg="Migration successfully executed" id="Add column frequency" duration=7.248371ms 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.034890542Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.043106724Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=8.218175ms 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.085320667Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.091807539Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=6.468182ms 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.119015046Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.120902882Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=1.887089ms 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.161001907Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.16105185Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=52.236µs 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.202971996Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.203021052Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=51.352µs 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.228345103Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-20T19:19:45.250 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.229890955Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=1.547896ms 2024-09-20T19:19:45.251 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.253574498Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.255773188Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=2.197686ms 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.295582276Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.297206198Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=1.617472ms 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.337647787Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.339237009Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=1.591005ms 2024-09-20T19:19:45.510 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.363289648Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.365031286Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=1.741124ms 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.388510455Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.395172495Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=6.661477ms 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.422112604Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.428697486Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=6.584078ms 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.47240529Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.472771783Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=366.93µs 2024-09-20T19:19:45.511 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.514362999Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-20T19:19:45.779 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.516080403Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=1.719854ms 2024-09-20T19:19:45.779 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.539655554Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-20T19:19:45.779 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.541307375Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=1.651275ms 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.564818594Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.5715828Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=6.763273ms 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.598481373Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.598635962Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=155.145µs 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.623731558Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.625396526Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=1.666815ms 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.648997437Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.650833546Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=1.835652ms 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.674265199Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.674474844Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=208.134µs 2024-09-20T19:19:45.780 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.716327753Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-20T19:19:45.781 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.718217309Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=1.884202ms 2024-09-20T19:19:45.781 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.741519779Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-20T19:19:45.781 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.743306378Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=1.787359ms 2024-09-20T19:19:45.781 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.783605329Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-20T19:19:46.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.785433032Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=1.826593ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.808933171Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.810645371Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=1.711297ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.83411926Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.836007852Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=1.887595ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.859463671Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.861384113Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=1.914965ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.88469282Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.884757416Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=63.32µs 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.909856734Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.917143021Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=7.282957ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.960477603Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.962301026Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=1.825063ms 2024-09-20T19:19:46.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.985512517Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-20T19:19:46.055 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:45 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:45.992688893Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=7.17584ms 2024-09-20T19:19:46.055 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.019297729Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-20T19:19:46.055 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.020753942Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=1.455539ms 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.110780+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.112976+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.238718+0000 osd.7 (osd.7) 17 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:19:46.056 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:44.240454+0000 osd.7 (osd.7) 18 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:19:46.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.035266+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.d scrub starts 2024-09-20T19:19:46.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:45 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.038368+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2024-09-20T19:19:46.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.061348046Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T19:19:46.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.063231752Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=1.882889ms 2024-09-20T19:19:46.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.103374271Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T19:19:46.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.105233667Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=1.85971ms 2024-09-20T19:19:46.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.145498484Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.16644784Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=20.948237ms 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.187452276Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.189079484Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=1.622278ms 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.212688028Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.214504338Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=1.83413ms 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.238223231Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.238949051Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=728.897µs 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.263451114Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.264690055Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=1.237425ms 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.288743276Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-20T19:19:46.311 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.289207968Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=461.892µs 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.314040664Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.323148785Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=9.106047ms 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.364388237Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.371663078Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=7.272991ms 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.398023573Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.399936462Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=1.912245ms 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.423331132Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.425080792Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=1.749083ms 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.44861859Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.449139179Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=519.942µs 2024-09-20T19:19:46.571 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.473855956Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.481479587Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=7.617508ms 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.524321771Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.526320365Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=1.997251ms 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.549639356Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.550050602Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=413.433µs 2024-09-20T19:19:46.572 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.574998327Z level=info msg="Executing migration" id="Move region to single row" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.575866085Z level=info msg="Migration successfully executed" id="Move region to single row" duration=867.585µs 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.600212359Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.602021516Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=1.808603ms 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.642184171Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.643976771Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=1.792273ms 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.667507172Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.669494793Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=1.988511ms 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.692721255Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.694540865Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=1.818446ms 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.734620851Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T19:19:46.840 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.736359307Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=1.738653ms 2024-09-20T19:19:46.841 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.776702757Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-20T19:19:46.841 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.778497307Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=1.789773ms 2024-09-20T19:19:46.841 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.818755045Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-20T19:19:46.841 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.81893423Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=181.489µs 2024-09-20T19:19:46.841 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.844124209Z level=info msg="Executing migration" id="create test_data table" 2024-09-20T19:19:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:19:47.092 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.845871546Z level=info msg="Migration successfully executed" id="create test_data table" duration=1.747607ms 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.869298386Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.870958384Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=1.660884ms 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.91133174Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.913045011Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=1.711814ms 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.953515346Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.955313742Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=1.797947ms 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.978861764Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:46.979253154Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=391.48µs 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.0207139Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.02175178Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=1.039966ms 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.071596845Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.07173414Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=138.699µs 2024-09-20T19:19:47.093 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.097000442Z level=info msg="Executing migration" id="create team table" 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:46 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.098542481Z level=info msg="Migration successfully executed" id="create team table" duration=1.541162ms 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.122261504Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.124228659Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=1.966195ms 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.139201232Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.141019525Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=1.816997ms 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.156110426Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.164095241Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=7.980134ms 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.181281819Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.181703625Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=422.042µs 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.198235689Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-20T19:19:47.371 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.200075022Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=1.839819ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.215232919Z level=info msg="Executing migration" id="create team member table" 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.216755215Z level=info msg="Migration successfully executed" id="create team member table" duration=1.517126ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.232153864Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.23393642Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=1.781623ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.257412112Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.259208562Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=1.795049ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.282669868Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.284433848Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=1.763363ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.324653893Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.33376597Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=9.11304ms 2024-09-20T19:19:47.372 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.375107708Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-20T19:19:47.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.239875+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.244476+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.827973+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.828093+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.896460+0000 mgr.y (mgr.24491) 58 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:19:47.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:45.935309+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.032517+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.036131+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.073202+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:46 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.074882+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-20T19:19:47.648 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.383460989Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=8.346721ms 2024-09-20T19:19:47.648 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.433766399Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-20T19:19:47.649 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.442023592Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=8.256739ms 2024-09-20T19:19:47.649 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.467475219Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-20T19:19:47.650 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.469375831Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=1.900349ms 2024-09-20T19:19:47.650 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.492722318Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-20T19:19:47.650 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.494500788Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=1.778094ms 2024-09-20T19:19:47.650 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.509594719Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T19:19:47.650 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.511506941Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=1.910096ms 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.526535943Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.528457402Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=1.919026ms 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.543587176Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.545362496Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=1.774777ms 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.56051468Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.562281153Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=1.765876ms 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.602526534Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-20T19:19:47.651 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.604345787Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=1.818336ms 2024-09-20T19:19:47.652 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.652967716Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.654890628Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=1.921839ms 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.678165282Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.679247192Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=1.081437ms 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.72029379Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.720823615Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=529.282µs 2024-09-20T19:19:47.901 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.762251025Z level=info msg="Executing migration" id="create tag table" 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.763761184Z level=info msg="Migration successfully executed" id="create tag table" duration=1.509513ms 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.804238996Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.806106069Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=1.86616ms 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.829510595Z level=info msg="Executing migration" id="create login attempt table" 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.831010131Z level=info msg="Migration successfully executed" id="create login attempt table" duration=1.498482ms 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.855092141Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.85691128Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=1.818209ms 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.880321874Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.882089557Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=1.767257ms 2024-09-20T19:19:47.902 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.905530976Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T19:19:48.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.172 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.932989921Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=27.457908ms 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.97274844Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:47 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:47.974522397Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=1.773896ms 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.014619223Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.016485119Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=1.864063ms 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.056681787Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-20T19:19:48.173 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.057522289Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=839.368µs 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.098718305Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.100080662Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=1.36204ms 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.124061516Z level=info msg="Executing migration" id="create user auth table" 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.125662775Z level=info msg="Migration successfully executed" id="create user auth table" duration=1.601048ms 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.149202017Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T19:19:48.174 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.151077772Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=1.874779ms 2024-09-20T19:19:48.175 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.174543265Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-20T19:19:48.175 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.17469157Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=146.846µs 2024-09-20T19:19:48.422 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.199876713Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-20T19:19:48.422 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.209719804Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=9.844914ms 2024-09-20T19:19:48.422 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.233469881Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-20T19:19:48.422 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.242532759Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=9.062635ms 2024-09-20T19:19:48.422 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.266998136Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.276055736Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=9.062177ms 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.317374198Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.326571685Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=9.195594ms 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.351047798Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.352953257Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=1.906479ms 2024-09-20T19:19:48.423 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.426418046Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.388677+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.395388+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.787880+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.b scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:46.791583+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.b scrub ok 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.043153+0000 osd.2 (osd.2) 29 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:19:48.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:47 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.047804+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:19:48.674 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.435661806Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=9.24497ms 2024-09-20T19:19:48.674 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.476790692Z level=info msg="Executing migration" id="create server_lock table" 2024-09-20T19:19:48.674 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.478847533Z level=info msg="Migration successfully executed" id="create server_lock table" duration=2.056338ms 2024-09-20T19:19:48.674 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.502217449Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-20T19:19:48.674 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.504224807Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=2.001571ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.527404486Z level=info msg="Executing migration" id="create user auth token table" 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.529386587Z level=info msg="Migration successfully executed" id="create user auth token table" duration=1.981578ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.552933699Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.554783705Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=1.850603ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.577918834Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.579903479Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=1.986722ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.603209523Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.605342919Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=2.13417ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.628371126Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.638133141Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=9.761659ms 2024-09-20T19:19:48.675 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.678681867Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-20T19:19:48.926 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.680872969Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=2.192746ms 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.704019661Z level=info msg="Executing migration" id="create cache_data table" 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.705969757Z level=info msg="Migration successfully executed" id="create cache_data table" duration=1.949648ms 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.745941491Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.747805939Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=1.866116ms 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.771293952Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.773283609Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=1.987941ms 2024-09-20T19:19:48.927 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.813406799Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.815547759Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=2.1402ms 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.838696748Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.838863352Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=160.788µs 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.863920547Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.864142452Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=220.032µs 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.905886619Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.907637946Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=1.751093ms 2024-09-20T19:19:48.928 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.931242608Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-20T19:19:49.204 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.204 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.204 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.204 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.205 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[24557]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.933571179Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=2.327325ms 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.973379587Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:48 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:48.975427722Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=2.047571ms 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.015271117Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.015408736Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=138.192µs 2024-09-20T19:19:49.205 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.040617712Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.042515767Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=1.897476ms 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.065917597Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.06777115Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=1.853096ms 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.107875293Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.109880284Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=2.003617ms 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.133188954Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.135177439Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=1.987841ms 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.158344472Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.168684097Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=10.338556ms 2024-09-20T19:19:49.206 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.208719761Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-20T19:19:49.290 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.290 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.290 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.290 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[22706]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.021023+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.025899+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.291 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.210206+0000 osd.3 (osd.3) 33 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:19:49.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.212529+0000 osd.3 (osd.3) 34 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:19:49.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:48 smithi042 bash[32075]: cluster 2024-09-20T19:19:47.897910+0000 mgr.y (mgr.24491) 59 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:49.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.210866141Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=2.14522ms 2024-09-20T19:19:49.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.234009287Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-20T19:19:49.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.234240328Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=231.318µs 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.259309046Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.261178305Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=1.868329ms 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.301223498Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.303216642Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=1.992502ms 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.32650281Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T19:19:49.466 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.328499408Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=1.995895ms 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.368405997Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.368557905Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=150.682µs 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.41056401Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.412565314Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=2.000851ms 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.427634253Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-20T19:19:49.467 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.429639937Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=2.003684ms 2024-09-20T19:19:49.468 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.469442873Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.47146395Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=2.020178ms 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.494844511Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.496986164Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=2.138323ms 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.537011014Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.547462086Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=10.450368ms 2024-09-20T19:19:49.767 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.570515269Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.572410702Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=1.895517ms 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.595824684Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.597772023Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=1.947279ms 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.621094993Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.671170043Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=50.074811ms 2024-09-20T19:19:49.768 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.771764727Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-20T19:19:50.019 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.821719295Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=49.954338ms 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.847066219Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.848928861Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=1.863371ms 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.872283885Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.87417583Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=1.893401ms 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.897512736Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-20T19:19:50.020 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.906998761Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=9.486288ms 2024-09-20T19:19:50.021 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.931145677Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-20T19:19:50.021 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.940405413Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=9.260884ms 2024-09-20T19:19:50.021 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.981592411Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-20T19:19:50.021 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:49.983872393Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=2.279099ms 2024-09-20T19:19:50.021 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.023693908Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-20T19:19:50.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:49 smithi162 bash[24557]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.02593362Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=2.241945ms 2024-09-20T19:19:50.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.04897773Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-20T19:19:50.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.051025931Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=2.044511ms 2024-09-20T19:19:50.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.09090669Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T19:19:50.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.093122333Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=2.216009ms 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.132940301Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.1330934Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=153.512µs 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.158199931Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.168758268Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=10.557711ms 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.191761306Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.202363782Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=10.60077ms 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.225357856Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.235858044Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=10.498268ms 2024-09-20T19:19:50.272 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.275784109Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T19:19:50.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.021621+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.d scrub starts 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.026785+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.d scrub ok 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.279520+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.281815+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.431597+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.433906+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: audit 2024-09-20T19:19:48.882380+0000 mgr.y (mgr.24491) 60 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.994793+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:48.996987+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:19:50.523 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.277892149Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=2.110214ms 2024-09-20T19:19:50.523 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.301127644Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T19:19:50.523 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.303171485Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=2.044507ms 2024-09-20T19:19:50.523 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.343077407Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.353562428Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=10.486168ms 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.376689817Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.387249978Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=10.559001ms 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.418584777Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.420517352Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=1.936679ms 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.460630258Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.471162716Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=10.532029ms 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.494225382Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.504862339Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=10.637903ms 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.527874123Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-20T19:19:50.524 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.528040965Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=167.398µs 2024-09-20T19:19:50.784 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.553168329Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-20T19:19:50.784 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.55553534Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=2.366171ms 2024-09-20T19:19:50.784 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.578486611Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.58067683Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=2.200702ms 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.603682457Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.60629046Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=2.611423ms 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.645678785Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.645833201Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=153.142µs 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.670961105Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.676950744Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=5.989311ms 2024-09-20T19:19:50.785 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.721290508Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-20T19:19:50.786 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.732575305Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=11.284411ms 2024-09-20T19:19:50.786 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.755128464Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-20T19:19:50.786 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.766563589Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=11.433498ms 2024-09-20T19:19:50.786 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.788801504Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-20T19:19:51.044 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.044 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.045 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.045 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.045 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.045 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[24557]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.045 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.800083898Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=11.281667ms 2024-09-20T19:19:51.045 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.822441654Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-20T19:19:51.045 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.833508694Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=11.085296ms 2024-09-20T19:19:51.045 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.855972786Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-20T19:19:51.045 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.856135464Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=158.378µs 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.897898239Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.899532277Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=1.633308ms 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.92324286Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.934325143Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=11.081287ms 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.956781371Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.956996239Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=215.421µs 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:50 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:50.99890527Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.01052368Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=11.620944ms 2024-09-20T19:19:51.046 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.049328669Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-20T19:19:51.302 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.05164226Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=2.312595ms 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.074610904Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.085890495Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=11.279067ms 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.124864023Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.126499141Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=1.634621ms 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.150162098Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.152153402Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=1.990721ms 2024-09-20T19:19:51.303 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.175467661Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.18683287Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=11.36413ms 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.22585536Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.227434889Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=1.578951ms 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.251152816Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.25320836Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=2.054344ms 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.293120391Z level=info msg="Executing migration" id="create alert_image table" 2024-09-20T19:19:51.304 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.294876872Z level=info msg="Migration successfully executed" id="create alert_image table" duration=1.756321ms 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[22706]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.898643+0000 mgr.y (mgr.24491) 61 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:19:51.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.986996+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:19:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:50 smithi042 bash[32075]: cluster 2024-09-20T19:19:49.990046+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.33523468Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.337239171Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=2.004051ms 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.377252385Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.377419927Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=161.388µs 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.419113548Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.421162985Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=2.04901ms 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.444580516Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.446651103Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=2.074826ms 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.511436488Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.5122236Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.536669218Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-20T19:19:51.583 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.537672831Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=1.00474ms 2024-09-20T19:19:51.584 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.561989036Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-20T19:19:51.584 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.563953084Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=1.963652ms 2024-09-20T19:19:51.584 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.587271548Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-20T19:19:51.843 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.599355469Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=12.082667ms 2024-09-20T19:19:51.843 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.620976428Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-20T19:19:51.843 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.623069112Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=2.09199ms 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.662845272Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.664982438Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=2.135959ms 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.704803438Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.706520685Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=1.716368ms 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.730340251Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.732383225Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=2.045191ms 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.755578957Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.757626495Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=2.041665ms 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.780820324Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.780898586Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=79.213µs 2024-09-20T19:19:51.844 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.8227582Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-20T19:19:51.845 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.822910808Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=152.746µs 2024-09-20T19:19:51.845 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.848018948Z level=info msg="Executing migration" id="add library_element folder uid" 2024-09-20T19:19:52.015 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:51 smithi042 systemd[1]: Stopping Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:19:52.015 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.015 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.015 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.015 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.016 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.017 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.018 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.019 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.020 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.020 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.020 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:51 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.251027+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.253711+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.319325+0000 osd.7 (osd.7) 21 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.321509+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.857926+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:19:52.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.860275+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.963430+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.e scrub starts 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:50.966648+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.e scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.010870+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.012899+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:19:52.103 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.861390823Z level=info msg="Migration successfully executed" id="add library_element folder uid" duration=13.370921ms 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.889984644Z level=info msg="Executing migration" id="populate library_element folder_uid" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.890811783Z level=info msg="Migration successfully executed" id="populate library_element folder_uid" duration=828.042µs 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.932092593Z level=info msg="Executing migration" id="add index library_element org_id-folder_uid-name-kind" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.934299962Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_uid-name-kind" duration=2.206722ms 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.957388941Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.958027235Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=637.328µs 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:51 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:51.999498682Z level=info msg="Executing migration" id="create data_keys table" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.001705332Z level=info msg="Migration successfully executed" id="create data_keys table" duration=2.207623ms 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.024783745Z level=info msg="Executing migration" id="create secrets table" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.026645501Z level=info msg="Migration successfully executed" id="create secrets table" duration=1.861037ms 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.050059821Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-20T19:19:52.104 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.106624092Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=56.564743ms 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.150356866Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.160707302Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=10.349265ms 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.183978828Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.18427185Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=292.366µs 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.234453196Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.295534899Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=61.081869ms 2024-09-20T19:19:52.393 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.343172621Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-20T19:19:52.394 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.397570069Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=54.397595ms 2024-09-20T19:19:52.646 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.443573077Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-20T19:19:52.646 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.445063578Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=1.490165ms 2024-09-20T19:19:52.646 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.46875638Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.470645277Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=1.88564ms 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.494104993Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.494499694Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=392.784µs 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.519254059Z level=info msg="Executing migration" id="create permission table" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.520693641Z level=info msg="Migration successfully executed" id="create permission table" duration=1.44049ms 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.544606846Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.54631964Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=1.711888ms 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.56983306Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.571529743Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=1.695782ms 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.611860465Z level=info msg="Executing migration" id="create role table" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.613692218Z level=info msg="Migration successfully executed" id="create role table" duration=1.833059ms 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.637154161Z level=info msg="Executing migration" id="add column display_name" 2024-09-20T19:19:52.647 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.650822443Z level=info msg="Migration successfully executed" id="add column display_name" duration=13.669417ms 2024-09-20T19:19:52.935 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.679041124Z level=info msg="Executing migration" id="add column group_name" 2024-09-20T19:19:52.935 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.692663353Z level=info msg="Migration successfully executed" id="add column group_name" duration=13.615696ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.737579384Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.739708044Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=2.128176ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.771603565Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.773795524Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=2.191156ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.788631625Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.790870521Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=2.231155ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.830600571Z level=info msg="Executing migration" id="create team role table" 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.832597612Z level=info msg="Migration successfully executed" id="create team role table" duration=1.997004ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.847606061Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.84983254Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=2.225816ms 2024-09-20T19:19:52.936 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.872782822Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T19:19:52.937 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.875005338Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=2.221289ms 2024-09-20T19:19:52.937 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.898009628Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-20T19:19:52.937 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.900111202Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=2.100973ms 2024-09-20T19:19:52.937 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.940012663Z level=info msg="Executing migration" id="create user role table" 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.196 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.197 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[24557]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.941954559Z level=info msg="Migration successfully executed" id="create user role table" duration=1.943375ms 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.982096765Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:52 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:52.984715403Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=2.619825ms 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.02412645Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.02657593Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=2.449006ms 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.04938501Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.051574816Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=2.186193ms 2024-09-20T19:19:53.197 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.074635047Z level=info msg="Executing migration" id="create builtin role table" 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.076533925Z level=info msg="Migration successfully executed" id="create builtin role table" duration=1.898786ms 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.099859283Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.102028959Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=2.169243ms 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.141871342Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.143947589Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=2.07181ms 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.183761369Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-20T19:19:53.198 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.200409569Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=16.646644ms 2024-09-20T19:19:53.464 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.225649399Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.227727233Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=2.077444ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.267698474Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.269878754Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=2.1795ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.29299629Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.295047811Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=2.051368ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.335013299Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.337167885Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=2.154323ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.360207419Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.361871047Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=1.662851ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.385508199Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.387724205Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=2.223046ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.410783104Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.425032138Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=14.248521ms 2024-09-20T19:19:53.465 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.469344891Z level=info msg="Executing migration" id="permission kind migration" 2024-09-20T19:19:53.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.479 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[22706]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.252221+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.255863+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.339306+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.341768+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.406401+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.410587+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:52 smithi042 bash[32075]: cluster 2024-09-20T19:19:51.900061+0000 mgr.y (mgr.24491) 62 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.484280888Z level=info msg="Migration successfully executed" id="permission kind migration" duration=14.93446ms 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.561589015Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.575635844Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=14.047702ms 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.620309585Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.634376829Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=14.066848ms 2024-09-20T19:19:53.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.662274484Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.664408651Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=2.133333ms 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.687518034Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.689690453Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=2.172832ms 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.712731887Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.714831568Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=2.099464ms 2024-09-20T19:19:53.751 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.754854745Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-20T19:19:54.010 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:53 smithi042 bash[60443]: ts=2024-09-20T19:19:53.757Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:19:54.011 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:53 smithi042 bash[22706]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[24557]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[24557]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[24557]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[24557]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.052 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.756927325Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=2.071544ms 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.780416309Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.782625361Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=2.208352ms 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.805681621Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.805862423Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=184.346µs 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.830757116Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.830829395Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=76.219µs 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.856076868Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.857008603Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=933.928µs 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.89805521Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.899167897Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=1.114103ms 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.923322107Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.924634091Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=1.312085ms 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.948589703Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.949092125Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=502.489µs 2024-09-20T19:19:54.053 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.990630559Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-20T19:19:54.054 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:53 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:53.991819134Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=1.193632ms 2024-09-20T19:19:54.228 INFO:teuthology.orchestra.run.smithi042.stdout:231928234025 2024-09-20T19:19:54.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T19:19:54.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:52.808273+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:19:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:52.810380+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.116374599Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.118276168Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=1.900719ms 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.225112115Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.227591959Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=2.48056ms 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.333616581Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.348087265Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=14.470067ms 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.467338933Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-20T19:19:54.463 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.467509698Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=165.405µs 2024-09-20T19:19:54.721 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.492570412Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-20T19:19:54.721 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.494657527Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=2.084558ms 2024-09-20T19:19:54.721 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.517840565Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-20T19:19:54.721 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.52015115Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=2.310136ms 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.543178119Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.545409724Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=2.231055ms 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.585101324Z level=info msg="Executing migration" id="add correlation config column" 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.599749041Z level=info msg="Migration successfully executed" id="add correlation config column" duration=14.64697ms 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.626995575Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.629212258Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=2.216196ms 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.652341854Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.654502644Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=2.160533ms 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.677527008Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-20T19:19:54.722 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.725835596Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=48.30715ms 2024-09-20T19:19:54.724 INFO:teuthology.orchestra.run.smithi042.stdout:274877906974 2024-09-20T19:19:54.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T19:19:54.976 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.769557834Z level=info msg="Executing migration" id="create correlation v2" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.771983758Z level=info msg="Migration successfully executed" id="create correlation v2" duration=2.424514ms 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.795006925Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.797169742Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=2.162386ms 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.836896612Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.839058352Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=2.161387ms 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.862320824Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.864726665Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=2.407221ms 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.879172083Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.879755827Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=589.427µs 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.904366696Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.906174666Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=1.809166ms 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.946397072Z level=info msg="Executing migration" id="add provisioning column" 2024-09-20T19:19:54.977 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:54.961959849Z level=info msg="Migration successfully executed" id="add provisioning column" duration=15.561998ms 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:54 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.728584+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.728692+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.900689+0000 mgr.y (mgr.24491) 63 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:53.981242+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.027761+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:54 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.029927+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.005547333Z level=info msg="Executing migration" id="create entity_events table" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.007609881Z level=info msg="Migration successfully executed" id="create entity_events table" duration=2.0627ms 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.022518941Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.024701464Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=2.181569ms 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.039516128Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.040482056Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.064660335Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.065596933Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.106655939Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.108426302Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=1.770389ms 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.148788576Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-20T19:19:55.230 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.150945816Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=2.158267ms 2024-09-20T19:19:55.231 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.19073981Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:19:55.231 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.193003762Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=2.264499ms 2024-09-20T19:19:55.231 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.232905478Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.235307605Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=2.411371ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.258174341Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.260337491Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=2.16334ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.283355372Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.285745836Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=2.389491ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.358812481Z level=info msg="Executing migration" id="Drop public config table" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.360711606Z level=info msg="Migration successfully executed" id="Drop public config table" duration=1.899482ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.384180089Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.38657469Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=2.395797ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.409395293Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.411698308Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=2.304285ms 2024-09-20T19:19:55.489 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.451418579Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T19:19:55.490 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.453919582Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=2.500233ms 2024-09-20T19:19:55.490 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.493412048Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.495668723Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=2.258458ms 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.518735707Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.563169478Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=44.432992ms 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.585604826Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.599887978Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=14.286365ms 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.62754819Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-20T19:19:55.749 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.642901Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=15.352123ms 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.669429778Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.669987006Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=558.481µs 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.694725137Z level=info msg="Executing migration" id="add share column" 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.709851712Z level=info msg="Migration successfully executed" id="add share column" duration=15.120325ms 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.753468316Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-20T19:19:55.750 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.754047071Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=579.155µs 2024-09-20T19:19:56.010 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.778731266Z level=info msg="Executing migration" id="create file table" 2024-09-20T19:19:56.010 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.780743197Z level=info msg="Migration successfully executed" id="create file table" duration=2.013538ms 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.804037532Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.806299447Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=2.263728ms 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.829319978Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.831549574Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=2.230583ms 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.871238662Z level=info msg="Executing migration" id="create file_meta table" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.873105668Z level=info msg="Migration successfully executed" id="create file_meta table" duration=1.868146ms 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.896572114Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.89881489Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=2.243715ms 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.922134065Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.92228437Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=152.669µs 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.947309095Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.947489284Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=180.519µs 2024-09-20T19:19:56.011 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.972577468Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-20T19:19:56.012 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:55.973717384Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=1.141489ms 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:55 smithi162 bash[24557]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.014795762Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.015600115Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=807.169µs 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.040202861Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.043011935Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=2.807514ms 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.082137918Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-20T19:19:56.270 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.098058072Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=15.919144ms 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.124083645Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.124478644Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=395.189µs 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.149419617Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.151662109Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=2.241832ms 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.191363757Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.192180066Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=820.932µs 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.216633067Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.217181755Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=548.222µs 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.2418729Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.242870484Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=997.354µs 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.258853028Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-20T19:19:56.271 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.274516158Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=15.662847ms 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[22706]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.315 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.252907+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.255098+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.440458+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.a scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.441951+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.a scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.797167+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: cluster 2024-09-20T19:19:54.798876+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.916813+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]} : dispatch 2024-09-20T19:19:56.316 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.917298+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.917700+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-20T19:19:56.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:55 smithi042 bash[32075]: audit 2024-09-20T19:19:55.918115+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.292421997Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.30911739Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=16.694336ms 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.325999255Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.328151282Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=2.151257ms 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.368023384Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.47832616Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=110.303189ms 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.518383668Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-20T19:19:56.539 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.519634997Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=1.251721ms 2024-09-20T19:19:56.540 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.54376685Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-20T19:19:56.598 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:19:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:19:56.599 INFO:teuthology.orchestra.run.smithi042.stdout:81604378711 2024-09-20T19:19:56.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T19:19:56.802 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.545110847Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=1.347847ms 2024-09-20T19:19:56.802 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.569080668Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.598458199Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=29.377058ms 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.619386346Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.629863486Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=10.476368ms 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.652965983Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.653534935Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=568.585µs 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.678211683Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.678632816Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=418.113µs 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.720275082Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.720808444Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=531.225µs 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.745556795Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.746048158Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=487.299µs 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.787578238Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-20T19:19:56.803 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.788058714Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=480.102µs 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.829517918Z level=info msg="Executing migration" id="create folder table" 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.831995249Z level=info msg="Migration successfully executed" id="create folder table" duration=2.478034ms 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.854990369Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.857325002Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=2.334088ms 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.897179768Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-20T19:19:57.095 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.899474233Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=2.293555ms 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.939235777Z level=info msg="Executing migration" id="Update folder title length" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.939288583Z level=info msg="Migration successfully executed" id="Update folder title length" duration=55.675µs 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.956154498Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.958712111Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=2.556742ms 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.973163382Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:56 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:56.975386611Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=2.225398ms 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.023646816Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.026788356Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=3.140757ms 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.057439831Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.05845965Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=1.020207ms 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.099353578Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-20T19:19:57.096 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.099933001Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=578.926µs 2024-09-20T19:19:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[24557]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.178 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[62231]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[32075]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.303296+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:19:57.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.305558+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.844418+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.848150+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.901583+0000 mgr.y (mgr.24491) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.983877+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984103+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984199+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:55.984286+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T19:19:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: cluster 2024-09-20T19:19:55.999280+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:19:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[22706]: audit 2024-09-20T19:19:56.515341+0000 mon.a (mon.0) 912 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.141350575Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.143636658Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=2.285646ms 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.166663069Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.16902489Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=2.361051ms 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.208679352Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.21094931Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=2.269682ms 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.234021064Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.236404159Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=2.381958ms 2024-09-20T19:19:57.373 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.259231865Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.261474857Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=2.244276ms 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.284502691Z level=info msg="Executing migration" id="create anon_device table" 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.286372451Z level=info msg="Migration successfully executed" id="create anon_device table" duration=1.870716ms 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.301386223Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.303851503Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=2.465867ms 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.326820538Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.329204712Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=2.385344ms 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.343657737Z level=info msg="Executing migration" id="create signing_key table" 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.345546152Z level=info msg="Migration successfully executed" id="create signing_key table" duration=1.889522ms 2024-09-20T19:19:57.374 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.377315761Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-20T19:19:57.633 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.379682753Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=2.368489ms 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.402688987Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.405006002Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=2.319002ms 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.44473303Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.445423817Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=692.73µs 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.47000689Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.485888824Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=15.884518ms 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.528632817Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.53013245Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=1.502029ms 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.570740379Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.570796986Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=60.69µs 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.612692726Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.614975312Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=2.282176ms 2024-09-20T19:19:57.634 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.638082519Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-20T19:19:57.635 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.638121031Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=48.553µs 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.680010356Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.682490115Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=2.48011ms 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.70551041Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.707875701Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=2.364177ms 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.730875572Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.733114895Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=2.237672ms 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.772885178Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-20T19:19:57.886 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.774466964Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=1.581106ms 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.798074816Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.798722647Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=649.004µs 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.823342232Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.823505631Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=161.608µs 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.856954651Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.873148816Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=16.157966ms 2024-09-20T19:19:57.887 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.890601722Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-20T19:19:57.896 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:57 smithi042 bash[62743]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.907514391Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=16.911379ms 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.924179494Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.92494577Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=764.992µs 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=migrator t=2024-09-20T19:19:57.94104264Z level=info msg="migrations completed" performed=551 skipped=0 duration=19.159319713s 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=sqlstore t=2024-09-20T19:19:57.943651282Z level=info msg="Created default organization" 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:57 smithi162 bash[51183]: logger=secrets t=2024-09-20T19:19:57.967252291Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=plugin.store t=2024-09-20T19:19:58.013019884Z level=info msg="Loading plugins..." 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=local.finder t=2024-09-20T19:19:58.065068869Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=plugin.store t=2024-09-20T19:19:58.065093228Z level=info msg="Plugins loaded" count=55 duration=52.075828ms 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=query_data t=2024-09-20T19:19:58.067417655Z level=info msg="Query Service initialization" 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=live.push_http t=2024-09-20T19:19:58.071387332Z level=info msg="Live Push Gateway initialization" 2024-09-20T19:19:58.241 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.migration t=2024-09-20T19:19:58.116978349Z level=info msg=Starting 2024-09-20T19:19:58.242 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.migration t=2024-09-20T19:19:58.117865744Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-20T19:19:58.242 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.migration orgID=1 t=2024-09-20T19:19:58.118671823Z level=info msg="Migrating alerts for organisation" 2024-09-20T19:19:58.242 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.migration orgID=1 t=2024-09-20T19:19:58.119900558Z level=info msg="Alerts found to migrate" alerts=0 2024-09-20T19:19:58.242 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.migration t=2024-09-20T19:19:58.123439162Z level=info msg="Completed alerting migration" 2024-09-20T19:19:58.242 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.state.manager t=2024-09-20T19:19:58.245177718Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-20T19:19:58.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.258 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:58 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T19:19:58.258 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:58 smithi042 systemd[1]: Stopped Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:58.258 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:19:58 smithi042 systemd[1]: Started Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.258 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.259 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.340872+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.343243+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.423392+0000 osd.1 (osd.1) 53 : cluster [DBG] 5.c scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.425149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.c scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.853108+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.854457+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:56.993552+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:19:58.260 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.261 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.059500+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:19:58.261 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.261 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:58 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.060881+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:19:58.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=infra.usagestats.collector t=2024-09-20T19:19:58.24998549Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-20T19:19:58.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=provisioning.datasources t=2024-09-20T19:19:58.254923857Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T19:19:58.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=provisioning.datasources t=2024-09-20T19:19:58.310720065Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-20T19:19:58.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=provisioning.alerting t=2024-09-20T19:19:58.388544811Z level=info msg="starting to provision alerting" 2024-09-20T19:19:58.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=provisioning.alerting t=2024-09-20T19:19:58.388579805Z level=info msg="finished to provision alerting" 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=grafanaStorageLogger t=2024-09-20T19:19:58.389761844Z level=info msg="Storage starting" 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.state.manager t=2024-09-20T19:19:58.390300419Z level=info msg="Warming state cache for startup" 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.multiorg.alertmanager t=2024-09-20T19:19:58.390846519Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=http.server t=2024-09-20T19:19:58.393137807Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=http.server t=2024-09-20T19:19:58.393379217Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=sqlstore.transactions t=2024-09-20T19:19:58.400460186Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=grafana.update.checker t=2024-09-20T19:19:58.463205997Z level=info msg="Update check succeeded" duration=74.160812ms 2024-09-20T19:19:58.553 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=plugins.update.checker t=2024-09-20T19:19:58.470821113Z level=info msg="Update check succeeded" duration=81.784476ms 2024-09-20T19:19:58.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.state.manager t=2024-09-20T19:19:58.611708118Z level=info msg="State cache has been initialized" states=0 duration=221.368648ms 2024-09-20T19:19:58.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ngalert.scheduler t=2024-09-20T19:19:58.611765771Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-20T19:19:58.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=ticker t=2024-09-20T19:19:58.61187988Z level=info msg=starting first_tick=2024-09-20T19:20:00Z 2024-09-20T19:19:58.892 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[51183]: logger=provisioning.dashboard t=2024-09-20T19:19:58.6136553Z level=info msg="starting to provision dashboards" 2024-09-20T19:19:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:19:58 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.157 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[24557]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[22706]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.894215+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:19:59.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.895918+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:57.902331+0000 mgr.y (mgr.24491) 65 : cluster [DBG] pgmap v45: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.000727+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.026712+0000 osd.3 (osd.3) 41 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cluster 2024-09-20T19:19:58.028767+0000 osd.3 (osd.3) 42 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: audit 2024-09-20T19:19:58.098699+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: audit 2024-09-20T19:19:58.103895+0000 mon.a (mon.0) 916 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:19:59.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cephadm 2024-09-20T19:19:58.105938+0000 mgr.y (mgr.24491) 66 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:19:59.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:19:59 smithi042 bash[32075]: cephadm 2024-09-20T19:19:58.370878+0000 mgr.y (mgr.24491) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi162 2024-09-20T19:19:59.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[51183]: logger=grafana-apiserver t=2024-09-20T19:19:59.161788567Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-20T19:19:59.552 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:19:59 smithi162 bash[51183]: logger=grafana-apiserver t=2024-09-20T19:19:59.162380949Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-20T19:19:59.571 INFO:teuthology.orchestra.run.smithi042.stdout:150323855423 2024-09-20T19:19:59.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T19:20:00.042 INFO:teuthology.orchestra.run.smithi042.stdout:34359738479 2024-09-20T19:20:00.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T19:20:00.050 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.050 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.050 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.051 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.052 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.052 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.052 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.052 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.052 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: audit 2024-09-20T19:19:58.891959+0000 mgr.y (mgr.24491) 68 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000106+0000 mon.a (mon.0) 917 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000163+0000 mon.a (mon.0) 918 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000195+0000 mon.a (mon.0) 919 : cluster [WRN] pg 2.5 is stuck peering for 111s, current state peering, last acting [2,0,4] 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:00.000280+0000 mon.a (mon.0) 920 : cluster [WRN] pg 4.7 is stuck peering for 103s, current state peering, last acting [2,5,0] 2024-09-20T19:20:00.603 INFO:teuthology.orchestra.run.smithi042.stdout:120259084361 2024-09-20T19:20:00.604 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T19:20:01.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:01 smithi162 bash[24557]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[32075]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.332970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.335032+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.549878+0000 osd.1 (osd.1) 55 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.552031+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[22706]: cluster 2024-09-20T19:19:59.903295+0000 mgr.y (mgr.24491) 69 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:20:01.474 INFO:teuthology.orchestra.run.smithi042.stdout:55834574948 2024-09-20T19:20:01.475 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T19:20:01.590 INFO:teuthology.orchestra.run.smithi042.stdout:188978561076 2024-09-20T19:20:01.591 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.818Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.818Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.820Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.42 port=9094 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.820Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.849Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.850Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.852Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-20T19:20:02.073 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:01 smithi042 bash[62791]: ts=2024-09-20T19:20:01.852Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T19:20:02.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:02 smithi162 bash[24557]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:02 smithi162 bash[24557]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:02 smithi162 bash[24557]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:02 smithi162 bash[24557]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[32075]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[32075]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[32075]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[32075]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[22706]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[22706]: cluster 2024-09-20T19:20:02.020607+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:20:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[22706]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:02 smithi042 bash[22706]: cluster 2024-09-20T19:20:02.020649+0000 mon.a (mon.0) 922 : cluster [INF] Cluster is now healthy 2024-09-20T19:20:02.802 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:20:02 smithi162 bash[51183]: logger=provisioning.dashboard t=2024-09-20T19:20:02.555336423Z level=info msg="finished to provision dashboards" 2024-09-20T19:20:03.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:03 smithi042 bash[32075]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:03 smithi042 bash[32075]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:03 smithi042 bash[22706]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:03 smithi042 bash[22706]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[24557]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[24557]: cluster 2024-09-20T19:20:01.904670+0000 mgr.y (mgr.24491) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 83 B/s, 1 objects/s recovering 2024-09-20T19:20:03.878 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 systemd[1]: Stopping Ceph prometheus.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.714Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.715Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.716Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.716Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-20T19:20:03.879 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:03 smithi162 bash[49971]: ts=2024-09-20T19:20:03.716Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-20T19:20:04.179 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:03 smithi042 bash[62791]: ts=2024-09-20T19:20:03.821Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000317675s 2024-09-20T19:20:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:04.364 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:04.364 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:04.364 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:04.470 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:04 smithi162 bash[51809]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:20:04.471 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:04 smithi162 bash[51870]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:20:04.471 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:04 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T19:20:04.471 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:04 smithi162 systemd[1]: Stopped Ceph prometheus.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:20:04.471 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:04 smithi162 systemd[1]: Started Ceph prometheus.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:20:04.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Bus STOPPING 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Bus STOPPED 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Bus STARTING 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Serving on http://:::9283 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Bus STARTED 2024-09-20T19:20:05.082 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:04 smithi042 bash[22958]: [20/Sep/2024:19:20:04] ENGINE Bus STOPPING 2024-09-20T19:20:05.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[32075]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.339 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.340 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22706]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.389 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.325133+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.328518+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:03.905363+0000 mgr.y (mgr.24491) 71 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 1 objects/s recovering 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:04.095422+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: cluster 2024-09-20T19:20:04.097827+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.384311+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.394278+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.400937+0000 mon.a (mon.0) 925 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.401439+0000 mgr.y (mgr.24491) 72 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.402656+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.403022+0000 mgr.y (mgr.24491) 73 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi042.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.407693+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.416224+0000 mon.a (mon.0) 928 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-20T19:20:05.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.416718+0000 mgr.y (mgr.24491) 74 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.422013+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"} : dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.422466+0000 mgr.y (mgr.24491) 75 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.426379+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.434425+0000 mon.a (mon.0) 931 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.434852+0000 mgr.y (mgr.24491) 76 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.439319+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"} : dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.439669+0000 mgr.y (mgr.24491) 77 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi162.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.443499+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.717374+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[24557]: audit 2024-09-20T19:20:04.853888+0000 mon.a (mon.0) 935 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STOPPED 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STARTING 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Serving on http://:::9283 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STARTED 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STOPPING 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STOPPED 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STARTING 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Serving on http://:::9283 2024-09-20T19:20:05.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:05 smithi042 bash[22958]: [20/Sep/2024:19:20:05] ENGINE Bus STARTED 2024-09-20T19:20:05.802 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.394Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-20T19:20:05.802 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.394Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.394Z caller=main.go:623 level=info host_details="(Linux 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 smithi162 (none))" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.394Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.394Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.397Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.397Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.399Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.399Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.402Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.402Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.879µs 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.402Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.402Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.403Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T19:20:05.803 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.403Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.838µs wal_replay_duration=803.707µs wbl_replay_duration=119ns total_replay_duration=849.738µs 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.405Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.405Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.405Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.422Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.656974ms db_storage=922ns remote_storage=1.112µs web_handler=338ns query_engine=469ns scrape=242.625µs scrape_sd=144.316µs notify=25.669µs notify_sd=18.225µs rules=16.734482ms tracing=5.98µs 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.422Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-20T19:20:05.804 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:20:05 smithi162 bash[51898]: ts=2024-09-20T19:20:05.422Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:20:06.399 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:06.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[22706]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:06 smithi042 bash[32075]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:04.719145+0000 mgr.y (mgr.24491) 78 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.332684+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.334206+0000 mgr.y (mgr.24491) 79 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.449763+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:06 smithi162 bash[24557]: audit 2024-09-20T19:20:05.450724+0000 mgr.y (mgr.24491) 80 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[22706]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:07 smithi042 bash[32075]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.280734+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.282287+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:07 smithi162 bash[24557]: cluster 2024-09-20T19:20:05.905976+0000 mgr.y (mgr.24491) 81 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 66 B/s, 1 objects/s recovering 2024-09-20T19:20:08.331 INFO:teuthology.orchestra.run.smithi042.stdout:34359738482 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:08 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:06.281074+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:06.283241+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.129392+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:20:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.131525+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:20:08.931 INFO:teuthology.orchestra.run.smithi042.stdout:274877906977 2024-09-20T19:20:08.945 INFO:teuthology.orchestra.run.smithi042.stdout:231928234028 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.201 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[22706]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:09 smithi042 bash[32075]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:08 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: cluster 2024-09-20T19:20:07.907486+0000 mgr.y (mgr.24491) 82 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 60 B/s, 1 objects/s recovering 2024-09-20T19:20:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.331432+0000 mon.a (mon.0) 938 : audit [DBG] from='client.? v1:172.21.15.42:0/3117649352' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:20:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.931090+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.42:0/3941919104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:20:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:09 smithi162 bash[24557]: audit 2024-09-20T19:20:08.944729+0000 mon.a (mon.0) 939 : audit [DBG] from='client.? v1:172.21.15.42:0/3892913196' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:20:09.503 INFO:teuthology.orchestra.run.smithi042.stdout:81604378714 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[32075]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[32075]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[32075]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[32075]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[22706]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[22706]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[22706]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:10 smithi042 bash[22706]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:10 smithi162 bash[24557]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:10 smithi162 bash[24557]: audit 2024-09-20T19:20:08.902757+0000 mgr.y (mgr.24491) 83 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:10 smithi162 bash[24557]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:10 smithi162 bash[24557]: audit 2024-09-20T19:20:09.503056+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? v1:172.21.15.42:0/223276811' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:20:10.977 INFO:teuthology.orchestra.run.smithi042.stdout:55834574950 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.315 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.317 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[32075]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[22706]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738479 got 34359738482 for osd.0 2024-09-20T19:20:11.320 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.341541+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.341648+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:09.908177+0000 mgr.y (mgr.24491) 84 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: cluster 2024-09-20T19:20:10.112582+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.794174+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.799035+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:11 smithi162 bash[24557]: audit 2024-09-20T19:20:10.976708+0000 mon.a (mon.0) 944 : audit [DBG] from='client.? v1:172.21.15.42:0/3143960958' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:20:11.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234025 got 231928234028 for osd.6 2024-09-20T19:20:11.875 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:12.116 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:20:11 smithi042 bash[62791]: ts=2024-09-20T19:20:11.824Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.0036233s 2024-09-20T19:20:12.163 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:12 smithi042 bash[22706]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:12.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:12 smithi042 bash[22706]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:12.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:12 smithi042 bash[32075]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:12.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:12 smithi042 bash[32075]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:12.199 INFO:teuthology.orchestra.run.smithi042.stdout:150323855427 2024-09-20T19:20:12.355 INFO:teuthology.orchestra.run.smithi042.stdout:120259084365 2024-09-20T19:20:12.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906974 got 274877906977 for osd.7 2024-09-20T19:20:12.405 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:12 smithi162 bash[24557]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:12 smithi162 bash[24557]: audit 2024-09-20T19:20:11.528236+0000 mon.a (mon.0) 945 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:13.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[22706]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:13 smithi042 bash[32075]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: cluster 2024-09-20T19:20:11.908844+0000 mgr.y (mgr.24491) 85 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: audit 2024-09-20T19:20:12.199421+0000 mon.a (mon.0) 946 : audit [DBG] from='client.? v1:172.21.15.42:0/1013288541' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:13 smithi162 bash[24557]: audit 2024-09-20T19:20:12.354512+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.42:0/694704828' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:20:13.671 INFO:teuthology.orchestra.run.smithi042.stdout:188978561079 2024-09-20T19:20:14.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[32075]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:14.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:14 smithi042 bash[22706]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:14.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574948 got 55834574950 for osd.1 2024-09-20T19:20:14.421 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:14.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378711 got 81604378714 for osd.2 2024-09-20T19:20:14.425 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: cluster 2024-09-20T19:20:12.272151+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: cluster 2024-09-20T19:20:12.275506+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:14 smithi162 bash[24557]: audit 2024-09-20T19:20:13.671477+0000 mon.a (mon.0) 947 : audit [DBG] from='client.? v1:172.21.15.42:0/1051709666' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:20:15.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:15 smithi042 bash[32075]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:15.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:15 smithi042 bash[32075]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:15.166 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:15 smithi042 bash[22706]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:15.166 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:15 smithi042 bash[22706]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:15.166 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855423 got 150323855427 for osd.4 2024-09-20T19:20:15.167 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:15.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084361 got 120259084365 for osd.3 2024-09-20T19:20:15.234 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:15.321 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561076 got 188978561079 for osd.5 2024-09-20T19:20:15.321 DEBUG:teuthology.parallel:result is None 2024-09-20T19:20:15.322 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T19:20:15.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:20:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:15 smithi162 bash[24557]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:15 smithi162 bash[24557]: cluster 2024-09-20T19:20:13.909490+0000 mgr.y (mgr.24491) 86 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:16.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:16 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:16 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:16 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.000369+0000 osd.3 (osd.3) 47 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:20:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:16 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:16 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.003437+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[32075]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:17 smithi042 bash[22706]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:15.909910+0000 mgr.y (mgr.24491) 87 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.024771+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.026385+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.339311+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.345422+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.848960+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.852440+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.961676+0000 mon.a (mon.0) 950 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.962485+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:17 smithi162 bash[24557]: audit 2024-09-20T19:20:16.969693+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[22706]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:18 smithi042 bash[32075]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:18 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:18 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.654757+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:20:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:18 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:18 smithi162 bash[24557]: cluster 2024-09-20T19:20:16.656947+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:20:19.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:18 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:19 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.205936+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.206114+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.519363+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:20:19.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.521936+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:20:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:17.911325+0000 mgr.y (mgr.24491) 88 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:19 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.156755+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:20:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:20 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.221231+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:18.221345+0000 osd.6 (osd.6) 34 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: audit 2024-09-20T19:20:18.913967+0000 mgr.y (mgr.24491) 89 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:20 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.157064+0000 mon.a (mon.0) 954 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:20:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[32075]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:21 smithi042 bash[22706]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:21.527 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:20:21.529 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.563468+0000 osd.1 (osd.1) 59 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.566994+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:21 smithi162 bash[24557]: cluster 2024-09-20T19:20:19.912007+0000 mgr.y (mgr.24491) 90 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:22.385 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-20T19:20:19.911608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":976,"num_read_kb":833,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244172,"kb_used_data":7164,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717044,"statfs":{"total":767960285184,"available":767710253056,"internally_reserved":0,"allocated":7335936,"data_stored":4334025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003231"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112278+0000","last_change":"2024-09-20T19:19:26.193045+0000","last_active":"2024-09-20T19:20:10.112278+0000","last_peered":"2024-09-20T19:20:10.112278+0000","last_clean":"2024-09-20T19:20:10.112278+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:20:10.112278+0000","last_undegraded":"2024-09-20T19:20:10.112278+0000","last_fullsized":"2024-09-20T19:20:10.112278+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.192962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.192962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.220702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.118400+0000","last_change":"2024-09-20T19:19:17.698496+0000","last_active":"2024-09-20T19:20:10.118400+0000","last_peered":"2024-09-20T19:20:10.118400+0000","last_clean":"2024-09-20T19:20:10.118400+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:20:10.118400+0000","last_undegraded":"2024-09-20T19:20:10.118400+0000","last_fullsized":"2024-09-20T19:20:10.118400+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.698427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.698427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.779691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":107,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113403+0000","last_change":"2024-09-20T19:19:45.244647+0000","last_active":"2024-09-20T19:20:10.113403+0000","last_peered":"2024-09-20T19:20:10.113403+0000","last_clean":"2024-09-20T19:20:10.113403+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:20:10.113403+0000","last_undegraded":"2024-09-20T19:20:10.113403+0000","last_fullsized":"2024-09-20T19:20:10.113403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:45.244486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.244486+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.299507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.003898+0000","last_change":"2024-09-20T19:19:13.217834+0000","last_active":"2024-09-20T19:20:11.003898+0000","last_peered":"2024-09-20T19:20:11.003898+0000","last_clean":"2024-09-20T19:20:11.003898+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:20:11.003898+0000","last_undegraded":"2024-09-20T19:20:11.003898+0000","last_fullsized":"2024-09-20T19:20:11.003898+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.217715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.217715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:32.923722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113497+0000","last_change":"2024-09-20T19:19:11.241620+0000","last_active":"2024-09-20T19:20:10.113497+0000","last_peered":"2024-09-20T19:20:10.113497+0000","last_clean":"2024-09-20T19:20:10.113497+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:20:10.113497+0000","last_undegraded":"2024-09-20T19:20:10.113497+0000","last_fullsized":"2024-09-20T19:20:10.113497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:11.241540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:11.241540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:13.499402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.003918+0000","last_change":"2024-09-20T19:19:12.206330+0000","last_active":"2024-09-20T19:20:11.003918+0000","last_peered":"2024-09-20T19:20:11.003918+0000","last_clean":"2024-09-20T19:20:11.003918+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:20:11.003918+0000","last_undegraded":"2024-09-20T19:20:11.003918+0000","last_fullsized":"2024-09-20T19:20:11.003918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:12.206186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:12.206186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.210390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.441540+0000","last_change":"2024-09-20T19:19:30.514806+0000","last_active":"2024-09-20T19:20:10.441540+0000","last_peered":"2024-09-20T19:20:10.441540+0000","last_clean":"2024-09-20T19:20:10.441540+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:20:10.441540+0000","last_undegraded":"2024-09-20T19:20:10.441540+0000","last_fullsized":"2024-09-20T19:20:10.441540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:30.514674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:30.514674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:35.302233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004274+0000","last_change":"2024-09-20T19:19:32.223697+0000","last_active":"2024-09-20T19:20:11.004274+0000","last_peered":"2024-09-20T19:20:11.004274+0000","last_clean":"2024-09-20T19:20:11.004274+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:20:11.004274+0000","last_undegraded":"2024-09-20T19:20:11.004274+0000","last_fullsized":"2024-09-20T19:20:11.004274+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.223538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.223538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.453072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115865+0000","last_change":"2024-09-20T19:19:59.335157+0000","last_active":"2024-09-20T19:20:10.115865+0000","last_peered":"2024-09-20T19:20:10.115865+0000","last_clean":"2024-09-20T19:20:10.115865+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:20:10.115865+0000","last_undegraded":"2024-09-20T19:20:10.115865+0000","last_fullsized":"2024-09-20T19:20:10.115865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.335040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.335040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.153396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":92,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004350+0000","last_change":"2024-09-20T19:19:15.268253+0000","last_active":"2024-09-20T19:20:11.004350+0000","last_peered":"2024-09-20T19:20:11.004350+0000","last_clean":"2024-09-20T19:20:11.004350+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:20:11.004350+0000","last_undegraded":"2024-09-20T19:20:11.004350+0000","last_fullsized":"2024-09-20T19:20:11.004350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:19:15.268070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:15.268070+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.887074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115165+0000","last_change":"2024-09-20T19:19:43.205222+0000","last_active":"2024-09-20T19:20:10.115165+0000","last_peered":"2024-09-20T19:20:10.115165+0000","last_clean":"2024-09-20T19:20:10.115165+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:20:10.115165+0000","last_undegraded":"2024-09-20T19:20:10.115165+0000","last_fullsized":"2024-09-20T19:20:10.115165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.205146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.205146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.365222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":73,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256897+0000","last_change":"2024-09-20T19:19:32.084443+0000","last_active":"2024-09-20T19:20:10.256897+0000","last_peered":"2024-09-20T19:20:10.256897+0000","last_clean":"2024-09-20T19:20:10.256897+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:20:10.256897+0000","last_undegraded":"2024-09-20T19:20:10.256897+0000","last_fullsized":"2024-09-20T19:20:10.256897+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.084319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.084319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.681633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.116077+0000","last_change":"2024-09-20T19:19:41.142126+0000","last_active":"2024-09-20T19:20:10.116077+0000","last_peered":"2024-09-20T19:20:10.116077+0000","last_clean":"2024-09-20T19:20:10.116077+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:20:10.116077+0000","last_undegraded":"2024-09-20T19:20:10.116077+0000","last_fullsized":"2024-09-20T19:20:10.116077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.141975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.141975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.007349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004706+0000","last_change":"2024-09-20T19:19:42.188043+0000","last_active":"2024-09-20T19:20:11.004706+0000","last_peered":"2024-09-20T19:20:11.004706+0000","last_clean":"2024-09-20T19:20:11.004706+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:20:11.004706+0000","last_undegraded":"2024-09-20T19:20:11.004706+0000","last_fullsized":"2024-09-20T19:20:11.004706+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:42.187756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:42.187756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.911434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":77,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256986+0000","last_change":"2024-09-20T19:19:40.029794+0000","last_active":"2024-09-20T19:20:10.256986+0000","last_peered":"2024-09-20T19:20:10.256986+0000","last_clean":"2024-09-20T19:20:10.256986+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:20:10.256986+0000","last_undegraded":"2024-09-20T19:20:10.256986+0000","last_fullsized":"2024-09-20T19:20:10.256986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.029674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.029674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.995856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112714+0000","last_change":"2024-09-20T19:19:54.798943+0000","last_active":"2024-09-20T19:20:10.112714+0000","last_peered":"2024-09-20T19:20:10.112714+0000","last_clean":"2024-09-20T19:20:10.112714+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:20:10.112714+0000","last_undegraded":"2024-09-20T19:20:10.112714+0000","last_fullsized":"2024-09-20T19:20:10.112714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.798882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.798882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.457870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"77'17","reported_seq":110,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005591+0000","last_change":"2024-09-20T19:19:48.026945+0000","last_active":"2024-09-20T19:20:11.005591+0000","last_peered":"2024-09-20T19:20:11.005591+0000","last_clean":"2024-09-20T19:20:11.005591+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:20:11.005591+0000","last_undegraded":"2024-09-20T19:20:11.005591+0000","last_fullsized":"2024-09-20T19:20:11.005591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:19:48.026795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.026795+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.794994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115692+0000","last_change":"2024-09-20T19:20:05.282366+0000","last_active":"2024-09-20T19:20:10.115692+0000","last_peered":"2024-09-20T19:20:10.115692+0000","last_clean":"2024-09-20T19:20:10.115692+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:20:10.115692+0000","last_undegraded":"2024-09-20T19:20:10.115692+0000","last_fullsized":"2024-09-20T19:20:10.115692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:05.282293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:05.282293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.281996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442690+0000","last_change":"2024-09-20T19:19:56.425232+0000","last_active":"2024-09-20T19:20:10.442690+0000","last_peered":"2024-09-20T19:20:10.442690+0000","last_clean":"2024-09-20T19:20:10.442690+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:20:10.442690+0000","last_undegraded":"2024-09-20T19:20:10.442690+0000","last_fullsized":"2024-09-20T19:20:10.442690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.425156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.425156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:21.486886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":72,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.999056+0000","last_change":"2024-09-20T19:19:46.074958+0000","last_active":"2024-09-20T19:20:10.999056+0000","last_peered":"2024-09-20T19:20:10.999056+0000","last_clean":"2024-09-20T19:20:10.999056+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:20:10.999056+0000","last_undegraded":"2024-09-20T19:20:10.999056+0000","last_fullsized":"2024-09-20T19:20:10.999056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:46.074887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.074887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.317322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"77'10","reported_seq":97,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004986+0000","last_change":"2024-09-20T19:19:43.146526+0000","last_active":"2024-09-20T19:20:11.004986+0000","last_peered":"2024-09-20T19:20:11.004986+0000","last_clean":"2024-09-20T19:20:11.004986+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:20:11.004986+0000","last_undegraded":"2024-09-20T19:20:11.004986+0000","last_fullsized":"2024-09-20T19:20:11.004986+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:43.146459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.146459+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.154934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442673+0000","last_change":"2024-09-20T19:19:54.442006+0000","last_active":"2024-09-20T19:20:10.442673+0000","last_peered":"2024-09-20T19:20:10.442673+0000","last_clean":"2024-09-20T19:20:10.442673+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:20:10.442673+0000","last_undegraded":"2024-09-20T19:20:10.442673+0000","last_fullsized":"2024-09-20T19:20:10.442673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.441955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.441955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.241390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":115,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.999045+0000","last_change":"2024-09-20T19:19:45.038512+0000","last_active":"2024-09-20T19:20:10.999045+0000","last_peered":"2024-09-20T19:20:10.999045+0000","last_clean":"2024-09-20T19:20:10.999045+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:20:10.999045+0000","last_undegraded":"2024-09-20T19:20:10.999045+0000","last_fullsized":"2024-09-20T19:20:10.999045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:45.038377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.038377+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.979665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005112+0000","last_change":"2024-09-20T19:19:38.210845+0000","last_active":"2024-09-20T19:20:11.005112+0000","last_peered":"2024-09-20T19:20:11.005112+0000","last_clean":"2024-09-20T19:20:11.005112+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:20:11.005112+0000","last_undegraded":"2024-09-20T19:20:11.005112+0000","last_fullsized":"2024-09-20T19:20:11.005112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:38.210713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:38.210713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.317890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"77'15","reported_seq":107,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443361+0000","last_change":"2024-09-20T19:19:15.487623+0000","last_active":"2024-09-20T19:20:10.443361+0000","last_peered":"2024-09-20T19:20:10.443361+0000","last_clean":"2024-09-20T19:20:10.443361+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:20:10.443361+0000","last_undegraded":"2024-09-20T19:20:10.443361+0000","last_fullsized":"2024-09-20T19:20:10.443361+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:15.487541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:15.487541+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:22.627466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.441807+0000","last_change":"2024-09-20T19:19:36.378407+0000","last_active":"2024-09-20T19:20:10.441807+0000","last_peered":"2024-09-20T19:20:10.441807+0000","last_clean":"2024-09-20T19:20:10.441807+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:20:10.441807+0000","last_undegraded":"2024-09-20T19:20:10.441807+0000","last_fullsized":"2024-09-20T19:20:10.441807+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.378285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.378285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:40.689679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":114,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005421+0000","last_change":"2024-09-20T19:19:50.966826+0000","last_active":"2024-09-20T19:20:11.005421+0000","last_peered":"2024-09-20T19:20:11.005421+0000","last_clean":"2024-09-20T19:20:11.005421+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:20:11.005421+0000","last_undegraded":"2024-09-20T19:20:11.005421+0000","last_fullsized":"2024-09-20T19:20:11.005421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:50.966659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.966659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.209609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":73,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257293+0000","last_change":"2024-09-20T19:19:34.013906+0000","last_active":"2024-09-20T19:20:10.257293+0000","last_peered":"2024-09-20T19:20:10.257293+0000","last_clean":"2024-09-20T19:20:10.257293+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:20:10.257293+0000","last_undegraded":"2024-09-20T19:20:10.257293+0000","last_fullsized":"2024-09-20T19:20:10.257293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.013787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.013787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.149084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"77'11","reported_seq":101,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005540+0000","last_change":"2024-09-20T19:19:26.201077+0000","last_active":"2024-09-20T19:20:11.005540+0000","last_peered":"2024-09-20T19:20:11.005540+0000","last_clean":"2024-09-20T19:20:11.005540+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:20:11.005540+0000","last_undegraded":"2024-09-20T19:20:11.005540+0000","last_fullsized":"2024-09-20T19:20:11.005540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:26.200992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.200992+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.983769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115066+0000","last_change":"2024-09-20T19:19:48.281994+0000","last_active":"2024-09-20T19:20:10.115066+0000","last_peered":"2024-09-20T19:20:10.115066+0000","last_clean":"2024-09-20T19:20:10.115066+0000","last_became_active":"2024-09-20T19:18:07.269527+0000","last_became_peered":"2024-09-20T19:18:07.269527+0000","last_unstale":"2024-09-20T19:20:10.115066+0000","last_undegraded":"2024-09-20T19:20:10.115066+0000","last_fullsized":"2024-09-20T19:20:10.115066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.281832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.281832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.535302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":77,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257559+0000","last_change":"2024-09-20T19:19:34.966265+0000","last_active":"2024-09-20T19:20:10.257559+0000","last_peered":"2024-09-20T19:20:10.257559+0000","last_clean":"2024-09-20T19:20:10.257559+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:20:10.257559+0000","last_undegraded":"2024-09-20T19:20:10.257559+0000","last_fullsized":"2024-09-20T19:20:10.257559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.966141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.966141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.384654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:16.026450+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:20:16.026450+0000","last_peered":"2024-09-20T19:20:16.026450+0000","last_clean":"2024-09-20T19:20:16.026450+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:20:16.026450+0000","last_undegraded":"2024-09-20T19:20:16.026450+0000","last_fullsized":"2024-09-20T19:20:16.026450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.064548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"77'14","reported_seq":104,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004453+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:20:11.004453+0000","last_peered":"2024-09-20T19:20:11.004453+0000","last_clean":"2024-09-20T19:20:11.004453+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:20:11.004453+0000","last_undegraded":"2024-09-20T19:20:11.004453+0000","last_fullsized":"2024-09-20T19:20:11.004453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.837736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":94,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:16.657319+0000","last_change":"2024-09-20T19:20:16.657110+0000","last_active":"2024-09-20T19:20:16.657319+0000","last_peered":"2024-09-20T19:20:16.657319+0000","last_clean":"2024-09-20T19:20:16.657319+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:20:16.657319+0000","last_undegraded":"2024-09-20T19:20:16.657319+0000","last_fullsized":"2024-09-20T19:20:16.657319+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.656960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.656960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:43.026972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":115,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:12.426630+0000","last_change":"2024-09-20T19:19:37.178863+0000","last_active":"2024-09-20T19:20:12.426630+0000","last_peered":"2024-09-20T19:20:12.426630+0000","last_clean":"2024-09-20T19:20:12.426630+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:20:12.426630+0000","last_undegraded":"2024-09-20T19:20:12.426630+0000","last_fullsized":"2024-09-20T19:20:12.426630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.178724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.178724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.727526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115465+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:20:10.115465+0000","last_peered":"2024-09-20T19:20:10.115465+0000","last_clean":"2024-09-20T19:20:10.115465+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:20:10.115465+0000","last_undegraded":"2024-09-20T19:20:10.115465+0000","last_fullsized":"2024-09-20T19:20:10.115465+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:18.443029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"77'18","reported_seq":115,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.110944+0000","last_change":"2024-09-20T19:19:22.198542+0000","last_active":"2024-09-20T19:20:10.110944+0000","last_peered":"2024-09-20T19:20:10.110944+0000","last_clean":"2024-09-20T19:20:10.110944+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:20:10.110944+0000","last_undegraded":"2024-09-20T19:20:10.110944+0000","last_fullsized":"2024-09-20T19:20:10.110944+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:19:22.198447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:22.198447+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.994468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":88,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443494+0000","last_change":"2024-09-20T19:19:27.488807+0000","last_active":"2024-09-20T19:20:10.443494+0000","last_peered":"2024-09-20T19:20:10.443494+0000","last_clean":"2024-09-20T19:20:10.443494+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:20:10.443494+0000","last_undegraded":"2024-09-20T19:20:10.443494+0000","last_fullsized":"2024-09-20T19:20:10.443494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:27.488644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.488644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.323330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004188+0000","last_change":"2024-09-20T19:19:44.113106+0000","last_active":"2024-09-20T19:20:11.004188+0000","last_peered":"2024-09-20T19:20:11.004188+0000","last_clean":"2024-09-20T19:20:11.004188+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:20:11.004188+0000","last_undegraded":"2024-09-20T19:20:11.004188+0000","last_fullsized":"2024-09-20T19:20:11.004188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.112985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.112985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.242746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004746+0000","last_change":"2024-09-20T19:19:48.997145+0000","last_active":"2024-09-20T19:20:11.004746+0000","last_peered":"2024-09-20T19:20:11.004746+0000","last_clean":"2024-09-20T19:20:11.004746+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:20:11.004746+0000","last_undegraded":"2024-09-20T19:20:11.004746+0000","last_fullsized":"2024-09-20T19:20:11.004746+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.996997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.996997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.494961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"77'19","reported_seq":119,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115105+0000","last_change":"2024-09-20T19:19:27.727451+0000","last_active":"2024-09-20T19:20:10.115105+0000","last_peered":"2024-09-20T19:20:10.115105+0000","last_clean":"2024-09-20T19:20:10.115105+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:20:10.115105+0000","last_undegraded":"2024-09-20T19:20:10.115105+0000","last_fullsized":"2024-09-20T19:20:10.115105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.727294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.727294+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.582205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998822+0000","last_change":"2024-09-20T19:19:56.998769+0000","last_active":"2024-09-20T19:20:10.998822+0000","last_peered":"2024-09-20T19:20:10.998822+0000","last_clean":"2024-09-20T19:20:10.998822+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:20:10.998822+0000","last_undegraded":"2024-09-20T19:20:10.998822+0000","last_fullsized":"2024-09-20T19:20:10.998822+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.169199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.169199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.901847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.705276+0000","last_change":"2024-09-20T19:19:41.871990+0000","last_active":"2024-09-20T19:20:10.705276+0000","last_peered":"2024-09-20T19:20:10.705276+0000","last_clean":"2024-09-20T19:20:10.705276+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:20:10.705276+0000","last_undegraded":"2024-09-20T19:20:10.705276+0000","last_fullsized":"2024-09-20T19:20:10.705276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.871854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.871854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.298167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.441567+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:20:10.441567+0000","last_peered":"2024-09-20T19:20:10.441567+0000","last_clean":"2024-09-20T19:20:10.441567+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:20:10.441567+0000","last_undegraded":"2024-09-20T19:20:10.441567+0000","last_fullsized":"2024-09-20T19:20:10.441567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:29.401329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"77'10","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442509+0000","last_change":"2024-09-20T19:19:40.461604+0000","last_active":"2024-09-20T19:20:10.442509+0000","last_peered":"2024-09-20T19:20:10.442509+0000","last_clean":"2024-09-20T19:20:10.442509+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:20:10.442509+0000","last_undegraded":"2024-09-20T19:20:10.442509+0000","last_fullsized":"2024-09-20T19:20:10.442509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:40.461532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.461532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:08.703461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443173+0000","last_change":"2024-09-20T19:19:34.429306+0000","last_active":"2024-09-20T19:20:10.443173+0000","last_peered":"2024-09-20T19:20:10.443173+0000","last_clean":"2024-09-20T19:20:10.443173+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:20:10.443173+0000","last_undegraded":"2024-09-20T19:20:10.443173+0000","last_fullsized":"2024-09-20T19:20:10.443173+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.429125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.429125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.671437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":115,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:12.426593+0000","last_change":"2024-09-20T19:19:37.807674+0000","last_active":"2024-09-20T19:20:12.426593+0000","last_peered":"2024-09-20T19:20:12.426593+0000","last_clean":"2024-09-20T19:20:12.426593+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:20:12.426593+0000","last_undegraded":"2024-09-20T19:20:12.426593+0000","last_fullsized":"2024-09-20T19:20:12.426593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.807547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.807547+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.829611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112446+0000","last_change":"2024-09-20T19:19:31.793749+0000","last_active":"2024-09-20T19:20:10.112446+0000","last_peered":"2024-09-20T19:20:10.112446+0000","last_clean":"2024-09-20T19:20:10.112446+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:20:10.112446+0000","last_undegraded":"2024-09-20T19:20:10.112446+0000","last_fullsized":"2024-09-20T19:20:10.112446+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.128931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'30","reported_seq":159,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:12.426599+0000","last_change":"2024-09-20T19:19:46.395559+0000","last_active":"2024-09-20T19:20:12.426599+0000","last_peered":"2024-09-20T19:20:12.426599+0000","last_clean":"2024-09-20T19:20:12.426599+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:20:12.426599+0000","last_undegraded":"2024-09-20T19:20:12.426599+0000","last_fullsized":"2024-09-20T19:20:12.426599+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:19:46.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.395404+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.847884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257486+0000","last_change":"2024-09-20T19:19:33.057715+0000","last_active":"2024-09-20T19:20:10.257486+0000","last_peered":"2024-09-20T19:20:10.257486+0000","last_clean":"2024-09-20T19:20:10.257486+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:20:10.257486+0000","last_undegraded":"2024-09-20T19:20:10.257486+0000","last_fullsized":"2024-09-20T19:20:10.257486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.057567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.057567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.036699+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115870+0000","last_change":"2024-09-20T19:19:56.854517+0000","last_active":"2024-09-20T19:20:10.115870+0000","last_peered":"2024-09-20T19:20:10.115870+0000","last_clean":"2024-09-20T19:20:10.115870+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:20:10.115870+0000","last_undegraded":"2024-09-20T19:20:10.115870+0000","last_fullsized":"2024-09-20T19:20:10.115870+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.854461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.854461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.519787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.111420+0000","last_change":"2024-09-20T19:19:50.253935+0000","last_active":"2024-09-20T19:20:10.111420+0000","last_peered":"2024-09-20T19:20:10.111420+0000","last_clean":"2024-09-20T19:20:10.111420+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:20:10.111420+0000","last_undegraded":"2024-09-20T19:20:10.111420+0000","last_fullsized":"2024-09-20T19:20:10.111420+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.253727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.253727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.038762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"77'13","reported_seq":26,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998968+0000","last_change":"2024-09-20T19:19:57.013393+0000","last_active":"2024-09-20T19:20:10.998968+0000","last_peered":"2024-09-20T19:20:10.998968+0000","last_clean":"2024-09-20T19:20:10.998968+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:20:10.998968+0000","last_undegraded":"2024-09-20T19:20:10.998968+0000","last_fullsized":"2024-09-20T19:20:10.998968+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:19:51.410598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.410598+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:12.734213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":208,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:15.055434+0000","last_change":"2024-09-20T19:19:19.441522+0000","last_active":"2024-09-20T19:20:15.055434+0000","last_peered":"2024-09-20T19:20:15.055434+0000","last_clean":"2024-09-20T19:20:15.055434+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:20:15.055434+0000","last_undegraded":"2024-09-20T19:20:15.055434+0000","last_fullsized":"2024-09-20T19:20:15.055434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:19:19.441341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.441341+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:39.849970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":113,"num_read_kb":108,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998672+0000","last_change":"2024-09-20T19:19:49.990251+0000","last_active":"2024-09-20T19:20:10.998672+0000","last_peered":"2024-09-20T19:20:10.998672+0000","last_clean":"2024-09-20T19:20:10.998672+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:20:10.998672+0000","last_undegraded":"2024-09-20T19:20:10.998672+0000","last_fullsized":"2024-09-20T19:20:10.998672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:49.990058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:49.990058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.407192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998934+0000","last_change":"2024-09-20T19:19:25.169431+0000","last_active":"2024-09-20T19:20:10.998934+0000","last_peered":"2024-09-20T19:20:10.998934+0000","last_clean":"2024-09-20T19:20:10.998934+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:20:10.998934+0000","last_undegraded":"2024-09-20T19:20:10.998934+0000","last_fullsized":"2024-09-20T19:20:10.998934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:25.169306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:25.169306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:28.573693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115593+0000","last_change":"2024-09-20T19:19:55.305695+0000","last_active":"2024-09-20T19:20:10.115593+0000","last_peered":"2024-09-20T19:20:10.115593+0000","last_clean":"2024-09-20T19:20:10.115593+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:20:10.115593+0000","last_undegraded":"2024-09-20T19:20:10.115593+0000","last_fullsized":"2024-09-20T19:20:10.115593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:55.305573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.305573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.195185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'12","reported_seq":101,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.117736+0000","last_change":"2024-09-20T19:19:23.739790+0000","last_active":"2024-09-20T19:20:10.117736+0000","last_peered":"2024-09-20T19:20:10.117736+0000","last_clean":"2024-09-20T19:20:10.117736+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:20:10.117736+0000","last_undegraded":"2024-09-20T19:20:10.117736+0000","last_fullsized":"2024-09-20T19:20:10.117736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:23.739633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.739633+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:38.349636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":93,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115115+0000","last_change":"2024-09-20T19:19:19.752346+0000","last_active":"2024-09-20T19:20:10.115115+0000","last_peered":"2024-09-20T19:20:10.115115+0000","last_clean":"2024-09-20T19:20:10.115115+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:20:10.115115+0000","last_undegraded":"2024-09-20T19:20:10.115115+0000","last_fullsized":"2024-09-20T19:20:10.115115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:19:19.752214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.752214+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:22.507215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115171+0000","last_change":"2024-09-20T19:19:44.240584+0000","last_active":"2024-09-20T19:20:10.115171+0000","last_peered":"2024-09-20T19:20:10.115171+0000","last_clean":"2024-09-20T19:20:10.115171+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:20:10.115171+0000","last_undegraded":"2024-09-20T19:20:10.115171+0000","last_fullsized":"2024-09-20T19:20:10.115171+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.240462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.240462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.731548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":77,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257576+0000","last_change":"2024-09-20T19:19:18.140999+0000","last_active":"2024-09-20T19:20:10.257576+0000","last_peered":"2024-09-20T19:20:10.257576+0000","last_clean":"2024-09-20T19:20:10.257576+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:20:10.257576+0000","last_undegraded":"2024-09-20T19:20:10.257576+0000","last_fullsized":"2024-09-20T19:20:10.257576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:18.140785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:18.140785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.388246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442520+0000","last_change":"2024-09-20T19:19:24.381047+0000","last_active":"2024-09-20T19:20:10.442520+0000","last_peered":"2024-09-20T19:20:10.442520+0000","last_clean":"2024-09-20T19:20:10.442520+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:20:10.442520+0000","last_undegraded":"2024-09-20T19:20:10.442520+0000","last_fullsized":"2024-09-20T19:20:10.442520+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:24.380899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:24.380899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:30.202989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"77'16","reported_seq":129,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.704373+0000","last_change":"2024-09-20T19:19:39.901351+0000","last_active":"2024-09-20T19:20:10.704373+0000","last_peered":"2024-09-20T19:20:10.704373+0000","last_clean":"2024-09-20T19:20:10.704373+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:20:10.704373+0000","last_undegraded":"2024-09-20T19:20:10.704373+0000","last_fullsized":"2024-09-20T19:20:10.704373+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:19:39.901171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.901171+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.526242+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":94,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:15.003738+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:20:15.003738+0000","last_peered":"2024-09-20T19:20:15.003738+0000","last_clean":"2024-09-20T19:20:15.003738+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:20:15.003738+0000","last_undegraded":"2024-09-20T19:20:15.003738+0000","last_fullsized":"2024-09-20T19:20:15.003738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.492805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257252+0000","last_change":"2024-09-20T19:19:07.170908+0000","last_active":"2024-09-20T19:20:10.257252+0000","last_peered":"2024-09-20T19:20:10.257252+0000","last_clean":"2024-09-20T19:20:10.257252+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:20:10.257252+0000","last_undegraded":"2024-09-20T19:20:10.257252+0000","last_fullsized":"2024-09-20T19:20:10.257252+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:07.170760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:07.170760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.385612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":134,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:12.275658+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:20:12.275658+0000","last_peered":"2024-09-20T19:20:12.275658+0000","last_clean":"2024-09-20T19:20:12.275658+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:20:12.275658+0000","last_undegraded":"2024-09-20T19:20:12.275658+0000","last_fullsized":"2024-09-20T19:20:12.275658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:27.954958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115847+0000","last_change":"2024-09-20T19:19:26.163243+0000","last_active":"2024-09-20T19:20:10.115847+0000","last_peered":"2024-09-20T19:20:10.115847+0000","last_clean":"2024-09-20T19:20:10.115847+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:20:10.115847+0000","last_undegraded":"2024-09-20T19:20:10.115847+0000","last_fullsized":"2024-09-20T19:20:10.115847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.163172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.163172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.703757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":73,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257650+0000","last_change":"2024-09-20T19:19:17.120513+0000","last_active":"2024-09-20T19:20:10.257650+0000","last_peered":"2024-09-20T19:20:10.257650+0000","last_clean":"2024-09-20T19:20:10.257650+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:20:10.257650+0000","last_undegraded":"2024-09-20T19:20:10.257650+0000","last_fullsized":"2024-09-20T19:20:10.257650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.120471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.120471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.128195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"77'19","reported_seq":112,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.704328+0000","last_change":"2024-09-20T19:19:27.036130+0000","last_active":"2024-09-20T19:20:10.704328+0000","last_peered":"2024-09-20T19:20:10.704328+0000","last_clean":"2024-09-20T19:20:10.704328+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:20:10.704328+0000","last_undegraded":"2024-09-20T19:20:10.704328+0000","last_fullsized":"2024-09-20T19:20:10.704328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.036022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.036022+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.263159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":84,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.999151+0000","last_change":"2024-09-20T19:19:13.243819+0000","last_active":"2024-09-20T19:20:10.999151+0000","last_peered":"2024-09-20T19:20:10.999151+0000","last_clean":"2024-09-20T19:20:10.999151+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:20:10.999151+0000","last_undegraded":"2024-09-20T19:20:10.999151+0000","last_fullsized":"2024-09-20T19:20:10.999151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.243639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.243639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.750283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":76,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998541+0000","last_change":"2024-09-20T19:19:43.031914+0000","last_active":"2024-09-20T19:20:10.998541+0000","last_peered":"2024-09-20T19:20:10.998541+0000","last_clean":"2024-09-20T19:20:10.998541+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:20:10.998541+0000","last_undegraded":"2024-09-20T19:20:10.998541+0000","last_fullsized":"2024-09-20T19:20:10.998541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.031768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.031768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.442373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115152+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:20:10.115152+0000","last_peered":"2024-09-20T19:20:10.115152+0000","last_clean":"2024-09-20T19:20:10.115152+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:20:10.115152+0000","last_undegraded":"2024-09-20T19:20:10.115152+0000","last_fullsized":"2024-09-20T19:20:10.115152+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.126953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"77'9","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.114811+0000","last_change":"2024-09-20T19:19:46.791747+0000","last_active":"2024-09-20T19:20:10.114811+0000","last_peered":"2024-09-20T19:20:10.114811+0000","last_clean":"2024-09-20T19:20:10.114811+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:20:10.114811+0000","last_undegraded":"2024-09-20T19:20:10.114811+0000","last_fullsized":"2024-09-20T19:20:10.114811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.791592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.791592+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.994746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443608+0000","last_change":"2024-09-20T19:19:35.390225+0000","last_active":"2024-09-20T19:20:10.443608+0000","last_peered":"2024-09-20T19:20:10.443608+0000","last_clean":"2024-09-20T19:20:10.443608+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:20:10.443608+0000","last_undegraded":"2024-09-20T19:20:10.443608+0000","last_fullsized":"2024-09-20T19:20:10.443608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.390064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.390064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.228348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":76,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998459+0000","last_change":"2024-09-20T19:19:41.044575+0000","last_active":"2024-09-20T19:20:10.998459+0000","last_peered":"2024-09-20T19:20:10.998459+0000","last_clean":"2024-09-20T19:20:10.998459+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:20:10.998459+0000","last_undegraded":"2024-09-20T19:20:10.998459+0000","last_fullsized":"2024-09-20T19:20:10.998459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.044427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.044427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.962646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.114618+0000","last_change":"2024-09-20T19:19:52.810445+0000","last_active":"2024-09-20T19:20:10.114618+0000","last_peered":"2024-09-20T19:20:10.114618+0000","last_clean":"2024-09-20T19:20:10.114618+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:20:10.114618+0000","last_undegraded":"2024-09-20T19:20:10.114618+0000","last_fullsized":"2024-09-20T19:20:10.114618+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:19:52.810385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:52.810385+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.122928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"77'15","reported_seq":106,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256598+0000","last_change":"2024-09-20T19:19:44.046132+0000","last_active":"2024-09-20T19:20:10.256598+0000","last_peered":"2024-09-20T19:20:10.256598+0000","last_clean":"2024-09-20T19:20:10.256598+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:20:10.256598+0000","last_undegraded":"2024-09-20T19:20:10.256598+0000","last_fullsized":"2024-09-20T19:20:10.256598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:44.045925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.045925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.259093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":92,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.999140+0000","last_change":"2024-09-20T19:19:21.207668+0000","last_active":"2024-09-20T19:20:10.999140+0000","last_peered":"2024-09-20T19:20:10.999140+0000","last_clean":"2024-09-20T19:20:10.999140+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:20:10.999140+0000","last_undegraded":"2024-09-20T19:20:10.999140+0000","last_fullsized":"2024-09-20T19:20:10.999140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:21.207433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:21.207433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.475369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":114,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115098+0000","last_change":"2024-09-20T19:19:17.113356+0000","last_active":"2024-09-20T19:20:10.115098+0000","last_peered":"2024-09-20T19:20:10.115098+0000","last_clean":"2024-09-20T19:20:10.115098+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:20:10.115098+0000","last_undegraded":"2024-09-20T19:20:10.115098+0000","last_fullsized":"2024-09-20T19:20:10.115098+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:17.113285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.113285+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:18.739733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":73,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256951+0000","last_change":"2024-09-20T19:19:36.943615+0000","last_active":"2024-09-20T19:20:10.256951+0000","last_peered":"2024-09-20T19:20:10.256951+0000","last_clean":"2024-09-20T19:20:10.256951+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:20:10.256951+0000","last_undegraded":"2024-09-20T19:20:10.256951+0000","last_fullsized":"2024-09-20T19:20:10.256951+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.943527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.943527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.968462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"77'12","reported_seq":132,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:16.852586+0000","last_change":"2024-09-20T19:20:16.852586+0000","last_active":"2024-09-20T19:20:16.852586+0000","last_peered":"2024-09-20T19:20:16.852586+0000","last_clean":"2024-09-20T19:20:16.852586+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:20:16.852586+0000","last_undegraded":"2024-09-20T19:20:16.852586+0000","last_fullsized":"2024-09-20T19:20:16.852586+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:16.852450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.852450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.371990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":102,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.006403+0000","last_change":"2024-09-20T19:19:33.263120+0000","last_active":"2024-09-20T19:20:11.006403+0000","last_peered":"2024-09-20T19:20:11.006403+0000","last_clean":"2024-09-20T19:20:11.006403+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:20:11.006403+0000","last_undegraded":"2024-09-20T19:20:11.006403+0000","last_fullsized":"2024-09-20T19:20:11.006403+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:19:33.262915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.262915+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.795568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":76,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998390+0000","last_change":"2024-09-20T19:19:51.013009+0000","last_active":"2024-09-20T19:20:10.998390+0000","last_peered":"2024-09-20T19:20:10.998390+0000","last_clean":"2024-09-20T19:20:10.998390+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:20:10.998390+0000","last_undegraded":"2024-09-20T19:20:10.998390+0000","last_fullsized":"2024-09-20T19:20:10.998390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.012906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.012906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.429741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":66,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113043+0000","last_change":"2024-09-20T19:18:13.370848+0000","last_active":"2024-09-20T19:20:10.113043+0000","last_peered":"2024-09-20T19:20:10.113043+0000","last_clean":"2024-09-20T19:20:10.113043+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:20:10.113043+0000","last_undegraded":"2024-09-20T19:20:10.113043+0000","last_fullsized":"2024-09-20T19:20:10.113043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:18:12.302527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.117369+0000","last_change":"2024-09-20T19:19:24.694785+0000","last_active":"2024-09-20T19:20:10.117369+0000","last_peered":"2024-09-20T19:20:10.117369+0000","last_clean":"2024-09-20T19:20:10.117369+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:20:10.117369+0000","last_undegraded":"2024-09-20T19:20:10.117369+0000","last_fullsized":"2024-09-20T19:20:10.117369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:24.694641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:24.694641+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.496424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":84,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998572+0000","last_change":"2024-09-20T19:19:28.174295+0000","last_active":"2024-09-20T19:20:10.998572+0000","last_peered":"2024-09-20T19:20:10.998572+0000","last_clean":"2024-09-20T19:20:10.998572+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:20:10.998572+0000","last_undegraded":"2024-09-20T19:20:10.998572+0000","last_fullsized":"2024-09-20T19:20:10.998572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.174157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.174157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.549082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112976+0000","last_change":"2024-09-20T19:19:57.060943+0000","last_active":"2024-09-20T19:20:10.112976+0000","last_peered":"2024-09-20T19:20:10.112976+0000","last_clean":"2024-09-20T19:20:10.112976+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:20:10.112976+0000","last_undegraded":"2024-09-20T19:20:10.112976+0000","last_fullsized":"2024-09-20T19:20:10.112976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.060887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.060887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.058027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":72,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998627+0000","last_change":"2024-09-20T19:19:54.030064+0000","last_active":"2024-09-20T19:20:10.998627+0000","last_peered":"2024-09-20T19:20:10.998627+0000","last_clean":"2024-09-20T19:20:10.998627+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:20:10.998627+0000","last_undegraded":"2024-09-20T19:20:10.998627+0000","last_fullsized":"2024-09-20T19:20:10.998627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.029937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.029937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.274260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.705034+0000","last_change":"2024-09-20T19:19:13.019741+0000","last_active":"2024-09-20T19:20:10.705034+0000","last_peered":"2024-09-20T19:20:10.705034+0000","last_clean":"2024-09-20T19:20:10.705034+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:20:10.705034+0000","last_undegraded":"2024-09-20T19:20:10.705034+0000","last_fullsized":"2024-09-20T19:20:10.705034+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.019586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.019586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.203383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":91,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113405+0000","last_change":"2024-09-20T19:19:29.203589+0000","last_active":"2024-09-20T19:20:10.113405+0000","last_peered":"2024-09-20T19:20:10.113405+0000","last_clean":"2024-09-20T19:20:10.113405+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:20:10.113405+0000","last_undegraded":"2024-09-20T19:20:10.113405+0000","last_fullsized":"2024-09-20T19:20:10.113405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:19:29.203410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:29.203410+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.144800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":77,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257149+0000","last_change":"2024-09-20T19:19:43.040517+0000","last_active":"2024-09-20T19:20:10.257149+0000","last_peered":"2024-09-20T19:20:10.257149+0000","last_clean":"2024-09-20T19:20:10.257149+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:20:10.257149+0000","last_undegraded":"2024-09-20T19:20:10.257149+0000","last_fullsized":"2024-09-20T19:20:10.257149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.040460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.040460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:10.254103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115586+0000","last_change":"2024-09-20T19:19:56.343379+0000","last_active":"2024-09-20T19:20:10.115586+0000","last_peered":"2024-09-20T19:20:10.115586+0000","last_clean":"2024-09-20T19:20:10.115586+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:20:10.115586+0000","last_undegraded":"2024-09-20T19:20:10.115586+0000","last_fullsized":"2024-09-20T19:20:10.115586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.343254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.343254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.336373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":97,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112938+0000","last_change":"2024-09-20T19:19:13.215224+0000","last_active":"2024-09-20T19:20:10.112938+0000","last_peered":"2024-09-20T19:20:10.112938+0000","last_clean":"2024-09-20T19:20:10.112938+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:20:10.112938+0000","last_undegraded":"2024-09-20T19:20:10.112938+0000","last_fullsized":"2024-09-20T19:20:10.112938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:13.215055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.215055+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:18.941299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257357+0000","last_change":"2024-09-20T19:19:28.086132+0000","last_active":"2024-09-20T19:20:10.257357+0000","last_peered":"2024-09-20T19:20:10.257357+0000","last_clean":"2024-09-20T19:20:10.257357+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:20:10.257357+0000","last_undegraded":"2024-09-20T19:20:10.257357+0000","last_fullsized":"2024-09-20T19:20:10.257357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.086004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.086004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.650446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":113,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257504+0000","last_change":"2024-09-20T19:19:27.113335+0000","last_active":"2024-09-20T19:20:10.257504+0000","last_peered":"2024-09-20T19:20:10.257504+0000","last_clean":"2024-09-20T19:20:10.257504+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:20:10.257504+0000","last_undegraded":"2024-09-20T19:20:10.257504+0000","last_fullsized":"2024-09-20T19:20:10.257504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.113211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.113211+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.889024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.116457+0000","last_change":"2024-09-20T19:19:40.871513+0000","last_active":"2024-09-20T19:20:10.116457+0000","last_peered":"2024-09-20T19:20:10.116457+0000","last_clean":"2024-09-20T19:20:10.116457+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:20:10.116457+0000","last_undegraded":"2024-09-20T19:20:10.116457+0000","last_fullsized":"2024-09-20T19:20:10.116457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.871447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.871447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.175662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":97,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256641+0000","last_change":"2024-09-20T19:19:46.036288+0000","last_active":"2024-09-20T19:20:10.256641+0000","last_peered":"2024-09-20T19:20:10.256641+0000","last_clean":"2024-09-20T19:20:10.256641+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:20:10.256641+0000","last_undegraded":"2024-09-20T19:20:10.256641+0000","last_fullsized":"2024-09-20T19:20:10.256641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.036143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.036143+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.480915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115183+0000","last_change":"2024-09-20T19:19:50.860419+0000","last_active":"2024-09-20T19:20:10.115183+0000","last_peered":"2024-09-20T19:20:10.115183+0000","last_clean":"2024-09-20T19:20:10.115183+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:20:10.115183+0000","last_undegraded":"2024-09-20T19:20:10.115183+0000","last_fullsized":"2024-09-20T19:20:10.115183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.860284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.860284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.470934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":106,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112616+0000","last_change":"2024-09-20T19:18:11.330783+0000","last_active":"2024-09-20T19:20:10.112616+0000","last_peered":"2024-09-20T19:20:10.112616+0000","last_clean":"2024-09-20T19:20:10.112616+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:20:10.112616+0000","last_undegraded":"2024-09-20T19:20:10.112616+0000","last_fullsized":"2024-09-20T19:20:10.112616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:18:10.268062+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005889+0000","last_change":"2024-09-20T19:19:16.241092+0000","last_active":"2024-09-20T19:20:11.005889+0000","last_peered":"2024-09-20T19:20:11.005889+0000","last_clean":"2024-09-20T19:20:11.005889+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:20:11.005889+0000","last_undegraded":"2024-09-20T19:20:11.005889+0000","last_fullsized":"2024-09-20T19:20:11.005889+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:16.240975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:16.240975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.618347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442303+0000","last_change":"2024-09-20T19:19:09.454582+0000","last_active":"2024-09-20T19:20:10.442303+0000","last_peered":"2024-09-20T19:20:10.442303+0000","last_clean":"2024-09-20T19:20:10.442303+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:20:10.442303+0000","last_undegraded":"2024-09-20T19:20:10.442303+0000","last_fullsized":"2024-09-20T19:20:10.442303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:09.454436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:09.454436+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:18.594148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.705686+0000","last_change":"2024-09-20T19:19:19.061400+0000","last_active":"2024-09-20T19:20:10.705686+0000","last_peered":"2024-09-20T19:20:10.705686+0000","last_clean":"2024-09-20T19:20:10.705686+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:20:10.705686+0000","last_undegraded":"2024-09-20T19:20:10.705686+0000","last_fullsized":"2024-09-20T19:20:10.705686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:19.061323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.061323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:24.520151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113286+0000","last_change":"2024-09-20T19:19:47.212665+0000","last_active":"2024-09-20T19:20:10.113286+0000","last_peered":"2024-09-20T19:20:10.113286+0000","last_clean":"2024-09-20T19:20:10.113286+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:20:10.113286+0000","last_undegraded":"2024-09-20T19:20:10.113286+0000","last_fullsized":"2024-09-20T19:20:10.113286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:47.212542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.212542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.674953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112043+0000","last_change":"2024-09-20T19:19:57.895992+0000","last_active":"2024-09-20T19:20:10.112043+0000","last_peered":"2024-09-20T19:20:10.112043+0000","last_clean":"2024-09-20T19:20:10.112043+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:20:10.112043+0000","last_undegraded":"2024-09-20T19:20:10.112043+0000","last_fullsized":"2024-09-20T19:20:10.112043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.895923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.895923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.375254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":101,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004627+0000","last_change":"2024-09-20T19:19:27.179523+0000","last_active":"2024-09-20T19:20:11.004627+0000","last_peered":"2024-09-20T19:20:11.004627+0000","last_clean":"2024-09-20T19:20:11.004627+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:20:11.004627+0000","last_undegraded":"2024-09-20T19:20:11.004627+0000","last_fullsized":"2024-09-20T19:20:11.004627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.179398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.179398+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.909162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":94,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443487+0000","last_change":"2024-09-20T19:19:10.473684+0000","last_active":"2024-09-20T19:20:10.443487+0000","last_peered":"2024-09-20T19:20:10.443487+0000","last_clean":"2024-09-20T19:20:10.443487+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:20:10.443487+0000","last_undegraded":"2024-09-20T19:20:10.443487+0000","last_fullsized":"2024-09-20T19:20:10.443487+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:10.473507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:10.473507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:16.737439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.441967+0000","last_change":"2024-09-20T19:19:48.434045+0000","last_active":"2024-09-20T19:20:10.441967+0000","last_peered":"2024-09-20T19:20:10.441967+0000","last_clean":"2024-09-20T19:20:10.441967+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:20:10.441967+0000","last_undegraded":"2024-09-20T19:20:10.441967+0000","last_fullsized":"2024-09-20T19:20:10.441967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.433915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.433915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.655605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112263+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:20:10.112263+0000","last_peered":"2024-09-20T19:20:10.112263+0000","last_clean":"2024-09-20T19:20:10.112263+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:20:10.112263+0000","last_undegraded":"2024-09-20T19:20:10.112263+0000","last_fullsized":"2024-09-20T19:20:10.112263+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.984866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":88,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.110571+0000","last_change":"2024-09-20T19:19:33.242802+0000","last_active":"2024-09-20T19:20:10.110571+0000","last_peered":"2024-09-20T19:20:10.110571+0000","last_clean":"2024-09-20T19:20:10.110571+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:20:10.110571+0000","last_undegraded":"2024-09-20T19:20:10.110571+0000","last_fullsized":"2024-09-20T19:20:10.110571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:19:33.242651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.242651+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.383781+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.257430+0000","last_change":"2024-09-20T19:19:35.981274+0000","last_active":"2024-09-20T19:20:10.257430+0000","last_peered":"2024-09-20T19:20:10.257430+0000","last_clean":"2024-09-20T19:20:10.257430+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:20:10.257430+0000","last_undegraded":"2024-09-20T19:20:10.257430+0000","last_fullsized":"2024-09-20T19:20:10.257430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.981166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.981166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.523226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":77,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.704813+0000","last_change":"2024-09-20T19:19:29.014438+0000","last_active":"2024-09-20T19:20:10.704813+0000","last_peered":"2024-09-20T19:20:10.704813+0000","last_clean":"2024-09-20T19:20:10.704813+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:20:10.704813+0000","last_undegraded":"2024-09-20T19:20:10.704813+0000","last_fullsized":"2024-09-20T19:20:10.704813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:29.014364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:29.014364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.312931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115055+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:20:10.115055+0000","last_peered":"2024-09-20T19:20:10.115055+0000","last_clean":"2024-09-20T19:20:10.115055+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:20:10.115055+0000","last_undegraded":"2024-09-20T19:20:10.115055+0000","last_fullsized":"2024-09-20T19:20:10.115055+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.905322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":101,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112984+0000","last_change":"2024-09-20T19:19:28.173179+0000","last_active":"2024-09-20T19:20:10.112984+0000","last_peered":"2024-09-20T19:20:10.112984+0000","last_clean":"2024-09-20T19:20:10.112984+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:20:10.112984+0000","last_undegraded":"2024-09-20T19:20:10.112984+0000","last_fullsized":"2024-09-20T19:20:10.112984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:28.172944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.172944+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.576524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.706365+0000","last_change":"2024-09-20T19:19:11.981872+0000","last_active":"2024-09-20T19:20:10.706365+0000","last_peered":"2024-09-20T19:20:10.706365+0000","last_clean":"2024-09-20T19:20:10.706365+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:20:10.706365+0000","last_undegraded":"2024-09-20T19:20:10.706365+0000","last_fullsized":"2024-09-20T19:20:10.706365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:11.981739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:11.981739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:29.367984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.114827+0000","last_change":"2024-09-20T19:19:50.321638+0000","last_active":"2024-09-20T19:20:10.114827+0000","last_peered":"2024-09-20T19:20:10.114827+0000","last_clean":"2024-09-20T19:20:10.114827+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:20:10.114827+0000","last_undegraded":"2024-09-20T19:20:10.114827+0000","last_fullsized":"2024-09-20T19:20:10.114827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.321520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.321520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.084680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112787+0000","last_change":"2024-09-20T19:19:21.180046+0000","last_active":"2024-09-20T19:20:10.112787+0000","last_peered":"2024-09-20T19:20:10.112787+0000","last_clean":"2024-09-20T19:20:10.112787+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:20:10.112787+0000","last_undegraded":"2024-09-20T19:20:10.112787+0000","last_fullsized":"2024-09-20T19:20:10.112787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:21.179896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:21.179896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.033462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":76,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.115721+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:20:10.115721+0000","last_peered":"2024-09-20T19:20:10.115721+0000","last_clean":"2024-09-20T19:20:10.115721+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:20:10.115721+0000","last_undegraded":"2024-09-20T19:20:10.115721+0000","last_fullsized":"2024-09-20T19:20:10.115721+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.371707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.256749+0000","last_change":"2024-09-20T19:19:23.109142+0000","last_active":"2024-09-20T19:20:10.256749+0000","last_peered":"2024-09-20T19:20:10.256749+0000","last_clean":"2024-09-20T19:20:10.256749+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:20:10.256749+0000","last_undegraded":"2024-09-20T19:20:10.256749+0000","last_fullsized":"2024-09-20T19:20:10.256749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:23.109064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.109064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:42.414841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":98,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.117227+0000","last_change":"2024-09-20T19:19:55.848311+0000","last_active":"2024-09-20T19:20:10.117227+0000","last_peered":"2024-09-20T19:20:10.117227+0000","last_clean":"2024-09-20T19:20:10.117227+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:20:10.117227+0000","last_undegraded":"2024-09-20T19:20:10.117227+0000","last_fullsized":"2024-09-20T19:20:10.117227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:55.848164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.848164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:19.416599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":117,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:12.432707+0000","last_change":"2024-09-20T19:19:35.001831+0000","last_active":"2024-09-20T19:20:12.432707+0000","last_peered":"2024-09-20T19:20:12.432707+0000","last_clean":"2024-09-20T19:20:12.432707+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:20:12.432707+0000","last_undegraded":"2024-09-20T19:20:12.432707+0000","last_fullsized":"2024-09-20T19:20:12.432707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:35.001666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.001666+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.268895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.442688+0000","last_change":"2024-09-20T19:19:31.466386+0000","last_active":"2024-09-20T19:20:10.442688+0000","last_peered":"2024-09-20T19:20:10.442688+0000","last_clean":"2024-09-20T19:20:10.442688+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:20:10.442688+0000","last_undegraded":"2024-09-20T19:20:10.442688+0000","last_fullsized":"2024-09-20T19:20:10.442688+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.466247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.466247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.461700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113595+0000","last_change":"2024-09-20T19:19:16.214996+0000","last_active":"2024-09-20T19:20:10.113595+0000","last_peered":"2024-09-20T19:20:10.113595+0000","last_clean":"2024-09-20T19:20:10.113595+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:20:10.113595+0000","last_undegraded":"2024-09-20T19:20:10.113595+0000","last_fullsized":"2024-09-20T19:20:10.113595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:16.214940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:16.214940+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:41.495655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":120,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.706301+0000","last_change":"2024-09-20T19:19:57.028481+0000","last_active":"2024-09-20T19:20:10.706301+0000","last_peered":"2024-09-20T19:20:10.706301+0000","last_clean":"2024-09-20T19:20:10.706301+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:20:10.706301+0000","last_undegraded":"2024-09-20T19:20:10.706301+0000","last_fullsized":"2024-09-20T19:20:10.706301+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:40.905378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.905378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.869949+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.116964+0000","last_change":"2024-09-20T19:19:18.741633+0000","last_active":"2024-09-20T19:20:10.116964+0000","last_peered":"2024-09-20T19:20:10.116964+0000","last_clean":"2024-09-20T19:20:10.116964+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:20:10.116964+0000","last_undegraded":"2024-09-20T19:20:10.116964+0000","last_fullsized":"2024-09-20T19:20:10.116964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:18.741494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:18.741494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:21.585736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.005392+0000","last_change":"2024-09-20T19:19:39.226277+0000","last_active":"2024-09-20T19:20:11.005392+0000","last_peered":"2024-09-20T19:20:11.005392+0000","last_clean":"2024-09-20T19:20:11.005392+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:20:11.005392+0000","last_undegraded":"2024-09-20T19:20:11.005392+0000","last_fullsized":"2024-09-20T19:20:11.005392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:39.226160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.226160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.931829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.705471+0000","last_change":"2024-09-20T19:19:33.036193+0000","last_active":"2024-09-20T19:20:10.705471+0000","last_peered":"2024-09-20T19:20:10.705471+0000","last_clean":"2024-09-20T19:20:10.705471+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:20:10.705471+0000","last_undegraded":"2024-09-20T19:20:10.705471+0000","last_fullsized":"2024-09-20T19:20:10.705471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.036056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.036056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.896627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.998835+0000","last_change":"2024-09-20T19:19:47.048010+0000","last_active":"2024-09-20T19:20:10.998835+0000","last_peered":"2024-09-20T19:20:10.998835+0000","last_clean":"2024-09-20T19:20:10.998835+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:20:10.998835+0000","last_undegraded":"2024-09-20T19:20:10.998835+0000","last_fullsized":"2024-09-20T19:20:10.998835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:47.047816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.047816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.958306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.443165+0000","last_change":"2024-09-20T19:19:25.409927+0000","last_active":"2024-09-20T19:20:10.443165+0000","last_peered":"2024-09-20T19:20:10.443165+0000","last_clean":"2024-09-20T19:20:10.443165+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:20:10.443165+0000","last_undegraded":"2024-09-20T19:20:10.443165+0000","last_fullsized":"2024-09-20T19:20:10.443165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:25.409801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:25.409801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:27.236371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.112829+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:20:10.112829+0000","last_peered":"2024-09-20T19:20:10.112829+0000","last_clean":"2024-09-20T19:20:10.112829+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:20:10.112829+0000","last_undegraded":"2024-09-20T19:20:10.112829+0000","last_fullsized":"2024-09-20T19:20:10.112829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:20.115239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.113702+0000","last_change":"2024-09-20T19:19:58.028896+0000","last_active":"2024-09-20T19:20:10.113702+0000","last_peered":"2024-09-20T19:20:10.113702+0000","last_clean":"2024-09-20T19:20:10.113702+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:20:10.113702+0000","last_undegraded":"2024-09-20T19:20:10.113702+0000","last_fullsized":"2024-09-20T19:20:10.113702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:58.028776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:58.028776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.888058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:10.111237+0000","last_change":"2024-09-20T19:19:51.256022+0000","last_active":"2024-09-20T19:20:10.111237+0000","last_peered":"2024-09-20T19:20:10.111237+0000","last_clean":"2024-09-20T19:20:10.111237+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:20:10.111237+0000","last_undegraded":"2024-09-20T19:20:10.111237+0000","last_fullsized":"2024-09-20T19:20:10.111237+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:51.255877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.255877+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.975626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-20T19:20:11.004880+0000","last_change":"2024-09-20T19:19:28.186599+0000","last_active":"2024-09-20T19:20:11.004880+0000","last_peered":"2024-09-20T19:20:11.004880+0000","last_clean":"2024-09-20T19:20:11.004880+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:20:11.004880+0000","last_undegraded":"2024-09-20T19:20:11.004880+0000","last_fullsized":"2024-09-20T19:20:11.004880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.186507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.186507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.594221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":108,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877906979,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1286144,"data_stored":910629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234030,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716884,"statfs":{"total":95995035648,"available":95966089216,"internally_reserved":0,"allocated":1273856,"data_stored":909965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561080,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":320688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855428,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":692224,"data_stored":319675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084366,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":712704,"data_stored":320462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378716,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36024,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709128,"statfs":{"total":95995035648,"available":95958147072,"internally_reserved":0,"allocated":692224,"data_stored":320542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":704512,"data_stored":321169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36608,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708544,"statfs":{"total":95995035648,"available":95957549056,"internally_reserved":0,"allocated":1290240,"data_stored":910895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:20:22.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:20:22.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[32075]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:22.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:22 smithi042 bash[22706]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: cluster 2024-09-20T19:20:20.961194+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: cluster 2024-09-20T19:20:20.964250+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:22 smithi162 bash[24557]: audit 2024-09-20T19:20:21.513663+0000 mgr.y (mgr.24491) 91 : audit [DBG] from='client.14802 v1:172.21.15.42:0/4111573620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:23 smithi162 bash[24557]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:23 smithi162 bash[24557]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:23.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:23 smithi042 bash[22706]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:23 smithi042 bash[22706]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:23 smithi042 bash[32075]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:23 smithi042 bash[32075]: cluster 2024-09-20T19:20:21.913390+0000 mgr.y (mgr.24491) 92 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:25 smithi162 bash[24557]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:25 smithi162 bash[24557]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:25 smithi042 bash[32075]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:25 smithi042 bash[32075]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:25 smithi042 bash[22706]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:25 smithi042 bash[22706]: cluster 2024-09-20T19:20:23.914116+0000 mgr.y (mgr.24491) 93 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:26.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:26 smithi162 bash[24557]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:26 smithi162 bash[24557]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:26 smithi162 bash[24557]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:26.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:26 smithi162 bash[24557]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:26.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[32075]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[32075]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[32075]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[32075]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[22706]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[22706]: cluster 2024-09-20T19:20:24.875107+0000 osd.0 (osd.0) 49 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[22706]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:26.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:26 smithi042 bash[22706]: cluster 2024-09-20T19:20:24.877385+0000 osd.0 (osd.0) 50 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:20:27.199 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:27 smithi162 bash[24557]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[22706]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.225481+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.228311+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.850018+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.853414+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:20:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: cluster 2024-09-20T19:20:25.914798+0000 mgr.y (mgr.24491) 94 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:27 smithi042 bash[32075]: audit 2024-09-20T19:20:26.556259+0000 mon.a (mon.0) 955 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:28.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:28 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:28 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:28 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.002427+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.b scrub starts 2024-09-20T19:20:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:28 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:28 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.004772+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.b scrub ok 2024-09-20T19:20:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:20:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:20:29.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:28 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:29 smithi162 bash[24557]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[22706]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:27.916261+0000 mgr.y (mgr.24491) 95 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.697335+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:29 smithi042 bash[32075]: cluster 2024-09-20T19:20:28.699907+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:20:29.589 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-20T19:20:27.915099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":986,"num_read_kb":843,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244372,"kb_used_data":7356,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716844,"statfs":{"total":767960285184,"available":767710048256,"internally_reserved":0,"allocated":7532544,"data_stored":4438633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.766492"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726340+0000","last_change":"2024-09-20T19:19:26.193045+0000","last_active":"2024-09-20T19:20:19.726340+0000","last_peered":"2024-09-20T19:20:19.726340+0000","last_clean":"2024-09-20T19:20:19.726340+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:20:19.726340+0000","last_undegraded":"2024-09-20T19:20:19.726340+0000","last_fullsized":"2024-09-20T19:20:19.726340+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.192962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.192962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.220702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729809+0000","last_change":"2024-09-20T19:19:17.698496+0000","last_active":"2024-09-20T19:20:19.729809+0000","last_peered":"2024-09-20T19:20:19.729809+0000","last_clean":"2024-09-20T19:20:19.729809+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:20:19.729809+0000","last_undegraded":"2024-09-20T19:20:19.729809+0000","last_fullsized":"2024-09-20T19:20:19.729809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.698427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.698427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.779691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":111,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725798+0000","last_change":"2024-09-20T19:19:45.244647+0000","last_active":"2024-09-20T19:20:19.725798+0000","last_peered":"2024-09-20T19:20:19.725798+0000","last_clean":"2024-09-20T19:20:19.725798+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:20:19.725798+0000","last_undegraded":"2024-09-20T19:20:19.725798+0000","last_fullsized":"2024-09-20T19:20:19.725798+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:45.244486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.244486+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.299507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.158785+0000","last_change":"2024-09-20T19:19:13.217834+0000","last_active":"2024-09-20T19:20:19.158785+0000","last_peered":"2024-09-20T19:20:19.158785+0000","last_clean":"2024-09-20T19:20:19.158785+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:20:19.158785+0000","last_undegraded":"2024-09-20T19:20:19.158785+0000","last_fullsized":"2024-09-20T19:20:19.158785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.217715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.217715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:32.923722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729842+0000","last_change":"2024-09-20T19:19:11.241620+0000","last_active":"2024-09-20T19:20:19.729842+0000","last_peered":"2024-09-20T19:20:19.729842+0000","last_clean":"2024-09-20T19:20:19.729842+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:20:19.729842+0000","last_undegraded":"2024-09-20T19:20:19.729842+0000","last_fullsized":"2024-09-20T19:20:19.729842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:11.241540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:11.241540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.158895+0000","last_change":"2024-09-20T19:19:12.206330+0000","last_active":"2024-09-20T19:20:19.158895+0000","last_peered":"2024-09-20T19:20:19.158895+0000","last_clean":"2024-09-20T19:20:19.158895+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:20:19.158895+0000","last_undegraded":"2024-09-20T19:20:19.158895+0000","last_fullsized":"2024-09-20T19:20:19.158895+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:12.206186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:12.206186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.210390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154350+0000","last_change":"2024-09-20T19:19:30.514806+0000","last_active":"2024-09-20T19:20:19.154350+0000","last_peered":"2024-09-20T19:20:19.154350+0000","last_clean":"2024-09-20T19:20:19.154350+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:20:19.154350+0000","last_undegraded":"2024-09-20T19:20:19.154350+0000","last_fullsized":"2024-09-20T19:20:19.154350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:30.514674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:30.514674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:35.302233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159268+0000","last_change":"2024-09-20T19:19:32.223697+0000","last_active":"2024-09-20T19:20:19.159268+0000","last_peered":"2024-09-20T19:20:19.159268+0000","last_clean":"2024-09-20T19:20:19.159268+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:20:19.159268+0000","last_undegraded":"2024-09-20T19:20:19.159268+0000","last_fullsized":"2024-09-20T19:20:19.159268+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.223538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.223538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.453072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.732198+0000","last_change":"2024-09-20T19:19:59.335157+0000","last_active":"2024-09-20T19:20:19.732198+0000","last_peered":"2024-09-20T19:20:19.732198+0000","last_clean":"2024-09-20T19:20:19.732198+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:20:19.732198+0000","last_undegraded":"2024-09-20T19:20:19.732198+0000","last_fullsized":"2024-09-20T19:20:19.732198+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.335040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.335040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.153396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159217+0000","last_change":"2024-09-20T19:19:15.268253+0000","last_active":"2024-09-20T19:20:19.159217+0000","last_peered":"2024-09-20T19:20:19.159217+0000","last_clean":"2024-09-20T19:20:19.159217+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:20:19.159217+0000","last_undegraded":"2024-09-20T19:20:19.159217+0000","last_fullsized":"2024-09-20T19:20:19.159217+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:19:15.268070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:15.268070+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.887074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729680+0000","last_change":"2024-09-20T19:19:43.205222+0000","last_active":"2024-09-20T19:20:19.729680+0000","last_peered":"2024-09-20T19:20:19.729680+0000","last_clean":"2024-09-20T19:20:19.729680+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:20:19.729680+0000","last_undegraded":"2024-09-20T19:20:19.729680+0000","last_fullsized":"2024-09-20T19:20:19.729680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.205146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.205146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.365222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":77,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313296+0000","last_change":"2024-09-20T19:19:32.084443+0000","last_active":"2024-09-20T19:20:19.313296+0000","last_peered":"2024-09-20T19:20:19.313296+0000","last_clean":"2024-09-20T19:20:19.313296+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:20:19.313296+0000","last_undegraded":"2024-09-20T19:20:19.313296+0000","last_fullsized":"2024-09-20T19:20:19.313296+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.084319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.084319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.681633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731836+0000","last_change":"2024-09-20T19:19:41.142126+0000","last_active":"2024-09-20T19:20:19.731836+0000","last_peered":"2024-09-20T19:20:19.731836+0000","last_clean":"2024-09-20T19:20:19.731836+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:20:19.731836+0000","last_undegraded":"2024-09-20T19:20:19.731836+0000","last_fullsized":"2024-09-20T19:20:19.731836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.141975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.141975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.007349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159557+0000","last_change":"2024-09-20T19:19:42.188043+0000","last_active":"2024-09-20T19:20:19.159557+0000","last_peered":"2024-09-20T19:20:19.159557+0000","last_clean":"2024-09-20T19:20:19.159557+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:20:19.159557+0000","last_undegraded":"2024-09-20T19:20:19.159557+0000","last_fullsized":"2024-09-20T19:20:19.159557+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:42.187756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:42.187756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.911434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313487+0000","last_change":"2024-09-20T19:19:40.029794+0000","last_active":"2024-09-20T19:20:19.313487+0000","last_peered":"2024-09-20T19:20:19.313487+0000","last_clean":"2024-09-20T19:20:19.313487+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:20:19.313487+0000","last_undegraded":"2024-09-20T19:20:19.313487+0000","last_fullsized":"2024-09-20T19:20:19.313487+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.029674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.029674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.995856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725053+0000","last_change":"2024-09-20T19:19:54.798943+0000","last_active":"2024-09-20T19:20:19.725053+0000","last_peered":"2024-09-20T19:20:19.725053+0000","last_clean":"2024-09-20T19:20:19.725053+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:20:19.725053+0000","last_undegraded":"2024-09-20T19:20:19.725053+0000","last_fullsized":"2024-09-20T19:20:19.725053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.798882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.798882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.457870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"77'17","reported_seq":115,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160627+0000","last_change":"2024-09-20T19:19:48.026945+0000","last_active":"2024-09-20T19:20:19.160627+0000","last_peered":"2024-09-20T19:20:19.160627+0000","last_clean":"2024-09-20T19:20:19.160627+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:20:19.160627+0000","last_undegraded":"2024-09-20T19:20:19.160627+0000","last_fullsized":"2024-09-20T19:20:19.160627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:19:48.026795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.026795+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.794994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731857+0000","last_change":"2024-09-20T19:20:05.282366+0000","last_active":"2024-09-20T19:20:19.731857+0000","last_peered":"2024-09-20T19:20:19.731857+0000","last_clean":"2024-09-20T19:20:19.731857+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:20:19.731857+0000","last_undegraded":"2024-09-20T19:20:19.731857+0000","last_fullsized":"2024-09-20T19:20:19.731857+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:05.282293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:05.282293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.281996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.157952+0000","last_change":"2024-09-20T19:19:56.425232+0000","last_active":"2024-09-20T19:20:19.157952+0000","last_peered":"2024-09-20T19:20:19.157952+0000","last_clean":"2024-09-20T19:20:19.157952+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:20:19.157952+0000","last_undegraded":"2024-09-20T19:20:19.157952+0000","last_fullsized":"2024-09-20T19:20:19.157952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.425156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.425156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:21.486886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.158008+0000","last_change":"2024-09-20T19:19:46.074958+0000","last_active":"2024-09-20T19:20:19.158008+0000","last_peered":"2024-09-20T19:20:19.158008+0000","last_clean":"2024-09-20T19:20:19.158008+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:20:19.158008+0000","last_undegraded":"2024-09-20T19:20:19.158008+0000","last_fullsized":"2024-09-20T19:20:19.158008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:46.074887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.074887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.317322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"77'10","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160085+0000","last_change":"2024-09-20T19:19:43.146526+0000","last_active":"2024-09-20T19:20:19.160085+0000","last_peered":"2024-09-20T19:20:19.160085+0000","last_clean":"2024-09-20T19:20:19.160085+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:20:19.160085+0000","last_undegraded":"2024-09-20T19:20:19.160085+0000","last_fullsized":"2024-09-20T19:20:19.160085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:43.146459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.146459+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.154934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.155383+0000","last_change":"2024-09-20T19:19:54.442006+0000","last_active":"2024-09-20T19:20:19.155383+0000","last_peered":"2024-09-20T19:20:19.155383+0000","last_clean":"2024-09-20T19:20:19.155383+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:20:19.155383+0000","last_undegraded":"2024-09-20T19:20:19.155383+0000","last_fullsized":"2024-09-20T19:20:19.155383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.441955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.441955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.241390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":122,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:22.434766+0000","last_change":"2024-09-20T19:19:45.038512+0000","last_active":"2024-09-20T19:20:22.434766+0000","last_peered":"2024-09-20T19:20:22.434766+0000","last_clean":"2024-09-20T19:20:22.434766+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:20:22.434766+0000","last_undegraded":"2024-09-20T19:20:22.434766+0000","last_fullsized":"2024-09-20T19:20:22.434766+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:45.038377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.038377+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.979665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160337+0000","last_change":"2024-09-20T19:19:38.210845+0000","last_active":"2024-09-20T19:20:19.160337+0000","last_peered":"2024-09-20T19:20:19.160337+0000","last_clean":"2024-09-20T19:20:19.160337+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:20:19.160337+0000","last_undegraded":"2024-09-20T19:20:19.160337+0000","last_fullsized":"2024-09-20T19:20:19.160337+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:38.210713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:38.210713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.317890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"77'15","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.156401+0000","last_change":"2024-09-20T19:19:15.487623+0000","last_active":"2024-09-20T19:20:19.156401+0000","last_peered":"2024-09-20T19:20:19.156401+0000","last_clean":"2024-09-20T19:20:19.156401+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:20:19.156401+0000","last_undegraded":"2024-09-20T19:20:19.156401+0000","last_fullsized":"2024-09-20T19:20:19.156401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:15.487541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:15.487541+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154899+0000","last_change":"2024-09-20T19:19:36.378407+0000","last_active":"2024-09-20T19:20:19.154899+0000","last_peered":"2024-09-20T19:20:19.154899+0000","last_clean":"2024-09-20T19:20:19.154899+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:20:19.154899+0000","last_undegraded":"2024-09-20T19:20:19.154899+0000","last_fullsized":"2024-09-20T19:20:19.154899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.378285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.378285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:40.689679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":120,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160734+0000","last_change":"2024-09-20T19:19:50.966826+0000","last_active":"2024-09-20T19:20:19.160734+0000","last_peered":"2024-09-20T19:20:19.160734+0000","last_clean":"2024-09-20T19:20:19.160734+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:20:19.160734+0000","last_undegraded":"2024-09-20T19:20:19.160734+0000","last_fullsized":"2024-09-20T19:20:19.160734+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:50.966659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.966659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.209609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":77,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314025+0000","last_change":"2024-09-20T19:19:34.013906+0000","last_active":"2024-09-20T19:20:19.314025+0000","last_peered":"2024-09-20T19:20:19.314025+0000","last_clean":"2024-09-20T19:20:19.314025+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:20:19.314025+0000","last_undegraded":"2024-09-20T19:20:19.314025+0000","last_fullsized":"2024-09-20T19:20:19.314025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.013787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.013787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.149084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"77'11","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160702+0000","last_change":"2024-09-20T19:19:26.201077+0000","last_active":"2024-09-20T19:20:19.160702+0000","last_peered":"2024-09-20T19:20:19.160702+0000","last_clean":"2024-09-20T19:20:19.160702+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:20:19.160702+0000","last_undegraded":"2024-09-20T19:20:19.160702+0000","last_fullsized":"2024-09-20T19:20:19.160702+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:26.200992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.200992+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.983769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.730891+0000","last_change":"2024-09-20T19:19:48.281994+0000","last_active":"2024-09-20T19:20:19.730891+0000","last_peered":"2024-09-20T19:20:19.730891+0000","last_clean":"2024-09-20T19:20:19.730891+0000","last_became_active":"2024-09-20T19:18:07.269527+0000","last_became_peered":"2024-09-20T19:18:07.269527+0000","last_unstale":"2024-09-20T19:20:19.730891+0000","last_undegraded":"2024-09-20T19:20:19.730891+0000","last_fullsized":"2024-09-20T19:20:19.730891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.281832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.281832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.535302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314844+0000","last_change":"2024-09-20T19:19:34.966265+0000","last_active":"2024-09-20T19:20:19.314844+0000","last_peered":"2024-09-20T19:20:19.314844+0000","last_clean":"2024-09-20T19:20:19.314844+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:20:19.314844+0000","last_undegraded":"2024-09-20T19:20:19.314844+0000","last_fullsized":"2024-09-20T19:20:19.314844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.966141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.966141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.384654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.728292+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:20:19.728292+0000","last_peered":"2024-09-20T19:20:19.728292+0000","last_clean":"2024-09-20T19:20:19.728292+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:20:19.728292+0000","last_undegraded":"2024-09-20T19:20:19.728292+0000","last_fullsized":"2024-09-20T19:20:19.728292+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.064548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"77'14","reported_seq":108,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159557+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:20:19.159557+0000","last_peered":"2024-09-20T19:20:19.159557+0000","last_clean":"2024-09-20T19:20:19.159557+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:20:19.159557+0000","last_undegraded":"2024-09-20T19:20:19.159557+0000","last_fullsized":"2024-09-20T19:20:19.159557+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.837736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.568713+0000","last_change":"2024-09-20T19:20:16.657110+0000","last_active":"2024-09-20T19:20:19.568713+0000","last_peered":"2024-09-20T19:20:19.568713+0000","last_clean":"2024-09-20T19:20:19.568713+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:20:19.568713+0000","last_undegraded":"2024-09-20T19:20:19.568713+0000","last_fullsized":"2024-09-20T19:20:19.568713+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.656960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.656960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:43.026972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725297+0000","last_change":"2024-09-20T19:19:37.178863+0000","last_active":"2024-09-20T19:20:19.725297+0000","last_peered":"2024-09-20T19:20:19.725297+0000","last_clean":"2024-09-20T19:20:19.725297+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:20:19.725297+0000","last_undegraded":"2024-09-20T19:20:19.725297+0000","last_fullsized":"2024-09-20T19:20:19.725297+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.178724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.178724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.727526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731487+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:20:19.731487+0000","last_peered":"2024-09-20T19:20:19.731487+0000","last_clean":"2024-09-20T19:20:19.731487+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:20:19.731487+0000","last_undegraded":"2024-09-20T19:20:19.731487+0000","last_fullsized":"2024-09-20T19:20:19.731487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:18.443029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"77'18","reported_seq":119,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726189+0000","last_change":"2024-09-20T19:19:22.198542+0000","last_active":"2024-09-20T19:20:19.726189+0000","last_peered":"2024-09-20T19:20:19.726189+0000","last_clean":"2024-09-20T19:20:19.726189+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:20:19.726189+0000","last_undegraded":"2024-09-20T19:20:19.726189+0000","last_fullsized":"2024-09-20T19:20:19.726189+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:19:22.198447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:22.198447+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.994468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.156162+0000","last_change":"2024-09-20T19:19:27.488807+0000","last_active":"2024-09-20T19:20:19.156162+0000","last_peered":"2024-09-20T19:20:19.156162+0000","last_clean":"2024-09-20T19:20:19.156162+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:20:19.156162+0000","last_undegraded":"2024-09-20T19:20:19.156162+0000","last_fullsized":"2024-09-20T19:20:19.156162+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:27.488644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.488644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.323330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159507+0000","last_change":"2024-09-20T19:19:44.113106+0000","last_active":"2024-09-20T19:20:19.159507+0000","last_peered":"2024-09-20T19:20:19.159507+0000","last_clean":"2024-09-20T19:20:19.159507+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:20:19.159507+0000","last_undegraded":"2024-09-20T19:20:19.159507+0000","last_fullsized":"2024-09-20T19:20:19.159507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.112985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.112985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.242746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159471+0000","last_change":"2024-09-20T19:19:48.997145+0000","last_active":"2024-09-20T19:20:19.159471+0000","last_peered":"2024-09-20T19:20:19.159471+0000","last_clean":"2024-09-20T19:20:19.159471+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:20:19.159471+0000","last_undegraded":"2024-09-20T19:20:19.159471+0000","last_fullsized":"2024-09-20T19:20:19.159471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.996997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.996997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.494961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"77'19","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.727010+0000","last_change":"2024-09-20T19:19:27.727451+0000","last_active":"2024-09-20T19:20:19.727010+0000","last_peered":"2024-09-20T19:20:19.727010+0000","last_clean":"2024-09-20T19:20:19.727010+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:20:19.727010+0000","last_undegraded":"2024-09-20T19:20:19.727010+0000","last_fullsized":"2024-09-20T19:20:19.727010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.727294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.727294+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.582205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.153386+0000","last_change":"2024-09-20T19:19:56.998769+0000","last_active":"2024-09-20T19:20:19.153386+0000","last_peered":"2024-09-20T19:20:19.153386+0000","last_clean":"2024-09-20T19:20:19.153386+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:20:19.153386+0000","last_undegraded":"2024-09-20T19:20:19.153386+0000","last_fullsized":"2024-09-20T19:20:19.153386+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.169199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.169199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.901847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.566456+0000","last_change":"2024-09-20T19:19:41.871990+0000","last_active":"2024-09-20T19:20:19.566456+0000","last_peered":"2024-09-20T19:20:19.566456+0000","last_clean":"2024-09-20T19:20:19.566456+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:20:19.566456+0000","last_undegraded":"2024-09-20T19:20:19.566456+0000","last_fullsized":"2024-09-20T19:20:19.566456+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.871854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.871854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.298167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154041+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:20:19.154041+0000","last_peered":"2024-09-20T19:20:19.154041+0000","last_clean":"2024-09-20T19:20:19.154041+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:20:19.154041+0000","last_undegraded":"2024-09-20T19:20:19.154041+0000","last_fullsized":"2024-09-20T19:20:19.154041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:29.401329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"77'10","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154976+0000","last_change":"2024-09-20T19:19:40.461604+0000","last_active":"2024-09-20T19:20:19.154976+0000","last_peered":"2024-09-20T19:20:19.154976+0000","last_clean":"2024-09-20T19:20:19.154976+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:20:19.154976+0000","last_undegraded":"2024-09-20T19:20:19.154976+0000","last_fullsized":"2024-09-20T19:20:19.154976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:40.461532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.461532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:08.703461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.157746+0000","last_change":"2024-09-20T19:19:34.429306+0000","last_active":"2024-09-20T19:20:19.157746+0000","last_peered":"2024-09-20T19:20:19.157746+0000","last_clean":"2024-09-20T19:20:19.157746+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:20:19.157746+0000","last_undegraded":"2024-09-20T19:20:19.157746+0000","last_fullsized":"2024-09-20T19:20:19.157746+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.429125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.429125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.671437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725887+0000","last_change":"2024-09-20T19:19:37.807674+0000","last_active":"2024-09-20T19:20:19.725887+0000","last_peered":"2024-09-20T19:20:19.725887+0000","last_clean":"2024-09-20T19:20:19.725887+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:20:19.725887+0000","last_undegraded":"2024-09-20T19:20:19.725887+0000","last_fullsized":"2024-09-20T19:20:19.725887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.807547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.807547+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.829611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724753+0000","last_change":"2024-09-20T19:19:31.793749+0000","last_active":"2024-09-20T19:20:19.724753+0000","last_peered":"2024-09-20T19:20:19.724753+0000","last_clean":"2024-09-20T19:20:19.724753+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:20:19.724753+0000","last_undegraded":"2024-09-20T19:20:19.724753+0000","last_fullsized":"2024-09-20T19:20:19.724753+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.128931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'30","reported_seq":165,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.157649+0000","last_change":"2024-09-20T19:19:46.395559+0000","last_active":"2024-09-20T19:20:19.157649+0000","last_peered":"2024-09-20T19:20:19.157649+0000","last_clean":"2024-09-20T19:20:19.157649+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:20:19.157649+0000","last_undegraded":"2024-09-20T19:20:19.157649+0000","last_fullsized":"2024-09-20T19:20:19.157649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:19:46.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.395404+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.847884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314157+0000","last_change":"2024-09-20T19:19:33.057715+0000","last_active":"2024-09-20T19:20:19.314157+0000","last_peered":"2024-09-20T19:20:19.314157+0000","last_clean":"2024-09-20T19:20:19.314157+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:20:19.314157+0000","last_undegraded":"2024-09-20T19:20:19.314157+0000","last_fullsized":"2024-09-20T19:20:19.314157+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.057567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.057567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.036699+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724237+0000","last_change":"2024-09-20T19:19:56.854517+0000","last_active":"2024-09-20T19:20:19.724237+0000","last_peered":"2024-09-20T19:20:19.724237+0000","last_clean":"2024-09-20T19:20:19.724237+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:20:19.724237+0000","last_undegraded":"2024-09-20T19:20:19.724237+0000","last_fullsized":"2024-09-20T19:20:19.724237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.854461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.854461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.519787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729482+0000","last_change":"2024-09-20T19:19:50.253935+0000","last_active":"2024-09-20T19:20:19.729482+0000","last_peered":"2024-09-20T19:20:19.729482+0000","last_clean":"2024-09-20T19:20:19.729482+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:20:19.729482+0000","last_undegraded":"2024-09-20T19:20:19.729482+0000","last_fullsized":"2024-09-20T19:20:19.729482+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.253727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.253727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.038762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"77'13","reported_seq":30,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.157496+0000","last_change":"2024-09-20T19:19:57.013393+0000","last_active":"2024-09-20T19:20:19.157496+0000","last_peered":"2024-09-20T19:20:19.157496+0000","last_clean":"2024-09-20T19:20:19.157496+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:20:19.157496+0000","last_undegraded":"2024-09-20T19:20:19.157496+0000","last_fullsized":"2024-09-20T19:20:19.157496+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:19:51.410598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.410598+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:12.734213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":222,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:25.077302+0000","last_change":"2024-09-20T19:19:19.441522+0000","last_active":"2024-09-20T19:20:25.077302+0000","last_peered":"2024-09-20T19:20:25.077302+0000","last_clean":"2024-09-20T19:20:25.077302+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:20:25.077302+0000","last_undegraded":"2024-09-20T19:20:25.077302+0000","last_fullsized":"2024-09-20T19:20:25.077302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:19:19.441341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.441341+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:39.849970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":118,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.153563+0000","last_change":"2024-09-20T19:19:49.990251+0000","last_active":"2024-09-20T19:20:19.153563+0000","last_peered":"2024-09-20T19:20:19.153563+0000","last_clean":"2024-09-20T19:20:19.153563+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:20:19.153563+0000","last_undegraded":"2024-09-20T19:20:19.153563+0000","last_fullsized":"2024-09-20T19:20:19.153563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:49.990058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:49.990058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.407192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.156509+0000","last_change":"2024-09-20T19:19:25.169431+0000","last_active":"2024-09-20T19:20:19.156509+0000","last_peered":"2024-09-20T19:20:19.156509+0000","last_clean":"2024-09-20T19:20:19.156509+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:20:19.156509+0000","last_undegraded":"2024-09-20T19:20:19.156509+0000","last_fullsized":"2024-09-20T19:20:19.156509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:25.169306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:25.169306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:28.573693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731924+0000","last_change":"2024-09-20T19:19:55.305695+0000","last_active":"2024-09-20T19:20:19.731924+0000","last_peered":"2024-09-20T19:20:19.731924+0000","last_clean":"2024-09-20T19:20:19.731924+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:20:19.731924+0000","last_undegraded":"2024-09-20T19:20:19.731924+0000","last_fullsized":"2024-09-20T19:20:19.731924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:55.305573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.305573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.195185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'12","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.728971+0000","last_change":"2024-09-20T19:19:23.739790+0000","last_active":"2024-09-20T19:20:19.728971+0000","last_peered":"2024-09-20T19:20:19.728971+0000","last_clean":"2024-09-20T19:20:19.728971+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:20:19.728971+0000","last_undegraded":"2024-09-20T19:20:19.728971+0000","last_fullsized":"2024-09-20T19:20:19.728971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:23.739633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.739633+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:38.349636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:25.853565+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:20:25.853565+0000","last_peered":"2024-09-20T19:20:25.853565+0000","last_clean":"2024-09-20T19:20:25.853565+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:20:25.853565+0000","last_undegraded":"2024-09-20T19:20:25.853565+0000","last_fullsized":"2024-09-20T19:20:25.853565+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.851755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.730748+0000","last_change":"2024-09-20T19:19:44.240584+0000","last_active":"2024-09-20T19:20:19.730748+0000","last_peered":"2024-09-20T19:20:19.730748+0000","last_clean":"2024-09-20T19:20:19.730748+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:20:19.730748+0000","last_undegraded":"2024-09-20T19:20:19.730748+0000","last_fullsized":"2024-09-20T19:20:19.730748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.240462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.240462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.731548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314955+0000","last_change":"2024-09-20T19:19:18.140999+0000","last_active":"2024-09-20T19:20:19.314955+0000","last_peered":"2024-09-20T19:20:19.314955+0000","last_clean":"2024-09-20T19:20:19.314955+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:20:19.314955+0000","last_undegraded":"2024-09-20T19:20:19.314955+0000","last_fullsized":"2024-09-20T19:20:19.314955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:18.140785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:18.140785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.388246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.156935+0000","last_change":"2024-09-20T19:19:24.381047+0000","last_active":"2024-09-20T19:20:19.156935+0000","last_peered":"2024-09-20T19:20:19.156935+0000","last_clean":"2024-09-20T19:20:19.156935+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:20:19.156935+0000","last_undegraded":"2024-09-20T19:20:19.156935+0000","last_fullsized":"2024-09-20T19:20:19.156935+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:24.380899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:24.380899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:30.202989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"77'16","reported_seq":136,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:21.904357+0000","last_change":"2024-09-20T19:19:39.901351+0000","last_active":"2024-09-20T19:20:21.904357+0000","last_peered":"2024-09-20T19:20:21.904357+0000","last_clean":"2024-09-20T19:20:21.904357+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:20:21.904357+0000","last_undegraded":"2024-09-20T19:20:21.904357+0000","last_fullsized":"2024-09-20T19:20:21.904357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:19:39.901171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.901171+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.526242+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725756+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:20:19.725756+0000","last_peered":"2024-09-20T19:20:19.725756+0000","last_clean":"2024-09-20T19:20:19.725756+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:20:19.725756+0000","last_undegraded":"2024-09-20T19:20:19.725756+0000","last_fullsized":"2024-09-20T19:20:19.725756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.492805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313962+0000","last_change":"2024-09-20T19:19:07.170908+0000","last_active":"2024-09-20T19:20:19.313962+0000","last_peered":"2024-09-20T19:20:19.313962+0000","last_clean":"2024-09-20T19:20:19.313962+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:20:19.313962+0000","last_undegraded":"2024-09-20T19:20:19.313962+0000","last_fullsized":"2024-09-20T19:20:19.313962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:07.170760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:07.170760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.385612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":138,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731118+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:20:19.731118+0000","last_peered":"2024-09-20T19:20:19.731118+0000","last_clean":"2024-09-20T19:20:19.731118+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:20:19.731118+0000","last_undegraded":"2024-09-20T19:20:19.731118+0000","last_fullsized":"2024-09-20T19:20:19.731118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:27.954958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729888+0000","last_change":"2024-09-20T19:19:26.163243+0000","last_active":"2024-09-20T19:20:19.729888+0000","last_peered":"2024-09-20T19:20:19.729888+0000","last_clean":"2024-09-20T19:20:19.729888+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:20:19.729888+0000","last_undegraded":"2024-09-20T19:20:19.729888+0000","last_fullsized":"2024-09-20T19:20:19.729888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.163172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.163172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.703757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.315194+0000","last_change":"2024-09-20T19:19:17.120513+0000","last_active":"2024-09-20T19:20:19.315194+0000","last_peered":"2024-09-20T19:20:19.315194+0000","last_clean":"2024-09-20T19:20:19.315194+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:20:19.315194+0000","last_undegraded":"2024-09-20T19:20:19.315194+0000","last_fullsized":"2024-09-20T19:20:19.315194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.120471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.120471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.128195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"77'19","reported_seq":117,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.567884+0000","last_change":"2024-09-20T19:19:27.036130+0000","last_active":"2024-09-20T19:20:19.567884+0000","last_peered":"2024-09-20T19:20:19.567884+0000","last_clean":"2024-09-20T19:20:19.567884+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:20:19.567884+0000","last_undegraded":"2024-09-20T19:20:19.567884+0000","last_fullsized":"2024-09-20T19:20:19.567884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.036022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.036022+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.263159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.155524+0000","last_change":"2024-09-20T19:19:13.243819+0000","last_active":"2024-09-20T19:20:19.155524+0000","last_peered":"2024-09-20T19:20:19.155524+0000","last_clean":"2024-09-20T19:20:19.155524+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:20:19.155524+0000","last_undegraded":"2024-09-20T19:20:19.155524+0000","last_fullsized":"2024-09-20T19:20:19.155524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.243639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.243639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.750283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154335+0000","last_change":"2024-09-20T19:19:43.031914+0000","last_active":"2024-09-20T19:20:19.154335+0000","last_peered":"2024-09-20T19:20:19.154335+0000","last_clean":"2024-09-20T19:20:19.154335+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:20:19.154335+0000","last_undegraded":"2024-09-20T19:20:19.154335+0000","last_fullsized":"2024-09-20T19:20:19.154335+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.031768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.031768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.442373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731225+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:20:19.731225+0000","last_peered":"2024-09-20T19:20:19.731225+0000","last_clean":"2024-09-20T19:20:19.731225+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:20:19.731225+0000","last_undegraded":"2024-09-20T19:20:19.731225+0000","last_fullsized":"2024-09-20T19:20:19.731225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.126953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"77'9","reported_seq":103,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726864+0000","last_change":"2024-09-20T19:19:46.791747+0000","last_active":"2024-09-20T19:20:19.726864+0000","last_peered":"2024-09-20T19:20:19.726864+0000","last_clean":"2024-09-20T19:20:19.726864+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:20:19.726864+0000","last_undegraded":"2024-09-20T19:20:19.726864+0000","last_fullsized":"2024-09-20T19:20:19.726864+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.791592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.791592+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.994746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159356+0000","last_change":"2024-09-20T19:19:35.390225+0000","last_active":"2024-09-20T19:20:19.159356+0000","last_peered":"2024-09-20T19:20:19.159356+0000","last_clean":"2024-09-20T19:20:19.159356+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:20:19.159356+0000","last_undegraded":"2024-09-20T19:20:19.159356+0000","last_fullsized":"2024-09-20T19:20:19.159356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.390064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.390064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.228348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.153869+0000","last_change":"2024-09-20T19:19:41.044575+0000","last_active":"2024-09-20T19:20:19.153869+0000","last_peered":"2024-09-20T19:20:19.153869+0000","last_clean":"2024-09-20T19:20:19.153869+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:20:19.153869+0000","last_undegraded":"2024-09-20T19:20:19.153869+0000","last_fullsized":"2024-09-20T19:20:19.153869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.044427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.044427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.962646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":80,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.727538+0000","last_change":"2024-09-20T19:19:52.810445+0000","last_active":"2024-09-20T19:20:19.727538+0000","last_peered":"2024-09-20T19:20:19.727538+0000","last_clean":"2024-09-20T19:20:19.727538+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:20:19.727538+0000","last_undegraded":"2024-09-20T19:20:19.727538+0000","last_fullsized":"2024-09-20T19:20:19.727538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:19:52.810385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:52.810385+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.122928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"77'15","reported_seq":111,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313072+0000","last_change":"2024-09-20T19:19:44.046132+0000","last_active":"2024-09-20T19:20:19.313072+0000","last_peered":"2024-09-20T19:20:19.313072+0000","last_clean":"2024-09-20T19:20:19.313072+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:20:19.313072+0000","last_undegraded":"2024-09-20T19:20:19.313072+0000","last_fullsized":"2024-09-20T19:20:19.313072+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:44.045925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.045925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.259093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":97,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.156804+0000","last_change":"2024-09-20T19:19:21.207668+0000","last_active":"2024-09-20T19:20:19.156804+0000","last_peered":"2024-09-20T19:20:19.156804+0000","last_clean":"2024-09-20T19:20:19.156804+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:20:19.156804+0000","last_undegraded":"2024-09-20T19:20:19.156804+0000","last_fullsized":"2024-09-20T19:20:19.156804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:21.207433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:21.207433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.475369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:25.228601+0000","last_change":"2024-09-20T19:20:25.228464+0000","last_active":"2024-09-20T19:20:25.228601+0000","last_peered":"2024-09-20T19:20:25.228601+0000","last_clean":"2024-09-20T19:20:25.228601+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:20:25.228601+0000","last_undegraded":"2024-09-20T19:20:25.228601+0000","last_fullsized":"2024-09-20T19:20:25.228601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:25.228325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.228325+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.437018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313456+0000","last_change":"2024-09-20T19:19:36.943615+0000","last_active":"2024-09-20T19:20:19.313456+0000","last_peered":"2024-09-20T19:20:19.313456+0000","last_clean":"2024-09-20T19:20:19.313456+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:20:19.313456+0000","last_undegraded":"2024-09-20T19:20:19.313456+0000","last_fullsized":"2024-09-20T19:20:19.313456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.943527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.943527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.968462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"77'12","reported_seq":137,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159337+0000","last_change":"2024-09-20T19:20:16.852586+0000","last_active":"2024-09-20T19:20:19.159337+0000","last_peered":"2024-09-20T19:20:19.159337+0000","last_clean":"2024-09-20T19:20:19.159337+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:20:19.159337+0000","last_undegraded":"2024-09-20T19:20:19.159337+0000","last_fullsized":"2024-09-20T19:20:19.159337+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:16.852450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.852450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.371990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.161297+0000","last_change":"2024-09-20T19:19:33.263120+0000","last_active":"2024-09-20T19:20:19.161297+0000","last_peered":"2024-09-20T19:20:19.161297+0000","last_clean":"2024-09-20T19:20:19.161297+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:20:19.161297+0000","last_undegraded":"2024-09-20T19:20:19.161297+0000","last_fullsized":"2024-09-20T19:20:19.161297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:19:33.262915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.262915+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:36.795568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154616+0000","last_change":"2024-09-20T19:19:51.013009+0000","last_active":"2024-09-20T19:20:19.154616+0000","last_peered":"2024-09-20T19:20:19.154616+0000","last_clean":"2024-09-20T19:20:19.154616+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:20:19.154616+0000","last_undegraded":"2024-09-20T19:20:19.154616+0000","last_fullsized":"2024-09-20T19:20:19.154616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.012906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.012906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.429741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":71,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726973+0000","last_change":"2024-09-20T19:18:13.370848+0000","last_active":"2024-09-20T19:20:19.726973+0000","last_peered":"2024-09-20T19:20:19.726973+0000","last_clean":"2024-09-20T19:20:19.726973+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:20:19.726973+0000","last_undegraded":"2024-09-20T19:20:19.726973+0000","last_fullsized":"2024-09-20T19:20:19.726973+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:18:12.302527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.728694+0000","last_change":"2024-09-20T19:19:24.694785+0000","last_active":"2024-09-20T19:20:19.728694+0000","last_peered":"2024-09-20T19:20:19.728694+0000","last_clean":"2024-09-20T19:20:19.728694+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:20:19.728694+0000","last_undegraded":"2024-09-20T19:20:19.728694+0000","last_fullsized":"2024-09-20T19:20:19.728694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:24.694641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:24.694641+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:33.496424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.153360+0000","last_change":"2024-09-20T19:19:28.174295+0000","last_active":"2024-09-20T19:20:19.153360+0000","last_peered":"2024-09-20T19:20:19.153360+0000","last_clean":"2024-09-20T19:20:19.153360+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:20:19.153360+0000","last_undegraded":"2024-09-20T19:20:19.153360+0000","last_fullsized":"2024-09-20T19:20:19.153360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.174157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.174157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.549082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724880+0000","last_change":"2024-09-20T19:19:57.060943+0000","last_active":"2024-09-20T19:20:19.724880+0000","last_peered":"2024-09-20T19:20:19.724880+0000","last_clean":"2024-09-20T19:20:19.724880+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:20:19.724880+0000","last_undegraded":"2024-09-20T19:20:19.724880+0000","last_fullsized":"2024-09-20T19:20:19.724880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.060887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.060887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.058027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154045+0000","last_change":"2024-09-20T19:19:54.030064+0000","last_active":"2024-09-20T19:20:19.154045+0000","last_peered":"2024-09-20T19:20:19.154045+0000","last_clean":"2024-09-20T19:20:19.154045+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:20:19.154045+0000","last_undegraded":"2024-09-20T19:20:19.154045+0000","last_fullsized":"2024-09-20T19:20:19.154045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.029937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.029937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.274260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.567539+0000","last_change":"2024-09-20T19:19:13.019741+0000","last_active":"2024-09-20T19:20:19.567539+0000","last_peered":"2024-09-20T19:20:19.567539+0000","last_clean":"2024-09-20T19:20:19.567539+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:20:19.567539+0000","last_undegraded":"2024-09-20T19:20:19.567539+0000","last_fullsized":"2024-09-20T19:20:19.567539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:13.019586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.019586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.203383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726898+0000","last_change":"2024-09-20T19:19:29.203589+0000","last_active":"2024-09-20T19:20:19.726898+0000","last_peered":"2024-09-20T19:20:19.726898+0000","last_clean":"2024-09-20T19:20:19.726898+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:20:19.726898+0000","last_undegraded":"2024-09-20T19:20:19.726898+0000","last_fullsized":"2024-09-20T19:20:19.726898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:19:29.203410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:29.203410+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.144800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314370+0000","last_change":"2024-09-20T19:19:43.040517+0000","last_active":"2024-09-20T19:20:19.314370+0000","last_peered":"2024-09-20T19:20:19.314370+0000","last_clean":"2024-09-20T19:20:19.314370+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:20:19.314370+0000","last_undegraded":"2024-09-20T19:20:19.314370+0000","last_fullsized":"2024-09-20T19:20:19.314370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.040460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.040460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:10.254103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731562+0000","last_change":"2024-09-20T19:19:56.343379+0000","last_active":"2024-09-20T19:20:19.731562+0000","last_peered":"2024-09-20T19:20:19.731562+0000","last_clean":"2024-09-20T19:20:19.731562+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:20:19.731562+0000","last_undegraded":"2024-09-20T19:20:19.731562+0000","last_fullsized":"2024-09-20T19:20:19.731562+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.343254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.343254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.336373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":101,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729052+0000","last_change":"2024-09-20T19:19:13.215224+0000","last_active":"2024-09-20T19:20:19.729052+0000","last_peered":"2024-09-20T19:20:19.729052+0000","last_clean":"2024-09-20T19:20:19.729052+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:20:19.729052+0000","last_undegraded":"2024-09-20T19:20:19.729052+0000","last_fullsized":"2024-09-20T19:20:19.729052+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:13.215055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:13.215055+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313836+0000","last_change":"2024-09-20T19:19:28.086132+0000","last_active":"2024-09-20T19:20:19.313836+0000","last_peered":"2024-09-20T19:20:19.313836+0000","last_clean":"2024-09-20T19:20:19.313836+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:20:19.313836+0000","last_undegraded":"2024-09-20T19:20:19.313836+0000","last_fullsized":"2024-09-20T19:20:19.313836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.086004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.086004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.650446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":119,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314528+0000","last_change":"2024-09-20T19:19:27.113335+0000","last_active":"2024-09-20T19:20:19.314528+0000","last_peered":"2024-09-20T19:20:19.314528+0000","last_clean":"2024-09-20T19:20:19.314528+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:20:19.314528+0000","last_undegraded":"2024-09-20T19:20:19.314528+0000","last_fullsized":"2024-09-20T19:20:19.314528+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.113211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.113211+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.889024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.728059+0000","last_change":"2024-09-20T19:19:40.871513+0000","last_active":"2024-09-20T19:20:19.728059+0000","last_peered":"2024-09-20T19:20:19.728059+0000","last_clean":"2024-09-20T19:20:19.728059+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:20:19.728059+0000","last_undegraded":"2024-09-20T19:20:19.728059+0000","last_fullsized":"2024-09-20T19:20:19.728059+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.871447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.871447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.175662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":101,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314410+0000","last_change":"2024-09-20T19:19:46.036288+0000","last_active":"2024-09-20T19:20:19.314410+0000","last_peered":"2024-09-20T19:20:19.314410+0000","last_clean":"2024-09-20T19:20:19.314410+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:20:19.314410+0000","last_undegraded":"2024-09-20T19:20:19.314410+0000","last_fullsized":"2024-09-20T19:20:19.314410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.036143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.036143+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.480915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.727311+0000","last_change":"2024-09-20T19:19:50.860419+0000","last_active":"2024-09-20T19:20:19.727311+0000","last_peered":"2024-09-20T19:20:19.727311+0000","last_clean":"2024-09-20T19:20:19.727311+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:20:19.727311+0000","last_undegraded":"2024-09-20T19:20:19.727311+0000","last_fullsized":"2024-09-20T19:20:19.727311+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.860284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.860284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.470934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":120,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:20.964678+0000","last_change":"2024-09-20T19:20:20.964445+0000","last_active":"2024-09-20T19:20:20.964678+0000","last_peered":"2024-09-20T19:20:20.964678+0000","last_clean":"2024-09-20T19:20:20.964678+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:20:20.964678+0000","last_undegraded":"2024-09-20T19:20:20.964678+0000","last_fullsized":"2024-09-20T19:20:20.964678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:20.964261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:20.964261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.805123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160778+0000","last_change":"2024-09-20T19:19:16.241092+0000","last_active":"2024-09-20T19:20:19.160778+0000","last_peered":"2024-09-20T19:20:19.160778+0000","last_clean":"2024-09-20T19:20:19.160778+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:20:19.160778+0000","last_undegraded":"2024-09-20T19:20:19.160778+0000","last_fullsized":"2024-09-20T19:20:19.160778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:16.240975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:16.240975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.618347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":110,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.567151+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:20:19.567151+0000","last_peered":"2024-09-20T19:20:19.567151+0000","last_clean":"2024-09-20T19:20:19.567151+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:20:19.567151+0000","last_undegraded":"2024-09-20T19:20:19.567151+0000","last_fullsized":"2024-09-20T19:20:19.567151+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:32.267574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.568290+0000","last_change":"2024-09-20T19:19:19.061400+0000","last_active":"2024-09-20T19:20:19.568290+0000","last_peered":"2024-09-20T19:20:19.568290+0000","last_clean":"2024-09-20T19:20:19.568290+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:20:19.568290+0000","last_undegraded":"2024-09-20T19:20:19.568290+0000","last_fullsized":"2024-09-20T19:20:19.568290+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:19.061323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.061323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:24.520151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724772+0000","last_change":"2024-09-20T19:19:47.212665+0000","last_active":"2024-09-20T19:20:19.724772+0000","last_peered":"2024-09-20T19:20:19.724772+0000","last_clean":"2024-09-20T19:20:19.724772+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:20:19.724772+0000","last_undegraded":"2024-09-20T19:20:19.724772+0000","last_fullsized":"2024-09-20T19:20:19.724772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:47.212542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.212542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.674953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724446+0000","last_change":"2024-09-20T19:19:57.895992+0000","last_active":"2024-09-20T19:20:19.724446+0000","last_peered":"2024-09-20T19:20:19.724446+0000","last_clean":"2024-09-20T19:20:19.724446+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:20:19.724446+0000","last_undegraded":"2024-09-20T19:20:19.724446+0000","last_fullsized":"2024-09-20T19:20:19.724446+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.895923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.895923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.375254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159303+0000","last_change":"2024-09-20T19:19:27.179523+0000","last_active":"2024-09-20T19:20:19.159303+0000","last_peered":"2024-09-20T19:20:19.159303+0000","last_clean":"2024-09-20T19:20:19.159303+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:20:19.159303+0000","last_undegraded":"2024-09-20T19:20:19.159303+0000","last_fullsized":"2024-09-20T19:20:19.159303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.179398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.179398+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.909162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159164+0000","last_change":"2024-09-20T19:20:17.522125+0000","last_active":"2024-09-20T19:20:19.159164+0000","last_peered":"2024-09-20T19:20:19.159164+0000","last_clean":"2024-09-20T19:20:19.159164+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:20:19.159164+0000","last_undegraded":"2024-09-20T19:20:19.159164+0000","last_fullsized":"2024-09-20T19:20:19.159164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:17.521947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:17.521947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.763307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154625+0000","last_change":"2024-09-20T19:19:48.434045+0000","last_active":"2024-09-20T19:20:19.154625+0000","last_peered":"2024-09-20T19:20:19.154625+0000","last_clean":"2024-09-20T19:20:19.154625+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:20:19.154625+0000","last_undegraded":"2024-09-20T19:20:19.154625+0000","last_fullsized":"2024-09-20T19:20:19.154625+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.433915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.433915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.655605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.724613+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:20:19.724613+0000","last_peered":"2024-09-20T19:20:19.724613+0000","last_clean":"2024-09-20T19:20:19.724613+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:20:19.724613+0000","last_undegraded":"2024-09-20T19:20:19.724613+0000","last_fullsized":"2024-09-20T19:20:19.724613+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.984866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":93,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726184+0000","last_change":"2024-09-20T19:19:33.242802+0000","last_active":"2024-09-20T19:20:19.726184+0000","last_peered":"2024-09-20T19:20:19.726184+0000","last_clean":"2024-09-20T19:20:19.726184+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:20:19.726184+0000","last_undegraded":"2024-09-20T19:20:19.726184+0000","last_fullsized":"2024-09-20T19:20:19.726184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:19:33.242651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.242651+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.383781+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.314629+0000","last_change":"2024-09-20T19:19:35.981274+0000","last_active":"2024-09-20T19:20:19.314629+0000","last_peered":"2024-09-20T19:20:19.314629+0000","last_clean":"2024-09-20T19:20:19.314629+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:20:19.314629+0000","last_undegraded":"2024-09-20T19:20:19.314629+0000","last_fullsized":"2024-09-20T19:20:19.314629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.981166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.981166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.523226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":81,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.566459+0000","last_change":"2024-09-20T19:19:29.014438+0000","last_active":"2024-09-20T19:20:19.566459+0000","last_peered":"2024-09-20T19:20:19.566459+0000","last_clean":"2024-09-20T19:20:19.566459+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:20:19.566459+0000","last_undegraded":"2024-09-20T19:20:19.566459+0000","last_fullsized":"2024-09-20T19:20:19.566459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:29.014364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:29.014364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:37.312931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.730730+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:20:19.730730+0000","last_peered":"2024-09-20T19:20:19.730730+0000","last_clean":"2024-09-20T19:20:19.730730+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:20:19.730730+0000","last_undegraded":"2024-09-20T19:20:19.730730+0000","last_fullsized":"2024-09-20T19:20:19.730730+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.905322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725370+0000","last_change":"2024-09-20T19:19:28.173179+0000","last_active":"2024-09-20T19:20:19.725370+0000","last_peered":"2024-09-20T19:20:19.725370+0000","last_clean":"2024-09-20T19:20:19.725370+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:20:19.725370+0000","last_undegraded":"2024-09-20T19:20:19.725370+0000","last_fullsized":"2024-09-20T19:20:19.725370+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:28.172944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.172944+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.576524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.568545+0000","last_change":"2024-09-20T19:19:11.981872+0000","last_active":"2024-09-20T19:20:19.568545+0000","last_peered":"2024-09-20T19:20:19.568545+0000","last_clean":"2024-09-20T19:20:19.568545+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:20:19.568545+0000","last_undegraded":"2024-09-20T19:20:19.568545+0000","last_fullsized":"2024-09-20T19:20:19.568545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:11.981739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:11.981739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:29.367984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.729630+0000","last_change":"2024-09-20T19:19:50.321638+0000","last_active":"2024-09-20T19:20:19.729630+0000","last_peered":"2024-09-20T19:20:19.729630+0000","last_clean":"2024-09-20T19:20:19.729630+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:20:19.729630+0000","last_undegraded":"2024-09-20T19:20:19.729630+0000","last_fullsized":"2024-09-20T19:20:19.729630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.321520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.321520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.084680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726601+0000","last_change":"2024-09-20T19:19:21.180046+0000","last_active":"2024-09-20T19:20:19.726601+0000","last_peered":"2024-09-20T19:20:19.726601+0000","last_clean":"2024-09-20T19:20:19.726601+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:20:19.726601+0000","last_undegraded":"2024-09-20T19:20:19.726601+0000","last_fullsized":"2024-09-20T19:20:19.726601+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:21.179896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:21.179896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.033462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":80,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.731343+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:20:19.731343+0000","last_peered":"2024-09-20T19:20:19.731343+0000","last_clean":"2024-09-20T19:20:19.731343+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:20:19.731343+0000","last_undegraded":"2024-09-20T19:20:19.731343+0000","last_fullsized":"2024-09-20T19:20:19.731343+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.371707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.313040+0000","last_change":"2024-09-20T19:19:23.109142+0000","last_active":"2024-09-20T19:20:19.313040+0000","last_peered":"2024-09-20T19:20:19.313040+0000","last_clean":"2024-09-20T19:20:19.313040+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:20:19.313040+0000","last_undegraded":"2024-09-20T19:20:19.313040+0000","last_fullsized":"2024-09-20T19:20:19.313040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:23.109064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.109064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:42.414841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":102,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.728411+0000","last_change":"2024-09-20T19:19:55.848311+0000","last_active":"2024-09-20T19:20:19.728411+0000","last_peered":"2024-09-20T19:20:19.728411+0000","last_clean":"2024-09-20T19:20:19.728411+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:20:19.728411+0000","last_undegraded":"2024-09-20T19:20:19.728411+0000","last_fullsized":"2024-09-20T19:20:19.728411+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:55.848164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.848164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:19.416599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.566713+0000","last_change":"2024-09-20T19:19:35.001831+0000","last_active":"2024-09-20T19:20:19.566713+0000","last_peered":"2024-09-20T19:20:19.566713+0000","last_clean":"2024-09-20T19:20:19.566713+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:20:19.566713+0000","last_undegraded":"2024-09-20T19:20:19.566713+0000","last_fullsized":"2024-09-20T19:20:19.566713+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:35.001666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.001666+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.268895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.158118+0000","last_change":"2024-09-20T19:19:31.466386+0000","last_active":"2024-09-20T19:20:19.158118+0000","last_peered":"2024-09-20T19:20:19.158118+0000","last_clean":"2024-09-20T19:20:19.158118+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:20:19.158118+0000","last_undegraded":"2024-09-20T19:20:19.158118+0000","last_fullsized":"2024-09-20T19:20:19.158118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.466247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.466247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.461700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726238+0000","last_change":"2024-09-20T19:19:16.214996+0000","last_active":"2024-09-20T19:20:19.726238+0000","last_peered":"2024-09-20T19:20:19.726238+0000","last_clean":"2024-09-20T19:20:19.726238+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:20:19.726238+0000","last_undegraded":"2024-09-20T19:20:19.726238+0000","last_fullsized":"2024-09-20T19:20:19.726238+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:16.214940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:16.214940+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:41.495655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":124,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.568542+0000","last_change":"2024-09-20T19:19:57.028481+0000","last_active":"2024-09-20T19:20:19.568542+0000","last_peered":"2024-09-20T19:20:19.568542+0000","last_clean":"2024-09-20T19:20:19.568542+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:20:19.568542+0000","last_undegraded":"2024-09-20T19:20:19.568542+0000","last_fullsized":"2024-09-20T19:20:19.568542+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:40.905378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.905378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.869949+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:24.877567+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:20:24.877567+0000","last_peered":"2024-09-20T19:20:24.877567+0000","last_clean":"2024-09-20T19:20:24.877567+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:20:24.877567+0000","last_undegraded":"2024-09-20T19:20:24.877567+0000","last_fullsized":"2024-09-20T19:20:24.877567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.632034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.160425+0000","last_change":"2024-09-20T19:19:39.226277+0000","last_active":"2024-09-20T19:20:19.160425+0000","last_peered":"2024-09-20T19:20:19.160425+0000","last_clean":"2024-09-20T19:20:19.160425+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:20:19.160425+0000","last_undegraded":"2024-09-20T19:20:19.160425+0000","last_fullsized":"2024-09-20T19:20:19.160425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:39.226160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.226160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.931829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.567984+0000","last_change":"2024-09-20T19:19:33.036193+0000","last_active":"2024-09-20T19:20:19.567984+0000","last_peered":"2024-09-20T19:20:19.567984+0000","last_clean":"2024-09-20T19:20:19.567984+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:20:19.567984+0000","last_undegraded":"2024-09-20T19:20:19.567984+0000","last_fullsized":"2024-09-20T19:20:19.567984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.036056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.036056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:44.896627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":109,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.154782+0000","last_change":"2024-09-20T19:19:47.048010+0000","last_active":"2024-09-20T19:20:19.154782+0000","last_peered":"2024-09-20T19:20:19.154782+0000","last_clean":"2024-09-20T19:20:19.154782+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:20:19.154782+0000","last_undegraded":"2024-09-20T19:20:19.154782+0000","last_fullsized":"2024-09-20T19:20:19.154782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:47.047816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.047816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.958306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.158750+0000","last_change":"2024-09-20T19:19:25.409927+0000","last_active":"2024-09-20T19:20:19.158750+0000","last_peered":"2024-09-20T19:20:19.158750+0000","last_clean":"2024-09-20T19:20:19.158750+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:20:19.158750+0000","last_undegraded":"2024-09-20T19:20:19.158750+0000","last_fullsized":"2024-09-20T19:20:19.158750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:25.409801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:25.409801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:27.236371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.726650+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:20:19.726650+0000","last_peered":"2024-09-20T19:20:19.726650+0000","last_clean":"2024-09-20T19:20:19.726650+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:20:19.726650+0000","last_undegraded":"2024-09-20T19:20:19.726650+0000","last_fullsized":"2024-09-20T19:20:19.726650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:20.115239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.727311+0000","last_change":"2024-09-20T19:19:58.028896+0000","last_active":"2024-09-20T19:20:19.727311+0000","last_peered":"2024-09-20T19:20:19.727311+0000","last_clean":"2024-09-20T19:20:19.727311+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:20:19.727311+0000","last_undegraded":"2024-09-20T19:20:19.727311+0000","last_fullsized":"2024-09-20T19:20:19.727311+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:58.028776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:58.028776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.888058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":109,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.725428+0000","last_change":"2024-09-20T19:19:51.256022+0000","last_active":"2024-09-20T19:20:19.725428+0000","last_peered":"2024-09-20T19:20:19.725428+0000","last_clean":"2024-09-20T19:20:19.725428+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:20:19.725428+0000","last_undegraded":"2024-09-20T19:20:19.725428+0000","last_fullsized":"2024-09-20T19:20:19.725428+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:51.255877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.255877+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.975626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-20T19:20:19.159881+0000","last_change":"2024-09-20T19:19:28.186599+0000","last_active":"2024-09-20T19:20:19.159881+0000","last_peered":"2024-09-20T19:20:19.159881+0000","last_clean":"2024-09-20T19:20:19.159881+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:20:19.159881+0000","last_undegraded":"2024-09-20T19:20:19.159881+0000","last_fullsized":"2024-09-20T19:20:19.159881+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.186507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.186507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.594221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877906981,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1310720,"data_stored":923705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234032,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1298432,"data_stored":923041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561082,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":708608,"data_stored":333764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855430,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31884,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713268,"statfs":{"total":95995035648,"available":95962386432,"internally_reserved":0,"allocated":716800,"data_stored":332751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084368,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":737280,"data_stored":333538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378718,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36044,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709108,"statfs":{"total":95995035648,"available":95958126592,"internally_reserved":0,"allocated":716800,"data_stored":333618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":729088,"data_stored":334245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708524,"statfs":{"total":95995035648,"available":95957528576,"internally_reserved":0,"allocated":1314816,"data_stored":923971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:20:29.594 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T19:20:29.594 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T19:20:29.594 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T19:20:29.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:30 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:30.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: audit 2024-09-20T19:20:28.867970+0000 mgr.y (mgr.24491) 96 : audit [DBG] from='client.14808 v1:172.21.15.42:0/3505323489' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: audit 2024-09-20T19:20:28.925253+0000 mgr.y (mgr.24491) 97 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.722843+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:30.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:30 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.725185+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:31 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:29.916939+0000 mgr.y (mgr.24491) 98 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:30.144187+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:30.146796+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.065694+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:31 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.067952+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:20:32.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:32 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:32 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:32 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:32.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:32 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:32.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.055855+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:32.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:32 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.059592+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:33 smithi162 bash[24557]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[32075]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:31.918243+0000 mgr.y (mgr.24491) 99 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.489722+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.f scrub starts 2024-09-20T19:20:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:33 smithi042 bash[22706]: cluster 2024-09-20T19:20:32.494007+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.f scrub ok 2024-09-20T19:20:34.445 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:34.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:34.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:34 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.480210+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:20:34.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.482596+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:20:34.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.798550+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:20:34.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:34.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:34 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.800782+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:35 smithi162 bash[24557]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[22706]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.799824+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.803224+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:33.919056+0000 mgr.y (mgr.24491) 100 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:34.772584+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:20:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:35.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:35 smithi042 bash[32075]: cluster 2024-09-20T19:20:34.776377+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:20:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:20:36.200 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T19:20:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:36 smithi162 bash[24557]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:36.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[22706]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.229566+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.231767+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-09-20T19:20:36.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:36.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:36 smithi042 bash[32075]: audit 2024-09-20T19:20:36.199707+0000 mon.a (mon.0) 956 : audit [DBG] from='client.? v1:172.21.15.42:0/2312514629' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-20T19:20:37.006 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T19:20:37.007 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T19:20:37.007 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-20T19:20:37.017 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-20T19:20:37.018 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-20-1028'}, 'ceph': {'conf': {'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}}, 'workunit': {'branch': 'wip-guits-main-2024-09-20-1028', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}} 2024-09-20T19:20:37.018 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-20T19:20:37.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-20T19:20:37.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:37 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:37.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:37 smithi162 bash[24557]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:37.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:37 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:37.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:37 smithi042 bash[32075]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:37 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:37 smithi042 bash[22706]: cluster 2024-09-20T19:20:35.919499+0000 mgr.y (mgr.24491) 101 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:38.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.740825+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.743175+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.781721+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.785439+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.041940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:38.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:38 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.044466+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:20:39.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:38 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:39.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:39 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:37.920942+0000 mgr.y (mgr.24491) 102 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.742891+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:38.745082+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.072923+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:39 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.076217+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:40 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:40.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: audit 2024-09-20T19:20:38.936608+0000 mgr.y (mgr.24491) 103 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.063102+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:20:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:40 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.067992+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[22706]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:41 smithi042 bash[32075]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.801895+0000 osd.5 (osd.5) 43 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.803837+0000 osd.5 (osd.5) 44 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:39.921635+0000 mgr.y (mgr.24491) 104 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.759114+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.e scrub starts 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:41 smithi162 bash[24557]: cluster 2024-09-20T19:20:40.763187+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.e scrub ok 2024-09-20T19:20:41.834 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:42 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.499702+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.502035+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: audit 2024-09-20T19:20:41.585310+0000 mon.a (mon.0) 957 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.800668+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:42.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:42 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.804137+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:20:43.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.671 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE Bus STOPPING 2024-09-20T19:20:43.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.782780+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.782884+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:41.922925+0000 mgr.y (mgr.24491) 105 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.328027+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.522042+0000 osd.1 (osd.1) 67 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.524387+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:20:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.065469+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:20:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:43 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.067410+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:20:43.929 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:20:43.929 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE Bus STOPPED 2024-09-20T19:20:43.929 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE Bus STARTING 2024-09-20T19:20:43.929 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE Serving on http://:::9283 2024-09-20T19:20:43.930 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:43 smithi042 bash[22958]: [20/Sep/2024:19:20:43] ENGINE Bus STARTED 2024-09-20T19:20:44.464 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-20T19:20:44.464 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T19:20:44.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:44.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:44 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:45.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:45.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.757417+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:20:45.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:45.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:42.759894+0000 osd.5 (osd.5) 48 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:20:45.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.661293+0000 mon.a (mon.0) 959 : audit [INF] from='client.24677 ' entity='client.admin' 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.681736+0000 mon.a (mon.0) 960 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.751786+0000 mon.a (mon.0) 961 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: audit 2024-09-20T19:20:43.752584+0000 mgr.y (mgr.24491) 106 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.859654+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.861686+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:45.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:43.923565+0000 mgr.y (mgr.24491) 107 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:45.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:45.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.047130+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:20:45.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:45.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:44 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.049333+0000 osd.3 (osd.3) 70 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:20:45.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[22706]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:45 smithi042 bash[32075]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:46.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:45 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:46.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:45 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.714959+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:20:46.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:45 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:46.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:45 smithi162 bash[24557]: cluster 2024-09-20T19:20:44.717830+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:20:46.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:20:47.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:46 smithi162 bash[24557]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:47.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:46 smithi162 bash[24557]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:47.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:46 smithi042 bash[32075]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:47.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:46 smithi042 bash[32075]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:47.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:46 smithi042 bash[22706]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:47.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:46 smithi042 bash[22706]: cluster 2024-09-20T19:20:45.924259+0000 mgr.y (mgr.24491) 108 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:47 smithi162 bash[24557]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[32075]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.531458+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.531565+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.658947+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.661177+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.688285+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.905582+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.910307+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:46.997921+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:47 smithi042 bash[22706]: cluster 2024-09-20T19:20:47.001309+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:20:48.773 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:48.947 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:48 smithi162 bash[24557]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:48.947 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:48 smithi162 bash[24557]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:49.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:48 smithi042 bash[32075]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:49.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:48 smithi042 bash[32075]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:49.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:48 smithi042 bash[22706]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:49.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:48 smithi042 bash[22706]: cluster 2024-09-20T19:20:47.925739+0000 mgr.y (mgr.24491) 109 : cluster [DBG] pgmap v75: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:49.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:48 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:49.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:49.983 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.984 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[22706]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:49.985 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:49.986 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:49 smithi042 bash[32075]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.663684+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.666570+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.830594+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:20:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.831665+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.934849+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: cluster 2024-09-20T19:20:48.936809+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:48.947349+0000 mgr.y (mgr.24491) 110 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.299482+0000 mon.a (mon.0) 963 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.305921+0000 mon.a (mon.0) 964 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.621001+0000 mon.a (mon.0) 965 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.631262+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.633625+0000 mon.a (mon.0) 967 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.635363+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:49 smithi162 bash[24557]: audit 2024-09-20T19:20:49.645614+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:20:50.344 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:20:50.345 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:50 smithi162 bash[24557]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.149 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.149 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.149 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.149 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[32075]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.150 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.659344+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.661410+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.886251+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.888383+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: cluster 2024-09-20T19:20:49.926409+0000 mgr.y (mgr.24491) 111 : cluster [DBG] pgmap v76: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.151 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:50 smithi042 bash[22706]: audit 2024-09-20T19:20:50.334423+0000 mgr.y (mgr.24491) 112 : audit [DBG] from='client.24683 v1:172.21.15.42:0/286643729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:51.152 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-20T19:20:49.926020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1006,"num_read_kb":863,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244500,"kb_used_data":7316,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716716,"statfs":{"total":767960285184,"available":767709917184,"internally_reserved":0,"allocated":7491584,"data_stored":4422731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004100"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.324594+0000","last_change":"2024-09-20T19:20:38.044596+0000","last_active":"2024-09-20T19:20:42.324594+0000","last_peered":"2024-09-20T19:20:42.324594+0000","last_clean":"2024-09-20T19:20:42.324594+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:20:42.324594+0000","last_undegraded":"2024-09-20T19:20:42.324594+0000","last_fullsized":"2024-09-20T19:20:42.324594+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.044475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.044475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.419278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685882+0000","last_change":"2024-09-20T19:19:17.698496+0000","last_active":"2024-09-20T19:20:46.685882+0000","last_peered":"2024-09-20T19:20:46.685882+0000","last_clean":"2024-09-20T19:20:46.685882+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:20:46.685882+0000","last_undegraded":"2024-09-20T19:20:46.685882+0000","last_fullsized":"2024-09-20T19:20:46.685882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.698427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.698427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.327077+0000","last_change":"2024-09-20T19:19:45.244647+0000","last_active":"2024-09-20T19:20:42.327077+0000","last_peered":"2024-09-20T19:20:42.327077+0000","last_clean":"2024-09-20T19:20:42.327077+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:20:42.327077+0000","last_undegraded":"2024-09-20T19:20:42.327077+0000","last_fullsized":"2024-09-20T19:20:42.327077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:45.244486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.244486+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.299507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732981+0000","last_change":"2024-09-20T19:20:33.800940+0000","last_active":"2024-09-20T19:20:46.732981+0000","last_peered":"2024-09-20T19:20:46.732981+0000","last_clean":"2024-09-20T19:20:46.732981+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:20:46.732981+0000","last_undegraded":"2024-09-20T19:20:46.732981+0000","last_fullsized":"2024-09-20T19:20:46.732981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.800794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.800794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.248084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":101,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329385+0000","last_change":"2024-09-20T19:20:31.068114+0000","last_active":"2024-09-20T19:20:42.329385+0000","last_peered":"2024-09-20T19:20:42.329385+0000","last_clean":"2024-09-20T19:20:42.329385+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:20:42.329385+0000","last_undegraded":"2024-09-20T19:20:42.329385+0000","last_fullsized":"2024-09-20T19:20:42.329385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:31.067964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:31.067964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:48.259004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732993+0000","last_change":"2024-09-20T19:20:34.776570+0000","last_active":"2024-09-20T19:20:46.732993+0000","last_peered":"2024-09-20T19:20:46.732993+0000","last_clean":"2024-09-20T19:20:46.732993+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:20:46.732993+0000","last_undegraded":"2024-09-20T19:20:46.732993+0000","last_fullsized":"2024-09-20T19:20:46.732993+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:34.776387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:34.776387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.596561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.901763+0000","last_change":"2024-09-20T19:20:42.524526+0000","last_active":"2024-09-20T19:20:46.901763+0000","last_peered":"2024-09-20T19:20:46.901763+0000","last_clean":"2024-09-20T19:20:46.901763+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:20:46.901763+0000","last_undegraded":"2024-09-20T19:20:46.901763+0000","last_fullsized":"2024-09-20T19:20:46.901763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.524397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.524397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:01.089547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733151+0000","last_change":"2024-09-20T19:19:32.223697+0000","last_active":"2024-09-20T19:20:46.733151+0000","last_peered":"2024-09-20T19:20:46.733151+0000","last_clean":"2024-09-20T19:20:46.733151+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:20:46.733151+0000","last_undegraded":"2024-09-20T19:20:46.733151+0000","last_fullsized":"2024-09-20T19:20:46.733151+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.223538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.223538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.453072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.691416+0000","last_change":"2024-09-20T19:19:59.335157+0000","last_active":"2024-09-20T19:20:46.691416+0000","last_peered":"2024-09-20T19:20:46.691416+0000","last_clean":"2024-09-20T19:20:46.691416+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:20:46.691416+0000","last_undegraded":"2024-09-20T19:20:46.691416+0000","last_fullsized":"2024-09-20T19:20:46.691416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.335040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.335040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.153396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733345+0000","last_change":"2024-09-20T19:20:41.804284+0000","last_active":"2024-09-20T19:20:46.733345+0000","last_peered":"2024-09-20T19:20:46.733345+0000","last_clean":"2024-09-20T19:20:46.733345+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:20:46.733345+0000","last_undegraded":"2024-09-20T19:20:46.733345+0000","last_fullsized":"2024-09-20T19:20:46.733345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:20:41.804146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.804146+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:55.773005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689572+0000","last_change":"2024-09-20T19:19:43.205222+0000","last_active":"2024-09-20T19:20:46.689572+0000","last_peered":"2024-09-20T19:20:46.689572+0000","last_clean":"2024-09-20T19:20:46.689572+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:20:46.689572+0000","last_undegraded":"2024-09-20T19:20:46.689572+0000","last_fullsized":"2024-09-20T19:20:46.689572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.205146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.205146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.462890+0000","last_change":"2024-09-20T19:19:32.084443+0000","last_active":"2024-09-20T19:20:42.462890+0000","last_peered":"2024-09-20T19:20:42.462890+0000","last_clean":"2024-09-20T19:20:42.462890+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:20:42.462890+0000","last_undegraded":"2024-09-20T19:20:42.462890+0000","last_fullsized":"2024-09-20T19:20:42.462890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.084319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.084319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.691272+0000","last_change":"2024-09-20T19:19:41.142126+0000","last_active":"2024-09-20T19:20:46.691272+0000","last_peered":"2024-09-20T19:20:46.691272+0000","last_clean":"2024-09-20T19:20:46.691272+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:20:46.691272+0000","last_undegraded":"2024-09-20T19:20:46.691272+0000","last_fullsized":"2024-09-20T19:20:46.691272+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.141975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.141975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.007349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":107,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733389+0000","last_change":"2024-09-20T19:19:42.188043+0000","last_active":"2024-09-20T19:20:46.733389+0000","last_peered":"2024-09-20T19:20:46.733389+0000","last_clean":"2024-09-20T19:20:46.733389+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:20:46.733389+0000","last_undegraded":"2024-09-20T19:20:46.733389+0000","last_fullsized":"2024-09-20T19:20:46.733389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:42.187756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:42.187756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.911434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.462084+0000","last_change":"2024-09-20T19:19:40.029794+0000","last_active":"2024-09-20T19:20:42.462084+0000","last_peered":"2024-09-20T19:20:42.462084+0000","last_clean":"2024-09-20T19:20:42.462084+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:20:42.462084+0000","last_undegraded":"2024-09-20T19:20:42.462084+0000","last_fullsized":"2024-09-20T19:20:42.462084+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.029674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.029674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.995856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684084+0000","last_change":"2024-09-20T19:19:54.798943+0000","last_active":"2024-09-20T19:20:46.684084+0000","last_peered":"2024-09-20T19:20:46.684084+0000","last_clean":"2024-09-20T19:20:46.684084+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:20:46.684084+0000","last_undegraded":"2024-09-20T19:20:46.684084+0000","last_fullsized":"2024-09-20T19:20:46.684084+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.798882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.798882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.457870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"77'17","reported_seq":119,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734044+0000","last_change":"2024-09-20T19:19:48.026945+0000","last_active":"2024-09-20T19:20:46.734044+0000","last_peered":"2024-09-20T19:20:46.734044+0000","last_clean":"2024-09-20T19:20:46.734044+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:20:46.734044+0000","last_undegraded":"2024-09-20T19:20:46.734044+0000","last_fullsized":"2024-09-20T19:20:46.734044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:19:48.026795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.026795+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.794994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690597+0000","last_change":"2024-09-20T19:20:05.282366+0000","last_active":"2024-09-20T19:20:46.690597+0000","last_peered":"2024-09-20T19:20:46.690597+0000","last_clean":"2024-09-20T19:20:46.690597+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:20:46.690597+0000","last_undegraded":"2024-09-20T19:20:46.690597+0000","last_fullsized":"2024-09-20T19:20:46.690597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:05.282293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:05.282293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.281996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902459+0000","last_change":"2024-09-20T19:19:56.425232+0000","last_active":"2024-09-20T19:20:46.902459+0000","last_peered":"2024-09-20T19:20:46.902459+0000","last_clean":"2024-09-20T19:20:46.902459+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:20:46.902459+0000","last_undegraded":"2024-09-20T19:20:46.902459+0000","last_fullsized":"2024-09-20T19:20:46.902459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.425156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.425156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:21.486886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690023+0000","last_change":"2024-09-20T19:19:46.074958+0000","last_active":"2024-09-20T19:20:46.690023+0000","last_peered":"2024-09-20T19:20:46.690023+0000","last_clean":"2024-09-20T19:20:46.690023+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:20:46.690023+0000","last_undegraded":"2024-09-20T19:20:46.690023+0000","last_fullsized":"2024-09-20T19:20:46.690023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:46.074887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.074887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.317322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"77'10","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734037+0000","last_change":"2024-09-20T19:19:43.146526+0000","last_active":"2024-09-20T19:20:46.734037+0000","last_peered":"2024-09-20T19:20:46.734037+0000","last_clean":"2024-09-20T19:20:46.734037+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:20:46.734037+0000","last_undegraded":"2024-09-20T19:20:46.734037+0000","last_fullsized":"2024-09-20T19:20:46.734037+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:43.146459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.146459+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903428+0000","last_change":"2024-09-20T19:19:54.442006+0000","last_active":"2024-09-20T19:20:46.903428+0000","last_peered":"2024-09-20T19:20:46.903428+0000","last_clean":"2024-09-20T19:20:46.903428+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:20:46.903428+0000","last_undegraded":"2024-09-20T19:20:46.903428+0000","last_fullsized":"2024-09-20T19:20:46.903428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.441955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.441955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.241390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":130,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689472+0000","last_change":"2024-09-20T19:19:45.038512+0000","last_active":"2024-09-20T19:20:46.689472+0000","last_peered":"2024-09-20T19:20:46.689472+0000","last_clean":"2024-09-20T19:20:46.689472+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:20:46.689472+0000","last_undegraded":"2024-09-20T19:20:46.689472+0000","last_fullsized":"2024-09-20T19:20:46.689472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:45.038377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.038377+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.979665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733847+0000","last_change":"2024-09-20T19:19:38.210845+0000","last_active":"2024-09-20T19:20:46.733847+0000","last_peered":"2024-09-20T19:20:46.733847+0000","last_clean":"2024-09-20T19:20:46.733847+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:20:46.733847+0000","last_undegraded":"2024-09-20T19:20:46.733847+0000","last_fullsized":"2024-09-20T19:20:46.733847+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:38.210713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:38.210713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.317890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"77'15","reported_seq":124,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903069+0000","last_change":"2024-09-20T19:20:32.494239+0000","last_active":"2024-09-20T19:20:46.903069+0000","last_peered":"2024-09-20T19:20:46.903069+0000","last_clean":"2024-09-20T19:20:46.903069+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:20:46.903069+0000","last_undegraded":"2024-09-20T19:20:46.903069+0000","last_fullsized":"2024-09-20T19:20:46.903069+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:20:32.494020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.494020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.833966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902023+0000","last_change":"2024-09-20T19:19:36.378407+0000","last_active":"2024-09-20T19:20:46.902023+0000","last_peered":"2024-09-20T19:20:46.902023+0000","last_clean":"2024-09-20T19:20:46.902023+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:20:46.902023+0000","last_undegraded":"2024-09-20T19:20:46.902023+0000","last_fullsized":"2024-09-20T19:20:46.902023+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.378285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.378285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":129,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734201+0000","last_change":"2024-09-20T19:19:50.966826+0000","last_active":"2024-09-20T19:20:46.734201+0000","last_peered":"2024-09-20T19:20:46.734201+0000","last_clean":"2024-09-20T19:20:46.734201+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:20:46.734201+0000","last_undegraded":"2024-09-20T19:20:46.734201+0000","last_fullsized":"2024-09-20T19:20:46.734201+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:50.966659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.966659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.209609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.464065+0000","last_change":"2024-09-20T19:19:34.013906+0000","last_active":"2024-09-20T19:20:42.464065+0000","last_peered":"2024-09-20T19:20:42.464065+0000","last_clean":"2024-09-20T19:20:42.464065+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:20:42.464065+0000","last_undegraded":"2024-09-20T19:20:42.464065+0000","last_fullsized":"2024-09-20T19:20:42.464065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.013787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.013787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.149084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"77'11","reported_seq":118,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734325+0000","last_change":"2024-09-20T19:20:40.763340+0000","last_active":"2024-09-20T19:20:46.734325+0000","last_peered":"2024-09-20T19:20:46.734325+0000","last_clean":"2024-09-20T19:20:46.734325+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:20:46.734325+0000","last_undegraded":"2024-09-20T19:20:46.734325+0000","last_fullsized":"2024-09-20T19:20:46.734325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:40.763197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.763197+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:47.787913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689754+0000","last_change":"2024-09-20T19:19:48.281994+0000","last_active":"2024-09-20T19:20:46.689754+0000","last_peered":"2024-09-20T19:20:46.689754+0000","last_clean":"2024-09-20T19:20:46.689754+0000","last_became_active":"2024-09-20T19:18:07.269527+0000","last_became_peered":"2024-09-20T19:18:07.269527+0000","last_unstale":"2024-09-20T19:20:46.689754+0000","last_undegraded":"2024-09-20T19:20:46.689754+0000","last_fullsized":"2024-09-20T19:20:46.689754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.281832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.281832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.535302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.465594+0000","last_change":"2024-09-20T19:19:34.966265+0000","last_active":"2024-09-20T19:20:42.465594+0000","last_peered":"2024-09-20T19:20:42.465594+0000","last_clean":"2024-09-20T19:20:42.465594+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:20:42.465594+0000","last_undegraded":"2024-09-20T19:20:42.465594+0000","last_fullsized":"2024-09-20T19:20:42.465594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.966141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.966141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.384654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.327889+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:20:42.327889+0000","last_peered":"2024-09-20T19:20:42.327889+0000","last_clean":"2024-09-20T19:20:42.327889+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:20:42.327889+0000","last_undegraded":"2024-09-20T19:20:42.327889+0000","last_fullsized":"2024-09-20T19:20:42.327889+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.064548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"77'14","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733599+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:20:46.733599+0000","last_peered":"2024-09-20T19:20:46.733599+0000","last_clean":"2024-09-20T19:20:46.733599+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:20:46.733599+0000","last_undegraded":"2024-09-20T19:20:46.733599+0000","last_fullsized":"2024-09-20T19:20:46.733599+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.837736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330785+0000","last_change":"2024-09-20T19:20:16.657110+0000","last_active":"2024-09-20T19:20:42.330785+0000","last_peered":"2024-09-20T19:20:42.330785+0000","last_clean":"2024-09-20T19:20:42.330785+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:20:42.330785+0000","last_undegraded":"2024-09-20T19:20:42.330785+0000","last_fullsized":"2024-09-20T19:20:42.330785+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.656960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.656960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:43.026972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":127,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.322951+0000","last_change":"2024-09-20T19:19:37.178863+0000","last_active":"2024-09-20T19:20:42.322951+0000","last_peered":"2024-09-20T19:20:42.322951+0000","last_clean":"2024-09-20T19:20:42.322951+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:20:42.322951+0000","last_undegraded":"2024-09-20T19:20:42.322951+0000","last_fullsized":"2024-09-20T19:20:42.322951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.178724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.178724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.727526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690224+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:20:46.690224+0000","last_peered":"2024-09-20T19:20:46.690224+0000","last_clean":"2024-09-20T19:20:46.690224+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:20:46.690224+0000","last_undegraded":"2024-09-20T19:20:46.690224+0000","last_fullsized":"2024-09-20T19:20:46.690224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:18.443029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"77'18","reported_seq":129,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.323188+0000","last_change":"2024-09-20T19:20:40.068159+0000","last_active":"2024-09-20T19:20:42.323188+0000","last_peered":"2024-09-20T19:20:42.323188+0000","last_clean":"2024-09-20T19:20:42.323188+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:20:42.323188+0000","last_undegraded":"2024-09-20T19:20:42.323188+0000","last_fullsized":"2024-09-20T19:20:42.323188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:20:40.068007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.068007+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.669287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":96,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903431+0000","last_change":"2024-09-20T19:19:27.488807+0000","last_active":"2024-09-20T19:20:46.903431+0000","last_peered":"2024-09-20T19:20:46.903431+0000","last_clean":"2024-09-20T19:20:46.903431+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:20:46.903431+0000","last_undegraded":"2024-09-20T19:20:46.903431+0000","last_fullsized":"2024-09-20T19:20:46.903431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:27.488644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.488644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732951+0000","last_change":"2024-09-20T19:19:44.113106+0000","last_active":"2024-09-20T19:20:46.732951+0000","last_peered":"2024-09-20T19:20:46.732951+0000","last_clean":"2024-09-20T19:20:46.732951+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:20:46.732951+0000","last_undegraded":"2024-09-20T19:20:46.732951+0000","last_fullsized":"2024-09-20T19:20:46.732951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.112985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.112985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.242746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733216+0000","last_change":"2024-09-20T19:19:48.997145+0000","last_active":"2024-09-20T19:20:46.733216+0000","last_peered":"2024-09-20T19:20:46.733216+0000","last_clean":"2024-09-20T19:20:46.733216+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:20:46.733216+0000","last_undegraded":"2024-09-20T19:20:46.733216+0000","last_fullsized":"2024-09-20T19:20:46.733216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.996997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.996997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.494961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"77'19","reported_seq":127,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685181+0000","last_change":"2024-09-20T19:19:27.727451+0000","last_active":"2024-09-20T19:20:46.685181+0000","last_peered":"2024-09-20T19:20:46.685181+0000","last_clean":"2024-09-20T19:20:46.685181+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:20:46.685181+0000","last_undegraded":"2024-09-20T19:20:46.685181+0000","last_fullsized":"2024-09-20T19:20:46.685181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.727294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.727294+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.582205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689376+0000","last_change":"2024-09-20T19:19:56.998769+0000","last_active":"2024-09-20T19:20:46.689376+0000","last_peered":"2024-09-20T19:20:46.689376+0000","last_clean":"2024-09-20T19:20:46.689376+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:20:46.689376+0000","last_undegraded":"2024-09-20T19:20:46.689376+0000","last_fullsized":"2024-09-20T19:20:46.689376+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.169199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.169199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.901847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329842+0000","last_change":"2024-09-20T19:19:41.871990+0000","last_active":"2024-09-20T19:20:42.329842+0000","last_peered":"2024-09-20T19:20:42.329842+0000","last_clean":"2024-09-20T19:20:42.329842+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:20:42.329842+0000","last_undegraded":"2024-09-20T19:20:42.329842+0000","last_fullsized":"2024-09-20T19:20:42.329842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.871854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.871854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.298167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.901551+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:20:46.901551+0000","last_peered":"2024-09-20T19:20:46.901551+0000","last_clean":"2024-09-20T19:20:46.901551+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:20:46.901551+0000","last_undegraded":"2024-09-20T19:20:46.901551+0000","last_fullsized":"2024-09-20T19:20:46.901551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:29.401329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"77'10","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902997+0000","last_change":"2024-09-20T19:19:40.461604+0000","last_active":"2024-09-20T19:20:46.902997+0000","last_peered":"2024-09-20T19:20:46.902997+0000","last_clean":"2024-09-20T19:20:46.902997+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:20:46.902997+0000","last_undegraded":"2024-09-20T19:20:46.902997+0000","last_fullsized":"2024-09-20T19:20:46.902997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:40.461532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.461532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:08.703461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903353+0000","last_change":"2024-09-20T19:19:34.429306+0000","last_active":"2024-09-20T19:20:46.903353+0000","last_peered":"2024-09-20T19:20:46.903353+0000","last_clean":"2024-09-20T19:20:46.903353+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:20:46.903353+0000","last_undegraded":"2024-09-20T19:20:46.903353+0000","last_fullsized":"2024-09-20T19:20:46.903353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.429125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.429125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.671437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":129,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684900+0000","last_change":"2024-09-20T19:19:37.807674+0000","last_active":"2024-09-20T19:20:46.684900+0000","last_peered":"2024-09-20T19:20:46.684900+0000","last_clean":"2024-09-20T19:20:46.684900+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:20:46.684900+0000","last_undegraded":"2024-09-20T19:20:46.684900+0000","last_fullsized":"2024-09-20T19:20:46.684900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.807547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.807547+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.829611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685049+0000","last_change":"2024-09-20T19:19:31.793749+0000","last_active":"2024-09-20T19:20:46.685049+0000","last_peered":"2024-09-20T19:20:46.685049+0000","last_clean":"2024-09-20T19:20:46.685049+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:20:46.685049+0000","last_undegraded":"2024-09-20T19:20:46.685049+0000","last_fullsized":"2024-09-20T19:20:46.685049+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.793611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.793611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'30","reported_seq":173,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903055+0000","last_change":"2024-09-20T19:19:46.395559+0000","last_active":"2024-09-20T19:20:46.903055+0000","last_peered":"2024-09-20T19:20:46.903055+0000","last_clean":"2024-09-20T19:20:46.903055+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:20:46.903055+0000","last_undegraded":"2024-09-20T19:20:46.903055+0000","last_fullsized":"2024-09-20T19:20:46.903055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:19:46.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.395404+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.463509+0000","last_change":"2024-09-20T19:19:33.057715+0000","last_active":"2024-09-20T19:20:42.463509+0000","last_peered":"2024-09-20T19:20:42.463509+0000","last_clean":"2024-09-20T19:20:42.463509+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:20:42.463509+0000","last_undegraded":"2024-09-20T19:20:42.463509+0000","last_fullsized":"2024-09-20T19:20:42.463509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.057567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.057567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:43.036699+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685329+0000","last_change":"2024-09-20T19:19:56.854517+0000","last_active":"2024-09-20T19:20:46.685329+0000","last_peered":"2024-09-20T19:20:46.685329+0000","last_clean":"2024-09-20T19:20:46.685329+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:20:46.685329+0000","last_undegraded":"2024-09-20T19:20:46.685329+0000","last_fullsized":"2024-09-20T19:20:46.685329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.854461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.854461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.519787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.328964+0000","last_change":"2024-09-20T19:19:50.253935+0000","last_active":"2024-09-20T19:20:42.328964+0000","last_peered":"2024-09-20T19:20:42.328964+0000","last_clean":"2024-09-20T19:20:42.328964+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:20:42.328964+0000","last_undegraded":"2024-09-20T19:20:42.328964+0000","last_fullsized":"2024-09-20T19:20:42.328964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.253727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.253727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.038762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"77'13","reported_seq":34,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689579+0000","last_change":"2024-09-20T19:19:57.013393+0000","last_active":"2024-09-20T19:20:46.689579+0000","last_peered":"2024-09-20T19:20:46.689579+0000","last_clean":"2024-09-20T19:20:46.689579+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:20:46.689579+0000","last_undegraded":"2024-09-20T19:20:46.689579+0000","last_fullsized":"2024-09-20T19:20:46.689579+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:19:51.410598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.410598+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:12.734213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":246,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.901975+0000","last_change":"2024-09-20T19:19:19.441522+0000","last_active":"2024-09-20T19:20:46.901975+0000","last_peered":"2024-09-20T19:20:46.901975+0000","last_clean":"2024-09-20T19:20:46.901975+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:20:46.901975+0000","last_undegraded":"2024-09-20T19:20:46.901975+0000","last_fullsized":"2024-09-20T19:20:46.901975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:19:19.441341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:19:19.441341+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688519+0000","last_change":"2024-09-20T19:19:49.990251+0000","last_active":"2024-09-20T19:20:46.688519+0000","last_peered":"2024-09-20T19:20:46.688519+0000","last_clean":"2024-09-20T19:20:46.688519+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:20:46.688519+0000","last_undegraded":"2024-09-20T19:20:46.688519+0000","last_fullsized":"2024-09-20T19:20:46.688519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:49.990058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:49.990058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.407192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689955+0000","last_change":"2024-09-20T19:20:30.146960+0000","last_active":"2024-09-20T19:20:46.689955+0000","last_peered":"2024-09-20T19:20:46.689955+0000","last_clean":"2024-09-20T19:20:46.689955+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:20:46.689955+0000","last_undegraded":"2024-09-20T19:20:46.689955+0000","last_fullsized":"2024-09-20T19:20:46.689955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:30.146806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:30.146806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.691081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690826+0000","last_change":"2024-09-20T19:19:55.305695+0000","last_active":"2024-09-20T19:20:46.690826+0000","last_peered":"2024-09-20T19:20:46.690826+0000","last_clean":"2024-09-20T19:20:46.690826+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:20:46.690826+0000","last_undegraded":"2024-09-20T19:20:46.690826+0000","last_fullsized":"2024-09-20T19:20:46.690826+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:55.305573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.305573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.195185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'12","reported_seq":110,"reported_epoch":107,"state":"active+clean+scrubbing","last_fresh":"2024-09-20T19:20:46.686049+0000","last_change":"2024-09-20T19:20:46.898291+0000","last_active":"2024-09-20T19:20:46.686049+0000","last_peered":"2024-09-20T19:20:46.686049+0000","last_clean":"2024-09-20T19:20:46.686049+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:20:46.686049+0000","last_undegraded":"2024-09-20T19:20:46.686049+0000","last_fullsized":"2024-09-20T19:20:46.686049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:23.739633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.739633+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684474+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:20:46.684474+0000","last_peered":"2024-09-20T19:20:46.684474+0000","last_clean":"2024-09-20T19:20:46.684474+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:20:46.684474+0000","last_undegraded":"2024-09-20T19:20:46.684474+0000","last_fullsized":"2024-09-20T19:20:46.684474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.851755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689832+0000","last_change":"2024-09-20T19:19:44.240584+0000","last_active":"2024-09-20T19:20:46.689832+0000","last_peered":"2024-09-20T19:20:46.689832+0000","last_clean":"2024-09-20T19:20:46.689832+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:20:46.689832+0000","last_undegraded":"2024-09-20T19:20:46.689832+0000","last_fullsized":"2024-09-20T19:20:46.689832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.240462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.240462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.731548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.760023+0000","last_change":"2024-09-20T19:20:42.760023+0000","last_active":"2024-09-20T19:20:42.760023+0000","last_peered":"2024-09-20T19:20:42.760023+0000","last_clean":"2024-09-20T19:20:42.760023+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:20:42.760023+0000","last_undegraded":"2024-09-20T19:20:42.760023+0000","last_fullsized":"2024-09-20T19:20:42.760023+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.759902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.759902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.827837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902189+0000","last_change":"2024-09-20T19:20:41.502208+0000","last_active":"2024-09-20T19:20:46.902189+0000","last_peered":"2024-09-20T19:20:46.902189+0000","last_clean":"2024-09-20T19:20:46.902189+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:20:46.902189+0000","last_undegraded":"2024-09-20T19:20:46.902189+0000","last_fullsized":"2024-09-20T19:20:46.902189+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:41.502048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.502048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:05.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"77'16","reported_seq":142,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330365+0000","last_change":"2024-09-20T19:19:39.901351+0000","last_active":"2024-09-20T19:20:42.330365+0000","last_peered":"2024-09-20T19:20:42.330365+0000","last_clean":"2024-09-20T19:20:42.330365+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:20:42.330365+0000","last_undegraded":"2024-09-20T19:20:42.330365+0000","last_fullsized":"2024-09-20T19:20:42.330365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:19:39.901171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.901171+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.526242+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.324172+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:20:42.324172+0000","last_peered":"2024-09-20T19:20:42.324172+0000","last_clean":"2024-09-20T19:20:42.324172+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:20:42.324172+0000","last_undegraded":"2024-09-20T19:20:42.324172+0000","last_fullsized":"2024-09-20T19:20:42.324172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.492805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.462640+0000","last_change":"2024-09-20T19:19:07.170908+0000","last_active":"2024-09-20T19:20:42.462640+0000","last_peered":"2024-09-20T19:20:42.462640+0000","last_clean":"2024-09-20T19:20:42.462640+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:20:42.462640+0000","last_undegraded":"2024-09-20T19:20:42.462640+0000","last_fullsized":"2024-09-20T19:20:42.462640+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:07.170760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:07.170760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":142,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690101+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:20:46.690101+0000","last_peered":"2024-09-20T19:20:46.690101+0000","last_clean":"2024-09-20T19:20:46.690101+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:20:46.690101+0000","last_undegraded":"2024-09-20T19:20:46.690101+0000","last_fullsized":"2024-09-20T19:20:46.690101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:27.954958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690535+0000","last_change":"2024-09-20T19:19:26.163243+0000","last_active":"2024-09-20T19:20:46.690535+0000","last_peered":"2024-09-20T19:20:46.690535+0000","last_clean":"2024-09-20T19:20:46.690535+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:20:46.690535+0000","last_undegraded":"2024-09-20T19:20:46.690535+0000","last_fullsized":"2024-09-20T19:20:46.690535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.163172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.163172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.703757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.465856+0000","last_change":"2024-09-20T19:20:39.803967+0000","last_active":"2024-09-20T19:20:42.465856+0000","last_peered":"2024-09-20T19:20:42.465856+0000","last_clean":"2024-09-20T19:20:42.465856+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:20:42.465856+0000","last_undegraded":"2024-09-20T19:20:42.465856+0000","last_fullsized":"2024-09-20T19:20:42.465856+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:39.803847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.803847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.060059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"77'19","reported_seq":119,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330464+0000","last_change":"2024-09-20T19:19:27.036130+0000","last_active":"2024-09-20T19:20:42.330464+0000","last_peered":"2024-09-20T19:20:42.330464+0000","last_clean":"2024-09-20T19:20:42.330464+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:20:42.330464+0000","last_undegraded":"2024-09-20T19:20:42.330464+0000","last_fullsized":"2024-09-20T19:20:42.330464+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.036022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.036022+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.263159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":101,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689670+0000","last_change":"2024-09-20T19:20:35.231905+0000","last_active":"2024-09-20T19:20:46.689670+0000","last_peered":"2024-09-20T19:20:46.689670+0000","last_clean":"2024-09-20T19:20:46.689670+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:20:46.689670+0000","last_undegraded":"2024-09-20T19:20:46.689670+0000","last_fullsized":"2024-09-20T19:20:46.689670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:35.231776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:35.231776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:03.117077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688349+0000","last_change":"2024-09-20T19:19:43.031914+0000","last_active":"2024-09-20T19:20:46.688349+0000","last_peered":"2024-09-20T19:20:46.688349+0000","last_clean":"2024-09-20T19:20:46.688349+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:20:46.688349+0000","last_undegraded":"2024-09-20T19:20:46.688349+0000","last_fullsized":"2024-09-20T19:20:46.688349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.031768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.031768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.442373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689989+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:20:46.689989+0000","last_peered":"2024-09-20T19:20:46.689989+0000","last_clean":"2024-09-20T19:20:46.689989+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:20:46.689989+0000","last_undegraded":"2024-09-20T19:20:46.689989+0000","last_fullsized":"2024-09-20T19:20:46.689989+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.126953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"77'9","reported_seq":107,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685058+0000","last_change":"2024-09-20T19:19:46.791747+0000","last_active":"2024-09-20T19:20:46.685058+0000","last_peered":"2024-09-20T19:20:46.685058+0000","last_clean":"2024-09-20T19:20:46.685058+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:20:46.685058+0000","last_undegraded":"2024-09-20T19:20:46.685058+0000","last_fullsized":"2024-09-20T19:20:46.685058+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.791592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.791592+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.994746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.904191+0000","last_change":"2024-09-20T19:19:35.390225+0000","last_active":"2024-09-20T19:20:46.904191+0000","last_peered":"2024-09-20T19:20:46.904191+0000","last_clean":"2024-09-20T19:20:46.904191+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:20:46.904191+0000","last_undegraded":"2024-09-20T19:20:46.904191+0000","last_fullsized":"2024-09-20T19:20:46.904191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.390064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.390064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.228348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688283+0000","last_change":"2024-09-20T19:19:41.044575+0000","last_active":"2024-09-20T19:20:46.688283+0000","last_peered":"2024-09-20T19:20:46.688283+0000","last_clean":"2024-09-20T19:20:46.688283+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:20:46.688283+0000","last_undegraded":"2024-09-20T19:20:46.688283+0000","last_fullsized":"2024-09-20T19:20:46.688283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.044427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.044427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.962646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":84,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684255+0000","last_change":"2024-09-20T19:19:52.810445+0000","last_active":"2024-09-20T19:20:46.684255+0000","last_peered":"2024-09-20T19:20:46.684255+0000","last_clean":"2024-09-20T19:20:46.684255+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:20:46.684255+0000","last_undegraded":"2024-09-20T19:20:46.684255+0000","last_fullsized":"2024-09-20T19:20:46.684255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:19:52.810385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:52.810385+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.122928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"77'15","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.458971+0000","last_change":"2024-09-20T19:19:44.046132+0000","last_active":"2024-09-20T19:20:42.458971+0000","last_peered":"2024-09-20T19:20:42.458971+0000","last_clean":"2024-09-20T19:20:42.458971+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:20:42.458971+0000","last_undegraded":"2024-09-20T19:20:42.458971+0000","last_fullsized":"2024-09-20T19:20:42.458971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:44.045925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.045925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.259093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":101,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690323+0000","last_change":"2024-09-20T19:19:21.207668+0000","last_active":"2024-09-20T19:20:46.690323+0000","last_peered":"2024-09-20T19:20:46.690323+0000","last_clean":"2024-09-20T19:20:46.690323+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:20:46.690323+0000","last_undegraded":"2024-09-20T19:20:46.690323+0000","last_fullsized":"2024-09-20T19:20:46.690323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:21.207433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:21.207433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":137,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689590+0000","last_change":"2024-09-20T19:20:25.228464+0000","last_active":"2024-09-20T19:20:46.689590+0000","last_peered":"2024-09-20T19:20:46.689590+0000","last_clean":"2024-09-20T19:20:46.689590+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:20:46.689590+0000","last_undegraded":"2024-09-20T19:20:46.689590+0000","last_fullsized":"2024-09-20T19:20:46.689590+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:25.228325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.228325+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.437018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.462600+0000","last_change":"2024-09-20T19:19:36.943615+0000","last_active":"2024-09-20T19:20:42.462600+0000","last_peered":"2024-09-20T19:20:42.462600+0000","last_clean":"2024-09-20T19:20:42.462600+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:20:42.462600+0000","last_undegraded":"2024-09-20T19:20:42.462600+0000","last_fullsized":"2024-09-20T19:20:42.462600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.943527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.943527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.968462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"77'12","reported_seq":141,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733507+0000","last_change":"2024-09-20T19:20:16.852586+0000","last_active":"2024-09-20T19:20:46.733507+0000","last_peered":"2024-09-20T19:20:46.733507+0000","last_clean":"2024-09-20T19:20:46.733507+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:20:46.733507+0000","last_undegraded":"2024-09-20T19:20:46.733507+0000","last_fullsized":"2024-09-20T19:20:46.733507+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:16.852450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.852450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.371990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":119,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734369+0000","last_change":"2024-09-20T19:20:37.785586+0000","last_active":"2024-09-20T19:20:46.734369+0000","last_peered":"2024-09-20T19:20:46.734369+0000","last_clean":"2024-09-20T19:20:46.734369+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:20:46.734369+0000","last_undegraded":"2024-09-20T19:20:46.734369+0000","last_fullsized":"2024-09-20T19:20:46.734369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:20:37.785449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.785449+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:49.952598+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.687955+0000","last_change":"2024-09-20T19:19:51.013009+0000","last_active":"2024-09-20T19:20:46.687955+0000","last_peered":"2024-09-20T19:20:46.687955+0000","last_clean":"2024-09-20T19:20:46.687955+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:20:46.687955+0000","last_undegraded":"2024-09-20T19:20:46.687955+0000","last_fullsized":"2024-09-20T19:20:46.687955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.012906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.012906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.429741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.324977+0000","last_change":"2024-09-20T19:20:28.004917+0000","last_active":"2024-09-20T19:20:42.324977+0000","last_peered":"2024-09-20T19:20:42.324977+0000","last_clean":"2024-09-20T19:20:42.324977+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:20:42.324977+0000","last_undegraded":"2024-09-20T19:20:42.324977+0000","last_fullsized":"2024-09-20T19:20:42.324977+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.004784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.004784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.093046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685941+0000","last_change":"2024-09-20T19:20:33.803376+0000","last_active":"2024-09-20T19:20:46.685941+0000","last_peered":"2024-09-20T19:20:46.685941+0000","last_clean":"2024-09-20T19:20:46.685941+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:20:46.685941+0000","last_undegraded":"2024-09-20T19:20:46.685941+0000","last_fullsized":"2024-09-20T19:20:46.685941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:33.803234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.803234+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:56.407084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688073+0000","last_change":"2024-09-20T19:19:28.174295+0000","last_active":"2024-09-20T19:20:46.688073+0000","last_peered":"2024-09-20T19:20:46.688073+0000","last_clean":"2024-09-20T19:20:46.688073+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:20:46.688073+0000","last_undegraded":"2024-09-20T19:20:46.688073+0000","last_fullsized":"2024-09-20T19:20:46.688073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.174157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.174157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.549082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329829+0000","last_change":"2024-09-20T19:19:57.060943+0000","last_active":"2024-09-20T19:20:42.329829+0000","last_peered":"2024-09-20T19:20:42.329829+0000","last_clean":"2024-09-20T19:20:42.329829+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:20:42.329829+0000","last_undegraded":"2024-09-20T19:20:42.329829+0000","last_fullsized":"2024-09-20T19:20:42.329829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.060887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.060887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.058027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688929+0000","last_change":"2024-09-20T19:19:54.030064+0000","last_active":"2024-09-20T19:20:46.688929+0000","last_peered":"2024-09-20T19:20:46.688929+0000","last_clean":"2024-09-20T19:20:46.688929+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:20:46.688929+0000","last_undegraded":"2024-09-20T19:20:46.688929+0000","last_fullsized":"2024-09-20T19:20:46.688929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.029937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.029937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.274260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329843+0000","last_change":"2024-09-20T19:20:37.743313+0000","last_active":"2024-09-20T19:20:42.329843+0000","last_peered":"2024-09-20T19:20:42.329843+0000","last_clean":"2024-09-20T19:20:42.329843+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:20:42.329843+0000","last_undegraded":"2024-09-20T19:20:42.329843+0000","last_fullsized":"2024-09-20T19:20:42.329843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:37.743188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.743188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:40.122912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330208+0000","last_change":"2024-09-20T19:19:29.203589+0000","last_active":"2024-09-20T19:20:42.330208+0000","last_peered":"2024-09-20T19:20:42.330208+0000","last_clean":"2024-09-20T19:20:42.330208+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:20:42.330208+0000","last_undegraded":"2024-09-20T19:20:42.330208+0000","last_fullsized":"2024-09-20T19:20:42.330208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:19:29.203410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:29.203410+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.144800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.464576+0000","last_change":"2024-09-20T19:19:43.040517+0000","last_active":"2024-09-20T19:20:42.464576+0000","last_peered":"2024-09-20T19:20:42.464576+0000","last_clean":"2024-09-20T19:20:42.464576+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:20:42.464576+0000","last_undegraded":"2024-09-20T19:20:42.464576+0000","last_fullsized":"2024-09-20T19:20:42.464576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.040460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.040460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:10.254103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690818+0000","last_change":"2024-09-20T19:19:56.343379+0000","last_active":"2024-09-20T19:20:46.690818+0000","last_peered":"2024-09-20T19:20:46.690818+0000","last_clean":"2024-09-20T19:20:46.690818+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:20:46.690818+0000","last_undegraded":"2024-09-20T19:20:46.690818+0000","last_fullsized":"2024-09-20T19:20:46.690818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.343254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.343254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.336373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330166+0000","last_change":"2024-09-20T19:20:32.059785+0000","last_active":"2024-09-20T19:20:42.330166+0000","last_peered":"2024-09-20T19:20:42.330166+0000","last_clean":"2024-09-20T19:20:42.330166+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:20:42.330166+0000","last_undegraded":"2024-09-20T19:20:42.330166+0000","last_fullsized":"2024-09-20T19:20:42.330166+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:32.059603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.059603+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.930875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.463434+0000","last_change":"2024-09-20T19:19:28.086132+0000","last_active":"2024-09-20T19:20:42.463434+0000","last_peered":"2024-09-20T19:20:42.463434+0000","last_clean":"2024-09-20T19:20:42.463434+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:20:42.463434+0000","last_undegraded":"2024-09-20T19:20:42.463434+0000","last_fullsized":"2024-09-20T19:20:42.463434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.086004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.086004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.650446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":126,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.464627+0000","last_change":"2024-09-20T19:19:27.113335+0000","last_active":"2024-09-20T19:20:42.464627+0000","last_peered":"2024-09-20T19:20:42.464627+0000","last_clean":"2024-09-20T19:20:42.464627+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:20:42.464627+0000","last_undegraded":"2024-09-20T19:20:42.464627+0000","last_fullsized":"2024-09-20T19:20:42.464627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.113211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.113211+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.889024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685348+0000","last_change":"2024-09-20T19:19:40.871513+0000","last_active":"2024-09-20T19:20:46.685348+0000","last_peered":"2024-09-20T19:20:46.685348+0000","last_clean":"2024-09-20T19:20:46.685348+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:20:46.685348+0000","last_undegraded":"2024-09-20T19:20:46.685348+0000","last_fullsized":"2024-09-20T19:20:46.685348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.871447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.871447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":104,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.464594+0000","last_change":"2024-09-20T19:19:46.036288+0000","last_active":"2024-09-20T19:20:42.464594+0000","last_peered":"2024-09-20T19:20:42.464594+0000","last_clean":"2024-09-20T19:20:42.464594+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:20:42.464594+0000","last_undegraded":"2024-09-20T19:20:42.464594+0000","last_fullsized":"2024-09-20T19:20:42.464594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.036143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.036143+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.480915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685440+0000","last_change":"2024-09-20T19:19:50.860419+0000","last_active":"2024-09-20T19:20:46.685440+0000","last_peered":"2024-09-20T19:20:46.685440+0000","last_clean":"2024-09-20T19:20:46.685440+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:20:46.685440+0000","last_undegraded":"2024-09-20T19:20:46.685440+0000","last_fullsized":"2024-09-20T19:20:46.685440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.860284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.860284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.470934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":127,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329468+0000","last_change":"2024-09-20T19:20:20.964445+0000","last_active":"2024-09-20T19:20:42.329468+0000","last_peered":"2024-09-20T19:20:42.329468+0000","last_clean":"2024-09-20T19:20:42.329468+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:20:42.329468+0000","last_undegraded":"2024-09-20T19:20:42.329468+0000","last_fullsized":"2024-09-20T19:20:42.329468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:20.964261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:20.964261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.805123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734139+0000","last_change":"2024-09-20T19:20:43.861815+0000","last_active":"2024-09-20T19:20:46.734139+0000","last_peered":"2024-09-20T19:20:46.734139+0000","last_clean":"2024-09-20T19:20:46.734139+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:20:46.734139+0000","last_undegraded":"2024-09-20T19:20:46.734139+0000","last_fullsized":"2024-09-20T19:20:46.734139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:43.861696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.861696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:11.874804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902714+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:20:46.902714+0000","last_peered":"2024-09-20T19:20:46.902714+0000","last_clean":"2024-09-20T19:20:46.902714+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:20:46.902714+0000","last_undegraded":"2024-09-20T19:20:46.902714+0000","last_fullsized":"2024-09-20T19:20:46.902714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:32.267574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330842+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:20:42.330842+0000","last_peered":"2024-09-20T19:20:42.330842+0000","last_clean":"2024-09-20T19:20:42.330842+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:20:42.330842+0000","last_undegraded":"2024-09-20T19:20:42.330842+0000","last_fullsized":"2024-09-20T19:20:42.330842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:33.835611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.325221+0000","last_change":"2024-09-20T19:19:47.212665+0000","last_active":"2024-09-20T19:20:42.325221+0000","last_peered":"2024-09-20T19:20:42.325221+0000","last_clean":"2024-09-20T19:20:42.325221+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:20:42.325221+0000","last_undegraded":"2024-09-20T19:20:42.325221+0000","last_fullsized":"2024-09-20T19:20:42.325221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:47.212542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.212542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.674953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.683637+0000","last_change":"2024-09-20T19:19:57.895992+0000","last_active":"2024-09-20T19:20:46.683637+0000","last_peered":"2024-09-20T19:20:46.683637+0000","last_clean":"2024-09-20T19:20:46.683637+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:20:46.683637+0000","last_undegraded":"2024-09-20T19:20:46.683637+0000","last_fullsized":"2024-09-20T19:20:46.683637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.895923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.895923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.375254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733527+0000","last_change":"2024-09-20T19:19:27.179523+0000","last_active":"2024-09-20T19:20:46.733527+0000","last_peered":"2024-09-20T19:20:46.733527+0000","last_clean":"2024-09-20T19:20:46.733527+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:20:46.733527+0000","last_undegraded":"2024-09-20T19:20:46.733527+0000","last_fullsized":"2024-09-20T19:20:46.733527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.179398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.179398+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.909162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":111,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.904086+0000","last_change":"2024-09-20T19:20:17.522125+0000","last_active":"2024-09-20T19:20:46.904086+0000","last_peered":"2024-09-20T19:20:46.904086+0000","last_clean":"2024-09-20T19:20:46.904086+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:20:46.904086+0000","last_undegraded":"2024-09-20T19:20:46.904086+0000","last_fullsized":"2024-09-20T19:20:46.904086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:17.521947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:17.521947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.763307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902242+0000","last_change":"2024-09-20T19:19:48.434045+0000","last_active":"2024-09-20T19:20:46.902242+0000","last_peered":"2024-09-20T19:20:46.902242+0000","last_clean":"2024-09-20T19:20:46.902242+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:20:46.902242+0000","last_undegraded":"2024-09-20T19:20:46.902242+0000","last_fullsized":"2024-09-20T19:20:46.902242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.433915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.433915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.655605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.322479+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:20:42.322479+0000","last_peered":"2024-09-20T19:20:42.322479+0000","last_clean":"2024-09-20T19:20:42.322479+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:20:42.322479+0000","last_undegraded":"2024-09-20T19:20:42.322479+0000","last_fullsized":"2024-09-20T19:20:42.322479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.984866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":103,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.322915+0000","last_change":"2024-09-20T19:20:39.076366+0000","last_active":"2024-09-20T19:20:42.322915+0000","last_peered":"2024-09-20T19:20:42.322915+0000","last_clean":"2024-09-20T19:20:42.322915+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:20:42.322915+0000","last_undegraded":"2024-09-20T19:20:42.322915+0000","last_fullsized":"2024-09-20T19:20:42.322915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:20:39.076230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.076230+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.693807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.465001+0000","last_change":"2024-09-20T19:19:35.981274+0000","last_active":"2024-09-20T19:20:42.465001+0000","last_peered":"2024-09-20T19:20:42.465001+0000","last_clean":"2024-09-20T19:20:42.465001+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:20:42.465001+0000","last_undegraded":"2024-09-20T19:20:42.465001+0000","last_fullsized":"2024-09-20T19:20:42.465001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.981166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.981166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.523226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329356+0000","last_change":"2024-09-20T19:20:38.745217+0000","last_active":"2024-09-20T19:20:42.329356+0000","last_peered":"2024-09-20T19:20:42.329356+0000","last_clean":"2024-09-20T19:20:42.329356+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:20:42.329356+0000","last_undegraded":"2024-09-20T19:20:42.329356+0000","last_fullsized":"2024-09-20T19:20:42.329356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.745091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.745091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:38.749886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689534+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:20:46.689534+0000","last_peered":"2024-09-20T19:20:46.689534+0000","last_clean":"2024-09-20T19:20:46.689534+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:20:46.689534+0000","last_undegraded":"2024-09-20T19:20:46.689534+0000","last_fullsized":"2024-09-20T19:20:46.689534+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.905322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":108,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.326658+0000","last_change":"2024-09-20T19:19:28.173179+0000","last_active":"2024-09-20T19:20:42.326658+0000","last_peered":"2024-09-20T19:20:42.326658+0000","last_clean":"2024-09-20T19:20:42.326658+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:20:42.326658+0000","last_undegraded":"2024-09-20T19:20:42.326658+0000","last_fullsized":"2024-09-20T19:20:42.326658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:28.172944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.172944+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:45.576524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330703+0000","last_change":"2024-09-20T19:20:29.725323+0000","last_active":"2024-09-20T19:20:42.330703+0000","last_peered":"2024-09-20T19:20:42.330703+0000","last_clean":"2024-09-20T19:20:42.330703+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:20:42.330703+0000","last_undegraded":"2024-09-20T19:20:42.330703+0000","last_fullsized":"2024-09-20T19:20:42.330703+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:29.725195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:29.725195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:39.235455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689328+0000","last_change":"2024-09-20T19:19:50.321638+0000","last_active":"2024-09-20T19:20:46.689328+0000","last_peered":"2024-09-20T19:20:46.689328+0000","last_clean":"2024-09-20T19:20:46.689328+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:20:46.689328+0000","last_undegraded":"2024-09-20T19:20:46.689328+0000","last_fullsized":"2024-09-20T19:20:46.689328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.321520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.321520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.084680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:44.049456+0000","last_change":"2024-09-20T19:20:44.049456+0000","last_active":"2024-09-20T19:20:44.049456+0000","last_peered":"2024-09-20T19:20:44.049456+0000","last_clean":"2024-09-20T19:20:44.049456+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:20:44.049456+0000","last_undegraded":"2024-09-20T19:20:44.049456+0000","last_fullsized":"2024-09-20T19:20:44.049456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:44.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.049342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:04.326689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690393+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:20:46.690393+0000","last_peered":"2024-09-20T19:20:46.690393+0000","last_clean":"2024-09-20T19:20:46.690393+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:20:46.690393+0000","last_undegraded":"2024-09-20T19:20:46.690393+0000","last_fullsized":"2024-09-20T19:20:46.690393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.371707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.459143+0000","last_change":"2024-09-20T19:19:23.109142+0000","last_active":"2024-09-20T19:20:42.459143+0000","last_peered":"2024-09-20T19:20:42.459143+0000","last_clean":"2024-09-20T19:20:42.459143+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:20:42.459143+0000","last_undegraded":"2024-09-20T19:20:42.459143+0000","last_fullsized":"2024-09-20T19:20:42.459143+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:23.109064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.109064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685663+0000","last_change":"2024-09-20T19:19:55.848311+0000","last_active":"2024-09-20T19:20:46.685663+0000","last_peered":"2024-09-20T19:20:46.685663+0000","last_clean":"2024-09-20T19:20:46.685663+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:20:46.685663+0000","last_undegraded":"2024-09-20T19:20:46.685663+0000","last_fullsized":"2024-09-20T19:20:46.685663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:55.848164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.848164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:19.416599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":137,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:44.717984+0000","last_change":"2024-09-20T19:20:44.717984+0000","last_active":"2024-09-20T19:20:44.717984+0000","last_peered":"2024-09-20T19:20:44.717984+0000","last_clean":"2024-09-20T19:20:44.717984+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:20:44.717984+0000","last_undegraded":"2024-09-20T19:20:44.717984+0000","last_fullsized":"2024-09-20T19:20:44.717984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:44.717840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.717840+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.300091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903180+0000","last_change":"2024-09-20T19:19:31.466386+0000","last_active":"2024-09-20T19:20:46.903180+0000","last_peered":"2024-09-20T19:20:46.903180+0000","last_clean":"2024-09-20T19:20:46.903180+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:20:46.903180+0000","last_undegraded":"2024-09-20T19:20:46.903180+0000","last_fullsized":"2024-09-20T19:20:46.903180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.466247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.466247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.461700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":102,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:43.067477+0000","last_change":"2024-09-20T19:20:43.067477+0000","last_active":"2024-09-20T19:20:43.067477+0000","last_peered":"2024-09-20T19:20:43.067477+0000","last_clean":"2024-09-20T19:20:43.067477+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:20:43.067477+0000","last_undegraded":"2024-09-20T19:20:43.067477+0000","last_fullsized":"2024-09-20T19:20:43.067477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:43.067414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.067414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.468070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":126,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.330550+0000","last_change":"2024-09-20T19:19:57.028481+0000","last_active":"2024-09-20T19:20:42.330550+0000","last_peered":"2024-09-20T19:20:42.330550+0000","last_clean":"2024-09-20T19:20:42.330550+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:20:42.330550+0000","last_undegraded":"2024-09-20T19:20:42.330550+0000","last_fullsized":"2024-09-20T19:20:42.330550+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:40.905378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.905378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.869949+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685651+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:20:46.685651+0000","last_peered":"2024-09-20T19:20:46.685651+0000","last_clean":"2024-09-20T19:20:46.685651+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:20:46.685651+0000","last_undegraded":"2024-09-20T19:20:46.685651+0000","last_fullsized":"2024-09-20T19:20:46.685651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.632034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733975+0000","last_change":"2024-09-20T19:19:39.226277+0000","last_active":"2024-09-20T19:20:46.733975+0000","last_peered":"2024-09-20T19:20:46.733975+0000","last_clean":"2024-09-20T19:20:46.733975+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:20:46.733975+0000","last_undegraded":"2024-09-20T19:20:46.733975+0000","last_fullsized":"2024-09-20T19:20:46.733975+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:39.226160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.226160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.931829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.661313+0000","last_change":"2024-09-20T19:20:46.661313+0000","last_active":"2024-09-20T19:20:46.661313+0000","last_peered":"2024-09-20T19:20:46.661313+0000","last_clean":"2024-09-20T19:20:46.661313+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:20:46.661313+0000","last_undegraded":"2024-09-20T19:20:46.661313+0000","last_fullsized":"2024-09-20T19:20:46.661313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:46.661186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:46.661186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:50.055994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689301+0000","last_change":"2024-09-20T19:19:47.048010+0000","last_active":"2024-09-20T19:20:46.689301+0000","last_peered":"2024-09-20T19:20:46.689301+0000","last_clean":"2024-09-20T19:20:46.689301+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:20:46.689301+0000","last_undegraded":"2024-09-20T19:20:46.689301+0000","last_fullsized":"2024-09-20T19:20:46.689301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:47.047816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.047816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.958306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903781+0000","last_change":"2024-09-20T19:20:33.482733+0000","last_active":"2024-09-20T19:20:46.903781+0000","last_peered":"2024-09-20T19:20:46.903781+0000","last_clean":"2024-09-20T19:20:46.903781+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:20:46.903781+0000","last_undegraded":"2024-09-20T19:20:46.903781+0000","last_fullsized":"2024-09-20T19:20:46.903781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.482605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.482605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:42.867843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.329787+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:20:42.329787+0000","last_peered":"2024-09-20T19:20:42.329787+0000","last_clean":"2024-09-20T19:20:42.329787+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:20:42.329787+0000","last_undegraded":"2024-09-20T19:20:42.329787+0000","last_fullsized":"2024-09-20T19:20:42.329787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:20.115239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.325670+0000","last_change":"2024-09-20T19:19:58.028896+0000","last_active":"2024-09-20T19:20:42.325670+0000","last_peered":"2024-09-20T19:20:42.325670+0000","last_clean":"2024-09-20T19:20:42.325670+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:20:42.325670+0000","last_undegraded":"2024-09-20T19:20:42.325670+0000","last_fullsized":"2024-09-20T19:20:42.325670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:58.028776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:58.028776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.888058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-20T19:20:42.328462+0000","last_change":"2024-09-20T19:19:51.256022+0000","last_active":"2024-09-20T19:20:42.328462+0000","last_peered":"2024-09-20T19:20:42.328462+0000","last_clean":"2024-09-20T19:20:42.328462+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:20:42.328462+0000","last_undegraded":"2024-09-20T19:20:42.328462+0000","last_fullsized":"2024-09-20T19:20:42.328462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:51.255877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.255877+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.975626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734019+0000","last_change":"2024-09-20T19:19:28.186599+0000","last_active":"2024-09-20T19:20:46.734019+0000","last_peered":"2024-09-20T19:20:46.734019+0000","last_clean":"2024-09-20T19:20:46.734019+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:20:46.734019+0000","last_undegraded":"2024-09-20T19:20:46.734019+0000","last_fullsized":"2024-09-20T19:20:46.734019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.186507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.186507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.594221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877906985,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1253376,"data_stored":896140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234036,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28308,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716844,"statfs":{"total":95995035648,"available":95966048256,"internally_reserved":0,"allocated":1310720,"data_stored":929579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561086,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":720896,"data_stored":340302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855434,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":729088,"data_stored":339289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084372,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":749568,"data_stored":340076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378722,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36060,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709092,"statfs":{"total":95995035648,"available":95958110208,"internally_reserved":0,"allocated":729088,"data_stored":340156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":741376,"data_stored":340783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36656,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708496,"statfs":{"total":95995035648,"available":95957499904,"internally_reserved":0,"allocated":1257472,"data_stored":896406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:20:51.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:53.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:52 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.301593+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.e scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.304353+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.e scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.893146+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.897009+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.c scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:51.927897+0000 mgr.y (mgr.24491) 113 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.002508+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:52 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.007095+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:54.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:54 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.313063+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.315257+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.382630+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:54.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.382778+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.713783+0000 osd.6 (osd.6) 47 : cluster [DBG] 4.a scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.718701+0000 osd.6 (osd.6) 48 : cluster [DBG] 4.a scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.720887+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.723113+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.910590+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:52.914112+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:54.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:54 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.010312+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-20T19:20:55.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:55 smithi162 bash[24557]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[22706]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.865842+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.868025+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.874529+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.876758+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:55 smithi042 bash[32075]: cluster 2024-09-20T19:20:53.928468+0000 mgr.y (mgr.24491) 114 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:56.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:56 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:56.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:56.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.720036+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.724108+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.738624+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.741142+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.812153+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.812241+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.885077+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:54.885169+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.029398+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.873084+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:56.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:56 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.875101+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:57 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.729808+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.732308+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.857807+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.860102+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:55.929127+0000 mgr.y (mgr.24491) 115 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: audit 2024-09-20T19:20:56.613531+0000 mon.a (mon.0) 972 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.703269+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:57 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.707667+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:20:57.725 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:20:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:20:58.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:58 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:58 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:58 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:58 smithi162 bash[24557]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[22706]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.862708+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:58 smithi042 bash[32075]: cluster 2024-09-20T19:20:56.865041+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:20:58.528 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-20T19:20:55.928741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1011,"num_read_kb":868,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244068,"kb_used_data":6908,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717148,"statfs":{"total":767960285184,"available":767710359552,"internally_reserved":0,"allocated":7073792,"data_stored":4218113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004162"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992467+0000","last_change":"2024-09-20T19:20:38.044596+0000","last_active":"2024-09-20T19:20:46.992467+0000","last_peered":"2024-09-20T19:20:46.992467+0000","last_clean":"2024-09-20T19:20:46.992467+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:20:46.992467+0000","last_undegraded":"2024-09-20T19:20:46.992467+0000","last_fullsized":"2024-09-20T19:20:46.992467+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.044475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.044475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.419278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685882+0000","last_change":"2024-09-20T19:19:17.698496+0000","last_active":"2024-09-20T19:20:46.685882+0000","last_peered":"2024-09-20T19:20:46.685882+0000","last_clean":"2024-09-20T19:20:46.685882+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:20:46.685882+0000","last_undegraded":"2024-09-20T19:20:46.685882+0000","last_fullsized":"2024-09-20T19:20:46.685882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:17.698427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:17.698427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":116,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993294+0000","last_change":"2024-09-20T19:19:45.244647+0000","last_active":"2024-09-20T19:20:46.993294+0000","last_peered":"2024-09-20T19:20:46.993294+0000","last_clean":"2024-09-20T19:20:46.993294+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:20:46.993294+0000","last_undegraded":"2024-09-20T19:20:46.993294+0000","last_fullsized":"2024-09-20T19:20:46.993294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:45.244486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.244486+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.299507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732981+0000","last_change":"2024-09-20T19:20:33.800940+0000","last_active":"2024-09-20T19:20:46.732981+0000","last_peered":"2024-09-20T19:20:46.732981+0000","last_clean":"2024-09-20T19:20:46.732981+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:20:46.732981+0000","last_undegraded":"2024-09-20T19:20:46.732981+0000","last_fullsized":"2024-09-20T19:20:46.732981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.800794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.800794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.248084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":103,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993640+0000","last_change":"2024-09-20T19:20:31.068114+0000","last_active":"2024-09-20T19:20:46.993640+0000","last_peered":"2024-09-20T19:20:46.993640+0000","last_clean":"2024-09-20T19:20:46.993640+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:20:46.993640+0000","last_undegraded":"2024-09-20T19:20:46.993640+0000","last_fullsized":"2024-09-20T19:20:46.993640+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:31.067964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:31.067964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:48.259004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732993+0000","last_change":"2024-09-20T19:20:34.776570+0000","last_active":"2024-09-20T19:20:46.732993+0000","last_peered":"2024-09-20T19:20:46.732993+0000","last_clean":"2024-09-20T19:20:46.732993+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:20:46.732993+0000","last_undegraded":"2024-09-20T19:20:46.732993+0000","last_fullsized":"2024-09-20T19:20:46.732993+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:34.776387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:34.776387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.596561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.901763+0000","last_change":"2024-09-20T19:20:42.524526+0000","last_active":"2024-09-20T19:20:46.901763+0000","last_peered":"2024-09-20T19:20:46.901763+0000","last_clean":"2024-09-20T19:20:46.901763+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:20:46.901763+0000","last_undegraded":"2024-09-20T19:20:46.901763+0000","last_fullsized":"2024-09-20T19:20:46.901763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.524397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.524397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:01.089547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733151+0000","last_change":"2024-09-20T19:19:32.223697+0000","last_active":"2024-09-20T19:20:46.733151+0000","last_peered":"2024-09-20T19:20:46.733151+0000","last_clean":"2024-09-20T19:20:46.733151+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:20:46.733151+0000","last_undegraded":"2024-09-20T19:20:46.733151+0000","last_fullsized":"2024-09-20T19:20:46.733151+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:32.223538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:32.223538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.453072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.691416+0000","last_change":"2024-09-20T19:19:59.335157+0000","last_active":"2024-09-20T19:20:46.691416+0000","last_peered":"2024-09-20T19:20:46.691416+0000","last_clean":"2024-09-20T19:20:46.691416+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:20:46.691416+0000","last_undegraded":"2024-09-20T19:20:46.691416+0000","last_fullsized":"2024-09-20T19:20:46.691416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.335040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.335040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.153396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733345+0000","last_change":"2024-09-20T19:20:41.804284+0000","last_active":"2024-09-20T19:20:46.733345+0000","last_peered":"2024-09-20T19:20:46.733345+0000","last_clean":"2024-09-20T19:20:46.733345+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:20:46.733345+0000","last_undegraded":"2024-09-20T19:20:46.733345+0000","last_fullsized":"2024-09-20T19:20:46.733345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:20:41.804146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.804146+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:55.773005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:48.831732+0000","last_change":"2024-09-20T19:20:48.831732+0000","last_active":"2024-09-20T19:20:48.831732+0000","last_peered":"2024-09-20T19:20:48.831732+0000","last_clean":"2024-09-20T19:20:48.831732+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:20:48.831732+0000","last_undegraded":"2024-09-20T19:20:48.831732+0000","last_fullsized":"2024-09-20T19:20:48.831732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:48.831670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.831670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:07.003294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":90,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:49.661540+0000","last_change":"2024-09-20T19:20:49.661540+0000","last_active":"2024-09-20T19:20:49.661540+0000","last_peered":"2024-09-20T19:20:49.661540+0000","last_clean":"2024-09-20T19:20:49.661540+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:20:49.661540+0000","last_undegraded":"2024-09-20T19:20:49.661540+0000","last_fullsized":"2024-09-20T19:20:49.661540+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:49.661422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:49.661422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:03.539487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.691272+0000","last_change":"2024-09-20T19:19:41.142126+0000","last_active":"2024-09-20T19:20:46.691272+0000","last_peered":"2024-09-20T19:20:46.691272+0000","last_clean":"2024-09-20T19:20:46.691272+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:20:46.691272+0000","last_undegraded":"2024-09-20T19:20:46.691272+0000","last_fullsized":"2024-09-20T19:20:46.691272+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.141975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.141975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.007349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":107,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733389+0000","last_change":"2024-09-20T19:19:42.188043+0000","last_active":"2024-09-20T19:20:46.733389+0000","last_peered":"2024-09-20T19:20:46.733389+0000","last_clean":"2024-09-20T19:20:46.733389+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:20:46.733389+0000","last_undegraded":"2024-09-20T19:20:46.733389+0000","last_fullsized":"2024-09-20T19:20:46.733389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:42.187756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:42.187756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.911434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":86,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.998570+0000","last_change":"2024-09-20T19:19:40.029794+0000","last_active":"2024-09-20T19:20:46.998570+0000","last_peered":"2024-09-20T19:20:46.998570+0000","last_clean":"2024-09-20T19:20:46.998570+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:20:46.998570+0000","last_undegraded":"2024-09-20T19:20:46.998570+0000","last_fullsized":"2024-09-20T19:20:46.998570+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.029674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.029674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.995856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684084+0000","last_change":"2024-09-20T19:19:54.798943+0000","last_active":"2024-09-20T19:20:46.684084+0000","last_peered":"2024-09-20T19:20:46.684084+0000","last_clean":"2024-09-20T19:20:46.684084+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:20:46.684084+0000","last_undegraded":"2024-09-20T19:20:46.684084+0000","last_fullsized":"2024-09-20T19:20:46.684084+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.798882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.798882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.457870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"77'17","reported_seq":119,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734044+0000","last_change":"2024-09-20T19:19:48.026945+0000","last_active":"2024-09-20T19:20:46.734044+0000","last_peered":"2024-09-20T19:20:46.734044+0000","last_clean":"2024-09-20T19:20:46.734044+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:20:46.734044+0000","last_undegraded":"2024-09-20T19:20:46.734044+0000","last_fullsized":"2024-09-20T19:20:46.734044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:19:48.026795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.026795+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.794994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690597+0000","last_change":"2024-09-20T19:20:05.282366+0000","last_active":"2024-09-20T19:20:46.690597+0000","last_peered":"2024-09-20T19:20:46.690597+0000","last_clean":"2024-09-20T19:20:46.690597+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:20:46.690597+0000","last_undegraded":"2024-09-20T19:20:46.690597+0000","last_fullsized":"2024-09-20T19:20:46.690597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:05.282293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:05.282293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.281996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902459+0000","last_change":"2024-09-20T19:19:56.425232+0000","last_active":"2024-09-20T19:20:46.902459+0000","last_peered":"2024-09-20T19:20:46.902459+0000","last_clean":"2024-09-20T19:20:46.902459+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:20:46.902459+0000","last_undegraded":"2024-09-20T19:20:46.902459+0000","last_fullsized":"2024-09-20T19:20:46.902459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.425156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.425156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:21.486886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690023+0000","last_change":"2024-09-20T19:19:46.074958+0000","last_active":"2024-09-20T19:20:46.690023+0000","last_peered":"2024-09-20T19:20:46.690023+0000","last_clean":"2024-09-20T19:20:46.690023+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:20:46.690023+0000","last_undegraded":"2024-09-20T19:20:46.690023+0000","last_fullsized":"2024-09-20T19:20:46.690023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:46.074887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.074887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.317322+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"77'10","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:51.897157+0000","last_change":"2024-09-20T19:20:51.897157+0000","last_active":"2024-09-20T19:20:51.897157+0000","last_peered":"2024-09-20T19:20:51.897157+0000","last_clean":"2024-09-20T19:20:51.897157+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:20:51.897157+0000","last_undegraded":"2024-09-20T19:20:51.897157+0000","last_fullsized":"2024-09-20T19:20:51.897157+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:51.897019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:51.897019+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:14.583589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903428+0000","last_change":"2024-09-20T19:19:54.442006+0000","last_active":"2024-09-20T19:20:46.903428+0000","last_peered":"2024-09-20T19:20:46.903428+0000","last_clean":"2024-09-20T19:20:46.903428+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:20:46.903428+0000","last_undegraded":"2024-09-20T19:20:46.903428+0000","last_fullsized":"2024-09-20T19:20:46.903428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.441955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.441955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.241390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":131,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689607+0000","last_change":"2024-09-20T19:19:45.038512+0000","last_active":"2024-09-20T19:20:46.689607+0000","last_peered":"2024-09-20T19:20:46.689607+0000","last_clean":"2024-09-20T19:20:46.689607+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:20:46.689607+0000","last_undegraded":"2024-09-20T19:20:46.689607+0000","last_fullsized":"2024-09-20T19:20:46.689607+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:45.038377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:45.038377+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.979665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733847+0000","last_change":"2024-09-20T19:19:38.210845+0000","last_active":"2024-09-20T19:20:46.733847+0000","last_peered":"2024-09-20T19:20:46.733847+0000","last_clean":"2024-09-20T19:20:46.733847+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:20:46.733847+0000","last_undegraded":"2024-09-20T19:20:46.733847+0000","last_fullsized":"2024-09-20T19:20:46.733847+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:38.210713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:38.210713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.317890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"77'15","reported_seq":124,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903069+0000","last_change":"2024-09-20T19:20:32.494239+0000","last_active":"2024-09-20T19:20:46.903069+0000","last_peered":"2024-09-20T19:20:46.903069+0000","last_clean":"2024-09-20T19:20:46.903069+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:20:46.903069+0000","last_undegraded":"2024-09-20T19:20:46.903069+0000","last_fullsized":"2024-09-20T19:20:46.903069+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:20:32.494020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.494020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.833966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902023+0000","last_change":"2024-09-20T19:19:36.378407+0000","last_active":"2024-09-20T19:20:46.902023+0000","last_peered":"2024-09-20T19:20:46.902023+0000","last_clean":"2024-09-20T19:20:46.902023+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:20:46.902023+0000","last_undegraded":"2024-09-20T19:20:46.902023+0000","last_fullsized":"2024-09-20T19:20:46.902023+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.378285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.378285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":130,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734248+0000","last_change":"2024-09-20T19:19:50.966826+0000","last_active":"2024-09-20T19:20:46.734248+0000","last_peered":"2024-09-20T19:20:46.734248+0000","last_clean":"2024-09-20T19:20:46.734248+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:20:46.734248+0000","last_undegraded":"2024-09-20T19:20:46.734248+0000","last_fullsized":"2024-09-20T19:20:46.734248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:50.966659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.966659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.209609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999076+0000","last_change":"2024-09-20T19:19:34.013906+0000","last_active":"2024-09-20T19:20:46.999076+0000","last_peered":"2024-09-20T19:20:46.999076+0000","last_clean":"2024-09-20T19:20:46.999076+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:20:46.999076+0000","last_undegraded":"2024-09-20T19:20:46.999076+0000","last_fullsized":"2024-09-20T19:20:46.999076+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.013787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.013787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.149084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"77'11","reported_seq":118,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734325+0000","last_change":"2024-09-20T19:20:40.763340+0000","last_active":"2024-09-20T19:20:46.734325+0000","last_peered":"2024-09-20T19:20:46.734325+0000","last_clean":"2024-09-20T19:20:46.734325+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:20:46.734325+0000","last_undegraded":"2024-09-20T19:20:46.734325+0000","last_fullsized":"2024-09-20T19:20:46.734325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:40.763197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.763197+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:47.787913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689754+0000","last_change":"2024-09-20T19:19:48.281994+0000","last_active":"2024-09-20T19:20:46.689754+0000","last_peered":"2024-09-20T19:20:46.689754+0000","last_clean":"2024-09-20T19:20:46.689754+0000","last_became_active":"2024-09-20T19:18:07.269527+0000","last_became_peered":"2024-09-20T19:18:07.269527+0000","last_unstale":"2024-09-20T19:20:46.689754+0000","last_undegraded":"2024-09-20T19:20:46.689754+0000","last_fullsized":"2024-09-20T19:20:46.689754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.281832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.281832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:11.535302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":86,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999794+0000","last_change":"2024-09-20T19:19:34.966265+0000","last_active":"2024-09-20T19:20:46.999794+0000","last_peered":"2024-09-20T19:20:46.999794+0000","last_clean":"2024-09-20T19:20:46.999794+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:20:46.999794+0000","last_undegraded":"2024-09-20T19:20:46.999794+0000","last_fullsized":"2024-09-20T19:20:46.999794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.966141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.966141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.384654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992608+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:20:46.992608+0000","last_peered":"2024-09-20T19:20:46.992608+0000","last_clean":"2024-09-20T19:20:46.992608+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:20:46.992608+0000","last_undegraded":"2024-09-20T19:20:46.992608+0000","last_fullsized":"2024-09-20T19:20:46.992608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.064548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"77'14","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733599+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:20:46.733599+0000","last_peered":"2024-09-20T19:20:46.733599+0000","last_clean":"2024-09-20T19:20:46.733599+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:20:46.733599+0000","last_undegraded":"2024-09-20T19:20:46.733599+0000","last_fullsized":"2024-09-20T19:20:46.733599+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.837736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001337+0000","last_change":"2024-09-20T19:20:16.657110+0000","last_active":"2024-09-20T19:20:47.001337+0000","last_peered":"2024-09-20T19:20:47.001337+0000","last_clean":"2024-09-20T19:20:47.001337+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:20:47.001337+0000","last_undegraded":"2024-09-20T19:20:47.001337+0000","last_fullsized":"2024-09-20T19:20:47.001337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.656960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.656960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:43.026972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":130,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992082+0000","last_change":"2024-09-20T19:19:37.178863+0000","last_active":"2024-09-20T19:20:46.992082+0000","last_peered":"2024-09-20T19:20:46.992082+0000","last_clean":"2024-09-20T19:20:46.992082+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:20:46.992082+0000","last_undegraded":"2024-09-20T19:20:46.992082+0000","last_fullsized":"2024-09-20T19:20:46.992082+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.178724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.178724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.727526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690224+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:20:46.690224+0000","last_peered":"2024-09-20T19:20:46.690224+0000","last_clean":"2024-09-20T19:20:46.690224+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:20:46.690224+0000","last_undegraded":"2024-09-20T19:20:46.690224+0000","last_fullsized":"2024-09-20T19:20:46.690224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:18.443029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"77'18","reported_seq":131,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992243+0000","last_change":"2024-09-20T19:20:40.068159+0000","last_active":"2024-09-20T19:20:46.992243+0000","last_peered":"2024-09-20T19:20:46.992243+0000","last_clean":"2024-09-20T19:20:46.992243+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:20:46.992243+0000","last_undegraded":"2024-09-20T19:20:46.992243+0000","last_fullsized":"2024-09-20T19:20:46.992243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:20:40.068007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.068007+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.669287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":96,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903431+0000","last_change":"2024-09-20T19:19:27.488807+0000","last_active":"2024-09-20T19:20:46.903431+0000","last_peered":"2024-09-20T19:20:46.903431+0000","last_clean":"2024-09-20T19:20:46.903431+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:20:46.903431+0000","last_undegraded":"2024-09-20T19:20:46.903431+0000","last_fullsized":"2024-09-20T19:20:46.903431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:19:27.488644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.488644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.732951+0000","last_change":"2024-09-20T19:19:44.113106+0000","last_active":"2024-09-20T19:20:46.732951+0000","last_peered":"2024-09-20T19:20:46.732951+0000","last_clean":"2024-09-20T19:20:46.732951+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:20:46.732951+0000","last_undegraded":"2024-09-20T19:20:46.732951+0000","last_fullsized":"2024-09-20T19:20:46.732951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.112985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.112985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.242746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733216+0000","last_change":"2024-09-20T19:19:48.997145+0000","last_active":"2024-09-20T19:20:46.733216+0000","last_peered":"2024-09-20T19:20:46.733216+0000","last_clean":"2024-09-20T19:20:46.733216+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:20:46.733216+0000","last_undegraded":"2024-09-20T19:20:46.733216+0000","last_fullsized":"2024-09-20T19:20:46.733216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.996997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.996997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.494961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"77'19","reported_seq":127,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685181+0000","last_change":"2024-09-20T19:19:27.727451+0000","last_active":"2024-09-20T19:20:46.685181+0000","last_peered":"2024-09-20T19:20:46.685181+0000","last_clean":"2024-09-20T19:20:46.685181+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:20:46.685181+0000","last_undegraded":"2024-09-20T19:20:46.685181+0000","last_fullsized":"2024-09-20T19:20:46.685181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.727294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.727294+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.582205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689376+0000","last_change":"2024-09-20T19:19:56.998769+0000","last_active":"2024-09-20T19:20:46.689376+0000","last_peered":"2024-09-20T19:20:46.689376+0000","last_clean":"2024-09-20T19:20:46.689376+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:20:46.689376+0000","last_undegraded":"2024-09-20T19:20:46.689376+0000","last_fullsized":"2024-09-20T19:20:46.689376+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.169199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.169199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.901847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":86,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999964+0000","last_change":"2024-09-20T19:19:41.871990+0000","last_active":"2024-09-20T19:20:46.999964+0000","last_peered":"2024-09-20T19:20:46.999964+0000","last_clean":"2024-09-20T19:20:46.999964+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:20:46.999964+0000","last_undegraded":"2024-09-20T19:20:46.999964+0000","last_fullsized":"2024-09-20T19:20:46.999964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.871854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.871854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.298167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.901551+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:20:46.901551+0000","last_peered":"2024-09-20T19:20:46.901551+0000","last_clean":"2024-09-20T19:20:46.901551+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:20:46.901551+0000","last_undegraded":"2024-09-20T19:20:46.901551+0000","last_fullsized":"2024-09-20T19:20:46.901551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:29.401329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"77'10","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902997+0000","last_change":"2024-09-20T19:19:40.461604+0000","last_active":"2024-09-20T19:20:46.902997+0000","last_peered":"2024-09-20T19:20:46.902997+0000","last_clean":"2024-09-20T19:20:46.902997+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:20:46.902997+0000","last_undegraded":"2024-09-20T19:20:46.902997+0000","last_fullsized":"2024-09-20T19:20:46.902997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:40.461532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.461532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:08.703461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903353+0000","last_change":"2024-09-20T19:19:34.429306+0000","last_active":"2024-09-20T19:20:46.903353+0000","last_peered":"2024-09-20T19:20:46.903353+0000","last_clean":"2024-09-20T19:20:46.903353+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:20:46.903353+0000","last_undegraded":"2024-09-20T19:20:46.903353+0000","last_fullsized":"2024-09-20T19:20:46.903353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:34.429125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:34.429125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.671437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":130,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685028+0000","last_change":"2024-09-20T19:19:37.807674+0000","last_active":"2024-09-20T19:20:46.685028+0000","last_peered":"2024-09-20T19:20:46.685028+0000","last_clean":"2024-09-20T19:20:46.685028+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:20:46.685028+0000","last_undegraded":"2024-09-20T19:20:46.685028+0000","last_fullsized":"2024-09-20T19:20:46.685028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:37.807547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:37.807547+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:04.829611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:49.888543+0000","last_change":"2024-09-20T19:20:49.888543+0000","last_active":"2024-09-20T19:20:49.888543+0000","last_peered":"2024-09-20T19:20:49.888543+0000","last_clean":"2024-09-20T19:20:49.888543+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:20:49.888543+0000","last_undegraded":"2024-09-20T19:20:49.888543+0000","last_fullsized":"2024-09-20T19:20:49.888543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:49.888394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:49.888394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:55.003131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'30","reported_seq":174,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903181+0000","last_change":"2024-09-20T19:19:46.395559+0000","last_active":"2024-09-20T19:20:46.903181+0000","last_peered":"2024-09-20T19:20:46.903181+0000","last_clean":"2024-09-20T19:20:46.903181+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:20:46.903181+0000","last_undegraded":"2024-09-20T19:20:46.903181+0000","last_fullsized":"2024-09-20T19:20:46.903181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:19:46.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.395404+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999317+0000","last_change":"2024-09-20T19:19:33.057715+0000","last_active":"2024-09-20T19:20:46.999317+0000","last_peered":"2024-09-20T19:20:46.999317+0000","last_clean":"2024-09-20T19:20:46.999317+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:20:46.999317+0000","last_undegraded":"2024-09-20T19:20:46.999317+0000","last_fullsized":"2024-09-20T19:20:46.999317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:33.057567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:33.057567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685329+0000","last_change":"2024-09-20T19:19:56.854517+0000","last_active":"2024-09-20T19:20:46.685329+0000","last_peered":"2024-09-20T19:20:46.685329+0000","last_clean":"2024-09-20T19:20:46.685329+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:20:46.685329+0000","last_undegraded":"2024-09-20T19:20:46.685329+0000","last_fullsized":"2024-09-20T19:20:46.685329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.854461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.854461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.519787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993122+0000","last_change":"2024-09-20T19:19:50.253935+0000","last_active":"2024-09-20T19:20:46.993122+0000","last_peered":"2024-09-20T19:20:46.993122+0000","last_clean":"2024-09-20T19:20:46.993122+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:20:46.993122+0000","last_undegraded":"2024-09-20T19:20:46.993122+0000","last_fullsized":"2024-09-20T19:20:46.993122+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.253727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.253727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.038762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"77'13","reported_seq":34,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689579+0000","last_change":"2024-09-20T19:19:57.013393+0000","last_active":"2024-09-20T19:20:46.689579+0000","last_peered":"2024-09-20T19:20:46.689579+0000","last_clean":"2024-09-20T19:20:46.689579+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:20:46.689579+0000","last_undegraded":"2024-09-20T19:20:46.689579+0000","last_fullsized":"2024-09-20T19:20:46.689579+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:19:51.410598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.410598+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:12.734213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":259,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:50.133996+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:20:50.133996+0000","last_peered":"2024-09-20T19:20:50.133996+0000","last_clean":"2024-09-20T19:20:50.133996+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:20:50.133996+0000","last_undegraded":"2024-09-20T19:20:50.133996+0000","last_fullsized":"2024-09-20T19:20:50.133996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:17.892259+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688519+0000","last_change":"2024-09-20T19:19:49.990251+0000","last_active":"2024-09-20T19:20:46.688519+0000","last_peered":"2024-09-20T19:20:46.688519+0000","last_clean":"2024-09-20T19:20:46.688519+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:20:46.688519+0000","last_undegraded":"2024-09-20T19:20:46.688519+0000","last_fullsized":"2024-09-20T19:20:46.688519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:49.990058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:49.990058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.407192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689955+0000","last_change":"2024-09-20T19:20:30.146960+0000","last_active":"2024-09-20T19:20:46.689955+0000","last_peered":"2024-09-20T19:20:46.689955+0000","last_clean":"2024-09-20T19:20:46.689955+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:20:46.689955+0000","last_undegraded":"2024-09-20T19:20:46.689955+0000","last_fullsized":"2024-09-20T19:20:46.689955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:30.146806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:30.146806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.691081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690826+0000","last_change":"2024-09-20T19:19:55.305695+0000","last_active":"2024-09-20T19:20:46.690826+0000","last_peered":"2024-09-20T19:20:46.690826+0000","last_clean":"2024-09-20T19:20:46.690826+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:20:46.690826+0000","last_undegraded":"2024-09-20T19:20:46.690826+0000","last_fullsized":"2024-09-20T19:20:46.690826+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:55.305573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.305573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.195185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'12","reported_seq":117,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.910460+0000","last_change":"2024-09-20T19:20:46.910460+0000","last_active":"2024-09-20T19:20:46.910460+0000","last_peered":"2024-09-20T19:20:46.910460+0000","last_clean":"2024-09-20T19:20:46.910460+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:20:46.910460+0000","last_undegraded":"2024-09-20T19:20:46.910460+0000","last_fullsized":"2024-09-20T19:20:46.910460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:46.910317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:46.910317+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:16.209696+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684474+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:20:46.684474+0000","last_peered":"2024-09-20T19:20:46.684474+0000","last_clean":"2024-09-20T19:20:46.684474+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:20:46.684474+0000","last_undegraded":"2024-09-20T19:20:46.684474+0000","last_fullsized":"2024-09-20T19:20:46.684474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.851755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689832+0000","last_change":"2024-09-20T19:19:44.240584+0000","last_active":"2024-09-20T19:20:46.689832+0000","last_peered":"2024-09-20T19:20:46.689832+0000","last_clean":"2024-09-20T19:20:46.689832+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:20:46.689832+0000","last_undegraded":"2024-09-20T19:20:46.689832+0000","last_fullsized":"2024-09-20T19:20:46.689832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:44.240462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.240462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.731548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999815+0000","last_change":"2024-09-20T19:20:42.760023+0000","last_active":"2024-09-20T19:20:46.999815+0000","last_peered":"2024-09-20T19:20:46.999815+0000","last_clean":"2024-09-20T19:20:46.999815+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:20:46.999815+0000","last_undegraded":"2024-09-20T19:20:46.999815+0000","last_fullsized":"2024-09-20T19:20:46.999815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.759902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.759902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.827837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902189+0000","last_change":"2024-09-20T19:20:41.502208+0000","last_active":"2024-09-20T19:20:46.902189+0000","last_peered":"2024-09-20T19:20:46.902189+0000","last_clean":"2024-09-20T19:20:46.902189+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:20:46.902189+0000","last_undegraded":"2024-09-20T19:20:46.902189+0000","last_fullsized":"2024-09-20T19:20:46.902189+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:41.502048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.502048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:05.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"77'16","reported_seq":145,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001185+0000","last_change":"2024-09-20T19:19:39.901351+0000","last_active":"2024-09-20T19:20:47.001185+0000","last_peered":"2024-09-20T19:20:47.001185+0000","last_clean":"2024-09-20T19:20:47.001185+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:20:47.001185+0000","last_undegraded":"2024-09-20T19:20:47.001185+0000","last_fullsized":"2024-09-20T19:20:47.001185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:19:39.901171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.901171+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.526242+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992675+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:20:46.992675+0000","last_peered":"2024-09-20T19:20:46.992675+0000","last_clean":"2024-09-20T19:20:46.992675+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:20:46.992675+0000","last_undegraded":"2024-09-20T19:20:46.992675+0000","last_fullsized":"2024-09-20T19:20:46.992675+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.492805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999332+0000","last_change":"2024-09-20T19:19:07.170908+0000","last_active":"2024-09-20T19:20:46.999332+0000","last_peered":"2024-09-20T19:20:46.999332+0000","last_clean":"2024-09-20T19:20:46.999332+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:20:46.999332+0000","last_undegraded":"2024-09-20T19:20:46.999332+0000","last_fullsized":"2024-09-20T19:20:46.999332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:07.170760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:07.170760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":142,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690101+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:20:46.690101+0000","last_peered":"2024-09-20T19:20:46.690101+0000","last_clean":"2024-09-20T19:20:46.690101+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:20:46.690101+0000","last_undegraded":"2024-09-20T19:20:46.690101+0000","last_fullsized":"2024-09-20T19:20:46.690101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:27.954958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690535+0000","last_change":"2024-09-20T19:19:26.163243+0000","last_active":"2024-09-20T19:20:46.690535+0000","last_peered":"2024-09-20T19:20:46.690535+0000","last_clean":"2024-09-20T19:20:46.690535+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:20:46.690535+0000","last_undegraded":"2024-09-20T19:20:46.690535+0000","last_fullsized":"2024-09-20T19:20:46.690535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:26.163172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:26.163172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:46.703757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":90,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.000206+0000","last_change":"2024-09-20T19:20:39.803967+0000","last_active":"2024-09-20T19:20:47.000206+0000","last_peered":"2024-09-20T19:20:47.000206+0000","last_clean":"2024-09-20T19:20:47.000206+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:20:47.000206+0000","last_undegraded":"2024-09-20T19:20:47.000206+0000","last_fullsized":"2024-09-20T19:20:47.000206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:39.803847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.803847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.060059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"77'19","reported_seq":121,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.000406+0000","last_change":"2024-09-20T19:19:27.036130+0000","last_active":"2024-09-20T19:20:47.000406+0000","last_peered":"2024-09-20T19:20:47.000406+0000","last_clean":"2024-09-20T19:20:47.000406+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:20:47.000406+0000","last_undegraded":"2024-09-20T19:20:47.000406+0000","last_fullsized":"2024-09-20T19:20:47.000406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:19:27.036022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.036022+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":101,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689670+0000","last_change":"2024-09-20T19:20:35.231905+0000","last_active":"2024-09-20T19:20:46.689670+0000","last_peered":"2024-09-20T19:20:46.689670+0000","last_clean":"2024-09-20T19:20:46.689670+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:20:46.689670+0000","last_undegraded":"2024-09-20T19:20:46.689670+0000","last_fullsized":"2024-09-20T19:20:46.689670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:35.231776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:35.231776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:03.117077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688349+0000","last_change":"2024-09-20T19:19:43.031914+0000","last_active":"2024-09-20T19:20:46.688349+0000","last_peered":"2024-09-20T19:20:46.688349+0000","last_clean":"2024-09-20T19:20:46.688349+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:20:46.688349+0000","last_undegraded":"2024-09-20T19:20:46.688349+0000","last_fullsized":"2024-09-20T19:20:46.688349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.031768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.031768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:55.442373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689989+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:20:46.689989+0000","last_peered":"2024-09-20T19:20:46.689989+0000","last_clean":"2024-09-20T19:20:46.689989+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:20:46.689989+0000","last_undegraded":"2024-09-20T19:20:46.689989+0000","last_fullsized":"2024-09-20T19:20:46.689989+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.126953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"77'9","reported_seq":107,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685058+0000","last_change":"2024-09-20T19:19:46.791747+0000","last_active":"2024-09-20T19:20:46.685058+0000","last_peered":"2024-09-20T19:20:46.685058+0000","last_clean":"2024-09-20T19:20:46.685058+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:20:46.685058+0000","last_undegraded":"2024-09-20T19:20:46.685058+0000","last_fullsized":"2024-09-20T19:20:46.685058+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.791592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.791592+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.994746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.904191+0000","last_change":"2024-09-20T19:19:35.390225+0000","last_active":"2024-09-20T19:20:46.904191+0000","last_peered":"2024-09-20T19:20:46.904191+0000","last_clean":"2024-09-20T19:20:46.904191+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:20:46.904191+0000","last_undegraded":"2024-09-20T19:20:46.904191+0000","last_fullsized":"2024-09-20T19:20:46.904191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.390064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.390064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.228348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688283+0000","last_change":"2024-09-20T19:19:41.044575+0000","last_active":"2024-09-20T19:20:46.688283+0000","last_peered":"2024-09-20T19:20:46.688283+0000","last_clean":"2024-09-20T19:20:46.688283+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:20:46.688283+0000","last_undegraded":"2024-09-20T19:20:46.688283+0000","last_fullsized":"2024-09-20T19:20:46.688283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:41.044427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:41.044427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:01.962646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":84,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.684255+0000","last_change":"2024-09-20T19:19:52.810445+0000","last_active":"2024-09-20T19:20:46.684255+0000","last_peered":"2024-09-20T19:20:46.684255+0000","last_clean":"2024-09-20T19:20:46.684255+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:20:46.684255+0000","last_undegraded":"2024-09-20T19:20:46.684255+0000","last_fullsized":"2024-09-20T19:20:46.684255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:19:52.810385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:52.810385+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.122928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"77'15","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.997829+0000","last_change":"2024-09-20T19:19:44.046132+0000","last_active":"2024-09-20T19:20:46.997829+0000","last_peered":"2024-09-20T19:20:46.997829+0000","last_clean":"2024-09-20T19:20:46.997829+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:20:46.997829+0000","last_undegraded":"2024-09-20T19:20:46.997829+0000","last_fullsized":"2024-09-20T19:20:46.997829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:44.045925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:44.045925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:51.304499+0000","last_change":"2024-09-20T19:20:51.304499+0000","last_active":"2024-09-20T19:20:51.304499+0000","last_peered":"2024-09-20T19:20:51.304499+0000","last_clean":"2024-09-20T19:20:51.304499+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:20:51.304499+0000","last_undegraded":"2024-09-20T19:20:51.304499+0000","last_fullsized":"2024-09-20T19:20:51.304499+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:51.304365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:51.304365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:00.834548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":138,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689783+0000","last_change":"2024-09-20T19:20:25.228464+0000","last_active":"2024-09-20T19:20:46.689783+0000","last_peered":"2024-09-20T19:20:46.689783+0000","last_clean":"2024-09-20T19:20:46.689783+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:20:46.689783+0000","last_undegraded":"2024-09-20T19:20:46.689783+0000","last_fullsized":"2024-09-20T19:20:46.689783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:25.228325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.228325+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.437018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.998355+0000","last_change":"2024-09-20T19:19:36.943615+0000","last_active":"2024-09-20T19:20:46.998355+0000","last_peered":"2024-09-20T19:20:46.998355+0000","last_clean":"2024-09-20T19:20:46.998355+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:20:46.998355+0000","last_undegraded":"2024-09-20T19:20:46.998355+0000","last_fullsized":"2024-09-20T19:20:46.998355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:36.943527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:36.943527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:48.968462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"77'12","reported_seq":141,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733507+0000","last_change":"2024-09-20T19:20:16.852586+0000","last_active":"2024-09-20T19:20:46.733507+0000","last_peered":"2024-09-20T19:20:46.733507+0000","last_clean":"2024-09-20T19:20:46.733507+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:20:46.733507+0000","last_undegraded":"2024-09-20T19:20:46.733507+0000","last_fullsized":"2024-09-20T19:20:46.733507+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:16.852450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.852450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.371990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":119,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734369+0000","last_change":"2024-09-20T19:20:37.785586+0000","last_active":"2024-09-20T19:20:46.734369+0000","last_peered":"2024-09-20T19:20:46.734369+0000","last_clean":"2024-09-20T19:20:46.734369+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:20:46.734369+0000","last_undegraded":"2024-09-20T19:20:46.734369+0000","last_fullsized":"2024-09-20T19:20:46.734369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:20:37.785449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.785449+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:49.952598+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.687955+0000","last_change":"2024-09-20T19:19:51.013009+0000","last_active":"2024-09-20T19:20:46.687955+0000","last_peered":"2024-09-20T19:20:46.687955+0000","last_clean":"2024-09-20T19:20:46.687955+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:20:46.687955+0000","last_undegraded":"2024-09-20T19:20:46.687955+0000","last_fullsized":"2024-09-20T19:20:46.687955+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.012906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.012906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:54.429741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992854+0000","last_change":"2024-09-20T19:20:28.004917+0000","last_active":"2024-09-20T19:20:46.992854+0000","last_peered":"2024-09-20T19:20:46.992854+0000","last_clean":"2024-09-20T19:20:46.992854+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:20:46.992854+0000","last_undegraded":"2024-09-20T19:20:46.992854+0000","last_fullsized":"2024-09-20T19:20:46.992854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.004784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.004784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.093046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685941+0000","last_change":"2024-09-20T19:20:33.803376+0000","last_active":"2024-09-20T19:20:46.685941+0000","last_peered":"2024-09-20T19:20:46.685941+0000","last_clean":"2024-09-20T19:20:46.685941+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:20:46.685941+0000","last_undegraded":"2024-09-20T19:20:46.685941+0000","last_fullsized":"2024-09-20T19:20:46.685941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:33.803234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.803234+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:56.407084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688073+0000","last_change":"2024-09-20T19:19:28.174295+0000","last_active":"2024-09-20T19:20:46.688073+0000","last_peered":"2024-09-20T19:20:46.688073+0000","last_clean":"2024-09-20T19:20:46.688073+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:20:46.688073+0000","last_undegraded":"2024-09-20T19:20:46.688073+0000","last_fullsized":"2024-09-20T19:20:46.688073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.174157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.174157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.549082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993709+0000","last_change":"2024-09-20T19:19:57.060943+0000","last_active":"2024-09-20T19:20:46.993709+0000","last_peered":"2024-09-20T19:20:46.993709+0000","last_clean":"2024-09-20T19:20:46.993709+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:20:46.993709+0000","last_undegraded":"2024-09-20T19:20:46.993709+0000","last_fullsized":"2024-09-20T19:20:46.993709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.060887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.060887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:03.058027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.688929+0000","last_change":"2024-09-20T19:19:54.030064+0000","last_active":"2024-09-20T19:20:46.688929+0000","last_peered":"2024-09-20T19:20:46.688929+0000","last_clean":"2024-09-20T19:20:46.688929+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:20:46.688929+0000","last_undegraded":"2024-09-20T19:20:46.688929+0000","last_fullsized":"2024-09-20T19:20:46.688929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.029937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.029937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:13.274260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999634+0000","last_change":"2024-09-20T19:20:37.743313+0000","last_active":"2024-09-20T19:20:46.999634+0000","last_peered":"2024-09-20T19:20:46.999634+0000","last_clean":"2024-09-20T19:20:46.999634+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:20:46.999634+0000","last_undegraded":"2024-09-20T19:20:46.999634+0000","last_fullsized":"2024-09-20T19:20:46.999634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:37.743188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.743188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:40.122912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":108,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001496+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:20:47.001496+0000","last_peered":"2024-09-20T19:20:47.001496+0000","last_clean":"2024-09-20T19:20:47.001496+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:20:47.001496+0000","last_undegraded":"2024-09-20T19:20:47.001496+0000","last_fullsized":"2024-09-20T19:20:47.001496+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:09.130493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":86,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.000144+0000","last_change":"2024-09-20T19:19:43.040517+0000","last_active":"2024-09-20T19:20:47.000144+0000","last_peered":"2024-09-20T19:20:47.000144+0000","last_clean":"2024-09-20T19:20:47.000144+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:20:47.000144+0000","last_undegraded":"2024-09-20T19:20:47.000144+0000","last_fullsized":"2024-09-20T19:20:47.000144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:43.040460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:43.040460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:10.254103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690818+0000","last_change":"2024-09-20T19:19:56.343379+0000","last_active":"2024-09-20T19:20:46.690818+0000","last_peered":"2024-09-20T19:20:46.690818+0000","last_clean":"2024-09-20T19:20:46.690818+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:20:46.690818+0000","last_undegraded":"2024-09-20T19:20:46.690818+0000","last_fullsized":"2024-09-20T19:20:46.690818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:56.343254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:56.343254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.336373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993860+0000","last_change":"2024-09-20T19:20:32.059785+0000","last_active":"2024-09-20T19:20:46.993860+0000","last_peered":"2024-09-20T19:20:46.993860+0000","last_clean":"2024-09-20T19:20:46.993860+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:20:46.993860+0000","last_undegraded":"2024-09-20T19:20:46.993860+0000","last_fullsized":"2024-09-20T19:20:46.993860+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:32.059603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.059603+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.930875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999080+0000","last_change":"2024-09-20T19:19:28.086132+0000","last_active":"2024-09-20T19:20:46.999080+0000","last_peered":"2024-09-20T19:20:46.999080+0000","last_clean":"2024-09-20T19:20:46.999080+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:20:46.999080+0000","last_undegraded":"2024-09-20T19:20:46.999080+0000","last_fullsized":"2024-09-20T19:20:46.999080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.086004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.086004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:51.650446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":129,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999657+0000","last_change":"2024-09-20T19:19:27.113335+0000","last_active":"2024-09-20T19:20:46.999657+0000","last_peered":"2024-09-20T19:20:46.999657+0000","last_clean":"2024-09-20T19:20:46.999657+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:20:46.999657+0000","last_undegraded":"2024-09-20T19:20:46.999657+0000","last_fullsized":"2024-09-20T19:20:46.999657+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.113211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.113211+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:50.889024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685348+0000","last_change":"2024-09-20T19:19:40.871513+0000","last_active":"2024-09-20T19:20:46.685348+0000","last_peered":"2024-09-20T19:20:46.685348+0000","last_clean":"2024-09-20T19:20:46.685348+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:20:46.685348+0000","last_undegraded":"2024-09-20T19:20:46.685348+0000","last_fullsized":"2024-09-20T19:20:46.685348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:40.871447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.871447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999801+0000","last_change":"2024-09-20T19:19:46.036288+0000","last_active":"2024-09-20T19:20:46.999801+0000","last_peered":"2024-09-20T19:20:46.999801+0000","last_clean":"2024-09-20T19:20:46.999801+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:20:46.999801+0000","last_undegraded":"2024-09-20T19:20:46.999801+0000","last_fullsized":"2024-09-20T19:20:46.999801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:19:46.036143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:46.036143+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.480915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685440+0000","last_change":"2024-09-20T19:19:50.860419+0000","last_active":"2024-09-20T19:20:46.685440+0000","last_peered":"2024-09-20T19:20:46.685440+0000","last_clean":"2024-09-20T19:20:46.685440+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:20:46.685440+0000","last_undegraded":"2024-09-20T19:20:46.685440+0000","last_fullsized":"2024-09-20T19:20:46.685440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.860284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.860284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:16.470934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":130,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993221+0000","last_change":"2024-09-20T19:20:20.964445+0000","last_active":"2024-09-20T19:20:46.993221+0000","last_peered":"2024-09-20T19:20:46.993221+0000","last_clean":"2024-09-20T19:20:46.993221+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:20:46.993221+0000","last_undegraded":"2024-09-20T19:20:46.993221+0000","last_fullsized":"2024-09-20T19:20:46.993221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:20.964261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:20.964261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.805123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734139+0000","last_change":"2024-09-20T19:20:43.861815+0000","last_active":"2024-09-20T19:20:46.734139+0000","last_peered":"2024-09-20T19:20:46.734139+0000","last_clean":"2024-09-20T19:20:46.734139+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:20:46.734139+0000","last_undegraded":"2024-09-20T19:20:46.734139+0000","last_fullsized":"2024-09-20T19:20:46.734139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:43.861696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.861696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:11.874804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":115,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902714+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:20:46.902714+0000","last_peered":"2024-09-20T19:20:46.902714+0000","last_clean":"2024-09-20T19:20:46.902714+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:20:46.902714+0000","last_undegraded":"2024-09-20T19:20:46.902714+0000","last_fullsized":"2024-09-20T19:20:46.902714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:32.267574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.000904+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:20:47.000904+0000","last_peered":"2024-09-20T19:20:47.000904+0000","last_clean":"2024-09-20T19:20:47.000904+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:20:47.000904+0000","last_undegraded":"2024-09-20T19:20:47.000904+0000","last_fullsized":"2024-09-20T19:20:47.000904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:33.835611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993124+0000","last_change":"2024-09-20T19:19:47.212665+0000","last_active":"2024-09-20T19:20:46.993124+0000","last_peered":"2024-09-20T19:20:46.993124+0000","last_clean":"2024-09-20T19:20:46.993124+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:20:46.993124+0000","last_undegraded":"2024-09-20T19:20:46.993124+0000","last_fullsized":"2024-09-20T19:20:46.993124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:47.212542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.212542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:56.674953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.683637+0000","last_change":"2024-09-20T19:19:57.895992+0000","last_active":"2024-09-20T19:20:46.683637+0000","last_peered":"2024-09-20T19:20:46.683637+0000","last_clean":"2024-09-20T19:20:46.683637+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:20:46.683637+0000","last_undegraded":"2024-09-20T19:20:46.683637+0000","last_fullsized":"2024-09-20T19:20:46.683637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:57.895923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:57.895923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:17.375254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733527+0000","last_change":"2024-09-20T19:19:27.179523+0000","last_active":"2024-09-20T19:20:46.733527+0000","last_peered":"2024-09-20T19:20:46.733527+0000","last_clean":"2024-09-20T19:20:46.733527+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:20:46.733527+0000","last_undegraded":"2024-09-20T19:20:46.733527+0000","last_fullsized":"2024-09-20T19:20:46.733527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:27.179398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:27.179398+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:47.909162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":111,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.904086+0000","last_change":"2024-09-20T19:20:17.522125+0000","last_active":"2024-09-20T19:20:46.904086+0000","last_peered":"2024-09-20T19:20:46.904086+0000","last_clean":"2024-09-20T19:20:46.904086+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:20:46.904086+0000","last_undegraded":"2024-09-20T19:20:46.904086+0000","last_fullsized":"2024-09-20T19:20:46.904086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:17.521947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:17.521947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.763307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.902242+0000","last_change":"2024-09-20T19:19:48.434045+0000","last_active":"2024-09-20T19:20:46.902242+0000","last_peered":"2024-09-20T19:20:46.902242+0000","last_clean":"2024-09-20T19:20:46.902242+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:20:46.902242+0000","last_undegraded":"2024-09-20T19:20:46.902242+0000","last_fullsized":"2024-09-20T19:20:46.902242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:48.433915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:48.433915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.655605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.991901+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:20:46.991901+0000","last_peered":"2024-09-20T19:20:46.991901+0000","last_clean":"2024-09-20T19:20:46.991901+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:20:46.991901+0000","last_undegraded":"2024-09-20T19:20:46.991901+0000","last_fullsized":"2024-09-20T19:20:46.991901+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.984866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":105,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.991953+0000","last_change":"2024-09-20T19:20:39.076366+0000","last_active":"2024-09-20T19:20:46.991953+0000","last_peered":"2024-09-20T19:20:46.991953+0000","last_clean":"2024-09-20T19:20:46.991953+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:20:46.991953+0000","last_undegraded":"2024-09-20T19:20:46.991953+0000","last_fullsized":"2024-09-20T19:20:46.991953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:20:39.076230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.076230+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.693807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.999919+0000","last_change":"2024-09-20T19:19:35.981274+0000","last_active":"2024-09-20T19:20:46.999919+0000","last_peered":"2024-09-20T19:20:46.999919+0000","last_clean":"2024-09-20T19:20:46.999919+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:20:46.999919+0000","last_undegraded":"2024-09-20T19:20:46.999919+0000","last_fullsized":"2024-09-20T19:20:46.999919+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:35.981166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:35.981166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:53.523226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.998917+0000","last_change":"2024-09-20T19:20:38.745217+0000","last_active":"2024-09-20T19:20:46.998917+0000","last_peered":"2024-09-20T19:20:46.998917+0000","last_clean":"2024-09-20T19:20:46.998917+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:20:46.998917+0000","last_undegraded":"2024-09-20T19:20:46.998917+0000","last_fullsized":"2024-09-20T19:20:46.998917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.745091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.745091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:38.749886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689534+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:20:46.689534+0000","last_peered":"2024-09-20T19:20:46.689534+0000","last_clean":"2024-09-20T19:20:46.689534+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:20:46.689534+0000","last_undegraded":"2024-09-20T19:20:46.689534+0000","last_fullsized":"2024-09-20T19:20:46.689534+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.905322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":118,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:48.936891+0000","last_change":"2024-09-20T19:20:48.936891+0000","last_active":"2024-09-20T19:20:48.936891+0000","last_peered":"2024-09-20T19:20:48.936891+0000","last_clean":"2024-09-20T19:20:48.936891+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:20:48.936891+0000","last_undegraded":"2024-09-20T19:20:48.936891+0000","last_fullsized":"2024-09-20T19:20:48.936891+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:48.936813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.936813+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:02.906803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001316+0000","last_change":"2024-09-20T19:20:29.725323+0000","last_active":"2024-09-20T19:20:47.001316+0000","last_peered":"2024-09-20T19:20:47.001316+0000","last_clean":"2024-09-20T19:20:47.001316+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:20:47.001316+0000","last_undegraded":"2024-09-20T19:20:47.001316+0000","last_fullsized":"2024-09-20T19:20:47.001316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:29.725195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:29.725195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:39.235455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689328+0000","last_change":"2024-09-20T19:19:50.321638+0000","last_active":"2024-09-20T19:20:46.689328+0000","last_peered":"2024-09-20T19:20:46.689328+0000","last_clean":"2024-09-20T19:20:46.689328+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:20:46.689328+0000","last_undegraded":"2024-09-20T19:20:46.689328+0000","last_fullsized":"2024-09-20T19:20:46.689328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:50.321520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:50.321520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:09.084680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.992567+0000","last_change":"2024-09-20T19:20:44.049456+0000","last_active":"2024-09-20T19:20:46.992567+0000","last_peered":"2024-09-20T19:20:46.992567+0000","last_clean":"2024-09-20T19:20:46.992567+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:20:46.992567+0000","last_undegraded":"2024-09-20T19:20:46.992567+0000","last_fullsized":"2024-09-20T19:20:46.992567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:44.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.049342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:04.326689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":85,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.690393+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:20:46.690393+0000","last_peered":"2024-09-20T19:20:46.690393+0000","last_clean":"2024-09-20T19:20:46.690393+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:20:46.690393+0000","last_undegraded":"2024-09-20T19:20:46.690393+0000","last_fullsized":"2024-09-20T19:20:46.690393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:22.371707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.998009+0000","last_change":"2024-09-20T19:19:23.109142+0000","last_active":"2024-09-20T19:20:46.998009+0000","last_peered":"2024-09-20T19:20:46.998009+0000","last_clean":"2024-09-20T19:20:46.998009+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:20:46.998009+0000","last_undegraded":"2024-09-20T19:20:46.998009+0000","last_fullsized":"2024-09-20T19:20:46.998009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:23.109064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:23.109064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685663+0000","last_change":"2024-09-20T19:19:55.848311+0000","last_active":"2024-09-20T19:20:46.685663+0000","last_peered":"2024-09-20T19:20:46.685663+0000","last_clean":"2024-09-20T19:20:46.685663+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:20:46.685663+0000","last_undegraded":"2024-09-20T19:20:46.685663+0000","last_fullsized":"2024-09-20T19:20:46.685663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:19:55.848164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:55.848164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:19.416599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":140,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001179+0000","last_change":"2024-09-20T19:20:44.717984+0000","last_active":"2024-09-20T19:20:47.001179+0000","last_peered":"2024-09-20T19:20:47.001179+0000","last_clean":"2024-09-20T19:20:47.001179+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:20:47.001179+0000","last_undegraded":"2024-09-20T19:20:47.001179+0000","last_fullsized":"2024-09-20T19:20:47.001179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:44.717840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.717840+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.300091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903180+0000","last_change":"2024-09-20T19:19:31.466386+0000","last_active":"2024-09-20T19:20:46.903180+0000","last_peered":"2024-09-20T19:20:46.903180+0000","last_clean":"2024-09-20T19:20:46.903180+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:20:46.903180+0000","last_undegraded":"2024-09-20T19:20:46.903180+0000","last_fullsized":"2024-09-20T19:20:46.903180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:31.466247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:31.466247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:59.461700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":104,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993096+0000","last_change":"2024-09-20T19:20:43.067477+0000","last_active":"2024-09-20T19:20:46.993096+0000","last_peered":"2024-09-20T19:20:46.993096+0000","last_clean":"2024-09-20T19:20:46.993096+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:20:46.993096+0000","last_undegraded":"2024-09-20T19:20:46.993096+0000","last_fullsized":"2024-09-20T19:20:46.993096+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:43.067414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.067414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.468070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":128,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.001161+0000","last_change":"2024-09-20T19:19:57.028481+0000","last_active":"2024-09-20T19:20:47.001161+0000","last_peered":"2024-09-20T19:20:47.001161+0000","last_clean":"2024-09-20T19:20:47.001161+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:20:47.001161+0000","last_undegraded":"2024-09-20T19:20:47.001161+0000","last_fullsized":"2024-09-20T19:20:47.001161+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:19:40.905378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:40.905378+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:49.869949+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.685651+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:20:46.685651+0000","last_peered":"2024-09-20T19:20:46.685651+0000","last_clean":"2024-09-20T19:20:46.685651+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:20:46.685651+0000","last_undegraded":"2024-09-20T19:20:46.685651+0000","last_fullsized":"2024-09-20T19:20:46.685651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.632034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.733975+0000","last_change":"2024-09-20T19:19:39.226277+0000","last_active":"2024-09-20T19:20:46.733975+0000","last_peered":"2024-09-20T19:20:46.733975+0000","last_clean":"2024-09-20T19:20:46.733975+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:20:46.733975+0000","last_undegraded":"2024-09-20T19:20:46.733975+0000","last_fullsized":"2024-09-20T19:20:46.733975+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:39.226160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:39.226160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:52.931829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:47.000655+0000","last_change":"2024-09-20T19:20:46.661313+0000","last_active":"2024-09-20T19:20:47.000655+0000","last_peered":"2024-09-20T19:20:47.000655+0000","last_clean":"2024-09-20T19:20:47.000655+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:20:47.000655+0000","last_undegraded":"2024-09-20T19:20:47.000655+0000","last_fullsized":"2024-09-20T19:20:47.000655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:46.661186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:46.661186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:50.055994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.689301+0000","last_change":"2024-09-20T19:19:47.048010+0000","last_active":"2024-09-20T19:20:46.689301+0000","last_peered":"2024-09-20T19:20:46.689301+0000","last_clean":"2024-09-20T19:20:46.689301+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:20:46.689301+0000","last_undegraded":"2024-09-20T19:20:46.689301+0000","last_fullsized":"2024-09-20T19:20:46.689301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:19:47.047816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.047816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:07.958306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.903781+0000","last_change":"2024-09-20T19:20:33.482733+0000","last_active":"2024-09-20T19:20:46.903781+0000","last_peered":"2024-09-20T19:20:46.903781+0000","last_clean":"2024-09-20T19:20:46.903781+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:20:46.903781+0000","last_undegraded":"2024-09-20T19:20:46.903781+0000","last_fullsized":"2024-09-20T19:20:46.903781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.482605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.482605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:42.867843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993194+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:20:46.993194+0000","last_peered":"2024-09-20T19:20:46.993194+0000","last_clean":"2024-09-20T19:20:46.993194+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:20:46.993194+0000","last_undegraded":"2024-09-20T19:20:46.993194+0000","last_fullsized":"2024-09-20T19:20:46.993194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:20.115239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993717+0000","last_change":"2024-09-20T19:19:58.028896+0000","last_active":"2024-09-20T19:20:46.993717+0000","last_peered":"2024-09-20T19:20:46.993717+0000","last_clean":"2024-09-20T19:20:46.993717+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:20:46.993717+0000","last_undegraded":"2024-09-20T19:20:46.993717+0000","last_fullsized":"2024-09-20T19:20:46.993717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:58.028776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:19:58.028776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:05.888058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.993253+0000","last_change":"2024-09-20T19:19:51.256022+0000","last_active":"2024-09-20T19:20:46.993253+0000","last_peered":"2024-09-20T19:20:46.993253+0000","last_clean":"2024-09-20T19:20:46.993253+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:20:46.993253+0000","last_undegraded":"2024-09-20T19:20:46.993253+0000","last_fullsized":"2024-09-20T19:20:46.993253+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:19:51.255877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.255877+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:14.975626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-20T19:20:46.734019+0000","last_change":"2024-09-20T19:19:28.186599+0000","last_active":"2024-09-20T19:20:46.734019+0000","last_peered":"2024-09-20T19:20:46.734019+0000","last_clean":"2024-09-20T19:20:46.734019+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:20:46.734019+0000","last_undegraded":"2024-09-20T19:20:46.734019+0000","last_fullsized":"2024-09-20T19:20:46.734019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:28.186507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:19:28.186507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:20:57.594221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877906986,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1253376,"data_stored":896140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234037,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28236,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716916,"statfs":{"total":95995035648,"available":95966121984,"internally_reserved":0,"allocated":1241088,"data_stored":895476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561087,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717492,"statfs":{"total":95995035648,"available":95966711808,"internally_reserved":0,"allocated":651264,"data_stored":306199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855435,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":659456,"data_stored":305186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":305973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378723,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":35988,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709164,"statfs":{"total":95995035648,"available":95958183936,"internally_reserved":0,"allocated":659456,"data_stored":306053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":306680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708580,"statfs":{"total":95995035648,"available":95957585920,"internally_reserved":0,"allocated":1257472,"data_stored":896406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:20:58.533 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T19:20:58.533 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump 2024-09-20T19:20:59.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:20:58 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:20:59.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:20:59 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.323956+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.326134+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: audit 2024-09-20T19:20:57.710611+0000 mgr.y (mgr.24491) 116 : audit [DBG] from='client.24689 v1:172.21.15.42:0/520427872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:20:59.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.741507+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.743496+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:57.930669+0000 mgr.y (mgr.24491) 117 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:20:59 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.709098+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.f scrub starts 2024-09-20T19:21:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:00.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:00.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:00 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.711463+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.f scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.723436+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.724828+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.812566+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:58.814916+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:21:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: audit 2024-09-20T19:20:58.958275+0000 mgr.y (mgr.24491) 118 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.814268+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:21:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:00 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.816501+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:21:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:01.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:20:59.931353+0000 mgr.y (mgr.24491) 119 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:01.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.847838+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.e scrub starts 2024-09-20T19:21:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.850225+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.e scrub ok 2024-09-20T19:21:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[22706]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:02 smithi042 bash[32075]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:02.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:02 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:02 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.307409+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:21:02.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:02 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:02.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:02 smithi162 bash[24557]: cluster 2024-09-20T19:21:00.309828+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:21:03.340 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:03.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[32075]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:03.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:03 smithi042 bash[22706]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:03.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.684258+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:21:03.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.687292+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.737405+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.a scrub starts 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.739482+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.a scrub ok 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:03.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:03 smithi162 bash[24557]: cluster 2024-09-20T19:21:01.932749+0000 mgr.y (mgr.24491) 120 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:04 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.682065+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.687216+0000 osd.1 (osd.1) 78 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.746456+0000 osd.5 (osd.5) 63 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:02.749631+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.021960+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:04 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.024065+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:21:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 109 2024-09-20T19:21:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:21:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-09-20T19:07:00.852407+0000 2024-09-20T19:21:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:modified 2024-09-20T19:20:55.011151+0000 2024-09-20T19:21:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:crush_version 18 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:full_ratio 0.95 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:backfillfull_ratio 0.9 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:nearfull_ratio 0.85 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:require_min_compat_client luminous 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:min_compat_client luminous 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:require_osd_release squid 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:stretch_mode_enabled false 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 79 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-20T19:21:05.000 INFO:teuthology.orchestra.run.smithi042.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 77 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:max_osd 8 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 up in weight 1 up_from 8 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6801/2265043130 v1:172.21.15.42:6802/2265043130 exists,up fb59ab36-8bbc-4e10-bc0f-b0cca84b317b 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6805/3513258758 v1:172.21.15.42:6806/3513258758 exists,up 1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2 up in weight 1 up_from 19 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6809/2620956901 v1:172.21.15.42:6810/2620956901 exists,up 12b15f1c-ba47-49cb-aefe-d99ff84d926d 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 up in weight 1 up_from 28 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6813/1507333597 v1:172.21.15.42:6814/1507333597 exists,up f0b69ee0-b93b-4ab6-8a35-783b18b9c73e 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4 up in weight 1 up_from 35 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6800/2898101120 v1:172.21.15.162:6801/2898101120 exists,up 90c5738d-14a9-415f-9249-eac0455635ad 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5 up in weight 1 up_from 44 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6804/4269525575 v1:172.21.15.162:6805/4269525575 exists,up e7648e52-3260-47ec-8778-35d4fa70ee27 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6 up in weight 1 up_from 54 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6808/1785075253 v1:172.21.15.162:6809/1785075253 exists,up 34b6c8df-9cb5-4fcb-955f-69fd6520465a 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7 up in weight 1 up_from 64 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6812/1845916231 v1:172.21.15.162:6813/1845916231 exists,up 6046a58c-ccf6-41ce-9888-b0c7a771205e 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 2.5 [7,2] 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 4.7 [1,2] 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 4.9 [1,5] 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 4.1f [1,2] 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/1743761075 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.001 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3575489582 expires 2024-09-21T19:07:29.716210+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6800/2546869704 expires 2024-09-21T19:07:29.716210+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/1361717385 expires 2024-09-21T19:08:07.715726+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/2117630454 expires 2024-09-21T19:08:07.715726+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/409595637 expires 2024-09-21T19:07:29.716210+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6800/2206296983 expires 2024-09-21T19:08:07.715726+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3240017151 expires 2024-09-21T19:08:07.715726+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6800/3764180564 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3774737654 expires 2024-09-21T19:07:29.716210+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3982292550 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/2784207196 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3505349280 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.002 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/368716713 expires 2024-09-21T19:18:55.853462+0000 2024-09-20T19:21:05.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[22706]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[22706]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[22706]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[32075]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[32075]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:05 smithi042 bash[32075]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:05 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:05 smithi162 bash[24557]: cluster 2024-09-20T19:21:03.933448+0000 mgr.y (mgr.24491) 121 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:05 smithi162 bash[24557]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:05 smithi162 bash[24557]: audit 2024-09-20T19:21:04.997331+0000 mon.a (mon.0) 973 : audit [DBG] from='client.? v1:172.21.15.42:0/1455670773' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-20T19:21:05.763 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6801/2265043130 v1:172.21.15.42:6802/2265043130 exists,up fb59ab36-8bbc-4e10-bc0f-b0cca84b317b', 'osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6805/3513258758 v1:172.21.15.42:6806/3513258758 exists,up 1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d', 'osd.2 up in weight 1 up_from 19 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6809/2620956901 v1:172.21.15.42:6810/2620956901 exists,up 12b15f1c-ba47-49cb-aefe-d99ff84d926d', 'osd.3 up in weight 1 up_from 28 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.42:6813/1507333597 v1:172.21.15.42:6814/1507333597 exists,up f0b69ee0-b93b-4ab6-8a35-783b18b9c73e', 'osd.4 up in weight 1 up_from 35 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6800/2898101120 v1:172.21.15.162:6801/2898101120 exists,up 90c5738d-14a9-415f-9249-eac0455635ad', 'osd.5 up in weight 1 up_from 44 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6804/4269525575 v1:172.21.15.162:6805/4269525575 exists,up e7648e52-3260-47ec-8778-35d4fa70ee27', 'osd.6 up in weight 1 up_from 54 up_thru 101 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6808/1785075253 v1:172.21.15.162:6809/1785075253 exists,up 34b6c8df-9cb5-4fcb-955f-69fd6520465a', 'osd.7 up in weight 1 up_from 64 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6812/1845916231 v1:172.21.15.162:6813/1845916231 exists,up 6046a58c-ccf6-41ce-9888-b0c7a771205e'] 2024-09-20T19:21:05.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon mon.a config show 2024-09-20T19:21:06.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:06 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:06.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:06 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:06 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.013529+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:21:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:06 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:06 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.014525+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:21:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:07.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:07 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.737950+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.d scrub starts 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.740267+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.d scrub ok 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.795686+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.797697+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.821637+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.825088+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:05.934051+0000 mgr.y (mgr.24491) 122 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.021782+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.023252+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.873687+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:07 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.877438+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:08.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:08 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.342927+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.345362+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.772296+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.774538+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:06.999594+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.001945+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:21:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.890191+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:21:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.892446+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:21:09.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:08 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:09.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:09 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:09.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.742517+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.744652+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.799861+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.801939+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.851002+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.855818+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:07.935484+0000 mgr.y (mgr.24491) 123 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.916053+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:09.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:09 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.918434+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:21:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:10 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.757130+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.759489+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.844555+0000 osd.0 (osd.0) 69 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:08.846649+0000 osd.0 (osd.0) 70 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: audit 2024-09-20T19:21:08.963670+0000 mgr.y (mgr.24491) 124 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.035472+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:10 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.037835+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:21:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:11.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:09.936176+0000 mgr.y (mgr.24491) 125 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.214332+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:11 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.216688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:21:11.801 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:21:11 smithi162 bash[51183]: logger=infra.usagestats t=2024-09-20T19:21:11.40571069Z level=info msg="Usage stats are ready to report" 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "mon.a", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster": "ceph", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "admin_socket_mode": "", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_client_required": "cephx, none", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_cluster_required": "cephx", 2024-09-20T19:21:12.087 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_debug": "true", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_service_required": "cephx", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "auth_supported": "", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_aio": "true", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_aio_poll_ms": "250", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_aio_reap_max": "16", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_async_discard_threads": "0", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_block_size": "4096", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_debug_aio": "false", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-20T19:21:12.088 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_debug_inflight_ios": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_enable_discard": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_flock_retry": "3", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_inject_crash": "0", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_ioring": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_ioring_hipri": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-20T19:21:12.089 INFO:teuthology.orchestra.run.smithi042.stdout: "bdev_type": "", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_alloc_size": "1048576", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_allocator": "hybrid", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_buffered_io": "true", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_check_for_zeros": "false", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_compact_log_sync": "false", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-20T19:21:12.090 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_max_log_runway": "4194304", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_max_prefetch": "1048576", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_min_flush_size": "524288", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_min_log_runway": "1048576", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_replay_recovery": "false", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluefs_sync_write": "false", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_allocation_from_file": "true", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_allocator": "hybrid", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_async_db_compaction": "true", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-20T19:21:12.091 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bdev_label_multi": "true", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_create": "true", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_db_create": "false", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_db_path": "", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_db_size": "0", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_path": "", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_preallocate_file": "false", 2024-09-20T19:21:12.092 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_size": "107374182400", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_wal_create": "false", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_wal_path": "", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_block_wal_size": "100663296", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bluefs": "true", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-20T19:21:12.093 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_autotune": "true", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_size": "0", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cache_type": "2q", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_clone_cow": "true", 2024-09-20T19:21:12.094 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_mode": "none", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_csum_type": "crc32c", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_freelist": "false", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-20T19:21:12.095 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_small_allocations": "0", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-20T19:21:12.096 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_default_buffered_read": "true", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_default_buffered_write": "false", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fail_eio": "false", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-20T19:21:12.097 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_mount": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_umount": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-20T19:21:12.098 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_ignore_data_csum": "false", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_log_op_age": "5.000000", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_alloc_size": "0", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_blob_size": "0", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_max_deferred_txc": "32", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_min_alloc_size": "0", 2024-09-20T19:21:12.099 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_nid_prealloc": "1024", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_retry_disk_reads": "3", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_rocksdb_cf": "true", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-20T19:21:12.100 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_spdk_mem": "512", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_tracing": "false", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-20T19:21:12.101 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_write_v2": "false", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_write_v2_random": "false", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "bluestore_zero_block_detection": "false", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-20T19:21:12.102 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_cluster_require_signatures": "false", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_cluster_require_version": "2", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_require_signatures": "false", 2024-09-20T19:21:12.103 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_require_version": "2", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_service_require_signatures": "false", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_service_require_version": "2", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "cephx_sign_messages": "true", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "chdir": "", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_acl_type": "", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_asio_thread_count": "2", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_cache_mid": "0.750000", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_cache_size": "16384", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_caps_release_delay": "5", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_check_pool_perm": "true", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_debug_force_sync_read": "false", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_debug_getattr_caps": "false", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_debug_inject_features": "", 2024-09-20T19:21:12.104 INFO:teuthology.orchestra.run.smithi042.stdout: "client_debug_inject_tick_delay": "0", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_die_on_failed_remount": "false", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_dirsize_rbytes": "true", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_force_lazyio": "false", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_fs": "", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_inject_release_failure": "false", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_max_inline_size": "4096", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_mds_namespace": "", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_metadata": "", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_mount_gid": "-1", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_mount_timeout": "300", 2024-09-20T19:21:12.105 INFO:teuthology.orchestra.run.smithi042.stdout: "client_mount_uid": "-1", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_mountpoint": "/", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_notify_timeout": "10", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc": "true", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc_max_dirty": "104857600", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc_max_objects": "1000", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc_size": "209715200", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_oc_target_dirty": "8388608", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_permissions": "true", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_quota": "true", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_quota_df": "true", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_readahead_max_bytes": "0", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_readahead_max_periods": "4", 2024-09-20T19:21:12.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_readahead_min": "131072", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_reconnect_stale": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_shutdown_timeout": "30", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_snapdir": ".snap", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_tick_interval": "1", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_trace": "", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_try_dentry_invalidate": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_use_faked_inos": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_use_random_mds": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_graylog": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_graylog_port": "12201", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_monitors": "default=true", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_syslog": "false", 2024-09-20T19:21:12.107 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "clog_to_syslog_level": "info", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "-", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_network": "", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_network_interface": "", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "compressor_zlib_isal": "false", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "compressor_zlib_level": "5", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "compressor_zlib_winsize": "-15", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "compressor_zstd_level": "1", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8a1a6edbc97129e01d4344a6578add33aaab808082c84049bd2e48e1f97b80d2", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_alien_op_num_threads": "6", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-20T19:21:12.108 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_osd_stat_interval": "0", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_seastar_cpu_cores": "", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crimson_seastar_num_threads": "0", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location_hook": "", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location_hook_timeout": "10", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "daemonize": "false", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "daos_pool": "tank", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_alienstore": "0/5", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_allow_any_pool_priority": "false", 2024-09-20T19:21:12.109 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_asok": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_asok_assert_abort": "false", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_asserts_on_shutdown": "true", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_auth": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_bdev": "1/3", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_bluefs": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_bluestore": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_buffer": "0/1", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_ceph_dedup": "0/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_ceph_exporter": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_cephfs_mirror": "0/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_cephsqlite": "0/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_client": "0/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_compressor": "1/5", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_context": "0/1", 2024-09-20T19:21:12.110 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_crimson_interrupt": "0/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_crush": "1/1", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_crypto": "1/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_cyanstore": "0/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_deliberately_leak_memory": "false", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_disable_randomized_ping": "false", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_dpdk": "1/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_eventtrace": "1/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_filer": "0/1", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_filestore": "1/3", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_finisher": "1/1", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_fuse": "1/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_heartbeat_testing_span": "0", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_heartbeatmap": "1/5", 2024-09-20T19:21:12.111 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_javaclient": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_journal": "1/3", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_journaler": "0/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_kstore": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_lockdep": "0/1", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mclock": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_balancer": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_locker": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_log": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_log_expire": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_migrator": "1/5", 2024-09-20T19:21:12.112 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mds_quiesce": "3/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_memstore": "1/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mgr": "2/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mgrc": "1/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_mon": "20/20", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_monc": "0/10", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_ms": "1/1", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_none": "0/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_objclass": "0/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_objectcacher": "0/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_objecter": "0/1", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_optracker": "0/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_osd": "1/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_paxos": "20/20", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_perfcounter": "1/5", 2024-09-20T19:21:12.113 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_prioritycache": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rados": "0/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rbd": "0/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rbd_mirror": "0/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rbd_pwl": "0/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rbd_replay": "0/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_refs": "0/0", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_reserver": "1/1", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_access": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_datacache": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_dbstore": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_flight": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_notification": "1/5", 2024-09-20T19:21:12.114 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rgw_sync": "1/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_rocksdb": "4/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_backref": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_cache": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_cleaner": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_device": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_epm": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_journal": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_lba": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_odata": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_omap": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_onode": "0/5", 2024-09-20T19:21:12.115 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_t": "0/5", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_seastore_tm": "0/5", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_striper": "0/1", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_test": "0/5", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_throttle": "1/1", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_timer": "0/1", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_tp": "0/5", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "debug_trace": "1/5", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "device_failure_prediction_mode": "none", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "err_to_graylog": "false", 2024-09-20T19:21:12.116 INFO:teuthology.orchestra.run.smithi042.stdout: "err_to_journald": "false", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "err_to_stderr": "true", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "err_to_syslog": "false", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "event_tracing": "false", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_addr": "0.0.0.0", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_cert_file": "", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_http_port": "9926", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_key_file": "", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_prio_limit": "5", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_sort_metrics": "true", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "exporter_stats_period": "5", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "fake_statfs_for_testing": "0", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "fatal_signal_handlers": "true", 2024-09-20T19:21:12.117 INFO:teuthology.orchestra.run.smithi042.stdout: "filer_max_purge_ops": "10", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filer_max_truncate_ops": "128", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_apply_finisher_threads": "1", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_blackhole": "false", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_btrfs_clone_range": "true", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_btrfs_snap": "true", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_caller_concurrency": "10", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_collect_device_partition_information": "true", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_commit_timeout": "600.000000", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_debug_inject_read_err": "false", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_debug_omap_check": "false", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_debug_verify_split": "false", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_dump_file": "", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-20T19:21:12.118 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fadvise": "true", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fail_eio": "true", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fd_cache_shards": "16", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fd_cache_size": "128", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fiemap": "false", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fiemap_threshold": "4096", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_inject_stall": "0", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_journal_parallel": "false", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_journal_trailing": "false", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_journal_writeahead": "false", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_kill_at": "0", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-20T19:21:12.119 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattrs": "0", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_xattr_value_size": "0", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_merge_threshold": "-10", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-20T19:21:12.120 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_odsync_write": "false", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_omap_backend": "rocksdb", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_omap_backend_path": "", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_op_thread_timeout": "60", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_op_threads": "2", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_punch_hole": "false", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-20T19:21:12.121 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_queue_max_ops": "50", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_seek_data_hole": "false", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_sloppy_crc": "false", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_splice": "false", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_split_multiple": "2", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_split_rand_factor": "20", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_update_to": "1000", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-20T19:21:12.122 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_enable": "true", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_xfs_extsize": "false", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "filestore_zfs_snap": "false", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "fio_dir": "/tmp/fio", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:21:12.123 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_allow_other": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_atomic_o_trunc": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_big_writes": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_debug": "false", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_default_permissions": "false", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_disable_pagecache": "false", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_max_write": "0", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_multithreaded": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_require_active_mds": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_set_user_groups": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_splice_move": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_splice_read": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_splice_write": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "fuse_use_invalidate_cb": "true", 2024-09-20T19:21:12.124 INFO:teuthology.orchestra.run.smithi042.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "gss_target_name": "ceph", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_file": "", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_inject_failure": "0", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_interval": "5", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "host": "smithi042", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-20T19:21:12.125 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "inject_early_sigterm": "false", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "jaeger_agent_port": "6799", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "jaeger_tracing_enable": "false", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_aio": "true", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_align_min_size": "65536", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_block_align": "true", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_block_size": "4096", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_dio": "true", 2024-09-20T19:21:12.126 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_discard": "false", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_force_aio": "false", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_ignore_corruption": "false", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_max_write_bytes": "10485760", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_max_write_entries": "100", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_replay_from": "0", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_write_header_frequency": "0", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journal_zero_on_create": "false", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journaler_prefetch_periods": "10", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journaler_prezero_periods": "5", 2024-09-20T19:21:12.127 INFO:teuthology.orchestra.run.smithi042.stdout: "journaler_write_head_interval": "15", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "key": "", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "keyfile": "", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_backend": "rocksdb", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_default_stripe_size": "65536", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_fsck_on_mount": "false", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_max_bytes": "67108864", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_max_ops": "512", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_nid_prealloc": "1024", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_onode_map_size": "1024", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-20T19:21:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_sync_submit_transaction": "false", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "kstore_sync_transaction": "false", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "librados_thread_count": "2", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "lockdep": "false", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "lockdep_force_backtrace": "false", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_coarse_timestamps": "true", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_flush_on_exit": "false", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_graylog_host": "127.0.0.1", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_graylog_port": "12201", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_max_new": "1000", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_max_recent": "10000", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_stderr_prefix": "debug ", 2024-09-20T19:21:12.129 INFO:teuthology.orchestra.run.smithi042.stdout: "log_stop_at_utilization": "0.970000", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "log_to_file": "true", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "log_to_graylog": "false", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "log_to_journald": "false", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "log_to_stderr": "false", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "log_to_syslog": "false", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "max_rotating_auth_attempts": "10", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_action_on_write_error": "1", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_alternate_name_max": "8192", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_asio_thread_count": "2", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_export_pin": "true", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_fragment_dirs": "true", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-20T19:21:12.130 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_fragment_interval": "5", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_interval": "10", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_max": "-1", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_max_until": "-1", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_merge_size": "50", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_midchunk": "0.300000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_min_start": "0.200000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_minchunk": "0.001000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_mode": "0", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_need_max": "1.200000", 2024-09-20T19:21:12.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_need_min": "0.800000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_overload_epochs": "2", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_split_bits": "3", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_split_size": "10000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_target_decay": "10.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_beacon_grace": "15.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_beacon_interval": "4.000000", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-20T19:21:12.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_mid": "0.700000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_quiesce_delay": "0", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_release_free_interval": "10", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_reservation": "0.050000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_trim_interval": "1", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cache_trim_threshold": "262144", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_client_prealloc_inos": "1000", 2024-09-20T19:21:12.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_connect_bootstrapping": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_damage_table_max_entries": "10000", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_debug_auth_pins": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_debug_frag": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_debug_scatterstat": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_debug_subtrees": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_decay_halflife": "5.000000", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_default_dir_hash": "2", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_defer_session_stale": "true", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_deny_all_reconnect": "false", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dir_keys_per_op": "16384", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dir_max_commit_size": "10", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dir_max_entries": "0", 2024-09-20T19:21:12.134 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dir_prefetch": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dump_cache_on_map": "false", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_early_reply": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_enable_op_tracker": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_enforce_unique_name": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_export_ephemeral_random": "true", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-20T19:21:12.135 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_health_summarize_threshold": "10", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_health_dummy": "false", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_migrator_session_race": "false", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-20T19:21:12.136 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_join_fs": "", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_journal_format": "1", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_create_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_dirfrag_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_export_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_import_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_journal_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_journal_expire_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_journal_replay_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_link_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_mdstable_at": "0", 2024-09-20T19:21:12.137 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_openc_at": "0", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_rename_at": "0", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_kill_shutdown_at": "0", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_event_large_threshold": "524288", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_events_per_segment": "1024", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_max_events": "-1", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_max_segments": "128", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_pause": "false", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_segment_size": "0", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-20T19:21:12.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_trim_threshold": "128", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_log_warn_factor": "2.000000", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_caps_per_client": "1048576", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_completed_flushes": "100000", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_completed_requests": "100000", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_export_size": "20971520", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_file_recover": "32", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_purge_files": "64", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_purge_ops": "8192", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_max_snaps_per_dir": "100", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_metrics_update_interval": "2", 2024-09-20T19:21:12.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_min_caps_per_client": "100", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_min_caps_working_set": "10000", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_numa_node": "-1", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_complaint_time": "30.000000", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_history_duration": "600", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_history_size": "20", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_history_slow_op_size": "20", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_op_log_threshold": "5", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_ping_grace": "15", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_ping_interval": "5", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-20T19:21:12.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_max_caps": "30000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_recall_warning_threshold": "262144", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_replay_interval": "1.000000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_root_ino_gid": "0", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_root_ino_uid": "0", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-20T19:21:12.141 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_scrub_stats_review_period": "1", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_shutdown_check": "0", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_skip_ino": "0", 2024-09-20T19:21:12.142 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_snap_max_uid": "4294967294", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_snap_min_uid": "0", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_snap_rstat": "false", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_standby_replay_damaged": "false", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_symlink_recovery": "true", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_thrash_exports": "0", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_thrash_fragments": "0", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_tick_interval": "5.000000", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_valgrind_exit": "false", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_verify_backtrace": "1", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_verify_scatter": "false", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-20T19:21:12.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_wipe_sessions": "false", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mempool_debug": "false", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "memstore_device_bytes": "1073741824", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "memstore_page_set": "false", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "memstore_page_size": "65536", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_client_bytes": "134217728", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_client_messages": "512", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_disabled_modules": "", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_enable_op_tracker": "true", 2024-09-20T19:21:12.144 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_max_pg_creating": "1024", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_max_pg_num_change": "128", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_mds_bytes": "134217728", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_mds_messages": "128", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_mon_bytes": "134217728", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_mon_messages": "128", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_history_duration": "600", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_history_size": "20", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-20T19:21:12.145 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_op_log_threshold": "5", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_osd_bytes": "536870912", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_osd_messages": "8192", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_pool": "true", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_standby_modules": "true", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_stats_period": "5", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_stats_threshold": "5", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_tick_period": "2", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_allow_pool_delete": "true", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_allow_pool_size_one": "false", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_auth_validate_all_caps": "true", 2024-09-20T19:21:12.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_bytes": "104857600", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_directed_command_retry": "2", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_hunt_parallel": "3", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_log_interval": "1.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_ping_interval": "10.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_client_target_rank": "-1", 2024-09-20T19:21:12.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_level": "debug", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_file": "true", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_journald": "false", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_stderr": "true", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_compact_on_bootstrap": "false", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_compact_on_start": "false", 2024-09-20T19:21:12.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_compact_on_trim": "true", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_cpu_threads": "4", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_crush_min_required_version": "hammer", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_daemon_bytes": "419430400", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_data_avail_crit": "5", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_data_avail_warn": "5", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_data_size_warn": "16106127360", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_dump_json": "false", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-20T19:21:12.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_dump_transactions": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_extra_checks": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_no_require_reef": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_no_require_squid": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_down_mkfs_grace": "60", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_down_uptime_grace": "60", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_election_default_strategy": "1", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_election_timeout": "5.000000", 2024-09-20T19:21:12.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_elector_ping_divisor": "2", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_enable_op_tracker": "true", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_fake_pool_delete": "false", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_force_quorum_join": "false", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_globalid_prealloc": "10000", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_detail_to_clog": "true", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_log_update_period": "5", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_max_detail": "50", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_to_clog": "true", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_to_clog_interval": "600", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-20T19:21:12.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_host": "172.21.15.42:6789/0", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_host_override": "", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_initial_members": "", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_lease": "5.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_log_full_interval": "50", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_log_max": "10000", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_log_max_summary": "50", 2024-09-20T19:21:12.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_log_epochs": "500", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_osd": "10000", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_pg_per_osd": "10000", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_pool_pg_num": "65536", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mds_force_trim_to": "0", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mds_skip_sanity": "false", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_memory_autotune": "true", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_memory_target": "2147483648", 2024-09-20T19:21:12.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_beacon_grace": "30", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_digest_period": "5", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_inactive_grace": "60", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_min_osdmap_epochs": "50", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_complaint_time": "30", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_history_duration": "600", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_history_size": "20", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_history_slow_op_size": "20", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-20T19:21:12.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_op_log_threshold": "5", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_auto_mark_in": "false", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_cache_size": "500", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_down_out_interval": "600", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-20T19:21:12.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_force_trim_to": "0", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_min_down_reporters": "2", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-20T19:21:12.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_report_timeout": "900", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_stuck_threshold": "60", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-20T19:21:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_probe_timeout": "2.000000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_reweight_max_change": "0.050000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_reweight_max_osds": "4", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_scrub_interval": "86400", 2024-09-20T19:21:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_scrub_max_keys": "100", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_scrub_timeout": "300", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_session_timeout": "300", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_smart_report_timeout": "5", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_stat_smooth_intervals": "6", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_stretch_pool_min_size": "2", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_stretch_pool_size": "4", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_debug": "false", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_provider_kill_at": "0", 2024-09-20T19:21:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_requester_kill_at": "0", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_sync_timeout": "60.000000", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_target_pg_per_osd": "100", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_tick_interval": "5", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_timecheck_interval": "300.000000", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_use_min_delay_socket": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_older_version_delay": "604800", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_misplaced": "false", 2024-09-20T19:21:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_older_version": "true", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": "", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-20T19:21:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "motr_tracing_enabled": "false", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_op_threads": "3", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_cm": "false", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_device_name": "", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_dscp": "96", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_local_gid": "", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_port_num": "1", 2024-09-20T19:21:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_sl": "3", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_support_srq": "true", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_rdma_type": "ib", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_async_reap_threshold": "5", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_before_connect": "false", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_ipv4": "true", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_ipv6": "false", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_msgr1": "true", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_msgr2": "false", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_port_max": "7568", 2024-09-20T19:21:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_port_min": "6800", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_retry_count": "3", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_bind_retry_delay": "5", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_blackhole_client": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_blackhole_mds": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_blackhole_mgr": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_blackhole_mon": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_blackhole_osd": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_client_mode": "crc secure", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_cluster_mode": "crc secure", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_cluster_type": "", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_compress_secure": "false", 2024-09-20T19:21:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_connection_idle_timeout": "900", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_connection_ready_timeout": "10", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_crc_data": "true", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_crc_header": "true", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_die_on_bad_msg": "false", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_die_on_bug": "true", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_die_on_old_message": "true", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_die_on_skipped_message": "false", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_coremask": "0xF", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_enable_tso": "true", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-20T19:21:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_hugepages": "", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_lro": "true", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_memory_channel": "4", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_pmd": "", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_port_id": "0", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_dump_on_send": "false", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_initial_backoff": "0.200000", 2024-09-20T19:21:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_delay_max": "1.000000", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_delay_type": "", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_network_congestion": "0", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_inject_socket_failures": "0", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_learn_addr_from_peer": "true", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_max_accept_failures": "4", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_max_backoff": "15.000000", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_mon_client_mode": "secure crc", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_mon_service_mode": "secure crc", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_osd_compress_min_size": "1024", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_osd_compress_mode": "none", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-20T19:21:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_pq_min_cost": "65536", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_public_type": "", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_service_mode": "crc secure", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_tcp_listen_backlog": "512", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_tcp_nodelay": "true", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_tcp_rcvbuf": "0", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "ms_type": "async+posix", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "no_config_file": "false", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_completion_locks_per_session": "32", 2024-09-20T19:21:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_inflight_ops": "1024", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_tick_interval": "5.000000", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objecter_timeout": "10.000000", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objectstore_blackhole": "false", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "openssl_engine_opts": "", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_delay_time": "5.000000", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_hist_halflife": "1000", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_max_low_ops": "2", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_max_ops": "4", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-20T19:21:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_agent_slop": "0.020000", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_asio_thread_count": "2", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_async_recovery_min_cost": "100", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backfill_scan_max": "512", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backfill_scan_min": "64", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backoff_on_degraded": "false", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backoff_on_peering": "false", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_backoff_on_unfound": "true", 2024-09-20T19:21:12.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_beacon_report_interval": "300", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_bench_duration": "30", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_bench_max_block_size": "67108864", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_blkin_trace_all": "false", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_check_for_log_corruption": "false", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-20T19:21:12.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_class_update_on_start": "true", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_client_message_cap": "256", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_client_message_size_cap": "524288000", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_client_op_priority": "63", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_client_watch_timeout": "30", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_command_max_records": "256", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_compact_on_start": "false", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_crush_update_on_start": "true", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_crush_update_weight_set": "true", 2024-09-20T19:21:12.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_feed_pullee": "-1", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_misdirected_ops": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_no_acting_change": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_no_purge_strays": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_op_order": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-20T19:21:12.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_shutdown": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_trim_objects": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_verify_snaps": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_keys": "1024", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-20T19:21:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_stride": "524288", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_default_notify_timeout": "30", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_delete_sleep": "0.000000", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_discard_disconnected_ops": "true", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_ec_partial_reads": "true", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_enable_op_tracker": "true", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-20T19:21:12.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_fast_info": "true", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_fast_shutdown": "true", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_function_tracing": "false", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_grace": "20", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_interval": "6", 2024-09-20T19:21:12.176 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_min_peers": "10", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_min_size": "2000", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_stale": "600", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_hit_set_max_size": "100000", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_hit_set_min_size": "1000", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_journal_size": "5120", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_kill_backfill_at": "0", 2024-09-20T19:21:12.177 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_map_cache_size": "50", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_map_dedup": "true", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_map_message_max": "40", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_map_share_max_epochs": "40", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_attr_name_len": "100", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_attr_size": "0", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_backfills": "1", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_markdown_count": "5", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_markdown_period": "600", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_object_name_len": "2048", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_object_namespace_len": "256", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_object_size": "134217728", 2024-09-20T19:21:12.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_pg_blocked_by": "16", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_pg_log_entries": "10000", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_pgls": "1024", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_push_cost": "8388608", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_push_objects": "10", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_scrubs": "3", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_trimming_pgs": "2", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_write_op_reply_len": "64", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_max_write_size": "90", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-20T19:21:12.179 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_profile": "balanced", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-20T19:21:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_base": "805306368", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_cache_min": "134217728", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_target": "4294967296", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_target_autotune": "false", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_min_pg_log_entries": "250", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_min_recovery_priority": "0", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-20T19:21:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mon_report_interval": "5", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_num_cache_shards": "32", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_num_op_tracker_shard": "32", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_numa_auto_affinity": "true", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_numa_node": "-1", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_numa_prefer_iface": "true", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_objecter_finishers": "1", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_objectstore": "bluestore", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_objectstore_fuse": "false", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_objectstore_tracing": "false", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_complaint_time": "30.000000", 2024-09-20T19:21:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_history_duration": "600", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_history_size": "20", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_history_slow_op_size": "20", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_log_threshold": "5", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_shards": "0", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_shards_hdd": "5", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_shards_ssd": "8", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_pq_min_cost": "65536", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_queue_cut_off": "high", 2024-09-20T19:21:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_op_thread_timeout": "15", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_open_classes_on_start": "true", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_os_flags": "0", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_peering_op_priority": "255", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_delete_cost": "1048576", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_delete_priority": "5", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_log_trim_max": "10000", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_log_trim_min": "100", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-20T19:21:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_crimson": "false", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-20T19:21:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_flags": "0", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_min_size": "0", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_pg_num": "32", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_pgp_num": "0", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_read_ratio": "70", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_size": "3", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_default_type": "replicated", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_push_per_object_cost": "1000", 2024-09-20T19:21:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recover_clone_overlap": "true", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_cost": "20971520", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_active": "0", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_max_single_start": "1", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_op_priority": "3", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_priority": "5", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-20T19:21:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_sleep": "0.000000", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_repair_during_recovery": "false", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_requested_scrub_priority": "5", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_auto_repair": "false", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_begin_hour": "0", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_begin_week_day": "0", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_chunk_max": "15", 2024-09-20T19:21:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_chunk_min": "5", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_cost": "52428800", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_during_recovery": "false", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_end_hour": "0", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_end_week_day": "0", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_event_cost": "4096", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_invalid_stats": "true", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_max_preemptions": "5", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_priority": "5", 2024-09-20T19:21:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_retry_delay": "30", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_retry_trimming": "10", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_scrub_sleep": "0.000000", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_skip_data_digest": "false", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_smart_report_timeout": "5", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_cost": "1048576", 2024-09-20T19:21:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_priority": "5", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_target_transaction_size": "30", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-20T19:21:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_tracing": "false", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_use_stale_snap": "false", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "osdc_blkin_trace_all": "false", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_kill_at": "0", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_max_join_drift": "10", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_min": "500", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_min_wait": "0.050000", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_propose_interval": "1.000000", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_service_trim_max": "500", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_service_trim_min": "10", 2024-09-20T19:21:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_stash_full_interval": "25", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_trim_max": "500", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "paxos_trim_min": "250", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "perf": "true", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "pid_file": "", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "-", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrv": "", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_bind_addr": "-", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_network": "", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_network_interface": "", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "qat_compressor_busy_polling": "false", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "qat_compressor_enabled": "false", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "qat_compressor_session_max_number": "256", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "rados_mon_op_timeout": "0", 2024-09-20T19:21:12.193 INFO:teuthology.orchestra.run.smithi042.stdout: "rados_osd_op_timeout": "0", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rados_tracing": "false", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_atime_update_interval": "60", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_balance_parent_reads": "false", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_balance_snap_reads": "false", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_blkin_trace_all": "false", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache": "true", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-20T19:21:12.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_policy": "writearound", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_size": "33554432", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_clone_copy_on_read": "false", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_compression_hint": "none", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_concurrent_management_ops": "10", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_clone_format": "auto", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_data_pool": "", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_features": "61", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_format": "2", 2024-09-20T19:21:12.195 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_map_options": "", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_order": "22", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_pool": "rbd", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_stripe_count": "0", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_default_stripe_unit": "0", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_enable_alloc_hint": "true", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_io_scheduler": "simple", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-20T19:21:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_order": "24", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_pool": "", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_journal_splay_width": "4", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_localize_parent_reads": "false", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_localize_snap_reads": "false", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-20T19:21:12.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-20T19:21:12.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-20T19:21:12.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_mtime_update_interval": "60", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_non_blocking_aio": "true", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_op_thread_timeout": "60", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_op_threads": "1", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_parent_cache_enabled": "false", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_plugins": "", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_bps_burst": "0", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_bps_limit": "0", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_exclude_ops": "0", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_iops_burst": "0", 2024-09-20T19:21:12.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_iops_limit": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-20T19:21:12.201 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_read_from_replica_policy": "default", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_skip_partial_discard": "true", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_tracing": "false", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_validate_names": "true", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd_validate_pool": "true", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "restapi_base_url": "", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "restapi_log_level": "", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-20T19:21:12.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_acl_grants_max_num": "100", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_admin_entry": "admin", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_asio_assert_yielding": "false", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_backend_store": "rados", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_barbican_url": "", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_beast_enable_async": "true", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_counters_cache": "false", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-20T19:21:12.203 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_cache_enabled": "true", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_cache_expiry_interval": "900", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_cache_lru_size": "25000", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_config_store": "rados", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_content_length_compat": "false", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_copy_obj_progress": "true", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_cors_rules_max_num": "100", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_cross_domain_policy": "", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_addr": "", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-20T19:21:12.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_password": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_kmip_username": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_require_ssl": "true", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-20T19:21:12.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_addr": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_auth": "token", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_namespace": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_prefix": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-20T19:21:12.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_token_file": "", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_curl_buffersize": "524288", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_curl_low_speed_time": "300", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-20T19:21:12.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_log_changes_size": "1000", 2024-09-20T19:21:12.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_log_num_shards": "128", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_log_window": "30", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-20T19:21:12.209 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_disable_s3select": "false", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-20T19:21:12.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dns_name": "", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dns_s3website_name": "", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dynamic_resharding": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_gc_threads": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_lc_threads": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_mdsearch": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_ops_log": "false", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_quota_threads": "true", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_static_website": "false", 2024-09-20T19:21:12.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enable_usage_log": "false", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_enforce_swift_acls": "true", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_exit_timeout_secs": "120", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_expose_bucket": "false", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_extended_http_attrs": "", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_filter": "none", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_frontends": "beast port=7480", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_deferred": "50", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_objs": "32", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-20T19:21:12.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_gc_processor_period": "3600", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_init_timeout": "300", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_kafka_connection_idle": "300", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-20T19:21:12.213 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_domain": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_password": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_password_path": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_project": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_tenant": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_token": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_token_path": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_admin_user": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_barbican_domain": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_barbican_password": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_barbican_project": "", 2024-09-20T19:21:12.214 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_barbican_user": "", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_url": "", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_debug_interval": "-1", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_lock_max_time": "90", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_max_objs": "32", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_max_rules": "1000", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_max_worker": "3", 2024-09-20T19:21:12.215 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lc_thread_delay": "0", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_dnattr": "uid", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_searchfilter": "", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ldap_uri": "", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_log_http_headers": "", 2024-09-20T19:21:12.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_log_object_name_utc": "false", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_attr_name_len": "0", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_attr_size": "0", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_chunk_size": "4194304", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_listing_results": "5000", 2024-09-20T19:21:12.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_notify_retries": "10", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_put_param_size": "1048576", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_put_size": "5368709120", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_max_slo_entries": "1000", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_md_log_max_shards": "64", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_mp_lock_max_time": "600", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-20T19:21:12.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_max_gc": "300", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_num_async_rados_threads": "32", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_num_control_oids": "8", 2024-09-20T19:21:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_numa_node": "-1", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_objexp_chunk_size": "100", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_objexp_gc_interval": "600", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_op_thread_timeout": "600", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_op_tracing": "false", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_opa_token": "", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_opa_url": "", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_opa_verify_ssl": "true", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-20T19:21:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ops_log_rados": "false", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_ops_log_socket_path": "", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_period_push_interval": "2.000000", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_cache_lanes": "3", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-20T19:21:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_cache_partitions": "3", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_print_continue": "true", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_rados_tracing": "false", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_realm": "", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_realm_id": "", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_region": "", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-20T19:21:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_request_uri": "", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_batch_size": "64", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_max_aio": "128", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_num_logs": "16", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshard_thread_interval": "600", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_resolve_cname": "false", 2024-09-20T19:21:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_run_sync_thread": "true", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_scheduler_type": "throttler", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_script_uri": "", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_service_provider_name": "", 2024-09-20T19:21:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_client_id": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_client_secret": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_entry": "sts", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_key": "sts", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_min_session_duration": "900", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sts_token_introspection_url": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_account_in_url": "false", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_auth_entry": "auth", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_auth_url": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_custom_header": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_need_stats": "true", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_tenant_name": "", 2024-09-20T19:21:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_token_expiration": "86400", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_url": "", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_url_prefix": "swift", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_lease_period": "120", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-20T19:21:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_thread_pool_size": "512", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_comment": "", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_createby": "", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_encoding": "", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_flag": "false", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_origin": "", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_torrent_tracker": "", 2024-09-20T19:21:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_trust_forwarded_https": "false", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_usage_max_shards": "32", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_usage_max_user_shards": "1", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_use_opa_authz": "false", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_counters_cache": "false", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_max_buckets": "1000", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_policies_max_num": "100", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-20T19:21:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_user_unique_email": "true", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_verify_ssl": "true", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zone": "", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zone_id": "", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zonegroup": "", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zonegroup_id": "", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgwlc_auto_session_clear": "true", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_block_size": "4096", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-20T19:21:12.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_size": "536870912", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_index_type": "binary_search", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_partition_filters": "false", 2024-09-20T19:21:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_perf": "false", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "run_dir": "/var/run/ceph", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_block_create": "true", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_cache_lru_size": "67108864", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_default_max_object_size": "16777216", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_device_size": "53687091200", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_full_integrity_check": "false", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_journal_batch_capacity": "16", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-20T19:21:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_main_device_type": "SSD", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "seastore_segment_size": "67108864", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "set_keepcaps": "false", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "setgroup": "ceph", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "setuser": "ceph", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "setuser_match_path": "", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-20T19:21:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "thp": "false", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "threadpool_default_timeout": "60", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "throttler_perf_counter": "true", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "tmp_dir": "/tmp", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "uadk_compressor_enabled": "false", 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-20T19:21:12.233 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[22706]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:12 smithi042 bash[32075]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.745314+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.749440+0000 osd.1 (osd.1) 88 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.850880+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: cluster 2024-09-20T19:21:10.853712+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:12 smithi162 bash[24557]: audit 2024-09-20T19:21:11.644060+0000 mon.a (mon.0) 974 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:12.852 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[22706]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:13 smithi042 bash[32075]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:13.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.937220+0000 mgr.y (mgr.24491) 126 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:13.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.943568+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:21:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:13 smithi162 bash[24557]: cluster 2024-09-20T19:21:11.945962+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:21:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[22706]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[22706]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[22706]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[22706]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[32075]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[32075]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[32075]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:14 smithi042 bash[32075]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:14.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:14 smithi162 bash[24557]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:14 smithi162 bash[24557]: cluster 2024-09-20T19:21:12.735506+0000 osd.6 (osd.6) 53 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:21:14.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:14 smithi162 bash[24557]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:14.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:14 smithi162 bash[24557]: cluster 2024-09-20T19:21:12.739902+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:21:15.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[32075]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:15.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:15 smithi042 bash[22706]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.713920+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:21:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.716319+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:21:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:15 smithi162 bash[24557]: cluster 2024-09-20T19:21:13.937840+0000 mgr.y (mgr.24491) 127 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:16.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:16 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:16.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:16 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:16 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.029567+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:21:16.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:16 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:16.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:16 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.032034+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:21:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:17 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:17 smithi042 bash[32075]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:17 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:17 smithi042 bash[22706]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:17 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:17 smithi162 bash[24557]: cluster 2024-09-20T19:21:15.938510+0000 mgr.y (mgr.24491) 128 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:18.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:18.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:18 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.652515+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.655451+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.690556+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.a scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.692750+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.a scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.845470+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:16.849162+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.108928+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.109034+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.150793+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.224254+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-09-20T19:21:18.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:18.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.226705+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-09-20T19:21:19.264 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a: {} 2024-09-20T19:21:19.264 INFO:teuthology.orchestra.run.smithi042.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:21:19.278 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b: {} 2024-09-20T19:21:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:21:19.289 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c: {} 2024-09-20T19:21:19.290 INFO:teuthology.orchestra.run.smithi042.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:21:19.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:19 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:19.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:19 smithi162 bash[24557]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:19.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:18 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:19.355 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:19 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:19.355 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:19 smithi042 bash[32075]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:19.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:19 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:19.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:19 smithi042 bash[22706]: cluster 2024-09-20T19:21:17.939467+0000 mgr.y (mgr.24491) 129 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:20.015 INFO:teuthology.run_tasks:Running task rados... 2024-09-20T19:21:20.026 INFO:tasks.rados:Beginning rados... 2024-09-20T19:21:20.027 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-20T19:21:20.027 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-20-1028'}, 'ceph': {'conf': {'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}}, 'workunit': {'branch': 'wip-guits-main-2024-09-20-1028', 'sha1': '88deadb8082f4a33832e228ccefc59cedba158a5'}} 2024-09-20T19:21:20.027 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-20T19:21:20.027 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-20T19:21:20.035 INFO:tasks.rados:joining rados 2024-09-20T19:21:20.035 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-20T19:21:20.035 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:21:20.035 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:21:20.035 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T19:21:20.036 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T19:21:20.036 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-20T19:21:20.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:20.036 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-20T19:21:20.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:21:20.037 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-20T19:21:20.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:21:20.037 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-20T19:21:20.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:21:20.038 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-09-20T19:21:20.038 INFO:tasks.rados:starting run 0 out of 1 2024-09-20T19:21:20.038 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-20T19:21:20.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-20T19:21:20.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:20.170 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:20.198 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:20.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:20.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:20 smithi162 bash[24557]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[22706]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: audit 2024-09-20T19:21:18.975095+0000 mgr.y (mgr.24491) 130 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.202181+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.204307+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-20T19:21:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:20 smithi042 bash[32075]: audit 2024-09-20T19:21:19.253587+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.42:0/4087543247' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:21:20.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:20.451 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:20.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:20.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:20.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:20.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:20.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:20.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:20.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:20.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:20.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:21.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:21.098 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:21.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:21.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:21.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:21.358 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:21.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[22706]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:21 smithi042 bash[32075]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:21.472 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.472 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.698184+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:21:21.472 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.472 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.700493+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:21:21.472 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:21 smithi162 bash[24557]: cluster 2024-09-20T19:21:19.940205+0000 mgr.y (mgr.24491) 131 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:21.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:21.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:21.616 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:21.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:21.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:21.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:21.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:21.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:22.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:22.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:22.252 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:22.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:22 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:22.409 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:22.512 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.038634+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.041095+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.220800+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:22 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.225269+0000 osd.2 (osd.2) 60 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:21:22.540 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:22.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:22.669 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:22.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:22.797 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:22.900 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:22.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:22.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:23.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:23.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:23.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:23.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:23.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:23 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:23.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.673769+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.676144+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:21.940991+0000 mgr.y (mgr.24491) 132 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.189241+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:23 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.193563+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:21:23.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:23.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:23.574 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:23.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:23.705 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:23.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:23.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:23.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:23.967 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:23.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:24.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:24.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:24.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:24.224 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:24.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:24 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:24.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.997830+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:22.999791+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.239186+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:21:24.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:24 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.240999+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:21:24.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:24.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:24.616 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:24.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:24.850 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:24.850 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:24.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.851 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:24.853 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:24.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:24.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:25.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:25.243 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:25.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:25 smithi162 bash[24557]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[22706]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:25.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:23.941738+0000 mgr.y (mgr.24491) 133 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.030804+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.033565+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.061072+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.063375+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.241227+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.e scrub starts 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:25 smithi042 bash[32075]: cluster 2024-09-20T19:21:24.243559+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.e scrub ok 2024-09-20T19:21:25.763 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:25.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:25.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:25.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:26.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:26.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:26.283 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:26.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:26.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:26 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:26 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:26 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:26 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.554 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.834351+0000 osd.1 (osd.1) 91 : cluster [DBG] 5.c scrub starts 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:26 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.836212+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2024-09-20T19:21:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:26.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:26.703 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:26.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:26.834 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:26.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:26.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:27.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:27.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:27.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:27.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.428 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[32075]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.481 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:27 smithi042 bash[22706]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.531 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.307841+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.f scrub starts 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.309301+0000 osd.2 (osd.2) 66 : cluster [DBG] 6.f scrub ok 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.911307+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.913810+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: cluster 2024-09-20T19:21:25.942481+0000 mgr.y (mgr.24491) 134 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:27 smithi162 bash[24557]: audit 2024-09-20T19:21:26.674259+0000 mon.a (mon.0) 976 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:27.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:27.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:27.786 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 110, 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-20T19:07:00.852407+0000", 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-20T19:21:17.140228+0000", 2024-09-20T19:21:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: "last_up_change": "2024-09-20T19:17:34.145990+0000", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "last_in_change": "2024-09-20T19:17:00.522021+0000", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_num": 5799936, 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_set": [ 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "pglog_hardlimit", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "purged_snapdirs", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "recovery_deletes", 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "sortbitwise" 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_version": 18, 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T19:21:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_snapshot": "", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_max": 6, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "max_osd": 8, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "require_min_compat_client": "luminous", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "min_compat_client": "luminous", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "require_osd_release": "squid", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "allow_crimson": false, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "pools": [ 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 1, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": ".mgr", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:12:46.311971+0000", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 1, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 1, 2024-09-20T19:21:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 1, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 1, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 1, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "22", 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.801 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.802 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "options": { 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_max": 32, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_min": 1 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.803 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": {} 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 7.8899998664855957, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 7.8899998664855957, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 0.37999999523162842, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 3, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 3, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.804 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 2, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": ".rgw.root", 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:05.791948+0000", 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.805 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.806 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "71", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.807 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.808 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:21:27.809 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.5, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.5, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.5, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.5, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 3, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "datapool", 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:06.729842+0000", 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 8193, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.811 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 3, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 3, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 3, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 3, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 3, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "79", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 2, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 79, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.813 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd": {} 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.815 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2.6500000953674316, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2.6500000953674316, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 4, 2024-09-20T19:21:27.816 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.log", 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:07.353972+0000", 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.817 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "73", 2024-09-20T19:21:27.818 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.819 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.820 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.821 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2.25, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2.25, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2.25, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2.25, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 5, 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.control", 2024-09-20T19:21:27.822 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:09.310321+0000", 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.823 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "75", 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.824 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.825 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.826 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.827 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.25, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.25, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.25, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.25, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 6, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.meta", 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:11.438339+0000", 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:21:27.828 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:21:27.829 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "77", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:21:27.830 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:21:27.831 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:21:27.832 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "options": { 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_bias": 4 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.75, 2024-09-20T19:21:27.833 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.75, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.75, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.75, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "osds": [ 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 0, 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b", 2024-09-20T19:21:27.834 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 8, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 75, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6801", 2024-09-20T19:21:27.835 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6802", 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6804", 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6803", 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.837 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6801/2265043130", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6802/2265043130", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6804/2265043130", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6803/2265043130", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 1, 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d", 2024-09-20T19:21:27.838 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 13, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 75, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.839 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6805", 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6806", 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.840 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6808", 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6807", 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:21:27.841 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6805/3513258758", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6806/3513258758", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6808/3513258758", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6807/3513258758", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 2, 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d", 2024-09-20T19:21:27.842 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 19, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 101, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6809", 2024-09-20T19:21:27.843 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6810", 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6812", 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6811", 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:21:27.845 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6809/2620956901", 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6810/2620956901", 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6812/2620956901", 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6811/2620956901", 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 3, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e", 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 28, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 75, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6813", 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6814", 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:21:27.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6816", 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6815", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6813/1507333597", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6814/1507333597", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6816/1507333597", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6815/1507333597", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 4, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "90c5738d-14a9-415f-9249-eac0455635ad", 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 35, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 101, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.851 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6800", 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6801", 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:21:27.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6803", 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.853 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6802", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6800/2898101120", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6801/2898101120", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6803/2898101120", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6802/2898101120", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 5, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27", 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 44, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 75, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.855 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6804", 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6805", 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:21:27.856 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6807", 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6806", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6804/4269525575", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6805/4269525575", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4269525575", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6806/4269525575", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.858 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 6, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a", 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 54, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 101, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.859 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6808", 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6809", 2024-09-20T19:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6811", 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.861 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6810", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6808/1785075253", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6809/1785075253", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6811/1785075253", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6810/1785075253", 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.862 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 7, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e", 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 64, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 75, 2024-09-20T19:21:27.863 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6812", 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.864 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6813", 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6815", 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6814", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6812/1845916231", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6813/1845916231", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1845916231", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6814/1845916231", 2024-09-20T19:21:27.866 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_xinfo": [ 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 0, 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:46.531570+0000", 2024-09-20T19:21:27.867 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 1, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:52.382787+0000", 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 2, 2024-09-20T19:21:27.868 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:54.885176+0000", 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 3, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.869 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:17.206122+0000", 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 4, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:54.812245+0000", 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 5, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:41.782889+0000", 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 6, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.871 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:20:18.221349+0000", 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 7, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:21:17.109039+0000", 2024-09-20T19:21:27.872 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap": [], 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap_items": [ 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "2.5", 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 7, 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.7", 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.9", 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:21:27.874 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 5 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.1f", 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap_primaries": [], 2024-09-20T19:21:27.875 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_temp": [], 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_temp": [], 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "blocklist": { 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/1743761075": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3575489582": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/2546869704": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/1361717385": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/2117630454": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/409595637": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/2206296983": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3240017151": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/3764180564": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3774737654": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3982292550": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/2784207196": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:21:27.876 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3505349280": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/368716713": "2024-09-21T19:18:55.853462+0000" 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "range_blocklist": {}, 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profiles": { 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "default": { 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "crush-failure-domain": "osd", 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "k": "2", 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "m": "1", 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "plugin": "jerasure", 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "technique": "reed_sol_van" 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps_queue": [], 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "new_removed_snaps": [], 2024-09-20T19:21:27.877 INFO:teuthology.orchestra.run.smithi042.stdout: "new_purged_snaps": [], 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_node_flags": {}, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "device_class_flags": {}, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": { 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode_enabled": false, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_bucket_count": 0, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "degraded_stretch_mode": 0, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "recovering_stretch_mode": 0, 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode_bucket": 0 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:27.878 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:21:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:27.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:27.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:27.957 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:21:27.958 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:21:27.975 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:21:27.975 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:21:27.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:27.994 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:21:27.994 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:21:27.994 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:21:27.994 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:21:28.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:28.012 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:21:28.012 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:21:28.030 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:21:28.031 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:21:28.047 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:21:28.047 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:21:28.068 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:21:28.068 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:21:28.085 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:21:28.086 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:21:28.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:28.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:28.265 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.734610+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.737576+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.945093+0000 osd.5 (osd.5) 73 : cluster [DBG] 6.d scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:26.947440+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.d scrub ok 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.275225+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.d scrub starts 2024-09-20T19:21:28.372 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.280606+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.405681+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.405794+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: audit 2024-09-20T19:21:27.792317+0000 mon.a (mon.0) 977 : audit [DBG] from='client.? v1:172.21.15.42:0/1767865575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: audit 2024-09-20T19:21:27.939950+0000 mon.a (mon.0) 978 : audit [DBG] from='client.? v1:172.21.15.42:0/1553691452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.081599+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub starts 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:28 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.083520+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.b scrub ok 2024-09-20T19:21:28.391 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:28.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:28.516 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:28.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:28.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:28.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:28.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:28.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:29.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:29.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:21:29.228 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'unique_pool_0' created 2024-09-20T19:21:29.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:28 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:29.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:29.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:29.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:29.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:29 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:29.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:27.943146+0000 mgr.y (mgr.24491) 135 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-09-20T19:21:29.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.520 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.225552+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.696248+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.42:0/3982520230' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.696548+0000 mon.a (mon.0) 980 : audit [INF] from='client.24725 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: audit 2024-09-20T19:21:28.921828+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:21:29.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.099014+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:21:29.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:29 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.100714+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:21:29.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:29.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:29.667 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:29.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:29.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:29.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:29.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:30.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:30.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:30.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:30 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:30.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.978023+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:28.981308+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:28.983771+0000 mgr.y (mgr.24491) 136 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.215907+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:29.217914+0000 mon.a (mon.0) 983 : audit [INF] from='client.24725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: audit 2024-09-20T19:21:29.217973+0000 mon.a (mon.0) 984 : audit [INF] from='client.? v1:172.21.15.42:0/2668947648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:30 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.223955+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-20T19:21:30.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:30.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:30.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:30.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:30.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:30.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:30.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:30.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:31.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.096 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:31.155 INFO:tasks.thrashosds.thrasher:j is {'epoch': 110, 'fsid': '49cb5aae-7783-11ef-baf5-efdc52797490', 'created': '2024-09-20T19:07:00.852407+0000', 'modified': '2024-09-20T19:21:17.140228+0000', 'last_up_change': '2024-09-20T19:17:34.145990+0000', 'last_in_change': '2024-09-20T19:17:00.522021+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T19:12:46.311971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T19:18:05.791948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-20T19:18:06.729842+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 79, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T19:18:07.353972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T19:18:09.310321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T19:18:11.438339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'fb59ab36-8bbc-4e10-bc0f-b0cca84b317b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6801', 'nonce': 2265043130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6802', 'nonce': 2265043130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6804', 'nonce': 2265043130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6803', 'nonce': 2265043130}]}, 'public_addr': '172.21.15.42:6801/2265043130', 'cluster_addr': '172.21.15.42:6802/2265043130', 'heartbeat_back_addr': '172.21.15.42:6804/2265043130', 'heartbeat_front_addr': '172.21.15.42:6803/2265043130', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6805', 'nonce': 3513258758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6806', 'nonce': 3513258758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6808', 'nonce': 3513258758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6807', 'nonce': 3513258758}]}, 'public_addr': '172.21.15.42:6805/3513258758', 'cluster_addr': '172.21.15.42:6806/3513258758', 'heartbeat_back_addr': '172.21.15.42:6808/3513258758', 'heartbeat_front_addr': '172.21.15.42:6807/3513258758', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '12b15f1c-ba47-49cb-aefe-d99ff84d926d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 101, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6809', 'nonce': 2620956901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6810', 'nonce': 2620956901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6812', 'nonce': 2620956901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6811', 'nonce': 2620956901}]}, 'public_addr': '172.21.15.42:6809/2620956901', 'cluster_addr': '172.21.15.42:6810/2620956901', 'heartbeat_back_addr': '172.21.15.42:6812/2620956901', 'heartbeat_front_addr': '172.21.15.42:6811/2620956901', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f0b69ee0-b93b-4ab6-8a35-783b18b9c73e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6813', 'nonce': 1507333597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6814', 'nonce': 1507333597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6816', 'nonce': 1507333597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6815', 'nonce': 1507333597}]}, 'public_addr': '172.21.15.42:6813/1507333597', 'cluster_addr': '172.21.15.42:6814/1507333597', 'heartbeat_back_addr': '172.21.15.42:6816/1507333597', 'heartbeat_front_addr': '172.21.15.42:6815/1507333597', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '90c5738d-14a9-415f-9249-eac0455635ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 101, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 2898101120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 2898101120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 2898101120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 2898101120}]}, 'public_addr': '172.21.15.162:6800/2898101120', 'cluster_addr': '172.21.15.162:6801/2898101120', 'heartbeat_back_addr': '172.21.15.162:6803/2898101120', 'heartbeat_front_addr': '172.21.15.162:6802/2898101120', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e7648e52-3260-47ec-8778-35d4fa70ee27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 4269525575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4269525575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4269525575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 4269525575}]}, 'public_addr': '172.21.15.162:6804/4269525575', 'cluster_addr': '172.21.15.162:6805/4269525575', 'heartbeat_back_addr': '172.21.15.162:6807/4269525575', 'heartbeat_front_addr': '172.21.15.162:6806/4269525575', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34b6c8df-9cb5-4fcb-955f-69fd6520465a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 101, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 1785075253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1785075253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1785075253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 1785075253}]}, 'public_addr': '172.21.15.162:6808/1785075253', 'cluster_addr': '172.21.15.162:6809/1785075253', 'heartbeat_back_addr': '172.21.15.162:6811/1785075253', 'heartbeat_front_addr': '172.21.15.162:6810/1785075253', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6046a58c-ccf6-41ce-9888-b0c7a771205e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 64, 'up_thru': 75, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 1845916231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1845916231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1845916231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 1845916231}]}, 'public_addr': '172.21.15.162:6812/1845916231', 'cluster_addr': '172.21.15.162:6813/1845916231', 'heartbeat_back_addr': '172.21.15.162:6815/1845916231', 'heartbeat_front_addr': '172.21.15.162:6814/1845916231', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:46.531570+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:52.382787+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:54.885176+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:17.206122+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:54.812245+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:41.782889+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:20:18.221349+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:21:17.109039+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.5', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.42:0/1743761075': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3575489582': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:6800/2546869704': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:0/1361717385': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/2117630454': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/409595637': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:6800/2206296983': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/3240017151': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:6800/3764180564': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3774737654': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:0/3982292550': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/2784207196': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3505349280': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/368716713': '2024-09-21T19:18:55.853462+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T19:21:31.155 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:21:31.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:31.229 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:31.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:31.359 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:31.461 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.469 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[32075]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:31 smithi042 bash[22706]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.489 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:29.943837+0000 mgr.y (mgr.24491) 137 : cluster [DBG] pgmap v101: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:30.228882+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:30.229988+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:31 smithi162 bash[24557]: cluster 2024-09-20T19:21:31.245805+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-20T19:21:31.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T19:21:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:31.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:31.744 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:31.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:31.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:31.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:32.007 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:32.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:32.139 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:32.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:32.288 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-20T19:21:32.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:32.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:32.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:32.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:32.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:32.801 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:32.831 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:32.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:32.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:32.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:33.064 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.090 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:33.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:33.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:33.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[22706]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:33 smithi042 bash[32075]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:33.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:31.945441+0000 mgr.y (mgr.24491) 138 : cluster [DBG] pgmap v104: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:32.904288+0000 osd.6 (osd.6) 55 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:33 smithi162 bash[24557]: cluster 2024-09-20T19:21:32.904432+0000 osd.6 (osd.6) 56 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:33.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:33.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:33.715 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:33.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:21:33.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:33.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:33.980 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:33.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:34.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:34.129 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:34.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:34.259 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:34.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:34.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:21:34.391 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:34.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:34.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:34 smithi042 bash[22706]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:34 smithi042 bash[22706]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:34 smithi042 bash[32075]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:34 smithi042 bash[32075]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.519 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:34.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:34 smithi162 bash[24557]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:34 smithi162 bash[24557]: cluster 2024-09-20T19:21:33.281012+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:21:34.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:34.649 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:34.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:34.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:34.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:35.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:35.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:35.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:35.168 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:35.271 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:35.298 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:35.401 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:35.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:35.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:35 smithi042 bash[32075]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:35 smithi042 bash[32075]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:35 smithi042 bash[22706]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:35 smithi042 bash[22706]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:35 smithi162 bash[24557]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:35 smithi162 bash[24557]: cluster 2024-09-20T19:21:33.946018+0000 mgr.y (mgr.24491) 139 : cluster [DBG] pgmap v106: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:35.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:35.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:35.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:35.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:35.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:35.952 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:35.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:36.054 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:36.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:36.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:36.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:36.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:36.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:36.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:36.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:36.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:36.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:36.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:36.831 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:36.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.965 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:36.986 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:36.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:37.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:37.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:37.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:37.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:37 smithi042 bash[22706]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:37 smithi042 bash[22706]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:37 smithi042 bash[32075]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.363 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:37 smithi042 bash[32075]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:37.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:37.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:37.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:37 smithi162 bash[24557]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:37 smithi162 bash[24557]: cluster 2024-09-20T19:21:35.946528+0000 mgr.y (mgr.24491) 140 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:37.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.638 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:37.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:37.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:37.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:37.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:37.989 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:38.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:38.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:38.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:38.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:38.260 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:38.287 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:38.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:38.416 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:38.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:38.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:38.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:38.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:38.804 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:38.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:38.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:38.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:39.010 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:39.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.063 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:39.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:39.192 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:39.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:38 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:39.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:39.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:39.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:39 smithi042 bash[22706]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:39 smithi042 bash[22706]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:39 smithi042 bash[32075]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:39 smithi042 bash[32075]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:39.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:39 smithi162 bash[24557]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:39 smithi162 bash[24557]: cluster 2024-09-20T19:21:37.948135+0000 mgr.y (mgr.24491) 141 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:21:39.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:39.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:39.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.715 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:39.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:39.845 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:39.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:39.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:40.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:40.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:40.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:40.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:40.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:40 smithi162 bash[24557]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:40 smithi162 bash[24557]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:40.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:40.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:40.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:40.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:40.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:40 smithi042 bash[22706]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:40 smithi042 bash[22706]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:40 smithi042 bash[32075]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.611 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:40 smithi042 bash[32075]: audit 2024-09-20T19:21:38.991974+0000 mgr.y (mgr.24491) 142 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:40.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:40.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:40.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:40.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:40.893 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:40.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:40.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:41.024 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:41.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:41.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:41.254 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:41.283 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:41.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:41.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:41 smithi162 bash[24557]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:41 smithi162 bash[24557]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:41.515 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:41.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:41 smithi042 bash[32075]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:41 smithi042 bash[32075]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:41 smithi042 bash[22706]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:41 smithi042 bash[22706]: cluster 2024-09-20T19:21:39.948964+0000 mgr.y (mgr.24491) 143 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:21:41.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:41.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:41.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:41.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:41.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:41.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:41.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:41.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:42.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:42.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:42.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:42.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:42.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:42.310 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:21:42.310 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:21:42.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:42.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:42.437 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:21:42.439 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:21:42.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:42.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:42.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:42.696 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:42.717 INFO:teuthology.orchestra.run.smithi042.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-20T19:21:42.767 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:21:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:21:42.785 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:21:42.785 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:21:42.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:42.802 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:21:42.802 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:21:42.818 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:21:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:21:42.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:42.836 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:21:42.837 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:21:42.856 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:21:42.856 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:21:42.874 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:21:42.874 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:21:42.895 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:21:42.895 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:42.909 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[22706]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:42.910 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:42.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:42.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:42 smithi042 bash[32075]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:42.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:42.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:41.708778+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:41.710878+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: cluster 2024-09-20T19:21:41.950595+0000 mgr.y (mgr.24491) 144 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.427332+0000 mgr.y (mgr.24491) 145 : audit [DBG] from='client.24737 v1:172.21.15.42:0/3096306845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:21:43.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:43.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.687491+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.42:0/1411868814' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:43.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:43.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:42 smithi162 bash[24557]: audit 2024-09-20T19:21:42.687858+0000 mon.a (mon.0) 992 : audit [INF] from='client.24743 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-20T19:21:43.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:43.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:43.181 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.206 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:43.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:43.436 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.462 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:43.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:43.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:43.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:43.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:43.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:43.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.967 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:43.967 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[32075]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:43.968 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:43.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:43.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:43.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:43.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:43 smithi042 bash[22706]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:43.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:43.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:44.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:44.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: audit 2024-09-20T19:21:42.711521+0000 mon.a (mon.0) 993 : audit [INF] from='client.24743 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:21:44.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:44.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: cluster 2024-09-20T19:21:42.717903+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:21:44.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:44.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:43 smithi162 bash[24557]: audit 2024-09-20T19:21:42.749928+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.42:0/3898504992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:44.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:44.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:44.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:44.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:44.371 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:44.474 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:44.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T19:21:44.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:44.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:44.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:44.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:44.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:44 smithi042 bash[22706]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:44 smithi042 bash[22706]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:44 smithi042 bash[32075]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:44 smithi042 bash[32075]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:44.874 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:44 smithi162 bash[24557]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.875 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:44 smithi162 bash[24557]: cluster 2024-09-20T19:21:43.951409+0000 mgr.y (mgr.24491) 146 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:21:44.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:44.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:44.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:45.022 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:45.125 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:45.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:45.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:45.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:45.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:45.525 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:45.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:45.743 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:21:45.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:45.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[22706]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:45 smithi042 bash[32075]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: cluster 2024-09-20T19:21:44.713619+0000 mon.a (mon.0) 995 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: audit 2024-09-20T19:21:45.442813+0000 mon.c (mon.2) 32 : audit [INF] from='client.? v1:172.21.15.42:0/3595891928' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:45 smithi162 bash[24557]: audit 2024-09-20T19:21:45.443328+0000 mon.a (mon.0) 996 : audit [INF] from='client.24755 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:21:45.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:45.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:46.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:46.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:46.077 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-20T19:21:41.949305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1061,"num_read_kb":918,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248860,"kb_used_data":7676,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712356,"statfs":{"total":767960285184,"available":767705452544,"internally_reserved":0,"allocated":7860224,"data_stored":4653113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005179"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717559+0000","last_change":"2024-09-20T19:20:38.044596+0000","last_active":"2024-09-20T19:21:33.717559+0000","last_peered":"2024-09-20T19:21:33.717559+0000","last_clean":"2024-09-20T19:21:33.717559+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:21:33.717559+0000","last_undegraded":"2024-09-20T19:21:33.717559+0000","last_fullsized":"2024-09-20T19:21:33.717559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.044475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.044475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.419278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282200+0000","last_change":"2024-09-20T19:20:58.815043+0000","last_active":"2024-09-20T19:21:33.282200+0000","last_peered":"2024-09-20T19:21:33.282200+0000","last_clean":"2024-09-20T19:21:33.282200+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:21:33.282200+0000","last_undegraded":"2024-09-20T19:21:33.282200+0000","last_fullsized":"2024-09-20T19:21:33.282200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.814925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.814925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:16.317816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":140,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717941+0000","last_change":"2024-09-20T19:20:52.007256+0000","last_active":"2024-09-20T19:21:33.717941+0000","last_peered":"2024-09-20T19:21:33.717941+0000","last_clean":"2024-09-20T19:21:33.717941+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:21:33.717941+0000","last_undegraded":"2024-09-20T19:21:33.717941+0000","last_fullsized":"2024-09-20T19:21:33.717941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:20:52.007105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.007105+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:08.466212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285087+0000","last_change":"2024-09-20T19:20:33.800940+0000","last_active":"2024-09-20T19:21:33.285087+0000","last_peered":"2024-09-20T19:21:33.285087+0000","last_clean":"2024-09-20T19:21:33.285087+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:21:33.285087+0000","last_undegraded":"2024-09-20T19:21:33.285087+0000","last_fullsized":"2024-09-20T19:21:33.285087+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.800794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.800794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.248084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.719564+0000","last_change":"2024-09-20T19:20:31.068114+0000","last_active":"2024-09-20T19:21:33.719564+0000","last_peered":"2024-09-20T19:21:33.719564+0000","last_clean":"2024-09-20T19:21:33.719564+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:21:33.719564+0000","last_undegraded":"2024-09-20T19:21:33.719564+0000","last_fullsized":"2024-09-20T19:21:33.719564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:31.067964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:31.067964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:48.259004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285262+0000","last_change":"2024-09-20T19:20:34.776570+0000","last_active":"2024-09-20T19:21:33.285262+0000","last_peered":"2024-09-20T19:21:33.285262+0000","last_clean":"2024-09-20T19:21:33.285262+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:21:33.285262+0000","last_undegraded":"2024-09-20T19:21:33.285262+0000","last_fullsized":"2024-09-20T19:21:33.285262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:34.776387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:34.776387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.596561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778542+0000","last_change":"2024-09-20T19:20:42.524526+0000","last_active":"2024-09-20T19:21:33.778542+0000","last_peered":"2024-09-20T19:21:33.778542+0000","last_clean":"2024-09-20T19:21:33.778542+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:21:33.778542+0000","last_undegraded":"2024-09-20T19:21:33.778542+0000","last_fullsized":"2024-09-20T19:21:33.778542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.524397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.524397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:01.089547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285739+0000","last_change":"2024-09-20T19:21:07.892598+0000","last_active":"2024-09-20T19:21:33.285739+0000","last_peered":"2024-09-20T19:21:33.285739+0000","last_clean":"2024-09-20T19:21:33.285739+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:21:33.285739+0000","last_undegraded":"2024-09-20T19:21:33.285739+0000","last_fullsized":"2024-09-20T19:21:33.285739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.892459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.892459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:13.431828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.891324+0000","last_change":"2024-09-20T19:21:21.041234+0000","last_active":"2024-09-20T19:21:33.891324+0000","last_peered":"2024-09-20T19:21:33.891324+0000","last_clean":"2024-09-20T19:21:33.891324+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:21:33.891324+0000","last_undegraded":"2024-09-20T19:21:33.891324+0000","last_fullsized":"2024-09-20T19:21:33.891324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:21.041106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.041106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:23.493108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285600+0000","last_change":"2024-09-20T19:20:41.804284+0000","last_active":"2024-09-20T19:21:33.285600+0000","last_peered":"2024-09-20T19:21:33.285600+0000","last_clean":"2024-09-20T19:21:33.285600+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:21:33.285600+0000","last_undegraded":"2024-09-20T19:21:33.285600+0000","last_fullsized":"2024-09-20T19:21:33.285600+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:20:41.804146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.804146+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:55.773005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890605+0000","last_change":"2024-09-20T19:20:48.831732+0000","last_active":"2024-09-20T19:21:33.890605+0000","last_peered":"2024-09-20T19:21:33.890605+0000","last_clean":"2024-09-20T19:21:33.890605+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:21:33.890605+0000","last_undegraded":"2024-09-20T19:21:33.890605+0000","last_fullsized":"2024-09-20T19:21:33.890605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:48.831670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.831670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:07.003294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961589+0000","last_change":"2024-09-20T19:20:49.661540+0000","last_active":"2024-09-20T19:21:33.961589+0000","last_peered":"2024-09-20T19:21:33.961589+0000","last_clean":"2024-09-20T19:21:33.961589+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:21:33.961589+0000","last_undegraded":"2024-09-20T19:21:33.961589+0000","last_fullsized":"2024-09-20T19:21:33.961589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:49.661422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:49.661422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:03.539487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.891209+0000","last_change":"2024-09-20T19:20:55.860228+0000","last_active":"2024-09-20T19:21:33.891209+0000","last_peered":"2024-09-20T19:21:33.891209+0000","last_clean":"2024-09-20T19:21:33.891209+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:21:33.891209+0000","last_undegraded":"2024-09-20T19:21:33.891209+0000","last_fullsized":"2024-09-20T19:21:33.891209+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:55.860111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:55.860111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:20.730406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286209+0000","last_change":"2024-09-20T19:21:06.877638+0000","last_active":"2024-09-20T19:21:33.286209+0000","last_peered":"2024-09-20T19:21:33.286209+0000","last_clean":"2024-09-20T19:21:33.286209+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:21:33.286209+0000","last_undegraded":"2024-09-20T19:21:33.286209+0000","last_fullsized":"2024-09-20T19:21:33.286209+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:21:06.877448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.877448+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:07.791256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961905+0000","last_change":"2024-09-20T19:21:07.802088+0000","last_active":"2024-09-20T19:21:33.961905+0000","last_peered":"2024-09-20T19:21:33.961905+0000","last_clean":"2024-09-20T19:21:33.961905+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:21:33.961905+0000","last_undegraded":"2024-09-20T19:21:33.961905+0000","last_fullsized":"2024-09-20T19:21:33.961905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.801948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.801948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:23.122195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278147+0000","last_change":"2024-09-20T19:21:08.846790+0000","last_active":"2024-09-20T19:21:33.278147+0000","last_peered":"2024-09-20T19:21:33.278147+0000","last_clean":"2024-09-20T19:21:33.278147+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:21:33.278147+0000","last_undegraded":"2024-09-20T19:21:33.278147+0000","last_fullsized":"2024-09-20T19:21:33.278147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.846665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.846665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:26.486168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.280559+0000","last_change":"2024-09-20T19:21:30.255300+0000","last_active":"2024-09-20T19:21:33.280559+0000","last_peered":"2024-09-20T19:21:33.280559+0000","last_clean":"2024-09-20T19:21:33.280559+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:21:33.280559+0000","last_undegraded":"2024-09-20T19:21:33.280559+0000","last_fullsized":"2024-09-20T19:21:33.280559+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:38.261571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"77'17","reported_seq":143,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286544+0000","last_change":"2024-09-20T19:21:27.280916+0000","last_active":"2024-09-20T19:21:33.286544+0000","last_peered":"2024-09-20T19:21:33.286544+0000","last_clean":"2024-09-20T19:21:33.286544+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:21:33.286544+0000","last_undegraded":"2024-09-20T19:21:33.286544+0000","last_fullsized":"2024-09-20T19:21:33.286544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:21:27.280627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:27.280627+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:45.924324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890969+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:21:33.890969+0000","last_peered":"2024-09-20T19:21:33.890969+0000","last_clean":"2024-09-20T19:21:33.890969+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:21:33.890969+0000","last_undegraded":"2024-09-20T19:21:33.890969+0000","last_fullsized":"2024-09-20T19:21:33.890969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:44.615171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779130+0000","last_change":"2024-09-20T19:21:25.836294+0000","last_active":"2024-09-20T19:21:33.779130+0000","last_peered":"2024-09-20T19:21:33.779130+0000","last_clean":"2024-09-20T19:21:33.779130+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:21:33.779130+0000","last_undegraded":"2024-09-20T19:21:33.779130+0000","last_fullsized":"2024-09-20T19:21:33.779130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.836217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.836217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.575754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282853+0000","last_change":"2024-09-20T19:21:25.309366+0000","last_active":"2024-09-20T19:21:33.282853+0000","last_peered":"2024-09-20T19:21:33.282853+0000","last_clean":"2024-09-20T19:21:33.282853+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:21:33.282853+0000","last_undegraded":"2024-09-20T19:21:33.282853+0000","last_fullsized":"2024-09-20T19:21:33.282853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.309306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.309306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:38.503428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278899+0000","last_change":"2024-09-20T19:21:30.242939+0000","last_active":"2024-09-20T19:21:33.278899+0000","last_peered":"2024-09-20T19:21:33.278899+0000","last_clean":"2024-09-20T19:21:33.278899+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:21:33.278899+0000","last_undegraded":"2024-09-20T19:21:33.278899+0000","last_fullsized":"2024-09-20T19:21:33.278899+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:58.473758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"77'10","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285953+0000","last_change":"2024-09-20T19:20:51.897157+0000","last_active":"2024-09-20T19:21:33.285953+0000","last_peered":"2024-09-20T19:21:33.285953+0000","last_clean":"2024-09-20T19:21:33.285953+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:21:33.285953+0000","last_undegraded":"2024-09-20T19:21:33.285953+0000","last_fullsized":"2024-09-20T19:21:33.285953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:51.897019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:51.897019+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:14.583589+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779699+0000","last_change":"2024-09-20T19:21:16.692876+0000","last_active":"2024-09-20T19:21:33.779699+0000","last_peered":"2024-09-20T19:21:33.779699+0000","last_clean":"2024-09-20T19:21:33.779699+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:21:33.779699+0000","last_undegraded":"2024-09-20T19:21:33.779699+0000","last_fullsized":"2024-09-20T19:21:33.779699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:16.692759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.692759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:21.218275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":165,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281009+0000","last_change":"2024-09-20T19:21:19.204377+0000","last_active":"2024-09-20T19:21:33.281009+0000","last_peered":"2024-09-20T19:21:33.281009+0000","last_clean":"2024-09-20T19:21:33.281009+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:21:33.281009+0000","last_undegraded":"2024-09-20T19:21:33.281009+0000","last_fullsized":"2024-09-20T19:21:33.281009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:19.204313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:19.204313+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:33.425102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286088+0000","last_change":"2024-09-20T19:21:00.850362+0000","last_active":"2024-09-20T19:21:33.286088+0000","last_peered":"2024-09-20T19:21:33.286088+0000","last_clean":"2024-09-20T19:21:33.286088+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:21:33.286088+0000","last_undegraded":"2024-09-20T19:21:33.286088+0000","last_fullsized":"2024-09-20T19:21:33.286088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.850236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.850236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:18.726421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052262+0000","last_change":"2024-09-20T19:21:30.255651+0000","last_active":"2024-09-20T19:21:34.052262+0000","last_peered":"2024-09-20T19:21:34.052262+0000","last_clean":"2024-09-20T19:21:34.052262+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:21:34.052262+0000","last_undegraded":"2024-09-20T19:21:34.052262+0000","last_fullsized":"2024-09-20T19:21:34.052262+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:54.267540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"77'15","reported_seq":140,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779538+0000","last_change":"2024-09-20T19:20:32.494239+0000","last_active":"2024-09-20T19:21:33.779538+0000","last_peered":"2024-09-20T19:21:33.779538+0000","last_clean":"2024-09-20T19:21:33.779538+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:21:33.779538+0000","last_undegraded":"2024-09-20T19:21:33.779538+0000","last_fullsized":"2024-09-20T19:21:33.779538+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:20:32.494020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.494020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.833966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778698+0000","last_change":"2024-09-20T19:20:58.724906+0000","last_active":"2024-09-20T19:21:33.778698+0000","last_peered":"2024-09-20T19:21:33.778698+0000","last_clean":"2024-09-20T19:21:33.778698+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:21:33.778698+0000","last_undegraded":"2024-09-20T19:21:33.778698+0000","last_fullsized":"2024-09-20T19:21:33.778698+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.724842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.724842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:27.471310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":164,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286584+0000","last_change":"2024-09-20T19:21:24.243650+0000","last_active":"2024-09-20T19:21:33.286584+0000","last_peered":"2024-09-20T19:21:33.286584+0000","last_clean":"2024-09-20T19:21:33.286584+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:21:33.286584+0000","last_undegraded":"2024-09-20T19:21:33.286584+0000","last_fullsized":"2024-09-20T19:21:33.286584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:24.243567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.243567+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:50.200693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961105+0000","last_change":"2024-09-20T19:21:26.947579+0000","last_active":"2024-09-20T19:21:33.961105+0000","last_peered":"2024-09-20T19:21:33.961105+0000","last_clean":"2024-09-20T19:21:33.961105+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:21:33.961105+0000","last_undegraded":"2024-09-20T19:21:33.961105+0000","last_fullsized":"2024-09-20T19:21:33.961105+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:26.947450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:26.947450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:50.365066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052930+0000","last_change":"2024-09-20T19:21:30.271310+0000","last_active":"2024-09-20T19:21:34.052930+0000","last_peered":"2024-09-20T19:21:34.052930+0000","last_clean":"2024-09-20T19:21:34.052930+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:21:34.052930+0000","last_undegraded":"2024-09-20T19:21:34.052930+0000","last_fullsized":"2024-09-20T19:21:34.052930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:54.342777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"77'11","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286434+0000","last_change":"2024-09-20T19:20:40.763340+0000","last_active":"2024-09-20T19:21:33.286434+0000","last_peered":"2024-09-20T19:21:33.286434+0000","last_clean":"2024-09-20T19:21:33.286434+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:21:33.286434+0000","last_undegraded":"2024-09-20T19:21:33.286434+0000","last_fullsized":"2024-09-20T19:21:33.286434+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:40.763197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.763197+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:47.787913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890582+0000","last_change":"2024-09-20T19:21:24.063529+0000","last_active":"2024-09-20T19:21:33.890582+0000","last_peered":"2024-09-20T19:21:33.890582+0000","last_clean":"2024-09-20T19:21:33.890582+0000","last_became_active":"2024-09-20T19:18:07.269527+0000","last_became_peered":"2024-09-20T19:18:07.269527+0000","last_unstale":"2024-09-20T19:21:33.890582+0000","last_undegraded":"2024-09-20T19:21:33.890582+0000","last_fullsized":"2024-09-20T19:21:33.890582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:24.063388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.063388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:26.111912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962568+0000","last_change":"2024-09-20T19:20:58.711594+0000","last_active":"2024-09-20T19:21:33.962568+0000","last_peered":"2024-09-20T19:21:33.962568+0000","last_clean":"2024-09-20T19:21:33.962568+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:21:33.962568+0000","last_undegraded":"2024-09-20T19:21:33.962568+0000","last_fullsized":"2024-09-20T19:21:33.962568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.711472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.711472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:22.916381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718293+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:21:33.718293+0000","last_peered":"2024-09-20T19:21:33.718293+0000","last_clean":"2024-09-20T19:21:33.718293+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:21:33.718293+0000","last_undegraded":"2024-09-20T19:21:33.718293+0000","last_fullsized":"2024-09-20T19:21:33.718293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:31.968046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717807+0000","last_change":"2024-09-20T19:21:30.266661+0000","last_active":"2024-09-20T19:21:33.717807+0000","last_peered":"2024-09-20T19:21:33.717807+0000","last_clean":"2024-09-20T19:21:33.717807+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:21:33.717807+0000","last_undegraded":"2024-09-20T19:21:33.717807+0000","last_fullsized":"2024-09-20T19:21:33.717807+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.987299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"77'14","reported_seq":128,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285414+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:21:33.285414+0000","last_peered":"2024-09-20T19:21:33.285414+0000","last_clean":"2024-09-20T19:21:33.285414+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:21:33.285414+0000","last_undegraded":"2024-09-20T19:21:33.285414+0000","last_fullsized":"2024-09-20T19:21:33.285414+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:31.247030+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053663+0000","last_change":"2024-09-20T19:20:16.657110+0000","last_active":"2024-09-20T19:21:34.053663+0000","last_peered":"2024-09-20T19:21:34.053663+0000","last_clean":"2024-09-20T19:21:34.053663+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:21:34.053663+0000","last_undegraded":"2024-09-20T19:21:34.053663+0000","last_fullsized":"2024-09-20T19:21:34.053663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.656960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.656960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:43.026972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":164,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717185+0000","last_change":"2024-09-20T19:21:07.002029+0000","last_active":"2024-09-20T19:21:33.717185+0000","last_peered":"2024-09-20T19:21:33.717185+0000","last_clean":"2024-09-20T19:21:33.717185+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:21:33.717185+0000","last_undegraded":"2024-09-20T19:21:33.717185+0000","last_fullsized":"2024-09-20T19:21:33.717185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:07.001951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.001951+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:14.042713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890084+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:21:33.890084+0000","last_peered":"2024-09-20T19:21:33.890084+0000","last_clean":"2024-09-20T19:21:33.890084+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:21:33.890084+0000","last_undegraded":"2024-09-20T19:21:33.890084+0000","last_fullsized":"2024-09-20T19:21:33.890084+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.719146+0000","last_change":"2024-09-20T19:21:30.257690+0000","last_active":"2024-09-20T19:21:33.719146+0000","last_peered":"2024-09-20T19:21:33.719146+0000","last_clean":"2024-09-20T19:21:33.719146+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:21:33.719146+0000","last_undegraded":"2024-09-20T19:21:33.719146+0000","last_fullsized":"2024-09-20T19:21:33.719146+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:31.099212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"77'18","reported_seq":147,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717411+0000","last_change":"2024-09-20T19:20:40.068159+0000","last_active":"2024-09-20T19:21:33.717411+0000","last_peered":"2024-09-20T19:21:33.717411+0000","last_clean":"2024-09-20T19:21:33.717411+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:21:33.717411+0000","last_undegraded":"2024-09-20T19:21:33.717411+0000","last_fullsized":"2024-09-20T19:21:33.717411+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:20:40.068007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:40.068007+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.669287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":120,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779686+0000","last_change":"2024-09-20T19:21:01.687454+0000","last_active":"2024-09-20T19:21:33.779686+0000","last_peered":"2024-09-20T19:21:33.779686+0000","last_clean":"2024-09-20T19:21:33.779686+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:21:33.779686+0000","last_undegraded":"2024-09-20T19:21:33.779686+0000","last_fullsized":"2024-09-20T19:21:33.779686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:21:01.687305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.687305+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.675133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285186+0000","last_change":"2024-09-20T19:20:55.875233+0000","last_active":"2024-09-20T19:21:33.285186+0000","last_peered":"2024-09-20T19:21:33.285186+0000","last_clean":"2024-09-20T19:21:33.285186+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:21:33.285186+0000","last_undegraded":"2024-09-20T19:21:33.285186+0000","last_fullsized":"2024-09-20T19:21:33.285186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:55.875113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:55.875113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:02.988541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286163+0000","last_change":"2024-09-20T19:21:15.032164+0000","last_active":"2024-09-20T19:21:33.286163+0000","last_peered":"2024-09-20T19:21:33.286163+0000","last_clean":"2024-09-20T19:21:33.286163+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:21:33.286163+0000","last_undegraded":"2024-09-20T19:21:33.286163+0000","last_fullsized":"2024-09-20T19:21:33.286163+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:15.032044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:15.032044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:33.740947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961331+0000","last_change":"2024-09-20T19:21:30.261284+0000","last_active":"2024-09-20T19:21:33.961331+0000","last_peered":"2024-09-20T19:21:33.961331+0000","last_clean":"2024-09-20T19:21:33.961331+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:21:33.961331+0000","last_undegraded":"2024-09-20T19:21:33.961331+0000","last_fullsized":"2024-09-20T19:21:33.961331+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:58.403791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"77'19","reported_seq":151,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282953+0000","last_change":"2024-09-20T19:21:07.855976+0000","last_active":"2024-09-20T19:21:33.282953+0000","last_peered":"2024-09-20T19:21:33.282953+0000","last_clean":"2024-09-20T19:21:33.282953+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:21:33.282953+0000","last_undegraded":"2024-09-20T19:21:33.282953+0000","last_fullsized":"2024-09-20T19:21:33.282953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:21:07.855828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.855828+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.893647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.280450+0000","last_change":"2024-09-20T19:20:57.326204+0000","last_active":"2024-09-20T19:21:33.280450+0000","last_peered":"2024-09-20T19:21:33.280450+0000","last_clean":"2024-09-20T19:21:33.280450+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:21:33.280450+0000","last_undegraded":"2024-09-20T19:21:33.280450+0000","last_fullsized":"2024-09-20T19:21:33.280450+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:57.326138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:57.326138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:08.993202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.051984+0000","last_change":"2024-09-20T19:20:57.743630+0000","last_active":"2024-09-20T19:21:34.051984+0000","last_peered":"2024-09-20T19:21:34.051984+0000","last_clean":"2024-09-20T19:21:34.051984+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:21:34.051984+0000","last_undegraded":"2024-09-20T19:21:34.051984+0000","last_fullsized":"2024-09-20T19:21:34.051984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:57.743505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:57.743505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:08.261508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778195+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:21:33.778195+0000","last_peered":"2024-09-20T19:21:33.778195+0000","last_clean":"2024-09-20T19:21:33.778195+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:21:33.778195+0000","last_undegraded":"2024-09-20T19:21:33.778195+0000","last_fullsized":"2024-09-20T19:21:33.778195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:30.773361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285874+0000","last_change":"2024-09-20T19:21:30.244576+0000","last_active":"2024-09-20T19:21:33.285874+0000","last_peered":"2024-09-20T19:21:33.285874+0000","last_clean":"2024-09-20T19:21:33.285874+0000","last_became_active":"2024-09-20T19:21:30.244233+0000","last_became_peered":"2024-09-20T19:21:30.244233+0000","last_unstale":"2024-09-20T19:21:33.285874+0000","last_undegraded":"2024-09-20T19:21:33.285874+0000","last_fullsized":"2024-09-20T19:21:33.285874+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:46.472041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"77'10","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779591+0000","last_change":"2024-09-20T19:21:10.749591+0000","last_active":"2024-09-20T19:21:33.779591+0000","last_peered":"2024-09-20T19:21:33.779591+0000","last_clean":"2024-09-20T19:21:33.779591+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:21:33.779591+0000","last_undegraded":"2024-09-20T19:21:33.779591+0000","last_fullsized":"2024-09-20T19:21:33.779591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:21:10.749450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.749450+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:14.713526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779814+0000","last_change":"2024-09-20T19:21:07.744803+0000","last_active":"2024-09-20T19:21:33.779814+0000","last_peered":"2024-09-20T19:21:33.779814+0000","last_clean":"2024-09-20T19:21:33.779814+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:21:33.779814+0000","last_undegraded":"2024-09-20T19:21:33.779814+0000","last_fullsized":"2024-09-20T19:21:33.779814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.744662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.744662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:16.079611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":164,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282871+0000","last_change":"2024-09-20T19:21:10.853855+0000","last_active":"2024-09-20T19:21:33.282871+0000","last_peered":"2024-09-20T19:21:33.282871+0000","last_clean":"2024-09-20T19:21:33.282871+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:21:33.282871+0000","last_undegraded":"2024-09-20T19:21:33.282871+0000","last_fullsized":"2024-09-20T19:21:33.282871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:10.853722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.853722+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:33.964201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.277688+0000","last_change":"2024-09-20T19:20:49.888543+0000","last_active":"2024-09-20T19:21:33.277688+0000","last_peered":"2024-09-20T19:21:33.277688+0000","last_clean":"2024-09-20T19:21:33.277688+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:21:33.277688+0000","last_undegraded":"2024-09-20T19:21:33.277688+0000","last_fullsized":"2024-09-20T19:21:33.277688+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:49.888394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:49.888394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:55.003131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053323+0000","last_change":"2024-09-20T19:21:30.245620+0000","last_active":"2024-09-20T19:21:34.053323+0000","last_peered":"2024-09-20T19:21:34.053323+0000","last_clean":"2024-09-20T19:21:34.053323+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:21:34.053323+0000","last_undegraded":"2024-09-20T19:21:34.053323+0000","last_fullsized":"2024-09-20T19:21:34.053323+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:41.979641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"77'30","reported_seq":208,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779780+0000","last_change":"2024-09-20T19:21:02.687387+0000","last_active":"2024-09-20T19:21:33.779780+0000","last_peered":"2024-09-20T19:21:33.779780+0000","last_clean":"2024-09-20T19:21:33.779780+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:21:33.779780+0000","last_undegraded":"2024-09-20T19:21:33.779780+0000","last_fullsized":"2024-09-20T19:21:33.779780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:21:02.687229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:02.687229+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:11.714867+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962093+0000","last_change":"2024-09-20T19:20:55.732504+0000","last_active":"2024-09-20T19:21:33.962093+0000","last_peered":"2024-09-20T19:21:33.962093+0000","last_clean":"2024-09-20T19:21:33.962093+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:21:33.962093+0000","last_undegraded":"2024-09-20T19:21:33.962093+0000","last_fullsized":"2024-09-20T19:21:33.962093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:55.732325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:55.732325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:56.655748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.279921+0000","last_change":"2024-09-20T19:21:13.716496+0000","last_active":"2024-09-20T19:21:33.279921+0000","last_peered":"2024-09-20T19:21:33.279921+0000","last_clean":"2024-09-20T19:21:33.279921+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:21:33.279921+0000","last_undegraded":"2024-09-20T19:21:33.279921+0000","last_fullsized":"2024-09-20T19:21:33.279921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:13.716329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:13.716329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:21.600072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.716623+0000","last_change":"2024-09-20T19:21:22.999854+0000","last_active":"2024-09-20T19:21:33.716623+0000","last_peered":"2024-09-20T19:21:33.716623+0000","last_clean":"2024-09-20T19:21:33.716623+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:21:33.716623+0000","last_undegraded":"2024-09-20T19:21:33.716623+0000","last_fullsized":"2024-09-20T19:21:33.716623+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:22.999796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:22.999796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:43.989969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278976+0000","last_change":"2024-09-20T19:21:30.255315+0000","last_active":"2024-09-20T19:21:33.278976+0000","last_peered":"2024-09-20T19:21:33.278976+0000","last_clean":"2024-09-20T19:21:33.278976+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:21:33.278976+0000","last_undegraded":"2024-09-20T19:21:33.278976+0000","last_fullsized":"2024-09-20T19:21:33.278976+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:59.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"77'13","reported_seq":58,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282562+0000","last_change":"2024-09-20T19:21:22.193770+0000","last_active":"2024-09-20T19:21:33.282562+0000","last_peered":"2024-09-20T19:21:33.282562+0000","last_clean":"2024-09-20T19:21:33.282562+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:21:33.282562+0000","last_undegraded":"2024-09-20T19:21:33.282562+0000","last_fullsized":"2024-09-20T19:21:33.282562+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:21:22.193573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:22.193573+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:42.700682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":325,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:40.274947+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:21:40.274947+0000","last_peered":"2024-09-20T19:21:40.274947+0000","last_clean":"2024-09-20T19:21:40.274947+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:21:40.274947+0000","last_undegraded":"2024-09-20T19:21:40.274947+0000","last_fullsized":"2024-09-20T19:21:40.274947+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:17.892259+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278219+0000","last_change":"2024-09-20T19:21:10.216852+0000","last_active":"2024-09-20T19:21:33.278219+0000","last_peered":"2024-09-20T19:21:33.278219+0000","last_clean":"2024-09-20T19:21:33.278219+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:21:33.278219+0000","last_undegraded":"2024-09-20T19:21:33.278219+0000","last_fullsized":"2024-09-20T19:21:33.278219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:10.216700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.216700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:25.466216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281172+0000","last_change":"2024-09-20T19:20:30.146960+0000","last_active":"2024-09-20T19:21:33.281172+0000","last_peered":"2024-09-20T19:21:33.281172+0000","last_clean":"2024-09-20T19:21:33.281172+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:21:33.281172+0000","last_undegraded":"2024-09-20T19:21:33.281172+0000","last_fullsized":"2024-09-20T19:21:33.281172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:30.146806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:30.146806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.691081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.891184+0000","last_change":"2024-09-20T19:21:03.024216+0000","last_active":"2024-09-20T19:21:33.891184+0000","last_peered":"2024-09-20T19:21:33.891184+0000","last_clean":"2024-09-20T19:21:33.891184+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:21:33.891184+0000","last_undegraded":"2024-09-20T19:21:33.891184+0000","last_fullsized":"2024-09-20T19:21:33.891184+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:03.024074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:03.024074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:28.483408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.891125+0000","last_change":"2024-09-20T19:21:30.249964+0000","last_active":"2024-09-20T19:21:33.891125+0000","last_peered":"2024-09-20T19:21:33.891125+0000","last_clean":"2024-09-20T19:21:33.891125+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:21:33.891125+0000","last_undegraded":"2024-09-20T19:21:33.891125+0000","last_fullsized":"2024-09-20T19:21:33.891125+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:58.372260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'12","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282418+0000","last_change":"2024-09-20T19:20:46.910460+0000","last_active":"2024-09-20T19:21:33.282418+0000","last_peered":"2024-09-20T19:21:33.282418+0000","last_clean":"2024-09-20T19:21:33.282418+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:21:33.282418+0000","last_undegraded":"2024-09-20T19:21:33.282418+0000","last_fullsized":"2024-09-20T19:21:33.282418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:46.910317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:46.910317+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:16.209696+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281764+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:21:33.281764+0000","last_peered":"2024-09-20T19:21:33.281764+0000","last_clean":"2024-09-20T19:21:33.281764+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:21:33.281764+0000","last_undegraded":"2024-09-20T19:21:33.281764+0000","last_fullsized":"2024-09-20T19:21:33.281764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:35.851755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890705+0000","last_change":"2024-09-20T19:20:56.865163+0000","last_active":"2024-09-20T19:21:33.890705+0000","last_peered":"2024-09-20T19:21:33.890705+0000","last_clean":"2024-09-20T19:21:33.890705+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:21:33.890705+0000","last_undegraded":"2024-09-20T19:21:33.890705+0000","last_fullsized":"2024-09-20T19:21:33.890705+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:56.865049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:56.865049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:15.417801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962577+0000","last_change":"2024-09-20T19:20:42.760023+0000","last_active":"2024-09-20T19:21:33.962577+0000","last_peered":"2024-09-20T19:21:33.962577+0000","last_clean":"2024-09-20T19:21:33.962577+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:21:33.962577+0000","last_undegraded":"2024-09-20T19:21:33.962577+0000","last_fullsized":"2024-09-20T19:21:33.962577+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:42.759902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:42.759902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:53.827837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778879+0000","last_change":"2024-09-20T19:20:41.502208+0000","last_active":"2024-09-20T19:21:33.778879+0000","last_peered":"2024-09-20T19:21:33.778879+0000","last_clean":"2024-09-20T19:21:33.778879+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:21:33.778879+0000","last_undegraded":"2024-09-20T19:21:33.778879+0000","last_fullsized":"2024-09-20T19:21:33.778879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:41.502048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:41.502048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:05.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.280160+0000","last_change":"2024-09-20T19:21:30.245233+0000","last_active":"2024-09-20T19:21:33.280160+0000","last_peered":"2024-09-20T19:21:33.280160+0000","last_clean":"2024-09-20T19:21:33.280160+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:21:33.280160+0000","last_undegraded":"2024-09-20T19:21:33.280160+0000","last_fullsized":"2024-09-20T19:21:33.280160+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:54.104390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"77'16","reported_seq":179,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053320+0000","last_change":"2024-09-20T19:21:12.740077+0000","last_active":"2024-09-20T19:21:34.053320+0000","last_peered":"2024-09-20T19:21:34.053320+0000","last_clean":"2024-09-20T19:21:34.053320+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:21:34.053320+0000","last_undegraded":"2024-09-20T19:21:34.053320+0000","last_fullsized":"2024-09-20T19:21:34.053320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:21:12.739915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:12.739915+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:36.865159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718703+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:21:33.718703+0000","last_peered":"2024-09-20T19:21:33.718703+0000","last_clean":"2024-09-20T19:21:33.718703+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:21:33.718703+0000","last_undegraded":"2024-09-20T19:21:33.718703+0000","last_fullsized":"2024-09-20T19:21:33.718703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:33.008883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962128+0000","last_change":"2024-09-20T19:20:52.723249+0000","last_active":"2024-09-20T19:21:33.962128+0000","last_peered":"2024-09-20T19:21:33.962128+0000","last_clean":"2024-09-20T19:21:33.962128+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:21:33.962128+0000","last_undegraded":"2024-09-20T19:21:33.962128+0000","last_fullsized":"2024-09-20T19:21:33.962128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.723121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.723121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:21.031548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":159,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890841+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:21:33.890841+0000","last_peered":"2024-09-20T19:21:33.890841+0000","last_clean":"2024-09-20T19:21:33.890841+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:21:33.890841+0000","last_undegraded":"2024-09-20T19:21:33.890841+0000","last_fullsized":"2024-09-20T19:21:33.890841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.891044+0000","last_change":"2024-09-20T19:20:53.868159+0000","last_active":"2024-09-20T19:21:33.891044+0000","last_peered":"2024-09-20T19:21:33.891044+0000","last_clean":"2024-09-20T19:21:33.891044+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:21:33.891044+0000","last_undegraded":"2024-09-20T19:21:33.891044+0000","last_fullsized":"2024-09-20T19:21:33.891044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.868034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.868034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:23.049747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961164+0000","last_change":"2024-09-20T19:20:39.803967+0000","last_active":"2024-09-20T19:21:33.961164+0000","last_peered":"2024-09-20T19:21:33.961164+0000","last_clean":"2024-09-20T19:21:33.961164+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:21:33.961164+0000","last_undegraded":"2024-09-20T19:21:33.961164+0000","last_fullsized":"2024-09-20T19:21:33.961164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:39.803847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.803847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.060059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.277762+0000","last_change":"2024-09-20T19:21:30.268769+0000","last_active":"2024-09-20T19:21:33.277762+0000","last_peered":"2024-09-20T19:21:33.277762+0000","last_clean":"2024-09-20T19:21:33.277762+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:21:33.277762+0000","last_undegraded":"2024-09-20T19:21:33.277762+0000","last_fullsized":"2024-09-20T19:21:33.277762+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:35.599514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"77'19","reported_seq":145,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052315+0000","last_change":"2024-09-20T19:20:52.718858+0000","last_active":"2024-09-20T19:21:34.052315+0000","last_peered":"2024-09-20T19:21:34.052315+0000","last_clean":"2024-09-20T19:21:34.052315+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:21:34.052315+0000","last_undegraded":"2024-09-20T19:21:34.052315+0000","last_fullsized":"2024-09-20T19:21:34.052315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:20:52.718711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.718711+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:18.643077+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281189+0000","last_change":"2024-09-20T19:20:35.231905+0000","last_active":"2024-09-20T19:21:33.281189+0000","last_peered":"2024-09-20T19:21:33.281189+0000","last_clean":"2024-09-20T19:21:33.281189+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:21:33.281189+0000","last_undegraded":"2024-09-20T19:21:33.281189+0000","last_fullsized":"2024-09-20T19:21:33.281189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:35.231776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:35.231776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:03.117077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.279579+0000","last_change":"2024-09-20T19:21:06.345504+0000","last_active":"2024-09-20T19:21:33.279579+0000","last_peered":"2024-09-20T19:21:33.279579+0000","last_clean":"2024-09-20T19:21:33.279579+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:21:33.279579+0000","last_undegraded":"2024-09-20T19:21:33.279579+0000","last_fullsized":"2024-09-20T19:21:33.279579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.345373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.345373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:26.417269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890083+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:21:33.890083+0000","last_peered":"2024-09-20T19:21:33.890083+0000","last_clean":"2024-09-20T19:21:33.890083+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:21:33.890083+0000","last_undegraded":"2024-09-20T19:21:33.890083+0000","last_fullsized":"2024-09-20T19:21:33.890083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.277392+0000","last_change":"2024-09-20T19:21:30.238309+0000","last_active":"2024-09-20T19:21:33.277392+0000","last_peered":"2024-09-20T19:21:33.277392+0000","last_clean":"2024-09-20T19:21:33.277392+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:21:33.277392+0000","last_undegraded":"2024-09-20T19:21:33.277392+0000","last_fullsized":"2024-09-20T19:21:33.277392+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.873494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"77'9","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282833+0000","last_change":"2024-09-20T19:21:05.825257+0000","last_active":"2024-09-20T19:21:33.282833+0000","last_peered":"2024-09-20T19:21:33.282833+0000","last_clean":"2024-09-20T19:21:33.282833+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:21:33.282833+0000","last_undegraded":"2024-09-20T19:21:33.282833+0000","last_fullsized":"2024-09-20T19:21:33.282833+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:21:05.825100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.825100+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:35.807778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779745+0000","last_change":"2024-09-20T19:21:05.740388+0000","last_active":"2024-09-20T19:21:33.779745+0000","last_peered":"2024-09-20T19:21:33.779745+0000","last_clean":"2024-09-20T19:21:33.779745+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:21:33.779745+0000","last_undegraded":"2024-09-20T19:21:33.779745+0000","last_fullsized":"2024-09-20T19:21:33.779745+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.740278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.740278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:19.446478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278824+0000","last_change":"2024-09-20T19:21:17.226835+0000","last_active":"2024-09-20T19:21:33.278824+0000","last_peered":"2024-09-20T19:21:33.278824+0000","last_clean":"2024-09-20T19:21:33.278824+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:21:33.278824+0000","last_undegraded":"2024-09-20T19:21:33.278824+0000","last_fullsized":"2024-09-20T19:21:33.278824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:17.226714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:17.226714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:24.198030+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281363+0000","last_change":"2024-09-20T19:21:16.655602+0000","last_active":"2024-09-20T19:21:33.281363+0000","last_peered":"2024-09-20T19:21:33.281363+0000","last_clean":"2024-09-20T19:21:33.281363+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:21:33.281363+0000","last_undegraded":"2024-09-20T19:21:33.281363+0000","last_fullsized":"2024-09-20T19:21:33.281363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:21:16.655463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.655463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:21.527404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778724+0000","last_change":"2024-09-20T19:21:30.256896+0000","last_active":"2024-09-20T19:21:33.778724+0000","last_peered":"2024-09-20T19:21:33.778724+0000","last_clean":"2024-09-20T19:21:33.778724+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:21:33.778724+0000","last_undegraded":"2024-09-20T19:21:33.778724+0000","last_fullsized":"2024-09-20T19:21:33.778724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.689001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"77'15","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961525+0000","last_change":"2024-09-20T19:20:56.707860+0000","last_active":"2024-09-20T19:21:33.961525+0000","last_peered":"2024-09-20T19:21:33.961525+0000","last_clean":"2024-09-20T19:21:33.961525+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:21:33.961525+0000","last_undegraded":"2024-09-20T19:21:33.961525+0000","last_fullsized":"2024-09-20T19:21:33.961525+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:20:56.707677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:56.707677+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:17.488249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281552+0000","last_change":"2024-09-20T19:20:51.304499+0000","last_active":"2024-09-20T19:21:33.281552+0000","last_peered":"2024-09-20T19:21:33.281552+0000","last_clean":"2024-09-20T19:21:33.281552+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:21:33.281552+0000","last_undegraded":"2024-09-20T19:21:33.281552+0000","last_fullsized":"2024-09-20T19:21:33.281552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:51.304365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:51.304365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:00.834548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":164,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890596+0000","last_change":"2024-09-20T19:20:25.228464+0000","last_active":"2024-09-20T19:21:33.890596+0000","last_peered":"2024-09-20T19:21:33.890596+0000","last_clean":"2024-09-20T19:21:33.890596+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:21:33.890596+0000","last_undegraded":"2024-09-20T19:21:33.890596+0000","last_fullsized":"2024-09-20T19:21:33.890596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:25.228325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.228325+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.437018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961071+0000","last_change":"2024-09-20T19:21:01.739612+0000","last_active":"2024-09-20T19:21:33.961071+0000","last_peered":"2024-09-20T19:21:33.961071+0000","last_clean":"2024-09-20T19:21:33.961071+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:21:33.961071+0000","last_undegraded":"2024-09-20T19:21:33.961071+0000","last_fullsized":"2024-09-20T19:21:33.961071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:01.739492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.739492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:18.911597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717528+0000","last_change":"2024-09-20T19:21:30.258423+0000","last_active":"2024-09-20T19:21:33.717528+0000","last_peered":"2024-09-20T19:21:33.717528+0000","last_clean":"2024-09-20T19:21:33.717528+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:21:33.717528+0000","last_undegraded":"2024-09-20T19:21:33.717528+0000","last_fullsized":"2024-09-20T19:21:33.717528+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:43.070928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"77'12","reported_seq":157,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285401+0000","last_change":"2024-09-20T19:20:16.852586+0000","last_active":"2024-09-20T19:21:33.285401+0000","last_peered":"2024-09-20T19:21:33.285401+0000","last_clean":"2024-09-20T19:21:33.285401+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:21:33.285401+0000","last_undegraded":"2024-09-20T19:21:33.285401+0000","last_fullsized":"2024-09-20T19:21:33.285401+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:20:16.852450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.852450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.371990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286919+0000","last_change":"2024-09-20T19:20:37.785586+0000","last_active":"2024-09-20T19:21:33.286919+0000","last_peered":"2024-09-20T19:21:33.286919+0000","last_clean":"2024-09-20T19:21:33.286919+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:21:33.286919+0000","last_undegraded":"2024-09-20T19:21:33.286919+0000","last_fullsized":"2024-09-20T19:21:33.286919+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:20:37.785449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.785449+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:49.952598+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.277990+0000","last_change":"2024-09-20T19:21:00.309978+0000","last_active":"2024-09-20T19:21:33.277990+0000","last_peered":"2024-09-20T19:21:33.277990+0000","last_clean":"2024-09-20T19:21:33.277990+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:21:33.277990+0000","last_undegraded":"2024-09-20T19:21:33.277990+0000","last_fullsized":"2024-09-20T19:21:33.277990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.309838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.309838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:13.081169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718020+0000","last_change":"2024-09-20T19:20:28.004917+0000","last_active":"2024-09-20T19:21:33.718020+0000","last_peered":"2024-09-20T19:21:33.718020+0000","last_clean":"2024-09-20T19:21:33.718020+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:21:33.718020+0000","last_undegraded":"2024-09-20T19:21:33.718020+0000","last_fullsized":"2024-09-20T19:21:33.718020+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.004784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.004784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.093046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.282247+0000","last_change":"2024-09-20T19:20:33.803376+0000","last_active":"2024-09-20T19:21:33.282247+0000","last_peered":"2024-09-20T19:21:33.282247+0000","last_clean":"2024-09-20T19:21:33.282247+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:21:33.282247+0000","last_undegraded":"2024-09-20T19:21:33.282247+0000","last_fullsized":"2024-09-20T19:21:33.282247+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:33.803234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.803234+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:56.407084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.278722+0000","last_change":"2024-09-20T19:20:52.315399+0000","last_active":"2024-09-20T19:21:33.278722+0000","last_peered":"2024-09-20T19:21:33.278722+0000","last_clean":"2024-09-20T19:21:33.278722+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:21:33.278722+0000","last_undegraded":"2024-09-20T19:21:33.278722+0000","last_fullsized":"2024-09-20T19:21:33.278722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.315272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.315272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.326277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.719875+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:21:33.719875+0000","last_peered":"2024-09-20T19:21:33.719875+0000","last_clean":"2024-09-20T19:21:33.719875+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:21:33.719875+0000","last_undegraded":"2024-09-20T19:21:33.719875+0000","last_fullsized":"2024-09-20T19:21:33.719875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:22.664352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.279690+0000","last_change":"2024-09-20T19:21:23.241066+0000","last_active":"2024-09-20T19:21:33.279690+0000","last_peered":"2024-09-20T19:21:33.279690+0000","last_clean":"2024-09-20T19:21:33.279690+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:21:33.279690+0000","last_undegraded":"2024-09-20T19:21:33.279690+0000","last_fullsized":"2024-09-20T19:21:33.279690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:23.241003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:23.241003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:44.184437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052576+0000","last_change":"2024-09-20T19:20:37.743313+0000","last_active":"2024-09-20T19:21:34.052576+0000","last_peered":"2024-09-20T19:21:34.052576+0000","last_clean":"2024-09-20T19:21:34.052576+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:21:34.052576+0000","last_undegraded":"2024-09-20T19:21:34.052576+0000","last_fullsized":"2024-09-20T19:21:34.052576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:37.743188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:37.743188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:40.122912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.720107+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:21:33.720107+0000","last_peered":"2024-09-20T19:21:33.720107+0000","last_clean":"2024-09-20T19:21:33.720107+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:21:33.720107+0000","last_undegraded":"2024-09-20T19:21:33.720107+0000","last_fullsized":"2024-09-20T19:21:33.720107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:09.130493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962638+0000","last_change":"2024-09-20T19:21:28.981452+0000","last_active":"2024-09-20T19:21:33.962638+0000","last_peered":"2024-09-20T19:21:33.962638+0000","last_clean":"2024-09-20T19:21:33.962638+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:21:33.962638+0000","last_undegraded":"2024-09-20T19:21:33.962638+0000","last_fullsized":"2024-09-20T19:21:33.962638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.981320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.981320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:42.283996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890225+0000","last_change":"2024-09-20T19:21:29.100795+0000","last_active":"2024-09-20T19:21:33.890225+0000","last_peered":"2024-09-20T19:21:33.890225+0000","last_clean":"2024-09-20T19:21:33.890225+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:21:33.890225+0000","last_undegraded":"2024-09-20T19:21:33.890225+0000","last_fullsized":"2024-09-20T19:21:33.890225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.100718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.100718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:41.514519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.719105+0000","last_change":"2024-09-20T19:20:32.059785+0000","last_active":"2024-09-20T19:21:33.719105+0000","last_peered":"2024-09-20T19:21:33.719105+0000","last_clean":"2024-09-20T19:21:33.719105+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:21:33.719105+0000","last_undegraded":"2024-09-20T19:21:33.719105+0000","last_fullsized":"2024-09-20T19:21:33.719105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:32.059603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.059603+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:41.930875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961913+0000","last_change":"2024-09-20T19:21:05.797825+0000","last_active":"2024-09-20T19:21:33.961913+0000","last_peered":"2024-09-20T19:21:33.961913+0000","last_clean":"2024-09-20T19:21:33.961913+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:21:33.961913+0000","last_undegraded":"2024-09-20T19:21:33.961913+0000","last_fullsized":"2024-09-20T19:21:33.961913+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.797707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.797707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:14.206559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":163,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962437+0000","last_change":"2024-09-20T19:21:02.749772+0000","last_active":"2024-09-20T19:21:33.962437+0000","last_peered":"2024-09-20T19:21:33.962437+0000","last_clean":"2024-09-20T19:21:33.962437+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:21:33.962437+0000","last_undegraded":"2024-09-20T19:21:33.962437+0000","last_fullsized":"2024-09-20T19:21:33.962437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:02.749641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:02.749641+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:02.793974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.279594+0000","last_change":"2024-09-20T19:20:53.876915+0000","last_active":"2024-09-20T19:21:33.279594+0000","last_peered":"2024-09-20T19:21:33.279594+0000","last_clean":"2024-09-20T19:21:33.279594+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:21:33.279594+0000","last_undegraded":"2024-09-20T19:21:33.279594+0000","last_fullsized":"2024-09-20T19:21:33.279594+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.876771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.876771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:59.986119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962405+0000","last_change":"2024-09-20T19:21:16.849353+0000","last_active":"2024-09-20T19:21:33.962405+0000","last_peered":"2024-09-20T19:21:33.962405+0000","last_clean":"2024-09-20T19:21:33.962405+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:21:33.962405+0000","last_undegraded":"2024-09-20T19:21:33.962405+0000","last_fullsized":"2024-09-20T19:21:33.962405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:21:16.849172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.849172+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:45.920790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.283189+0000","last_change":"2024-09-20T19:21:19.700650+0000","last_active":"2024-09-20T19:21:33.283189+0000","last_peered":"2024-09-20T19:21:33.283189+0000","last_clean":"2024-09-20T19:21:33.283189+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:21:33.283189+0000","last_undegraded":"2024-09-20T19:21:33.283189+0000","last_fullsized":"2024-09-20T19:21:33.283189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:19.700502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:19.700502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:19.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":156,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718714+0000","last_change":"2024-09-20T19:20:20.964445+0000","last_active":"2024-09-20T19:21:33.718714+0000","last_peered":"2024-09-20T19:21:33.718714+0000","last_clean":"2024-09-20T19:21:33.718714+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:21:33.718714+0000","last_undegraded":"2024-09-20T19:21:33.718714+0000","last_fullsized":"2024-09-20T19:21:33.718714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:20.964261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:20.964261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.805123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286610+0000","last_change":"2024-09-20T19:20:43.861815+0000","last_active":"2024-09-20T19:21:33.286610+0000","last_peered":"2024-09-20T19:21:33.286610+0000","last_clean":"2024-09-20T19:21:33.286610+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:21:33.286610+0000","last_undegraded":"2024-09-20T19:21:33.286610+0000","last_fullsized":"2024-09-20T19:21:33.286610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:43.861696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.861696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:11.874804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779216+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:21:33.779216+0000","last_peered":"2024-09-20T19:21:33.779216+0000","last_clean":"2024-09-20T19:21:33.779216+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:21:33.779216+0000","last_undegraded":"2024-09-20T19:21:33.779216+0000","last_fullsized":"2024-09-20T19:21:33.779216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052782+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:21:34.052782+0000","last_peered":"2024-09-20T19:21:34.052782+0000","last_clean":"2024-09-20T19:21:34.052782+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:21:34.052782+0000","last_undegraded":"2024-09-20T19:21:34.052782+0000","last_fullsized":"2024-09-20T19:21:34.052782+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:34.049346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718339+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:21:33.718339+0000","last_peered":"2024-09-20T19:21:33.718339+0000","last_clean":"2024-09-20T19:21:33.718339+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:21:33.718339+0000","last_undegraded":"2024-09-20T19:21:33.718339+0000","last_fullsized":"2024-09-20T19:21:33.718339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:29.912985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.277392+0000","last_change":"2024-09-20T19:21:21.676309+0000","last_active":"2024-09-20T19:21:33.277392+0000","last_peered":"2024-09-20T19:21:33.277392+0000","last_clean":"2024-09-20T19:21:33.277392+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:21:33.277392+0000","last_undegraded":"2024-09-20T19:21:33.277392+0000","last_fullsized":"2024-09-20T19:21:33.277392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:21.676157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.676157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:34.450323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285796+0000","last_change":"2024-09-20T19:20:52.914253+0000","last_active":"2024-09-20T19:21:33.285796+0000","last_peered":"2024-09-20T19:21:33.285796+0000","last_clean":"2024-09-20T19:21:33.285796+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:21:33.285796+0000","last_undegraded":"2024-09-20T19:21:33.285796+0000","last_fullsized":"2024-09-20T19:21:33.285796+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:52.914121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.914121+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:15.213678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":127,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779594+0000","last_change":"2024-09-20T19:20:17.522125+0000","last_active":"2024-09-20T19:21:33.779594+0000","last_peered":"2024-09-20T19:21:33.779594+0000","last_clean":"2024-09-20T19:21:33.779594+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:21:33.779594+0000","last_undegraded":"2024-09-20T19:21:33.779594+0000","last_fullsized":"2024-09-20T19:21:33.779594+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:17.521947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:17.521947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.763307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779370+0000","last_change":"2024-09-20T19:21:06.774687+0000","last_active":"2024-09-20T19:21:33.779370+0000","last_peered":"2024-09-20T19:21:33.779370+0000","last_clean":"2024-09-20T19:21:33.779370+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:21:33.779370+0000","last_undegraded":"2024-09-20T19:21:33.779370+0000","last_fullsized":"2024-09-20T19:21:33.779370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.774547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.774547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:23.401237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.716472+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:21:33.716472+0000","last_peered":"2024-09-20T19:21:33.716472+0000","last_clean":"2024-09-20T19:21:33.716472+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:21:33.716472+0000","last_undegraded":"2024-09-20T19:21:33.716472+0000","last_fullsized":"2024-09-20T19:21:33.716472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717347+0000","last_change":"2024-09-20T19:20:39.076366+0000","last_active":"2024-09-20T19:21:33.717347+0000","last_peered":"2024-09-20T19:21:33.717347+0000","last_clean":"2024-09-20T19:21:33.717347+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:21:33.717347+0000","last_undegraded":"2024-09-20T19:21:33.717347+0000","last_fullsized":"2024-09-20T19:21:33.717347+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:20:39.076230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.076230+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.693807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.962757+0000","last_change":"2024-09-20T19:21:25.913956+0000","last_active":"2024-09-20T19:21:33.962757+0000","last_peered":"2024-09-20T19:21:33.962757+0000","last_clean":"2024-09-20T19:21:33.962757+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:21:33.962757+0000","last_undegraded":"2024-09-20T19:21:33.962757+0000","last_fullsized":"2024-09-20T19:21:33.962757+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.913821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.913821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:34.240204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.051905+0000","last_change":"2024-09-20T19:20:38.745217+0000","last_active":"2024-09-20T19:21:34.051905+0000","last_peered":"2024-09-20T19:21:34.051905+0000","last_clean":"2024-09-20T19:21:34.051905+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:21:34.051905+0000","last_undegraded":"2024-09-20T19:21:34.051905+0000","last_fullsized":"2024-09-20T19:21:34.051905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.745091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.745091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:38.749886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890084+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:21:33.890084+0000","last_peered":"2024-09-20T19:21:33.890084+0000","last_clean":"2024-09-20T19:21:33.890084+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:21:33.890084+0000","last_undegraded":"2024-09-20T19:21:33.890084+0000","last_fullsized":"2024-09-20T19:21:33.890084+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.717459+0000","last_change":"2024-09-20T19:20:48.936891+0000","last_active":"2024-09-20T19:21:33.717459+0000","last_peered":"2024-09-20T19:21:33.717459+0000","last_clean":"2024-09-20T19:21:33.717459+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:21:33.717459+0000","last_undegraded":"2024-09-20T19:21:33.717459+0000","last_fullsized":"2024-09-20T19:21:33.717459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:48.936813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.936813+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:02.906803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053601+0000","last_change":"2024-09-20T19:20:29.725323+0000","last_active":"2024-09-20T19:21:34.053601+0000","last_peered":"2024-09-20T19:21:34.053601+0000","last_clean":"2024-09-20T19:21:34.053601+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:21:34.053601+0000","last_undegraded":"2024-09-20T19:21:34.053601+0000","last_fullsized":"2024-09-20T19:21:34.053601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:29.725195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:29.725195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:39.235455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890301+0000","last_change":"2024-09-20T19:21:11.946099+0000","last_active":"2024-09-20T19:21:33.890301+0000","last_peered":"2024-09-20T19:21:33.890301+0000","last_clean":"2024-09-20T19:21:33.890301+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:21:33.890301+0000","last_undegraded":"2024-09-20T19:21:33.890301+0000","last_fullsized":"2024-09-20T19:21:33.890301+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:11.945971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:11.945971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:32.017658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.716523+0000","last_change":"2024-09-20T19:20:44.049456+0000","last_active":"2024-09-20T19:21:33.716523+0000","last_peered":"2024-09-20T19:21:33.716523+0000","last_clean":"2024-09-20T19:21:33.716523+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:21:33.716523+0000","last_undegraded":"2024-09-20T19:21:33.716523+0000","last_fullsized":"2024-09-20T19:21:33.716523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:44.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.049342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:04.326689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.890124+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:21:33.890124+0000","last_peered":"2024-09-20T19:21:33.890124+0000","last_clean":"2024-09-20T19:21:33.890124+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:21:33.890124+0000","last_undegraded":"2024-09-20T19:21:33.890124+0000","last_fullsized":"2024-09-20T19:21:33.890124+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.961531+0000","last_change":"2024-09-20T19:20:54.741270+0000","last_active":"2024-09-20T19:21:33.961531+0000","last_peered":"2024-09-20T19:21:33.961531+0000","last_clean":"2024-09-20T19:21:33.961531+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:21:33.961531+0000","last_undegraded":"2024-09-20T19:21:33.961531+0000","last_fullsized":"2024-09-20T19:21:33.961531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:54.741154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:54.741154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:00.494352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281938+0000","last_change":"2024-09-20T19:21:26.737703+0000","last_active":"2024-09-20T19:21:33.281938+0000","last_peered":"2024-09-20T19:21:33.281938+0000","last_clean":"2024-09-20T19:21:33.281938+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:21:33.281938+0000","last_undegraded":"2024-09-20T19:21:33.281938+0000","last_fullsized":"2024-09-20T19:21:33.281938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:21:26.737598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:26.737598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:47.119800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":166,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053533+0000","last_change":"2024-09-20T19:20:44.717984+0000","last_active":"2024-09-20T19:21:34.053533+0000","last_peered":"2024-09-20T19:21:34.053533+0000","last_clean":"2024-09-20T19:21:34.053533+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:21:34.053533+0000","last_undegraded":"2024-09-20T19:21:34.053533+0000","last_fullsized":"2024-09-20T19:21:34.053533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:44.717840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:44.717840+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:45.300091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.778234+0000","last_change":"2024-09-20T19:21:08.759630+0000","last_active":"2024-09-20T19:21:33.778234+0000","last_peered":"2024-09-20T19:21:33.778234+0000","last_clean":"2024-09-20T19:21:33.778234+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:21:33.778234+0000","last_undegraded":"2024-09-20T19:21:33.778234+0000","last_fullsized":"2024-09-20T19:21:33.778234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.759504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.759504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:30.980559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":120,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718416+0000","last_change":"2024-09-20T19:20:43.067477+0000","last_active":"2024-09-20T19:21:33.718416+0000","last_peered":"2024-09-20T19:21:33.718416+0000","last_clean":"2024-09-20T19:21:33.718416+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:21:33.718416+0000","last_undegraded":"2024-09-20T19:21:33.718416+0000","last_fullsized":"2024-09-20T19:21:33.718416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:20:43.067414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:43.067414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:52.468070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.053462+0000","last_change":"2024-09-20T19:20:54.724274+0000","last_active":"2024-09-20T19:21:34.053462+0000","last_peered":"2024-09-20T19:21:34.053462+0000","last_clean":"2024-09-20T19:21:34.053462+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:21:34.053462+0000","last_undegraded":"2024-09-20T19:21:34.053462+0000","last_fullsized":"2024-09-20T19:21:34.053462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:54.724123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:54.724123+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:08.872114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.281672+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:21:33.281672+0000","last_peered":"2024-09-20T19:21:33.281672+0000","last_clean":"2024-09-20T19:21:33.281672+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:21:33.281672+0000","last_undegraded":"2024-09-20T19:21:33.281672+0000","last_fullsized":"2024-09-20T19:21:33.281672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:46.632034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.286353+0000","last_change":"2024-09-20T19:20:59.816653+0000","last_active":"2024-09-20T19:21:33.286353+0000","last_peered":"2024-09-20T19:21:33.286353+0000","last_clean":"2024-09-20T19:21:33.286353+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:21:33.286353+0000","last_undegraded":"2024-09-20T19:21:33.286353+0000","last_fullsized":"2024-09-20T19:21:33.286353+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:59.816510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:59.816510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:28.547857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:34.052749+0000","last_change":"2024-09-20T19:20:46.661313+0000","last_active":"2024-09-20T19:21:34.052749+0000","last_peered":"2024-09-20T19:21:34.052749+0000","last_clean":"2024-09-20T19:21:34.052749+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:21:34.052749+0000","last_undegraded":"2024-09-20T19:21:34.052749+0000","last_fullsized":"2024-09-20T19:21:34.052749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:46.661186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:46.661186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:50.055994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.280198+0000","last_change":"2024-09-20T19:21:21.225432+0000","last_active":"2024-09-20T19:21:33.280198+0000","last_peered":"2024-09-20T19:21:33.280198+0000","last_clean":"2024-09-20T19:21:33.280198+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:21:33.280198+0000","last_undegraded":"2024-09-20T19:21:33.280198+0000","last_fullsized":"2024-09-20T19:21:33.280198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:21:21.225282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.225282+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:30.142285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.779273+0000","last_change":"2024-09-20T19:20:33.482733+0000","last_active":"2024-09-20T19:21:33.779273+0000","last_peered":"2024-09-20T19:21:33.779273+0000","last_clean":"2024-09-20T19:21:33.779273+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:21:33.779273+0000","last_undegraded":"2024-09-20T19:21:33.779273+0000","last_fullsized":"2024-09-20T19:21:33.779273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.482605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.482605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:21:42.867843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.719731+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:21:33.719731+0000","last_peered":"2024-09-20T19:21:33.719731+0000","last_clean":"2024-09-20T19:21:33.719731+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:21:33.719731+0000","last_undegraded":"2024-09-20T19:21:33.719731+0000","last_fullsized":"2024-09-20T19:21:33.719731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718902+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:21:33.718902+0000","last_peered":"2024-09-20T19:21:33.718902+0000","last_clean":"2024-09-20T19:21:33.718902+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:21:33.718902+0000","last_undegraded":"2024-09-20T19:21:33.718902+0000","last_fullsized":"2024-09-20T19:21:33.718902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:19.716009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.718754+0000","last_change":"2024-09-20T19:21:24.033642+0000","last_active":"2024-09-20T19:21:33.718754+0000","last_peered":"2024-09-20T19:21:33.718754+0000","last_clean":"2024-09-20T19:21:33.718754+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:21:33.718754+0000","last_undegraded":"2024-09-20T19:21:33.718754+0000","last_fullsized":"2024-09-20T19:21:33.718754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:21:24.033570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.033570+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:47.158526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-20T19:21:33.285716+0000","last_change":"2024-09-20T19:21:08.918562+0000","last_active":"2024-09-20T19:21:33.285716+0000","last_peered":"2024-09-20T19:21:33.285716+0000","last_clean":"2024-09-20T19:21:33.285716+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:21:33.285716+0000","last_undegraded":"2024-09-20T19:21:33.285716+0000","last_fullsized":"2024-09-20T19:21:33.285716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.918443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.918443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:22:26.624961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877906996,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1351680,"data_stored":950515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234047,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28336,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716816,"statfs":{"total":95995035648,"available":95966019584,"internally_reserved":0,"allocated":1339392,"data_stored":949851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561097,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":749568,"data_stored":360574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855445,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31928,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713224,"statfs":{"total":95995035648,"available":95962341376,"internally_reserved":0,"allocated":757760,"data_stored":359561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084383,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":360348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378733,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36088,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709064,"statfs":{"total":95995035648,"available":95958081536,"internally_reserved":0,"allocated":757760,"data_stored":360428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":770048,"data_stored":361055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36672,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708480,"statfs":{"total":95995035648,"available":95957483520,"internally_reserved":0,"allocated":1355776,"data_stored":950781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:21:46.084 INFO:tasks.thrashosds.thrasher:Setting 2.8 to [2, 3, 1] 2024-09-20T19:21:46.084 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.8', '2', '3', '1'] 2024-09-20T19:21:46.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 2.8 2 3 1 2024-09-20T19:21:46.147 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:46.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:46.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:46.398 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:46.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:46.432 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:46.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:46.563 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:46.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:46.693 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:46.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:46.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:46.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:46.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:46.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:46.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:46.937 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:46.937 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:46.937 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:46.937 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:46.937 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:46 smithi162 bash[24557]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:46.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:47.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:47.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[22706]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:47.105 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.723420+0000 mon.a (mon.0) 997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: audit 2024-09-20T19:21:45.733993+0000 mon.a (mon.0) 998 : audit [INF] from='client.24755 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.741745+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:47.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:46 smithi042 bash[32075]: cluster 2024-09-20T19:21:45.952052+0000 mgr.y (mgr.24491) 147 : cluster [DBG] pgmap v114: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:21:47.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:47.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:47.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:47.344 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:47.416 DEBUG:teuthology.orchestra.run.smithi042:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-20T19:21:47.438 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight read -> 100 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight write -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight delete -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight snap_create -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight snap_remove -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight rollback -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight copy_from -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:adding op weight write_excl -> 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:ceph version 19.3.0-5099-g88deadb8 (88deadb8082f4a33832e228ccefc59cedba158a5) squid (dev) 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:Configuration: 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout: Number of operations: 4000 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout: Number of objects: 50 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout: Max in flight operations: 16 2024-09-20T19:21:47.439 INFO:tasks.rados.rados.0.smithi042.stdout: Object size (in bytes): 4000000 2024-09-20T19:21:47.440 INFO:tasks.rados.rados.0.smithi042.stdout: Write stride min: 400000 2024-09-20T19:21:47.440 INFO:tasks.rados.rados.0.smithi042.stdout: Write stride max: 800000 2024-09-20T19:21:47.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:47.475 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:47.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:47.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:47.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:47.730 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:47.748 INFO:tasks.rados.rados.0.smithi042.stdout:1: write initial oid 1 2024-09-20T19:21:47.748 INFO:tasks.rados.rados.0.smithi042.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-20T19:21:47.757 INFO:tasks.rados.rados.0.smithi042.stdout:1: writing smithi04268048-1 from 416807 to 892056 tid 1 2024-09-20T19:21:47.766 INFO:tasks.rados.rados.0.smithi042.stdout:1: writing smithi04268048-1 from 1542129 to 2016807 tid 2 2024-09-20T19:21:47.766 INFO:tasks.rados.rados.0.smithi042.stdout:2: write initial oid 2 2024-09-20T19:21:47.766 INFO:tasks.rados.rados.0.smithi042.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-20T19:21:47.773 INFO:tasks.rados.rados.0.smithi042.stdout:2: writing smithi04268048-2 from 416807 to 892056 tid 1 2024-09-20T19:21:47.782 INFO:tasks.rados.rados.0.smithi042.stdout:2: writing smithi04268048-2 from 1542129 to 2016807 tid 2 2024-09-20T19:21:47.782 INFO:tasks.rados.rados.0.smithi042.stdout:3: write initial oid 3 2024-09-20T19:21:47.782 INFO:tasks.rados.rados.0.smithi042.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-20T19:21:47.789 INFO:tasks.rados.rados.0.smithi042.stdout:3: writing smithi04268048-3 from 433614 to 984112 tid 1 2024-09-20T19:21:47.798 INFO:tasks.rados.rados.0.smithi042.stdout:3: writing smithi04268048-3 from 1600611 to 2033614 tid 2 2024-09-20T19:21:47.798 INFO:tasks.rados.rados.0.smithi042.stdout:4: write initial oid 4 2024-09-20T19:21:47.798 INFO:tasks.rados.rados.0.smithi042.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-20T19:21:47.806 INFO:tasks.rados.rados.0.smithi042.stdout:4: writing smithi04268048-4 from 450421 to 1076168 tid 1 2024-09-20T19:21:47.813 INFO:tasks.rados.rados.0.smithi042.stdout:4: writing smithi04268048-4 from 1659093 to 2050421 tid 2 2024-09-20T19:21:47.813 INFO:tasks.rados.rados.0.smithi042.stdout:5: write initial oid 5 2024-09-20T19:21:47.814 INFO:tasks.rados.rados.0.smithi042.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-20T19:21:47.821 INFO:tasks.rados.rados.0.smithi042.stdout:5: writing smithi04268048-5 from 467228 to 1168224 tid 1 2024-09-20T19:21:47.827 INFO:tasks.rados.rados.0.smithi042.stdout:5: writing smithi04268048-5 from 1717575 to 2067228 tid 2 2024-09-20T19:21:47.828 INFO:tasks.rados.rados.0.smithi042.stdout:6: write initial oid 6 2024-09-20T19:21:47.828 INFO:tasks.rados.rados.0.smithi042.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-20T19:21:47.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:47.836 INFO:tasks.rados.rados.0.smithi042.stdout:6: writing smithi04268048-6 from 484035 to 1260280 tid 1 2024-09-20T19:21:47.840 INFO:tasks.rados.rados.0.smithi042.stdout:6: writing smithi04268048-6 from 2059704 to 2084035 tid 2 2024-09-20T19:21:47.840 INFO:tasks.rados.rados.0.smithi042.stdout:7: write initial oid 7 2024-09-20T19:21:47.840 INFO:tasks.rados.rados.0.smithi042.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-20T19:21:47.845 INFO:tasks.rados.rados.0.smithi042.stdout:7: writing smithi04268048-7 from 500842 to 952336 tid 1 2024-09-20T19:21:47.852 INFO:tasks.rados.rados.0.smithi042.stdout:7: writing smithi04268048-7 from 1718186 to 2100842 tid 2 2024-09-20T19:21:47.852 INFO:tasks.rados.rados.0.smithi042.stdout:8: write initial oid 8 2024-09-20T19:21:47.852 INFO:tasks.rados.rados.0.smithi042.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-20T19:21:47.858 INFO:tasks.rados.rados.0.smithi042.stdout:8: writing smithi04268048-8 from 517649 to 1044392 tid 1 2024-09-20T19:21:47.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:47.865 INFO:tasks.rados.rados.0.smithi042.stdout:8: writing smithi04268048-8 from 1776668 to 2117649 tid 2 2024-09-20T19:21:47.865 INFO:tasks.rados.rados.0.smithi042.stdout:9: write initial oid 9 2024-09-20T19:21:47.865 INFO:tasks.rados.rados.0.smithi042.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-20T19:21:47.872 INFO:tasks.rados.rados.0.smithi042.stdout:9: writing smithi04268048-9 from 534456 to 1252801 tid 1 2024-09-20T19:21:47.877 INFO:tasks.rados.rados.0.smithi042.stdout:9: writing smithi04268048-9 from 1951503 to 2134456 tid 2 2024-09-20T19:21:47.877 INFO:tasks.rados.rados.0.smithi042.stdout:10: write initial oid 10 2024-09-20T19:21:47.877 INFO:tasks.rados.rados.0.smithi042.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-20T19:21:47.884 INFO:tasks.rados.rados.0.smithi042.stdout:10: writing smithi04268048-10 from 551263 to 1344857 tid 1 2024-09-20T19:21:47.890 INFO:tasks.rados.rados.0.smithi042.stdout:10: writing smithi04268048-10 from 1893632 to 2151263 tid 2 2024-09-20T19:21:47.890 INFO:tasks.rados.rados.0.smithi042.stdout:11: write initial oid 11 2024-09-20T19:21:47.890 INFO:tasks.rados.rados.0.smithi042.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-20T19:21:47.895 INFO:tasks.rados.rados.0.smithi042.stdout:11: writing smithi04268048-11 from 568070 to 1036913 tid 1 2024-09-20T19:21:47.901 INFO:tasks.rados.rados.0.smithi042.stdout:11: writing smithi04268048-11 from 1552114 to 2168070 tid 2 2024-09-20T19:21:47.901 INFO:tasks.rados.rados.0.smithi042.stdout:12: write initial oid 12 2024-09-20T19:21:47.902 INFO:tasks.rados.rados.0.smithi042.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-20T19:21:47.908 INFO:tasks.rados.rados.0.smithi042.stdout:12: writing smithi04268048-12 from 584877 to 1128969 tid 1 2024-09-20T19:21:47.914 INFO:tasks.rados.rados.0.smithi042.stdout:12: writing smithi04268048-12 from 1610596 to 2172599 tid 2 2024-09-20T19:21:47.914 INFO:tasks.rados.rados.0.smithi042.stdout:13: write initial oid 13 2024-09-20T19:21:47.914 INFO:tasks.rados.rados.0.smithi042.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-20T19:21:47.921 INFO:tasks.rados.rados.0.smithi042.stdout:13: writing smithi04268048-13 from 601684 to 1221025 tid 1 2024-09-20T19:21:47.926 INFO:tasks.rados.rados.0.smithi042.stdout:13: writing smithi04268048-13 from 1669078 to 2201684 tid 2 2024-09-20T19:21:47.927 INFO:tasks.rados.rados.0.smithi042.stdout:14: write initial oid 14 2024-09-20T19:21:47.927 INFO:tasks.rados.rados.0.smithi042.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-20T19:21:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:47 smithi042 bash[22706]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:47 smithi042 bash[22706]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:47 smithi042 bash[32075]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:47 smithi042 bash[32075]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.934 INFO:tasks.rados.rados.0.smithi042.stdout:14: writing smithi04268048-14 from 618491 to 1313081 tid 1 2024-09-20T19:21:47.939 INFO:tasks.rados.rados.0.smithi042.stdout:14: writing smithi04268048-14 from 2011207 to 2218491 tid 2 2024-09-20T19:21:47.939 INFO:tasks.rados.rados.0.smithi042.stdout:15: write initial oid 15 2024-09-20T19:21:47.939 INFO:tasks.rados.rados.0.smithi042.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-20T19:21:47.947 INFO:tasks.rados.rados.0.smithi042.stdout:15: writing smithi04268048-15 from 635298 to 1405137 tid 1 2024-09-20T19:21:47.952 INFO:tasks.rados.rados.0.smithi042.stdout:15: writing smithi04268048-15 from 2069689 to 2235298 tid 2 2024-09-20T19:21:47.952 INFO:tasks.rados.rados.0.smithi042.stdout:16: write initial oid 16 2024-09-20T19:21:47.952 INFO:tasks.rados.rados.0.smithi042.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-20T19:21:47.958 INFO:tasks.rados.rados.0.smithi042.stdout:16: writing smithi04268048-16 from 652105 to 1097193 tid 1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:16: writing smithi04268048-16 from 1728171 to 2181159 tid 2 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: finishing write tid 1 to smithi04268048-1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: finishing write tid 2 to smithi04268048-1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: finishing write tid 3 to smithi04268048-1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: finishing write tid 4 to smithi04268048-1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: oid 1 updating version 0 to 3 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: oid 1 version 3 is already newer than 1 2024-09-20T19:21:47.966 INFO:tasks.rados.rados.0.smithi042.stdout:1: oid 1 version 3 is already newer than 2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: finishing write tid 1 to smithi04268048-2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: finishing write tid 2 to smithi04268048-2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: finishing write tid 3 to smithi04268048-2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: finishing write tid 4 to smithi04268048-2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: oid 2 updating version 0 to 3 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: oid 2 version 3 is already newer than 1 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: oid 2 version 3 is already newer than 2 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:3: finishing write tid 1 to smithi04268048-3 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:3: finishing write tid 2 to smithi04268048-3 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:3: finishing write tid 3 to smithi04268048-3 2024-09-20T19:21:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:3: finishing write tid 4 to smithi04268048-3 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:3: oid 3 updating version 0 to 2 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:3: oid 3 version 2 is already newer than 1 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:3: oid 3 updating version 2 to 3 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: finishing write tid 1 to smithi04268048-4 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: finishing write tid 2 to smithi04268048-4 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: finishing write tid 3 to smithi04268048-4 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: finishing write tid 4 to smithi04268048-4 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: oid 4 updating version 0 to 1 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: oid 4 updating version 1 to 3 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: oid 4 version 3 is already newer than 2 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:21:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:5: finishing write tid 1 to smithi04268048-5 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: finishing write tid 2 to smithi04268048-5 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: finishing write tid 3 to smithi04268048-5 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: finishing write tid 4 to smithi04268048-5 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: oid 5 updating version 0 to 1 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: oid 5 updating version 1 to 2 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: oid 5 updating version 2 to 3 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: finishing write tid 1 to smithi04268048-6 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: finishing write tid 2 to smithi04268048-6 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: finishing write tid 3 to smithi04268048-6 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: finishing write tid 4 to smithi04268048-6 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: oid 6 updating version 0 to 5 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: oid 6 version 5 is already newer than 4 2024-09-20T19:21:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:6: oid 6 updating version 5 to 6 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:1: done (15 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2: done (14 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:3: done (13 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:4: done (12 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:5: done (11 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:6: done (10 left) 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:17: write initial oid 17 2024-09-20T19:21:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-20T19:21:47.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:47.972 INFO:tasks.rados.rados.0.smithi042.stdout:17: writing smithi04268048-17 from 668912 to 1305602 tid 1 2024-09-20T19:21:47.977 INFO:tasks.rados.rados.0.smithi042.stdout:17: writing smithi04268048-17 from 1903006 to 2268912 tid 2 2024-09-20T19:21:47.977 INFO:tasks.rados.rados.0.smithi042.stdout:18: write initial oid 18 2024-09-20T19:21:47.978 INFO:tasks.rados.rados.0.smithi042.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-20T19:21:47.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:47 smithi162 bash[24557]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:47 smithi162 bash[24557]: audit 2024-09-20T19:21:47.452102+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-20T19:21:47.985 INFO:tasks.rados.rados.0.smithi042.stdout:18: writing smithi04268048-18 from 685719 to 1397658 tid 1 2024-09-20T19:21:47.990 INFO:tasks.rados.rados.0.smithi042.stdout:18: writing smithi04268048-18 from 1845135 to 2285719 tid 2 2024-09-20T19:21:47.991 INFO:tasks.rados.rados.0.smithi042.stdout:19: write initial oid 19 2024-09-20T19:21:47.991 INFO:tasks.rados.rados.0.smithi042.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-20T19:21:47.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:47.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:47.999 INFO:tasks.rados.rados.0.smithi042.stdout:19: writing smithi04268048-19 from 702526 to 1489714 tid 1 2024-09-20T19:21:48.004 INFO:tasks.rados.rados.0.smithi042.stdout:19: writing smithi04268048-19 from 1903617 to 2302526 tid 2 2024-09-20T19:21:48.004 INFO:tasks.rados.rados.0.smithi042.stdout:20: write initial oid 20 2024-09-20T19:21:48.004 INFO:tasks.rados.rados.0.smithi042.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-20T19:21:48.010 INFO:tasks.rados.rados.0.smithi042.stdout:20: writing smithi04268048-20 from 719333 to 1181770 tid 1 2024-09-20T19:21:48.017 INFO:tasks.rados.rados.0.smithi042.stdout:20: writing smithi04268048-20 from 1962099 to 2319333 tid 2 2024-09-20T19:21:48.017 INFO:tasks.rados.rados.0.smithi042.stdout:21: write initial oid 21 2024-09-20T19:21:48.017 INFO:tasks.rados.rados.0.smithi042.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-20T19:21:48.024 INFO:tasks.rados.rados.0.smithi042.stdout:21: writing smithi04268048-21 from 736140 to 1273826 tid 1 2024-09-20T19:21:48.030 INFO:tasks.rados.rados.0.smithi042.stdout:21: writing smithi04268048-21 from 2020581 to 2336140 tid 2 2024-09-20T19:21:48.030 INFO:tasks.rados.rados.0.smithi042.stdout:22: write initial oid 22 2024-09-20T19:21:48.030 INFO:tasks.rados.rados.0.smithi042.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-20T19:21:48.038 INFO:tasks.rados.rados.0.smithi042.stdout:22: writing smithi04268048-22 from 752947 to 1365882 tid 1 2024-09-20T19:21:48.044 INFO:tasks.rados.rados.0.smithi042.stdout:22: writing smithi04268048-22 from 1962710 to 2352947 tid 2 2024-09-20T19:21:48.044 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.044 INFO:tasks.rados.rados.0.smithi042.stdout:7: finishing write tid 1 to smithi04268048-7 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: finishing write tid 2 to smithi04268048-7 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: finishing write tid 3 to smithi04268048-7 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: finishing write tid 4 to smithi04268048-7 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: oid 7 updating version 0 to 2 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: oid 7 version 2 is already newer than 1 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: oid 7 updating version 2 to 3 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: finishing write tid 1 to smithi04268048-8 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: finishing write tid 2 to smithi04268048-8 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: finishing write tid 3 to smithi04268048-8 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: finishing write tid 4 to smithi04268048-8 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: oid 8 updating version 0 to 7 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: oid 8 updating version 7 to 9 2024-09-20T19:21:48.045 INFO:tasks.rados.rados.0.smithi042.stdout:8: oid 8 version 9 is already newer than 8 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 9 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:7: done (15 left) 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:8: done (14 left) 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:23: write initial oid 23 2024-09-20T19:21:48.046 INFO:tasks.rados.rados.0.smithi042.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-20T19:21:48.052 INFO:tasks.rados.rados.0.smithi042.stdout:23: writing smithi04268048-23 from 769754 to 1457938 tid 1 2024-09-20T19:21:48.058 INFO:tasks.rados.rados.0.smithi042.stdout:23: writing smithi04268048-23 from 2021192 to 2369754 tid 2 2024-09-20T19:21:48.058 INFO:tasks.rados.rados.0.smithi042.stdout:24: write initial oid 24 2024-09-20T19:21:48.058 INFO:tasks.rados.rados.0.smithi042.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-20T19:21:48.065 INFO:tasks.rados.rados.0.smithi042.stdout:24: writing smithi04268048-24 from 786561 to 1266347 tid 1 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:24: writing smithi04268048-24 from 1796027 to 2263691 tid 2 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: finishing write tid 1 to smithi04268048-9 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: finishing write tid 2 to smithi04268048-9 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: finishing write tid 3 to smithi04268048-9 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: finishing write tid 4 to smithi04268048-9 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: oid 9 updating version 0 to 6 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: oid 9 version 6 is already newer than 5 2024-09-20T19:21:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:9: oid 9 version 6 is already newer than 4 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: finishing write tid 1 to smithi04268048-10 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: finishing write tid 2 to smithi04268048-10 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: finishing write tid 3 to smithi04268048-10 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: finishing write tid 4 to smithi04268048-10 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: oid 10 updating version 0 to 2 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: oid 10 version 2 is already newer than 1 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: oid 10 updating version 2 to 3 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:9: done (15 left) 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:10: done (14 left) 2024-09-20T19:21:48.072 INFO:tasks.rados.rados.0.smithi042.stdout:25: write initial oid 25 2024-09-20T19:21:48.073 INFO:tasks.rados.rados.0.smithi042.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-20T19:21:48.076 INFO:tasks.rados.rados.0.smithi042.stdout:25: writing smithi04268048-25 from 403368 to 958403 tid 1 2024-09-20T19:21:48.083 INFO:tasks.rados.rados.0.smithi042.stdout:25: writing smithi04268048-25 from 1454509 to 2182184 tid 2 2024-09-20T19:21:48.083 INFO:tasks.rados.rados.0.smithi042.stdout:26: write initial oid 26 2024-09-20T19:21:48.083 INFO:tasks.rados.rados.0.smithi042.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-20T19:21:48.089 INFO:tasks.rados.rados.0.smithi042.stdout:26: writing smithi04268048-26 from 420175 to 1050459 tid 1 2024-09-20T19:21:48.096 INFO:tasks.rados.rados.0.smithi042.stdout:26: writing smithi04268048-26 from 1796638 to 2267971 tid 2 2024-09-20T19:21:48.096 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.096 INFO:tasks.rados.rados.0.smithi042.stdout:11: finishing write tid 1 to smithi04268048-11 2024-09-20T19:21:48.096 INFO:tasks.rados.rados.0.smithi042.stdout:11: finishing write tid 2 to smithi04268048-11 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: finishing write tid 3 to smithi04268048-11 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: finishing write tid 4 to smithi04268048-11 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: oid 11 updating version 0 to 8 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: oid 11 version 8 is already newer than 7 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: oid 11 updating version 8 to 9 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: finishing write tid 1 to smithi04268048-12 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: finishing write tid 2 to smithi04268048-12 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: finishing write tid 3 to smithi04268048-12 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: finishing write tid 4 to smithi04268048-12 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: oid 12 updating version 0 to 3 2024-09-20T19:21:48.097 INFO:tasks.rados.rados.0.smithi042.stdout:12: oid 12 version 3 is already newer than 1 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:12: oid 12 version 3 is already newer than 2 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: finishing write tid 1 to smithi04268048-13 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: finishing write tid 2 to smithi04268048-13 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: finishing write tid 3 to smithi04268048-13 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: finishing write tid 4 to smithi04268048-13 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: oid 13 updating version 0 to 2 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: oid 13 version 2 is already newer than 1 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: oid 13 updating version 2 to 3 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:14: finishing write tid 1 to smithi04268048-14 2024-09-20T19:21:48.098 INFO:tasks.rados.rados.0.smithi042.stdout:14: finishing write tid 2 to smithi04268048-14 2024-09-20T19:21:48.099 INFO:tasks.rados.rados.0.smithi042.stdout:11: done (15 left) 2024-09-20T19:21:48.099 INFO:tasks.rados.rados.0.smithi042.stdout:12: done (14 left) 2024-09-20T19:21:48.099 INFO:tasks.rados.rados.0.smithi042.stdout:13: done (13 left) 2024-09-20T19:21:48.099 INFO:tasks.rados.rados.0.smithi042.stdout:27: write initial oid 27 2024-09-20T19:21:48.099 INFO:tasks.rados.rados.0.smithi042.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-20T19:21:48.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:48.103 INFO:tasks.rados.rados.0.smithi042.stdout:27: writing smithi04268048-27 from 436982 to 1142515 tid 1 2024-09-20T19:21:48.110 INFO:tasks.rados.rados.0.smithi042.stdout:27: writing smithi04268048-27 from 1855120 to 2436982 tid 2 2024-09-20T19:21:48.110 INFO:tasks.rados.rados.0.smithi042.stdout:28: write initial oid 28 2024-09-20T19:21:48.110 INFO:tasks.rados.rados.0.smithi042.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-20T19:21:48.117 INFO:tasks.rados.rados.0.smithi042.stdout:28: writing smithi04268048-28 from 453789 to 1234571 tid 1 2024-09-20T19:21:48.124 INFO:tasks.rados.rados.0.smithi042.stdout:28: writing smithi04268048-28 from 1913602 to 2388604 tid 2 2024-09-20T19:21:48.124 INFO:tasks.rados.rados.0.smithi042.stdout:29: write initial oid 29 2024-09-20T19:21:48.124 INFO:tasks.rados.rados.0.smithi042.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-20T19:21:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:48.129 INFO:tasks.rados.rados.0.smithi042.stdout:29: writing smithi04268048-29 from 470596 to 926627 tid 1 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:29: writing smithi04268048-29 from 1572084 to 2190744 tid 2 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: finishing write tid 3 to smithi04268048-14 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: finishing write tid 4 to smithi04268048-14 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: oid 14 updating version 0 to 5 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: oid 14 updating version 5 to 6 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: oid 14 version 6 is already newer than 4 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-20T19:21:48.137 INFO:tasks.rados.rados.0.smithi042.stdout:15: finishing write tid 1 to smithi04268048-15 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: finishing write tid 2 to smithi04268048-15 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: finishing write tid 3 to smithi04268048-15 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: finishing write tid 4 to smithi04268048-15 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: oid 15 updating version 0 to 3 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: oid 15 version 3 is already newer than 2 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: oid 15 version 3 is already newer than 1 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:16: finishing write tid 1 to smithi04268048-16 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:14: done (15 left) 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:15: done (14 left) 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:30: write initial oid 30 2024-09-20T19:21:48.138 INFO:tasks.rados.rados.0.smithi042.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-20T19:21:48.142 INFO:tasks.rados.rados.0.smithi042.stdout:30: writing smithi04268048-30 from 487403 to 1018683 tid 1 2024-09-20T19:21:48.148 INFO:tasks.rados.rados.0.smithi042.stdout:30: writing smithi04268048-30 from 1514213 to 1992884 tid 2 2024-09-20T19:21:48.148 INFO:tasks.rados.rados.0.smithi042.stdout:31: write initial oid 31 2024-09-20T19:21:48.148 INFO:tasks.rados.rados.0.smithi042.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-20T19:21:48.154 INFO:tasks.rados.rados.0.smithi042.stdout:31: writing smithi04268048-31 from 504210 to 1110739 tid 1 2024-09-20T19:21:48.161 INFO:tasks.rados.rados.0.smithi042.stdout:31: writing smithi04268048-31 from 1572695 to 2195024 tid 2 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: finishing write tid 2 to smithi04268048-16 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: finishing write tid 3 to smithi04268048-16 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: finishing write tid 4 to smithi04268048-16 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: oid 16 updating version 0 to 6 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: oid 16 version 6 is already newer than 4 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: oid 16 version 6 is already newer than 5 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:17: finishing write tid 1 to smithi04268048-17 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:17: finishing write tid 2 to smithi04268048-17 2024-09-20T19:21:48.162 INFO:tasks.rados.rados.0.smithi042.stdout:17: finishing write tid 3 to smithi04268048-17 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:17: finishing write tid 4 to smithi04268048-17 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:17: oid 17 updating version 0 to 6 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:17: oid 17 version 6 is already newer than 4 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:17: oid 17 version 6 is already newer than 5 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: finishing write tid 1 to smithi04268048-18 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: finishing write tid 2 to smithi04268048-18 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: finishing write tid 3 to smithi04268048-18 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: finishing write tid 4 to smithi04268048-18 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: oid 18 updating version 0 to 2 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: oid 18 version 2 is already newer than 1 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:18: oid 18 updating version 2 to 3 2024-09-20T19:21:48.163 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: finishing write tid 1 to smithi04268048-19 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: finishing write tid 2 to smithi04268048-19 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: finishing write tid 3 to smithi04268048-19 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: finishing write tid 4 to smithi04268048-19 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: oid 19 updating version 0 to 3 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: oid 19 version 3 is already newer than 1 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: oid 19 version 3 is already newer than 2 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:20: finishing write tid 1 to smithi04268048-20 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:20: finishing write tid 2 to smithi04268048-20 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:20: finishing write tid 3 to smithi04268048-20 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:16: done (15 left) 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:17: done (14 left) 2024-09-20T19:21:48.164 INFO:tasks.rados.rados.0.smithi042.stdout:18: done (13 left) 2024-09-20T19:21:48.165 INFO:tasks.rados.rados.0.smithi042.stdout:19: done (12 left) 2024-09-20T19:21:48.165 INFO:tasks.rados.rados.0.smithi042.stdout:32: write initial oid 32 2024-09-20T19:21:48.165 INFO:tasks.rados.rados.0.smithi042.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-20T19:21:48.168 INFO:tasks.rados.rados.0.smithi042.stdout:32: writing smithi04268048-32 from 521017 to 1319148 tid 1 2024-09-20T19:21:48.173 INFO:tasks.rados.rados.0.smithi042.stdout:32: writing smithi04268048-32 from 1747530 to 2229870 tid 2 2024-09-20T19:21:48.173 INFO:tasks.rados.rados.0.smithi042.stdout:33: write initial oid 33 2024-09-20T19:21:48.173 INFO:tasks.rados.rados.0.smithi042.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-20T19:21:48.179 INFO:tasks.rados.rados.0.smithi042.stdout:33: writing smithi04268048-33 from 537824 to 1011204 tid 1 2024-09-20T19:21:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:33: writing smithi04268048-33 from 1806012 to 2432010 tid 2 2024-09-20T19:21:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:34: write initial oid 34 2024-09-20T19:21:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-20T19:21:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:34: writing smithi04268048-34 from 554631 to 1103260 tid 1 2024-09-20T19:21:48.200 INFO:tasks.rados.rados.0.smithi042.stdout:34: writing smithi04268048-34 from 1748141 to 2234150 tid 2 2024-09-20T19:21:48.200 INFO:tasks.rados.rados.0.smithi042.stdout:35: write initial oid 35 2024-09-20T19:21:48.200 INFO:tasks.rados.rados.0.smithi042.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-20T19:21:48.207 INFO:tasks.rados.rados.0.smithi042.stdout:35: writing smithi04268048-35 from 571438 to 1195316 tid 1 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:35: writing smithi04268048-35 from 1806623 to 2436290 tid 2 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:20: finishing write tid 4 to smithi04268048-20 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:20: oid 20 updating version 0 to 2 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:20: oid 20 version 2 is already newer than 1 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:20: oid 20 updating version 2 to 3 2024-09-20T19:21:48.215 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: finishing write tid 1 to smithi04268048-21 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: finishing write tid 2 to smithi04268048-21 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: finishing write tid 3 to smithi04268048-21 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: finishing write tid 4 to smithi04268048-21 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: oid 21 updating version 0 to 5 2024-09-20T19:21:48.216 INFO:tasks.rados.rados.0.smithi042.stdout:21: oid 21 updating version 5 to 6 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:21: oid 21 version 6 is already newer than 4 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:22: finishing write tid 1 to smithi04268048-22 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:22: finishing write tid 2 to smithi04268048-22 2024-09-20T19:21:48.217 INFO:tasks.rados.rados.0.smithi042.stdout:22: finishing write tid 3 to smithi04268048-22 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:22: finishing write tid 4 to smithi04268048-22 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:22: oid 22 updating version 0 to 1 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:22: oid 22 updating version 1 to 3 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:22: oid 22 version 3 is already newer than 2 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:23: finishing write tid 1 to smithi04268048-23 2024-09-20T19:21:48.218 INFO:tasks.rados.rados.0.smithi042.stdout:23: finishing write tid 2 to smithi04268048-23 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: finishing write tid 3 to smithi04268048-23 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: finishing write tid 4 to smithi04268048-23 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: oid 23 updating version 0 to 9 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: oid 23 version 9 is already newer than 7 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: oid 23 version 9 is already newer than 8 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-20T19:21:48.219 INFO:tasks.rados.rados.0.smithi042.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: finishing write tid 1 to smithi04268048-24 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: finishing write tid 2 to smithi04268048-24 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: finishing write tid 3 to smithi04268048-24 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: finishing write tid 4 to smithi04268048-24 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: oid 24 updating version 0 to 9 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: oid 24 version 9 is already newer than 8 2024-09-20T19:21:48.220 INFO:tasks.rados.rados.0.smithi042.stdout:24: oid 24 version 9 is already newer than 7 2024-09-20T19:21:48.221 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-20T19:21:48.221 INFO:tasks.rados.rados.0.smithi042.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-20T19:21:48.221 INFO:tasks.rados.rados.0.smithi042.stdout:25: finishing write tid 1 to smithi04268048-25 2024-09-20T19:21:48.221 INFO:tasks.rados.rados.0.smithi042.stdout:25: finishing write tid 2 to smithi04268048-25 2024-09-20T19:21:48.221 INFO:tasks.rados.rados.0.smithi042.stdout:25: finishing write tid 3 to smithi04268048-25 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:25: finishing write tid 4 to smithi04268048-25 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:25: oid 25 updating version 0 to 8 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:25: oid 25 updating version 8 to 9 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:25: oid 25 version 9 is already newer than 7 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-20T19:21:48.222 INFO:tasks.rados.rados.0.smithi042.stdout:26: finishing write tid 1 to smithi04268048-26 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: finishing write tid 2 to smithi04268048-26 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: finishing write tid 3 to smithi04268048-26 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: finishing write tid 4 to smithi04268048-26 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: oid 26 updating version 0 to 4 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: oid 26 updating version 4 to 6 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: oid 26 version 6 is already newer than 5 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:27: finishing write tid 1 to smithi04268048-27 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:27: finishing write tid 2 to smithi04268048-27 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:27: finishing write tid 3 to smithi04268048-27 2024-09-20T19:21:48.223 INFO:tasks.rados.rados.0.smithi042.stdout:27: finishing write tid 4 to smithi04268048-27 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:27: oid 27 updating version 0 to 11 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:27: oid 27 version 11 is already newer than 10 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:27: oid 27 updating version 11 to 12 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: finishing write tid 1 to smithi04268048-28 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: finishing write tid 2 to smithi04268048-28 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: finishing write tid 3 to smithi04268048-28 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: finishing write tid 4 to smithi04268048-28 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: oid 28 updating version 0 to 5 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: oid 28 version 5 is already newer than 4 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:28: oid 28 updating version 5 to 6 2024-09-20T19:21:48.224 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:29: finishing write tid 1 to smithi04268048-29 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:20: done (15 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:21: done (14 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:22: done (13 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:23: done (12 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:24: done (11 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:25: done (10 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:26: done (9 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:27: done (8 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:28: done (7 left) 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:36: write initial oid 36 2024-09-20T19:21:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-20T19:21:48.226 INFO:tasks.rados.rados.0.smithi042.stdout:36: writing smithi04268048-36 from 588245 to 1287372 tid 1 2024-09-20T19:21:48.227 INFO:tasks.rados.rados.0.smithi042.stdout:36: writing smithi04268048-36 from 1865105 to 2354783 tid 2 2024-09-20T19:21:48.228 INFO:tasks.rados.rados.0.smithi042.stdout:37: write initial oid 37 2024-09-20T19:21:48.228 INFO:tasks.rados.rados.0.smithi042.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-20T19:21:48.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:48.235 INFO:tasks.rados.rados.0.smithi042.stdout:37: writing smithi04268048-37 from 605052 to 1379428 tid 1 2024-09-20T19:21:48.242 INFO:tasks.rados.rados.0.smithi042.stdout:37: writing smithi04268048-37 from 1923587 to 2556923 tid 2 2024-09-20T19:21:48.242 INFO:tasks.rados.rados.0.smithi042.stdout:38: write initial oid 38 2024-09-20T19:21:48.242 INFO:tasks.rados.rados.0.smithi042.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-20T19:21:48.248 INFO:tasks.rados.rados.0.smithi042.stdout:38: writing smithi04268048-38 from 621859 to 1071484 tid 1 2024-09-20T19:21:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:48.257 INFO:tasks.rados.rados.0.smithi042.stdout:38: writing smithi04268048-38 from 1865716 to 2621859 tid 2 2024-09-20T19:21:48.257 INFO:tasks.rados.rados.0.smithi042.stdout:39: write initial oid 39 2024-09-20T19:21:48.257 INFO:tasks.rados.rados.0.smithi042.stdout:29: finishing write tid 2 to smithi04268048-29 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: finishing write tid 3 to smithi04268048-29 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: finishing write tid 4 to smithi04268048-29 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: oid 29 updating version 0 to 4 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: oid 29 updating version 4 to 5 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: oid 29 updating version 5 to 6 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-20T19:21:48.258 INFO:tasks.rados.rados.0.smithi042.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-20T19:21:48.259 INFO:tasks.rados.rados.0.smithi042.stdout:30: finishing write tid 1 to smithi04268048-30 2024-09-20T19:21:48.259 INFO:tasks.rados.rados.0.smithi042.stdout:30: finishing write tid 2 to smithi04268048-30 2024-09-20T19:21:48.259 INFO:tasks.rados.rados.0.smithi042.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-20T19:21:48.263 INFO:tasks.rados.rados.0.smithi042.stdout:39: writing smithi04268048-39 from 638666 to 1163540 tid 1 2024-09-20T19:21:48.272 INFO:tasks.rados.rados.0.smithi042.stdout:39: writing smithi04268048-39 from 1924198 to 2561203 tid 2 2024-09-20T19:21:48.272 INFO:tasks.rados.rados.0.smithi042.stdout:29: done (10 left) 2024-09-20T19:21:48.272 INFO:tasks.rados.rados.0.smithi042.stdout:40: write initial oid 40 2024-09-20T19:21:48.272 INFO:tasks.rados.rados.0.smithi042.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-20T19:21:48.279 INFO:tasks.rados.rados.0.smithi042.stdout:40: writing smithi04268048-40 from 655473 to 1371949 tid 1 2024-09-20T19:21:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:40: writing smithi04268048-40 from 2099033 to 2655473 tid 2 2024-09-20T19:21:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:41: write initial oid 41 2024-09-20T19:21:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-20T19:21:48.295 INFO:tasks.rados.rados.0.smithi042.stdout:41: writing smithi04268048-41 from 672280 to 1464005 tid 1 2024-09-20T19:21:48.302 INFO:tasks.rados.rados.0.smithi042.stdout:41: writing smithi04268048-41 from 2157515 to 2672280 tid 2 2024-09-20T19:21:48.302 INFO:tasks.rados.rados.0.smithi042.stdout:42: write initial oid 42 2024-09-20T19:21:48.302 INFO:tasks.rados.rados.0.smithi042.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-20T19:21:48.308 INFO:tasks.rados.rados.0.smithi042.stdout:42: writing smithi04268048-42 from 689087 to 1156061 tid 1 2024-09-20T19:21:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:42: writing smithi04268048-42 from 1699644 to 2483976 tid 2 2024-09-20T19:21:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:43: write initial oid 43 2024-09-20T19:21:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-20T19:21:48.323 INFO:tasks.rados.rados.0.smithi042.stdout:43: writing smithi04268048-43 from 705894 to 1248117 tid 1 2024-09-20T19:21:48.329 INFO:tasks.rados.rados.0.smithi042.stdout:43: writing smithi04268048-43 from 1758126 to 2402469 tid 2 2024-09-20T19:21:48.330 INFO:tasks.rados.rados.0.smithi042.stdout:44: write initial oid 44 2024-09-20T19:21:48.330 INFO:tasks.rados.rados.0.smithi042.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-20T19:21:48.337 INFO:tasks.rados.rados.0.smithi042.stdout:44: writing smithi04268048-44 from 722701 to 1340173 tid 1 2024-09-20T19:21:48.344 INFO:tasks.rados.rados.0.smithi042.stdout:44: writing smithi04268048-44 from 1816608 to 2604609 tid 2 2024-09-20T19:21:48.344 INFO:tasks.rados.rados.0.smithi042.stdout:45: write initial oid 45 2024-09-20T19:21:48.344 INFO:tasks.rados.rados.0.smithi042.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-20T19:21:48.352 INFO:tasks.rados.rados.0.smithi042.stdout:45: writing smithi04268048-45 from 739508 to 1432229 tid 1 2024-09-20T19:21:48.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:45: writing smithi04268048-45 from 1875090 to 2523102 tid 2 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: finishing write tid 3 to smithi04268048-30 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: finishing write tid 4 to smithi04268048-30 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: oid 30 updating version 0 to 10 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: oid 30 updating version 10 to 12 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: oid 30 version 12 is already newer than 11 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-20T19:21:48.359 INFO:tasks.rados.rados.0.smithi042.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: finishing write tid 1 to smithi04268048-31 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: finishing write tid 2 to smithi04268048-31 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: finishing write tid 3 to smithi04268048-31 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: finishing write tid 4 to smithi04268048-31 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: oid 31 updating version 0 to 11 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: oid 31 version 11 is already newer than 10 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: oid 31 updating version 11 to 12 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:32: finishing write tid 1 to smithi04268048-32 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:32: finishing write tid 2 to smithi04268048-32 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:30: done (15 left) 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:31: done (14 left) 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:46: write initial oid 46 2024-09-20T19:21:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-20T19:21:48.367 INFO:tasks.rados.rados.0.smithi042.stdout:46: writing smithi04268048-46 from 756315 to 1524285 tid 1 2024-09-20T19:21:48.374 INFO:tasks.rados.rados.0.smithi042.stdout:46: writing smithi04268048-46 from 1933572 to 2725242 tid 2 2024-09-20T19:21:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:47: write initial oid 47 2024-09-20T19:21:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-20T19:21:48.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:48.383 INFO:tasks.rados.rados.0.smithi042.stdout:47: writing smithi04268048-47 from 773122 to 1332694 tid 1 2024-09-20T19:21:48.391 INFO:tasks.rados.rados.0.smithi042.stdout:47: writing smithi04268048-47 from 1992054 to 2643735 tid 2 2024-09-20T19:21:48.391 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.391 INFO:tasks.rados.rados.0.smithi042.stdout:32: finishing write tid 3 to smithi04268048-32 2024-09-20T19:21:48.391 INFO:tasks.rados.rados.0.smithi042.stdout:32: finishing write tid 4 to smithi04268048-32 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:32: oid 32 updating version 0 to 2 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:32: oid 32 updating version 2 to 3 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:32: oid 32 version 3 is already newer than 1 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:33: finishing write tid 1 to smithi04268048-33 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:32: done (15 left) 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:48: write initial oid 48 2024-09-20T19:21:48.392 INFO:tasks.rados.rados.0.smithi042.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-20T19:21:48.399 INFO:tasks.rados.rados.0.smithi042.stdout:48: writing smithi04268048-48 from 789929 to 1424750 tid 1 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:48: writing smithi04268048-48 from 2050536 to 2789929 tid 2 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: finishing write tid 2 to smithi04268048-33 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: finishing write tid 3 to smithi04268048-33 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: finishing write tid 4 to smithi04268048-33 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: oid 33 updating version 0 to 6 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: oid 33 version 6 is already newer than 4 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: oid 33 version 6 is already newer than 5 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-20T19:21:48.407 INFO:tasks.rados.rados.0.smithi042.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-20T19:21:48.408 INFO:tasks.rados.rados.0.smithi042.stdout:34: finishing write tid 1 to smithi04268048-34 2024-09-20T19:21:48.408 INFO:tasks.rados.rados.0.smithi042.stdout:34: finishing write tid 2 to smithi04268048-34 2024-09-20T19:21:48.408 INFO:tasks.rados.rados.0.smithi042.stdout:33: done (15 left) 2024-09-20T19:21:48.408 INFO:tasks.rados.rados.0.smithi042.stdout:49: write initial oid 49 2024-09-20T19:21:48.408 INFO:tasks.rados.rados.0.smithi042.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-20T19:21:48.413 INFO:tasks.rados.rados.0.smithi042.stdout:49: writing smithi04268048-49 from 406736 to 1116806 tid 1 2024-09-20T19:21:48.420 INFO:tasks.rados.rados.0.smithi042.stdout:49: writing smithi04268048-49 from 1709018 to 2364368 tid 2 2024-09-20T19:21:48.420 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: finishing write tid 3 to smithi04268048-34 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: finishing write tid 4 to smithi04268048-34 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: oid 34 updating version 0 to 1 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: oid 34 updating version 1 to 3 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: oid 34 version 3 is already newer than 2 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:35: finishing write tid 1 to smithi04268048-35 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:34: done (15 left) 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:50: write initial oid 50 2024-09-20T19:21:48.421 INFO:tasks.rados.rados.0.smithi042.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-20T19:21:48.427 INFO:tasks.rados.rados.0.smithi042.stdout:50: writing smithi04268048-50 from 423543 to 1208862 tid 1 2024-09-20T19:21:48.435 INFO:tasks.rados.rados.0.smithi042.stdout:50: writing smithi04268048-50 from 1767500 to 2566508 tid 2 2024-09-20T19:21:48.435 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: finishing write tid 2 to smithi04268048-35 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: finishing write tid 3 to smithi04268048-35 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: finishing write tid 4 to smithi04268048-35 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: oid 35 updating version 0 to 6 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: oid 35 version 6 is already newer than 5 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: oid 35 version 6 is already newer than 4 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:36: finishing write tid 1 to smithi04268048-36 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:36: finishing write tid 2 to smithi04268048-36 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:35: done (15 left) 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2024-09-20T19:21:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:36: finishing write tid 3 to smithi04268048-36 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: finishing write tid 4 to smithi04268048-36 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: oid 36 updating version 0 to 6 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: oid 36 version 6 is already newer than 5 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: oid 36 version 6 is already newer than 4 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:37: finishing write tid 1 to smithi04268048-37 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:36: done (15 left) 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:52: write oid 25 current snap is 0 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:37: finishing write tid 2 to smithi04268048-37 2024-09-20T19:21:48.437 INFO:tasks.rados.rados.0.smithi042.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-20T19:21:48.440 INFO:tasks.rados.rados.0.smithi042.stdout:52: writing smithi04268048-25 from 440350 to 900918 tid 1 2024-09-20T19:21:48.441 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:21:48.445 INFO:tasks.rados.rados.0.smithi042.stdout:52: writing smithi04268048-25 from 1309629 to 1852295 tid 2 2024-09-20T19:21:48.451 INFO:tasks.rados.rados.0.smithi042.stdout:52: writing smithi04268048-25 from 2323973 to 2840350 tid 3 2024-09-20T19:21:48.451 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.451 INFO:tasks.rados.rados.0.smithi042.stdout:37: finishing write tid 3 to smithi04268048-37 2024-09-20T19:21:48.451 INFO:tasks.rados.rados.0.smithi042.stdout:37: finishing write tid 4 to smithi04268048-37 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:37: oid 37 updating version 0 to 7 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:37: oid 37 updating version 7 to 9 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:37: oid 37 version 9 is already newer than 8 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:38: finishing write tid 1 to smithi04268048-38 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:37: done (15 left) 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:53: rollback oid 30 current snap is 0 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:53: done (15 left) 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:54: write (excl) oid 21 current snap is 0 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:38: finishing write tid 2 to smithi04268048-38 2024-09-20T19:21:48.452 INFO:tasks.rados.rados.0.smithi042.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-20T19:21:48.456 INFO:tasks.rados.rados.0.smithi042.stdout:54: writing smithi04268048-21 from 457157 to 992974 tid 1 2024-09-20T19:21:48.464 INFO:tasks.rados.rados.0.smithi042.stdout:54: writing smithi04268048-21 from 1768111 to 2170788 tid 2 2024-09-20T19:21:48.464 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:21:48.464 INFO:tasks.rados.rados.0.smithi042.stdout:38: finishing write tid 3 to smithi04268048-38 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: finishing write tid 4 to smithi04268048-38 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: oid 38 updating version 0 to 7 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: oid 38 updating version 7 to 9 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: oid 38 version 9 is already newer than 8 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:39: finishing write tid 1 to smithi04268048-39 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:38: done (15 left) 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:55: snap_create 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:39: finishing write tid 2 to smithi04268048-39 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:39: finishing write tid 3 to smithi04268048-39 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:39: finishing write tid 4 to smithi04268048-39 2024-09-20T19:21:48.465 INFO:tasks.rados.rados.0.smithi042.stdout:39: oid 39 updating version 0 to 8 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:39: oid 39 version 8 is already newer than 7 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:39: oid 39 updating version 8 to 9 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: finishing write tid 1 to smithi04268048-40 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: finishing write tid 2 to smithi04268048-40 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: finishing write tid 3 to smithi04268048-40 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: finishing write tid 4 to smithi04268048-40 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: oid 40 updating version 0 to 12 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: oid 40 version 12 is already newer than 10 2024-09-20T19:21:48.466 INFO:tasks.rados.rados.0.smithi042.stdout:40: oid 40 version 12 is already newer than 11 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: finishing write tid 1 to smithi04268048-41 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: finishing write tid 2 to smithi04268048-41 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: finishing write tid 3 to smithi04268048-41 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: finishing write tid 4 to smithi04268048-41 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: oid 41 updating version 0 to 11 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: oid 41 version 11 is already newer than 10 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: oid 41 updating version 11 to 12 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:42: finishing write tid 1 to smithi04268048-42 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:42: finishing write tid 2 to smithi04268048-42 2024-09-20T19:21:48.467 INFO:tasks.rados.rados.0.smithi042.stdout:42: finishing write tid 3 to smithi04268048-42 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:42: finishing write tid 4 to smithi04268048-42 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:42: oid 42 updating version 0 to 14 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:42: oid 42 version 14 is already newer than 13 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:42: oid 42 updating version 14 to 15 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 15 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: finishing write tid 1 to smithi04268048-43 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: finishing write tid 2 to smithi04268048-43 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: finishing write tid 3 to smithi04268048-43 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: finishing write tid 4 to smithi04268048-43 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: oid 43 updating version 0 to 5 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: oid 43 version 5 is already newer than 4 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:43: oid 43 updating version 5 to 6 2024-09-20T19:21:48.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: finishing write tid 1 to smithi04268048-44 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: finishing write tid 2 to smithi04268048-44 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: finishing write tid 3 to smithi04268048-44 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: finishing write tid 4 to smithi04268048-44 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: oid 44 updating version 0 to 5 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: oid 44 version 5 is already newer than 4 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: oid 44 updating version 5 to 6 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:45: finishing write tid 1 to smithi04268048-45 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:45: finishing write tid 2 to smithi04268048-45 2024-09-20T19:21:48.469 INFO:tasks.rados.rados.0.smithi042.stdout:45: finishing write tid 3 to smithi04268048-45 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:45: finishing write tid 4 to smithi04268048-45 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:45: oid 45 updating version 0 to 7 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:45: oid 45 updating version 7 to 9 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:45: oid 45 version 9 is already newer than 8 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:46: finishing write tid 1 to smithi04268048-46 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:46: finishing write tid 2 to smithi04268048-46 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:46: finishing write tid 3 to smithi04268048-46 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:46: finishing write tid 4 to smithi04268048-46 2024-09-20T19:21:48.470 INFO:tasks.rados.rados.0.smithi042.stdout:46: oid 46 updating version 0 to 12 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:46: oid 46 version 12 is already newer than 11 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:46: oid 46 version 12 is already newer than 10 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: finishing write tid 1 to smithi04268048-47 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: finishing write tid 2 to smithi04268048-47 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: finishing write tid 3 to smithi04268048-47 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: finishing write tid 4 to smithi04268048-47 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: oid 47 updating version 0 to 17 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: oid 47 updating version 17 to 18 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: oid 47 version 18 is already newer than 16 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 18 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:21:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: finishing write tid 1 to smithi04268048-48 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: finishing write tid 2 to smithi04268048-48 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: finishing write tid 3 to smithi04268048-48 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: finishing write tid 4 to smithi04268048-48 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: oid 48 updating version 0 to 13 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: oid 48 updating version 13 to 15 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: oid 48 version 15 is already newer than 14 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:49: finishing write tid 1 to smithi04268048-49 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:49: finishing write tid 2 to smithi04268048-49 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:49: finishing write tid 3 to smithi04268048-49 2024-09-20T19:21:48.472 INFO:tasks.rados.rados.0.smithi042.stdout:49: finishing write tid 4 to smithi04268048-49 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:49: oid 49 updating version 0 to 15 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:49: oid 49 version 15 is already newer than 14 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:49: oid 49 version 15 is already newer than 13 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: finishing write tid 1 to smithi04268048-50 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: finishing write tid 2 to smithi04268048-50 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: finishing write tid 3 to smithi04268048-50 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: finishing write tid 4 to smithi04268048-50 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: oid 50 updating version 0 to 20 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: oid 50 version 20 is already newer than 19 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:50: oid 50 updating version 20 to 21 2024-09-20T19:21:48.473 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: finishing write tid 1 to smithi04268048-25 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:51: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: finishing write tid 2 to smithi04268048-25 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:51: finishing copy_from to smithi04268048-2 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: finishing write tid 3 to smithi04268048-25 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: finishing write tid 4 to smithi04268048-25 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: finishing write tid 5 to smithi04268048-25 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: oid 25 updating version 0 to 17 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: oid 25 version 17 is already newer than 16 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: oid 25 updating version 17 to 18 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:52: oid 25 updating version 18 to 19 2024-09-20T19:21:48.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T19:21:48.475 INFO:tasks.rados.rados.0.smithi042.stdout:52: left oid 25 (ObjNum 50 snap 0 seq_num 50) 2024-09-20T19:21:48.475 INFO:tasks.rados.rados.0.smithi042.stdout:54: finishing write tid 1 to smithi04268048-21 2024-09-20T19:21:48.475 INFO:tasks.rados.rados.0.smithi042.stdout:54: finishing write tid 2 to smithi04268048-21 2024-09-20T19:21:48.475 INFO:tasks.rados.rados.0.smithi042.stdout:54: finishing write tid 3 to smithi04268048-21 2024-09-20T19:21:48.475 INFO:tasks.rados.rados.0.smithi042.stdout:54: finishing write tid 4 to smithi04268048-21 2024-09-20T19:21:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:54: oid 21 updating version 0 to 22 2024-09-20T19:21:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:54: oid 21 version 22 is already newer than 21 2024-09-20T19:21:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:54: oid 21 version 22 is already newer than 20 2024-09-20T19:21:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:21:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:54: left oid 21 (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:21:48.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:48.508 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:48.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:48.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:48.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:39: done (15 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:40: done (14 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:41: done (13 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:42: done (12 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:43: done (11 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:44: done (10 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:45: done (9 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:46: done (8 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:47: done (7 left) 2024-09-20T19:21:48.767 INFO:tasks.rados.rados.0.smithi042.stdout:48: done (6 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:49: done (5 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:50: done (4 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:51: done (3 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:52: done (2 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:54: done (1 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:55: done (0 left) 2024-09-20T19:21:48.768 INFO:tasks.rados.rados.0.smithi042.stdout:56: delete oid 34 current snap is 1 2024-09-20T19:21:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:48.774 INFO:tasks.rados.rados.0.smithi042.stdout:56: done (0 left) 2024-09-20T19:21:48.774 INFO:tasks.rados.rados.0.smithi042.stdout:57: snap_create 2024-09-20T19:21:48.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:48.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:48.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:49.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.004 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:48 smithi162 bash[24557]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[32075]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: audit 2024-09-20T19:21:47.748353+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? v1:172.21.15.42:0/1545680571' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: cluster 2024-09-20T19:21:47.755743+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-20T19:21:49.013 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.013 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:48 smithi042 bash[22706]: cluster 2024-09-20T19:21:47.952672+0000 mgr.y (mgr.24491) 148 : cluster [DBG] pgmap v116: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:21:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:49.051 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:49.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:21:49.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:49.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:49.260 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:49.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:49.507 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:49.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:49.766 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.768 INFO:tasks.rados.rados.0.smithi042.stdout:57: done (0 left) 2024-09-20T19:21:49.769 INFO:tasks.rados.rados.0.smithi042.stdout:58: write oid 33 current snap is 2 2024-09-20T19:21:49.769 INFO:tasks.rados.rados.0.smithi042.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-20T19:21:49.776 INFO:tasks.rados.rados.0.smithi042.stdout:58: writing smithi04268048-33 from 490771 to 1177086 tid 1 2024-09-20T19:21:49.785 INFO:tasks.rados.rados.0.smithi042.stdout:58: writing smithi04268048-33 from 1885075 to 2291421 tid 2 2024-09-20T19:21:49.785 INFO:tasks.rados.rados.0.smithi042.stdout:59: delete oid 3 current snap is 2 2024-09-20T19:21:49.790 INFO:tasks.rados.rados.0.smithi042.stdout:59: done (1 left) 2024-09-20T19:21:49.790 INFO:tasks.rados.rados.0.smithi042.stdout:60: write oid 12 current snap is 2 2024-09-20T19:21:49.790 INFO:tasks.rados.rados.0.smithi042.stdout:58: finishing write tid 1 to smithi04268048-33 2024-09-20T19:21:49.790 INFO:tasks.rados.rados.0.smithi042.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-20T19:21:49.794 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:49.797 INFO:tasks.rados.rados.0.smithi042.stdout:60: writing smithi04268048-12 from 524385 to 1077551 tid 1 2024-09-20T19:21:49.801 INFO:tasks.rados.rados.0.smithi042.stdout:60: writing smithi04268048-12 from 1602039 to 2012054 tid 2 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:60: writing smithi04268048-12 from 2577441 to 2924385 tid 3 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:61: rollback oid 47 current snap is 2 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 47 to 1 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:58: finishing write tid 2 to smithi04268048-33 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:58: finishing write tid 3 to smithi04268048-33 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:58: finishing write tid 4 to smithi04268048-33 2024-09-20T19:21:49.807 INFO:tasks.rados.rados.0.smithi042.stdout:58: oid 33 updating version 0 to 16 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:58: oid 33 updating version 16 to 17 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:58: oid 33 version 17 is already newer than 15 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 17 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:60: finishing write tid 1 to smithi04268048-12 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:60: finishing write tid 2 to smithi04268048-12 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:58: done (2 left) 2024-09-20T19:21:49.808 INFO:tasks.rados.rados.0.smithi042.stdout:62: snap_remove snap 0 2024-09-20T19:21:49.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:49.921 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:49.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:50.024 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.050 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:49 smithi162 bash[24557]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:49 smithi162 bash[24557]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:49 smithi162 bash[24557]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:49 smithi162 bash[24557]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:50.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:50.162 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[22706]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.162 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[22706]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[22706]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[22706]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[32075]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[32075]: cluster 2024-09-20T19:21:48.765695+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[32075]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.163 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:49 smithi042 bash[32075]: audit 2024-09-20T19:21:49.003384+0000 mgr.y (mgr.24491) 149 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:21:50.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:50.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.300 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:50.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:50.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:50.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:50.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:50.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:50.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:50.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.783 INFO:tasks.rados.rados.0.smithi042.stdout:62: done (2 left) 2024-09-20T19:21:50.784 INFO:tasks.rados.rados.0.smithi042.stdout:63: read oid 28 snap -1 2024-09-20T19:21:50.784 INFO:tasks.rados.rados.0.smithi042.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-20T19:21:50.784 INFO:tasks.rados.rados.0.smithi042.stdout:64: write oid 44 current snap is 2 2024-09-20T19:21:50.784 INFO:tasks.rados.rados.0.smithi042.stdout:64: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-20T19:21:50.791 INFO:tasks.rados.rados.0.smithi042.stdout:64: writing smithi04268048-44 from 541192 to 1169607 tid 1 2024-09-20T19:21:50.798 INFO:tasks.rados.rados.0.smithi042.stdout:64: writing smithi04268048-44 from 1660521 to 2214194 tid 2 2024-09-20T19:21:50.803 INFO:tasks.rados.rados.0.smithi042.stdout:64: writing smithi04268048-44 from 2888511 to 2941192 tid 3 2024-09-20T19:21:50.803 INFO:tasks.rados.rados.0.smithi042.stdout:65: write oid 36 current snap is 2 2024-09-20T19:21:50.803 INFO:tasks.rados.rados.0.smithi042.stdout:60: finishing write tid 3 to smithi04268048-12 2024-09-20T19:21:50.803 INFO:tasks.rados.rados.0.smithi042.stdout:60: finishing write tid 4 to smithi04268048-12 2024-09-20T19:21:50.803 INFO:tasks.rados.rados.0.smithi042.stdout:60: finishing write tid 5 to smithi04268048-12 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:60: oid 12 updating version 0 to 9 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:60: oid 12 updating version 9 to 10 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:60: oid 12 version 10 is already newer than 8 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:60: oid 12 updating version 10 to 11 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 11 (ObjNum 55 snap 2 seq_num 55) dirty exists 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:61: finishing rollback tid 0 to smithi04268048-47 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:61: finishing rollback tid 1 to smithi04268048-47 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:61: finishing rollback tid 2 to smithi04268048-47 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 25 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:64: finishing write tid 1 to smithi04268048-44 2024-09-20T19:21:50.804 INFO:tasks.rados.rados.0.smithi042.stdout:65: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-20T19:21:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:50.810 INFO:tasks.rados.rados.0.smithi042.stdout:65: writing smithi04268048-36 from 557999 to 1261663 tid 1 2024-09-20T19:21:50.815 INFO:tasks.rados.rados.0.smithi042.stdout:65: writing smithi04268048-36 from 1719003 to 2132687 tid 2 2024-09-20T19:21:50.820 INFO:tasks.rados.rados.0.smithi042.stdout:65: writing smithi04268048-36 from 2632287 to 2957999 tid 3 2024-09-20T19:21:50.820 INFO:tasks.rados.rados.0.smithi042.stdout:60: done (4 left) 2024-09-20T19:21:50.820 INFO:tasks.rados.rados.0.smithi042.stdout:61: done (3 left) 2024-09-20T19:21:50.820 INFO:tasks.rados.rados.0.smithi042.stdout:66: copy_from oid 35 from oid 21 current snap is 2 2024-09-20T19:21:50.820 INFO:tasks.rados.rados.0.smithi042.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-20T19:21:50.836 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 191707 2024-09-20T19:21:50.860 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: finishing write tid 2 to smithi04268048-44 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: finishing write tid 3 to smithi04268048-44 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: finishing write tid 4 to smithi04268048-44 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: finishing write tid 5 to smithi04268048-44 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: oid 44 updating version 0 to 17 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: oid 44 updating version 17 to 18 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: oid 44 updating version 18 to 19 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: oid 44 updating version 19 to 20 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 20 (ObjNum 56 snap 2 seq_num 56) dirty exists 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:64: left oid 44 (ObjNum 56 snap 2 seq_num 56) 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:65: finishing write tid 1 to smithi04268048-36 2024-09-20T19:21:50.887 INFO:tasks.rados.rados.0.smithi042.stdout:65: finishing write tid 2 to smithi04268048-36 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: finishing write tid 3 to smithi04268048-36 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: finishing write tid 4 to smithi04268048-36 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: finishing write tid 5 to smithi04268048-36 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: oid 36 updating version 0 to 8 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: oid 36 updating version 8 to 9 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: oid 36 updating version 9 to 10 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: oid 36 updating version 10 to 11 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 11 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: left oid 36 (ObjNum 57 snap 2 seq_num 57) 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:66: finishing copy_from racing read to smithi04268048-35 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:66: finishing copy_from to smithi04268048-35 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:63: done (3 left) 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:64: done (2 left) 2024-09-20T19:21:50.888 INFO:tasks.rados.rados.0.smithi042.stdout:65: done (1 left) 2024-09-20T19:21:50.889 INFO:tasks.rados.rados.0.smithi042.stdout:66: done (0 left) 2024-09-20T19:21:50.889 INFO:tasks.rados.rados.0.smithi042.stdout:67: snap_create 2024-09-20T19:21:50.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:50.920 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:50.920 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:50.921 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:50.921 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:50.921 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:50.921 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:50 smithi162 bash[24557]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:50.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:50.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:51.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:51.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:51.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[22706]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:51.177 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: cluster 2024-09-20T19:21:49.766547+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:21:51.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:51.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: cluster 2024-09-20T19:21:49.953366+0000 mgr.y (mgr.24491) 150 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:21:51.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:51.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:50 smithi042 bash[32075]: audit 2024-09-20T19:21:50.132234+0000 mon.a (mon.0) 1005 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:21:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:51.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:51.319 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:51.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:51.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:51.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:51.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:51.679 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:51.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:51.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:51.876 INFO:tasks.rados.rados.0.smithi042.stdout:67: done (0 left) 2024-09-20T19:21:51.876 INFO:tasks.rados.rados.0.smithi042.stdout:68: snap_remove snap 1 2024-09-20T19:21:51.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:51.951 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:51 smithi162 bash[24557]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:51.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:51.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[22706]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:52.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: cluster 2024-09-20T19:21:50.779617+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.865352+0000 mon.a (mon.0) 1007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.865992+0000 mon.a (mon.0) 1008 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:52.013 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:51 smithi042 bash[32075]: audit 2024-09-20T19:21:50.871818+0000 mon.a (mon.0) 1009 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:52.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:52.089 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:52.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94817798635392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:18:11.358660+0000", 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 220.83483434600001, 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.004151521, 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.196 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 57 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358660+0000", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358660+0000", 2024-09-20T19:21:52.197 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358663+0000", 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0760000000000001e-06 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358669+0000", 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0650000000000004e-06 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358679+0000", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0625000000000001e-05 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.358690+0000", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0319999999999999e-05 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.359291+0000", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00060188400000000001 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.359416+0000", 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000124091 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.360632+0000", 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012163510000000001 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.360717+0000", 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5084000000000001e-05 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.360851+0000", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000133993 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.361327+0000", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00047645400000000002 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.362324+0000", 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0009965830000000001 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.362552+0000", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022834099999999999 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.362642+0000", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9585000000000004e-05 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:11.362812+0000", 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017006899999999999 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.202 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.24418.0:1 3.1 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:18:14.350958+0000", 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 217.84253712899999, 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.008190559, 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.24418", 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.162:0/2294914623", 2024-09-20T19:21:52.203 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.350958+0000", 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.350958+0000", 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.350962+0000", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1239999999999998e-06 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.350976+0000", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3825999999999999e-05 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.351009+0000", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3064000000000001e-05 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.205 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.351035+0000", 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6543e-05 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.351170+0000", 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000134399 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.351179+0000", 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7569999999999999e-06 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.355372+0000", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0041926339999999998 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.355689+0000", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031677300000000001 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,4", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.355990+0000", 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00030097300000000001 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.356730+0000", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00074001100000000005 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.358040+0000", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001310667 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.359044+0000", 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010033150000000001 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.359099+0000", 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5198000000000002e-05 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.359148+0000", 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9274999999999998e-05 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14478.0:205 4.19 e76/71 4:9a6dd8c0:gc::gc.0:head v 76'1, mlcod=0'0)", 2024-09-20T19:21:52.209 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:18:14.352546+0000", 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 217.84094926099999, 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0040016060000000004, 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352546+0000", 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.210 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352546+0000", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352553+0000", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6899999999999992e-06 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352581+0000", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.741e-05 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352611+0000", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.027e-05 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.352631+0000", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9983000000000002e-05 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.353263+0000", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00063245900000000004 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.353282+0000", 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8479e-05 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.356494+0000", 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.003212103 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:14.356547+0000", 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3211999999999998e-05 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:1254 5.3 5:c609908c:::notify.5:head [watch ping cookie 94817798635392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:20:42.450408+0000", 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 69.743086977999994, 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0055368379999999997, 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1254 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.214 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450408+0000", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450408+0000", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450413+0000", 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2900000000000002e-06 2024-09-20T19:21:52.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450430+0000", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7142999999999998e-05 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450484+0000", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3560000000000002e-05 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.450508+0000", 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4046e-05 2024-09-20T19:21:52.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.451082+0000", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00057399700000000003 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.451100+0000", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7779e-05 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.455438+0000", 2024-09-20T19:21:52.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0043387030000000002 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.455651+0000", 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021310200000000001 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:20:42.455945+0000", 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029321799999999998 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:1326 5.3 5:c609908c:::notify.5:head [watch ping cookie 94817798635392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:17.474665+0000", 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 34.718829634000002, 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0048781220000000004, 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1326 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474665+0000", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474665+0000", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474670+0000", 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.0000000000000004e-06 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474686+0000", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.588e-05 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474737+0000", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.0816999999999998e-05 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.474772+0000", 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.5148999999999998e-05 2024-09-20T19:21:52.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.475002+0000", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022985599999999999 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.475019+0000", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7476e-05 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.479043+0000", 2024-09-20T19:21:52.222 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0040234809999999998 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.479258+0000", 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000215045 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:17.479543+0000", 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028541800000000001 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:13 7.5 e118/112 7:aa23a6e3:::smithi04268048-4:head v 118'1, mlcod=0'0)", 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.812831+0000", 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.3806635400000005, 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.004685463, 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.812831+0000", 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.812831+0000", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.812832+0000", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3399999999999999e-07 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.814321+0000", 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001488589 2024-09-20T19:21:52.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.814341+0000", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0571e-05 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.814352+0000", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0842000000000001e-05 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.814464+0000", 2024-09-20T19:21:52.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011209000000000001 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.814500+0000", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6050000000000002e-05 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.817413+0000", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0029132070000000001 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.817517+0000", 2024-09-20T19:21:52.227 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010348000000000001 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:17 7.f 7:f965d827:::smithi04268048-5:head [write 467228~700996,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.823319+0000", 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.370175422, 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0061177690000000003, 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.228 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 17 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.823319+0000", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.823319+0000", 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.823324+0000", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4800000000000003e-06 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.823976+0000", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00065182499999999997 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.824003+0000", 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7180000000000001e-05 2024-09-20T19:21:52.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.824017+0000", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4297000000000001e-05 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.824111+0000", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3664000000000003e-05 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.824271+0000", 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015989300000000001 2024-09-20T19:21:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.824442+0000", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017098300000000001 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.828421+0000", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039793579999999997 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.828513+0000", 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.2267000000000003e-05 2024-09-20T19:21:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829236+0000", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00072283999999999998 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829341+0000", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000105241 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829437+0000", 2024-09-20T19:21:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5741000000000004e-05 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:18 7.f 7:f965d827:::smithi04268048-5:head [write 1717575~349653] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.829022+0000", 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.3644724290000001, 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0038773319999999998, 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 18 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829022+0000", 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829022+0000", 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829026+0000", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4989999999999999e-06 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829484+0000", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00045766699999999998 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829498+0000", 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4154999999999999e-05 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829511+0000", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3551e-05 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829659+0000", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014794299999999999 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829758+0000", 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8814999999999995e-05 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829908+0000", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014957999999999999 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.831152+0000", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001244078 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.832059+0000", 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00090699400000000005 2024-09-20T19:21:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.832683+0000", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00062466599999999996 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.832814+0000", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013039200000000001 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.832900+0000", 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5991999999999999e-05 2024-09-20T19:21:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:20 7.f 7:f965d827:::smithi04268048-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.829671+0000", 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.3638235339999998, 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039402839999999996, 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 20 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829671+0000", 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829671+0000", 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829673+0000", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.423e-06 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829679+0000", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0000000000000002e-06 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829692+0000", 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3468e-05 2024-09-20T19:21:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.829707+0000", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5287000000000001e-05 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.830946+0000", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012380889999999999 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.831071+0000", 2024-09-20T19:21:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000125606 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.833429+0000", 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002358265 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.833504+0000", 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4525999999999998e-05 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.833612+0000", 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010762 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:28 7.6 7:6c48c6fe:::smithi04268048-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.853712+0000", 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.3397831199999999, 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0040131740000000004, 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.245 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 28 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853712+0000", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853712+0000", 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.246 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853712+0000", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7100000000000002e-07 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853717+0000", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5689999999999998e-06 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853724+0000", 2024-09-20T19:21:52.247 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.481e-06 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.853730+0000", 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2390000000000002e-06 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.856623+0000", 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002893058 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.248 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.856807+0000", 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018443499999999999 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.857518+0000", 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00071136699999999997 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.857606+0000", 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7341000000000003e-05 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.249 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.857725+0000", 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000119113 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:72 7.4 7:2a17ba7b:::smithi04268048-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:47.991961+0000", 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.2015342880000004, 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039183040000000001, 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.250 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 72 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991961+0000", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.251 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991961+0000", 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991962+0000", 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.19e-06 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991967+0000", 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9930000000000003e-06 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.252 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991989+0000", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2476e-05 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.991998+0000", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7839999999999992e-06 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.994408+0000", 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0024099270000000001 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.253 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.994613+0000", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00020507500000000001 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.995591+0000", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00097791700000000002 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.995707+0000", 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000116153 2024-09-20T19:21:52.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:47.995879+0000", 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017178899999999999 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:98 7.4 7:23ed6a02:::smithi04268048-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.084137+0000", 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.1093582670000002, 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045934030000000002, 2024-09-20T19:21:52.255 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 98 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084137+0000", 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084137+0000", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084140+0000", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4429999999999999e-06 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084550+0000", 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00040970399999999999 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084577+0000", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7526999999999999e-05 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084591+0000", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3883000000000001e-05 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084664+0000", 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3098999999999999e-05 2024-09-20T19:21:52.258 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084743+0000", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8500999999999999e-05 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084874+0000", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000131705 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.086599+0000", 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0017241380000000001 2024-09-20T19:21:52.259 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.087924+0000", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013250880000000001 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088550+0000", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00062618000000000005 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088661+0000", 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000111195 2024-09-20T19:21:52.260 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088730+0000", 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8940000000000001e-05 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:99 7.4 7:23ed6a02:::smithi04268048-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.084625+0000", 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.1088694199999996, 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0043446480000000004, 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.261 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 99 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084625+0000", 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.262 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084625+0000", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084627+0000", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.77e-06 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084634+0000", 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.2230000000000001e-06 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084645+0000", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0530000000000001e-05 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084653+0000", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.4759999999999998e-06 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.085595+0000", 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00094172400000000001 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.264 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.085752+0000", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000156379 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.085988+0000", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023694499999999999 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.086645+0000", 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000656929 2024-09-20T19:21:52.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088015+0000", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013698110000000001 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088814+0000", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00079915200000000002 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088892+0000", 2024-09-20T19:21:52.266 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.7905999999999996e-05 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.088970+0000", 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.7803e-05 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:100 7.4 7:23ed6a02:::smithi04268048-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.084675+0000", 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.1088201639999999, 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0046528860000000002, 2024-09-20T19:21:52.267 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 100 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084675+0000", 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084675+0000", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084676+0000", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0300000000000001e-06 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084680+0000", 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9400000000000004e-06 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084689+0000", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4630000000000007e-06 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.084697+0000", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8129999999999996e-06 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.086464+0000", 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001767455 2024-09-20T19:21:52.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.086578+0000", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011325 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.089074+0000", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0024964800000000001 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.089187+0000", 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000112799 2024-09-20T19:21:52.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.089328+0000", 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014065599999999999 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:101 7.6 7:6dcf8a02:::smithi04268048-26:head [write 420175~630284,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.090230+0000", 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.1032651700000002, 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039323889999999997, 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.272 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 101 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090230+0000", 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.273 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090230+0000", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090233+0000", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9000000000000002e-06 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090569+0000", 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00033619999999999999 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.274 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090596+0000", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6947000000000001e-05 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090616+0000", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9896000000000001e-05 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090768+0000", 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015192299999999999 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.090999+0000", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023138499999999999 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.091232+0000", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023346799999999999 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.093042+0000", 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0018095780000000001 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.093897+0000", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00085490100000000003 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.093992+0000", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4685999999999994e-05 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.094098+0000", 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010605 2024-09-20T19:21:52.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.094162+0000", 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4455000000000005e-05 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:104 7.6 7:6dcf8a02:::smithi04268048-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.097380+0000", 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 4.0961150079999999, 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0040676100000000001, 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.278 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 104 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097380+0000", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.279 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097380+0000", 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097380+0000", 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.5799999999999999e-07 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097383+0000", 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6989999999999999e-06 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.280 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097391+0000", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4390000000000003e-06 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.097398+0000", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.858e-06 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.099617+0000", 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002218678 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.281 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.099850+0000", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000232675 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.100977+0000", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001127156 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.101186+0000", 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000208552 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.282 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.101447+0000", 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000261895 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:141 7.9 7:9b44618a:::smithi04268048-36:head [write 588245~699127,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.222683+0000", 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.9708122980000002, 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0042460880000000003, 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.283 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 141 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.222683+0000", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.284 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.222683+0000", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.222685+0000", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7080000000000002e-06 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223041+0000", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00035566500000000002 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.285 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223060+0000", 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8604e-05 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223073+0000", 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3838e-05 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223225+0000", 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015195700000000001 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.286 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223487+0000", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026181899999999999 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.223752+0000", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026515799999999998 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.226138+0000", 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002385407 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.287 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.226166+0000", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8314e-05 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.226752+0000", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00058565199999999998 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.226849+0000", 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.6996000000000004e-05 2024-09-20T19:21:52.288 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.226929+0000", 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9969999999999995e-05 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:145 7.f 7:f6310c05:::smithi04268048-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.236467+0000", 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.9570276770000001, 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0041973979999999998, 2024-09-20T19:21:52.289 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 145 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236467+0000", 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236467+0000", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236469+0000", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5460000000000001e-06 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236676+0000", 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000207584 2024-09-20T19:21:52.291 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236689+0000", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2458e-05 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236697+0000", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5620000000000004e-06 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236776+0000", 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8645000000000003e-05 2024-09-20T19:21:52.292 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236875+0000", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8772999999999998e-05 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.236973+0000", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8284999999999998e-05 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.238580+0000", 2024-09-20T19:21:52.293 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016067950000000001 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.238860+0000", 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028057700000000001 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.240537+0000", 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001677003 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.294 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.240601+0000", 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3905999999999994e-05 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.240665+0000", 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3263999999999996e-05 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:204 7.4 7:23ed6a02:::smithi04268048-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.446375+0000", 2024-09-20T19:21:52.295 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.7471200499999999, 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039942700000000003, 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 204 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446375+0000", 2024-09-20T19:21:52.296 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446375+0000", 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446376+0000", 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6840000000000001e-06 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.297 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446587+0000", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000210316 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446606+0000", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8890999999999999e-05 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446614+0000", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6230000000000003e-06 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.298 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446673+0000", 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8424999999999997e-05 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446766+0000", 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3152000000000005e-05 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446913+0000", 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000146694 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.299 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.448547+0000", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016345960000000001 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.448950+0000", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00040276999999999999 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450249+0000", 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012992170000000001 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.300 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450317+0000", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7770999999999999e-05 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450369+0000", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2130999999999999e-05 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:201 7.7 e118/112 7:e8e5d7d7:::smithi04268048-2:head v 118'13, mlcod=118'11)", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:48.446515+0000", 2024-09-20T19:21:52.301 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.746980062, 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070655220000000003, 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446515+0000", 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446515+0000", 2024-09-20T19:21:52.302 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.446516+0000", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2860000000000001e-06 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450081+0000", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0035647840000000001 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450096+0000", 2024-09-20T19:21:52.303 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.472e-05 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450104+0000", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3270000000000001e-06 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450166+0000", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1943e-05 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.450188+0000", 2024-09-20T19:21:52.304 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2558e-05 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.453530+0000", 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0033418789999999999 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:48.453580+0000", 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.0025000000000003e-05 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:21:52.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:21:52.306 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:21:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:52.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:52.408 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:52.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:52.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:52.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:52.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:52.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:52.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:52.874 INFO:teuthology.orchestra.run.smithi042.stderr:set 2.8 pg_upmap mapping to [2,3,1] 2024-09-20T19:21:52.882 INFO:tasks.rados.rados.0.smithi042.stdout:68: done (0 left) 2024-09-20T19:21:52.882 INFO:tasks.rados.rados.0.smithi042.stdout:69: read oid 23 snap -1 2024-09-20T19:21:52.882 INFO:tasks.rados.rados.0.smithi042.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T19:21:52.882 INFO:tasks.rados.rados.0.smithi042.stdout:70: snap_remove snap 2 2024-09-20T19:21:52.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[32075]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:52.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:52.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:52.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:52.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:52 smithi042 bash[22706]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:52.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:52.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:52.992 INFO:tasks.rados.rados.0.smithi042.stdout:70: done (1 left) 2024-09-20T19:21:52.992 INFO:tasks.rados.rados.0.smithi042.stdout:71: read oid 18 snap -1 2024-09-20T19:21:52.992 INFO:tasks.rados.rados.0.smithi042.stdout:71: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-20T19:21:52.993 INFO:tasks.rados.rados.0.smithi042.stdout:72: read oid 39 snap -1 2024-09-20T19:21:52.993 INFO:tasks.rados.rados.0.smithi042.stdout:72: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T19:21:52.993 INFO:tasks.rados.rados.0.smithi042.stdout:73: snap_create 2024-09-20T19:21:52.993 INFO:tasks.rados.rados.0.smithi042.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T19:21:53.007 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2332468 2024-09-20T19:21:53.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:53.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:53.074 INFO:tasks.rados.rados.0.smithi042.stdout:71: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-20T19:21:53.074 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:21:53.127 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1230474 2024-09-20T19:21:53.138 INFO:tasks.rados.rados.0.smithi042.stdout:72: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T19:21:53.155 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:21:53.170 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:53.183 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:21:53.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:53.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:53.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: cluster 2024-09-20T19:21:51.876186+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-20T19:21:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: cluster 2024-09-20T19:21:51.954726+0000 mgr.y (mgr.24491) 151 : cluster [DBG] pgmap v122: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:52 smithi162 bash[24557]: audit 2024-09-20T19:21:52.695825+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]} : dispatch 2024-09-20T19:21:53.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:53.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:53.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:53.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:53.585 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:53.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:53.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:53.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:53.847 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:53.950 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:53.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:53.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:53.996 INFO:tasks.rados.rados.0.smithi042.stdout:69: done (3 left) 2024-09-20T19:21:53.996 INFO:tasks.rados.rados.0.smithi042.stdout:71: done (2 left) 2024-09-20T19:21:53.996 INFO:tasks.rados.rados.0.smithi042.stdout:72: done (1 left) 2024-09-20T19:21:53.996 INFO:tasks.rados.rados.0.smithi042.stdout:73: done (0 left) 2024-09-20T19:21:53.996 INFO:tasks.rados.rados.0.smithi042.stdout:74: snap_create 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:53 smithi162 bash[24557]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[22706]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.095 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: audit 2024-09-20T19:21:52.873910+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? v1:172.21.15.42:0/2460160717' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [2, 3, 1]}]': finished 2024-09-20T19:21:54.096 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.096 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: cluster 2024-09-20T19:21:52.878217+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-20T19:21:54.096 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.096 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:53 smithi042 bash[32075]: cluster 2024-09-20T19:21:52.991867+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-20T19:21:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:54.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:54.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:54.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:54.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:54.589 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:21:54.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:54.759 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.785 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:54.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:54.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:54.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:55.002 INFO:tasks.rados.rados.0.smithi042.stdout:74: done (0 left) 2024-09-20T19:21:55.003 INFO:tasks.rados.rados.0.smithi042.stdout:75: snap_create 2024-09-20T19:21:55.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:55.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:55.146 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:55.174 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:55.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[32075]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[32075]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[32075]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[32075]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[22706]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[22706]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[22706]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.289 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:55 smithi042 bash[22706]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:55 smithi162 bash[24557]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:55 smithi162 bash[24557]: cluster 2024-09-20T19:21:53.955464+0000 mgr.y (mgr.24491) 152 : cluster [DBG] pgmap v125: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 89 op/s 2024-09-20T19:21:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:55 smithi162 bash[24557]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:55 smithi162 bash[24557]: cluster 2024-09-20T19:21:53.995870+0000 mon.a (mon.0) 1015 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:21:55.306 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:55.410 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:55.435 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:55.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:55.564 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:55.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:21:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:21:55.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:55.873 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:21:55.873 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:21:55.890 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:21:55.891 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:21:55.908 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:21:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:21:55.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:55.924 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:21:55.925 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:21:55.944 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:21:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:21:55.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:55.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:55.962 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:21:55.962 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:21:55.981 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:21:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:21:56.003 INFO:tasks.rados.rados.0.smithi042.stdout:75: done (0 left) 2024-09-20T19:21:56.003 INFO:tasks.rados.rados.0.smithi042.stdout:76: snap_remove snap 3 2024-09-20T19:21:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:21:56.006 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:21:56.036 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:21:56.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.060 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:56.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: cluster 2024-09-20T19:21:55.001274+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: audit 2024-09-20T19:21:55.854277+0000 mon.c (mon.2) 33 : audit [DBG] from='client.? v1:172.21.15.42:0/418081614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:56 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.002904+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-20T19:21:56.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:56.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:56.315 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:21:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:21:56.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:56.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:56.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:56.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:56.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:56.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:56.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:56.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:56.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:56.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:56.986 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:56.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:57.002 INFO:tasks.rados.rados.0.smithi042.stdout:76: done (0 left) 2024-09-20T19:21:57.002 INFO:tasks.rados.rados.0.smithi042.stdout:77: snap_remove snap 5 2024-09-20T19:21:57.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:57.101 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.103 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:57 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.114 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:57.217 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:55.956110+0000 mgr.y (mgr.24491) 153 : cluster [DBG] pgmap v128: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 123 MiB data, 434 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.289711+0000 osd.5 (osd.5) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:56.289814+0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: audit 2024-09-20T19:21:56.746696+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: audit 2024-09-20T19:21:56.748824+0000 mon.a (mon.0) 1019 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:21:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:57 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.001789+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-20T19:21:57.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:57.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:57.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:57.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:57.740 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:57.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:57.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.898 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:57.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:58.001 INFO:tasks.rados.rados.0.smithi042.stdout:77: done (0 left) 2024-09-20T19:21:58.062 INFO:tasks.rados.rados.0.smithi042.stdout:78: copy_from oid 9 from oid 39 current snap is 6 2024-09-20T19:21:58.062 INFO:tasks.rados.rados.0.smithi042.stdout:79: read oid 50 snap 4 2024-09-20T19:21:58.062 INFO:tasks.rados.rados.0.smithi042.stdout:79: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-20T19:21:58.062 INFO:tasks.rados.rados.0.smithi042.stdout:80: copy_from oid 17 from oid 36 current snap is 6 2024-09-20T19:21:58.062 INFO:tasks.rados.rados.0.smithi042.stdout:81: delete oid 41 current snap is 6 2024-09-20T19:21:58.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:58.063 INFO:tasks.rados.rados.0.smithi042.stdout:81: done (3 left) 2024-09-20T19:21:58.064 INFO:tasks.rados.rados.0.smithi042.stdout:82: snap_create 2024-09-20T19:21:58.064 INFO:tasks.rados.rados.0.smithi042.stdout:79: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-20T19:21:58.064 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:21:58.087 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:21:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:78: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:78: finishing copy_from to smithi04268048-9 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:80: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:80: finishing copy_from to smithi04268048-17 2024-09-20T19:21:58.130 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 11 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-20T19:21:58.191 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:58.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:58.321 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:58.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.475 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:58.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:58.597 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:58.700 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:21:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:21:58.828 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:58.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:58.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:58.984 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:58.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:21:59.006 INFO:tasks.rados.rados.0.smithi042.stdout:78: done (3 left) 2024-09-20T19:21:59.006 INFO:tasks.rados.rados.0.smithi042.stdout:79: done (2 left) 2024-09-20T19:21:59.006 INFO:tasks.rados.rados.0.smithi042.stdout:80: done (1 left) 2024-09-20T19:21:59.006 INFO:tasks.rados.rados.0.smithi042.stdout:82: done (0 left) 2024-09-20T19:21:59.006 INFO:tasks.rados.rados.0.smithi042.stdout:83: snap_remove snap 4 2024-09-20T19:21:59.088 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.111 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:59.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:21:59.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[22706]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.245 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.246 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.246 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.246 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.246 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:58 smithi042 bash[32075]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:21:59.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:21:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:58 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:58 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.956841+0000 mgr.y (mgr.24491) 154 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:58 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:58 smithi162 bash[24557]: cluster 2024-09-20T19:21:57.999571+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:59 smithi162 bash[24557]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:59 smithi162 bash[24557]: audit 2024-09-20T19:21:58.750961+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.42:0/2117365411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:59 smithi162 bash[24557]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:21:59 smithi162 bash[24557]: audit 2024-09-20T19:21:58.751444+0000 mon.a (mon.0) 1022 : audit [INF] from='client.24779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:21:59.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:59.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:21:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:21:59.605 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:21:59.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:59.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:21:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:21:59.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.761 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:21:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:21:59.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:59.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:21:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:21:59.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:21:59.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:00.012 INFO:tasks.rados.rados.0.smithi042.stdout:83: done (0 left) 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:84: read oid 10 snap 6 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:85: copy_from oid 46 from oid 13 current snap is 7 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:86: copy_from oid 3 from oid 11 current snap is 7 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:87: copy_from oid 27 from oid 42 current snap is 7 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:88: read oid 45 snap -1 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:88: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:89: write oid 36 current snap is 7 2024-09-20T19:22:00.013 INFO:tasks.rados.rados.0.smithi042.stdout:89: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-20T19:22:00.018 INFO:tasks.rados.rados.0.smithi042.stdout:89: writing smithi04268048-36 from 591613 to 1045775 tid 1 2024-09-20T19:22:00.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:00.026 INFO:tasks.rados.rados.0.smithi042.stdout:89: writing smithi04268048-36 from 1719614 to 2136967 tid 2 2024-09-20T19:22:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:00.032 INFO:tasks.rados.rados.0.smithi042.stdout:89: writing smithi04268048-36 from 2570780 to 2991613 tid 3 2024-09-20T19:22:00.032 INFO:tasks.rados.rados.0.smithi042.stdout:90: read oid 19 snap -1 2024-09-20T19:22:00.032 INFO:tasks.rados.rados.0.smithi042.stdout:90: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-20T19:22:00.032 INFO:tasks.rados.rados.0.smithi042.stdout:91: write (excl) oid 32 current snap is 7 2024-09-20T19:22:00.032 INFO:tasks.rados.rados.0.smithi042.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-20T19:22:00.039 INFO:tasks.rados.rados.0.smithi042.stdout:91: writing smithi04268048-32 from 608420 to 1137831 tid 1 2024-09-20T19:22:00.046 INFO:tasks.rados.rados.0.smithi042.stdout:91: writing smithi04268048-32 from 1778096 to 2339107 tid 2 2024-09-20T19:22:00.051 INFO:tasks.rados.rados.0.smithi042.stdout:91: writing smithi04268048-32 from 2881850 to 3008420 tid 3 2024-09-20T19:22:00.051 INFO:tasks.rados.rados.0.smithi042.stdout:92: read oid 40 snap 6 2024-09-20T19:22:00.051 INFO:tasks.rados.rados.0.smithi042.stdout:92: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T19:22:00.051 INFO:tasks.rados.rados.0.smithi042.stdout:93: delete oid 7 current snap is 7 2024-09-20T19:22:00.051 INFO:tasks.rados.rados.0.smithi042.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1002900 2024-09-20T19:22:00.095 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1234109 2024-09-20T19:22:00.105 INFO:tasks.rados.rados.0.smithi042.stdout:88: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-20T19:22:00.116 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2730215 2024-09-20T19:22:00.127 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1160064 2024-09-20T19:22:00.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:00 smithi162 bash[24557]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.155 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:00.156 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2329565 2024-09-20T19:22:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:86: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:22:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:87: finishing copy_from racing read to smithi04268048-27 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: finishing write tid 1 to smithi04268048-36 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:85: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:87: finishing copy_from to smithi04268048-27 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 15 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: finishing write tid 2 to smithi04268048-36 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: finishing write tid 3 to smithi04268048-36 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: finishing write tid 4 to smithi04268048-36 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: finishing write tid 5 to smithi04268048-36 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: oid 36 updating version 0 to 17 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: oid 36 updating version 17 to 18 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: oid 36 version 18 is already newer than 15 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:89: oid 36 version 18 is already newer than 16 2024-09-20T19:22:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 18 (ObjNum 59 snap 7 seq_num 59) dirty exists 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:89: left oid 36 (ObjNum 59 snap 7 seq_num 59) 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:85: finishing copy_from to smithi04268048-46 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:91: finishing write tid 1 to smithi04268048-32 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:86: finishing copy_from to smithi04268048-3 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-20T19:22:00.162 INFO:tasks.rados.rados.0.smithi042.stdout:90: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-20T19:22:00.177 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 417508 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[32075]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: cluster 2024-09-20T19:21:58.991866+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: audit 2024-09-20T19:21:59.001521+0000 mon.a (mon.0) 1024 : audit [INF] from='client.24779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.006305+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:21:59 smithi042 bash[22706]: audit 2024-09-20T19:21:59.011841+0000 mgr.y (mgr.24491) 155 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:00.209 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2013511 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: finishing write tid 2 to smithi04268048-32 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: finishing write tid 3 to smithi04268048-32 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: finishing write tid 4 to smithi04268048-32 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: finishing write tid 5 to smithi04268048-32 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: oid 32 updating version 0 to 29 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: oid 32 version 29 is already newer than 27 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: oid 32 version 29 is already newer than 26 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: oid 32 version 29 is already newer than 28 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 29 (ObjNum 60 snap 7 seq_num 60) dirty exists 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:91: left oid 32 (ObjNum 60 snap 7 seq_num 60) 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:92: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T19:22:00.212 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:00.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:00.263 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1307480 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:84: done (9 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:85: done (8 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:86: done (7 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:87: done (6 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:88: done (5 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:89: done (4 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:90: done (3 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:91: done (2 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:92: done (1 left) 2024-09-20T19:22:00.277 INFO:tasks.rados.rados.0.smithi042.stdout:93: done (0 left) 2024-09-20T19:22:00.278 INFO:tasks.rados.rados.0.smithi042.stdout:94: rollback oid 23 current snap is 7 2024-09-20T19:22:00.278 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 6 2024-09-20T19:22:00.278 INFO:tasks.rados.rados.0.smithi042.stdout:95: snap_create 2024-09-20T19:22:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:00.282 INFO:tasks.rados.rados.0.smithi042.stdout:94: finishing rollback tid 0 to smithi04268048-23 2024-09-20T19:22:00.282 INFO:tasks.rados.rados.0.smithi042.stdout:94: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:22:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:00.283 INFO:tasks.rados.rados.0.smithi042.stdout:94: finishing rollback tid 2 to smithi04268048-23 2024-09-20T19:22:00.283 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 24 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-20T19:22:00.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:00.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:00.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:00.572 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:22:00.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T19:22:00.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:00.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:00.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:00.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:00.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:00.929 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:00.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:01.023 INFO:tasks.rados.rados.0.smithi042.stdout:94: done (1 left) 2024-09-20T19:22:01.023 INFO:tasks.rados.rados.0.smithi042.stdout:95: done (0 left) 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:96: rollback oid 10 current snap is 8 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 7 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:97: read oid 12 snap -1 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:97: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:98: read oid 35 snap 7 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:98: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:99: read oid 21 snap 6 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:99: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:100: read oid 29 snap -1 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:100: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:101: rollback oid 40 current snap is 8 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 6 2024-09-20T19:22:01.024 INFO:tasks.rados.rados.0.smithi042.stdout:102: delete oid 42 current snap is 8 2024-09-20T19:22:01.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.033 INFO:tasks.rados.rados.0.smithi042.stdout:98: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:22:01.042 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1040936 2024-09-20T19:22:01.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:01.119 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 816740 2024-09-20T19:22:01.153 INFO:tasks.rados.rados.0.smithi042.stdout:99: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:22:01.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:01.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.171 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:01.190 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:01.271 INFO:tasks.rados.rados.0.smithi042.stdout:96: finishing rollback tid 0 to smithi04268048-10 2024-09-20T19:22:01.271 INFO:tasks.rados.rados.0.smithi042.stdout:96: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:22:01.271 INFO:tasks.rados.rados.0.smithi042.stdout:96: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:22:01.271 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 16 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T19:22:01.272 INFO:tasks.rados.rados.0.smithi042.stdout:100: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-20T19:22:01.272 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:01.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.297 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:01.316 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:01.333 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1693086 2024-09-20T19:22:01.340 INFO:tasks.rados.rados.0.smithi042.stdout:101: finishing rollback tid 0 to smithi04268048-40 2024-09-20T19:22:01.340 INFO:tasks.rados.rados.0.smithi042.stdout:101: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:22:01.340 INFO:tasks.rados.rados.0.smithi042.stdout:101: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:22:01.340 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 20 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-20T19:22:01.341 INFO:tasks.rados.rados.0.smithi042.stdout:97: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:01.341 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:01.382 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:01.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.825107+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.826912+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:21:59.957359+0000 mgr.y (mgr.24491) 156 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 114 MiB data, 420 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 7 op/s 2024-09-20T19:22:01.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.011268+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:22:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.238451+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:22:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.240802+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:22:01.432 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1357297 2024-09-20T19:22:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:96: done (6 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:97: done (5 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:98: done (4 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:99: done (3 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:100: done (2 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:101: done (1 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:102: done (0 left) 2024-09-20T19:22:01.460 INFO:tasks.rados.rados.0.smithi042.stdout:103: snap_remove snap 7 2024-09-20T19:22:01.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:01.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:01.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:01.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:01.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:22:01.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:01.952 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:01.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:103: done (0 left) 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:104: read oid 14 snap 6 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:104: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:105: read oid 5 snap -1 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:105: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-20T19:22:02.037 INFO:tasks.rados.rados.0.smithi042.stdout:106: copy_from oid 49 from oid 48 current snap is 8 2024-09-20T19:22:02.038 INFO:tasks.rados.rados.0.smithi042.stdout:107: write (excl) oid 38 current snap is 8 2024-09-20T19:22:02.038 INFO:tasks.rados.rados.0.smithi042.stdout:107: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-20T19:22:02.038 INFO:tasks.rados.rados.0.smithi042.stdout:107: writing smithi04268048-38 from 658841 to 1130352 tid 1 2024-09-20T19:22:02.055 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:02.056 INFO:tasks.rados.rados.0.smithi042.stdout:107: writing smithi04268048-38 from 1553542 to 2261880 tid 2 2024-09-20T19:22:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:107: writing smithi04268048-38 from 2964119 to 3058841 tid 3 2024-09-20T19:22:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:108: write oid 46 current snap is 8 2024-09-20T19:22:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:108: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-20T19:22:02.068 INFO:tasks.rados.rados.0.smithi042.stdout:108: writing smithi04268048-46 from 675648 to 1222408 tid 1 2024-09-20T19:22:02.076 INFO:tasks.rados.rados.0.smithi042.stdout:108: writing smithi04268048-46 from 2012024 to 2580373 tid 2 2024-09-20T19:22:02.076 INFO:tasks.rados.rados.0.smithi042.stdout:109: rollback oid 41 current snap is 8 2024-09-20T19:22:02.076 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 6 2024-09-20T19:22:02.076 INFO:tasks.rados.rados.0.smithi042.stdout:104: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T19:22:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:02.091 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:02.132 INFO:tasks.rados.rados.0.smithi042.stdout:107: finishing write tid 1 to smithi04268048-38 2024-09-20T19:22:02.133 INFO:tasks.rados.rados.0.smithi042.stdout:105: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-20T19:22:02.137 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 816865 2024-09-20T19:22:02.152 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1556782 2024-09-20T19:22:02.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:02 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.183 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:02.184 INFO:tasks.rados.rados.0.smithi042.stdout:106: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:22:02.184 INFO:tasks.rados.rados.0.smithi042.stdout:107: finishing write tid 2 to smithi04268048-38 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: finishing write tid 3 to smithi04268048-38 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: finishing write tid 4 to smithi04268048-38 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:106: finishing copy_from to smithi04268048-49 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: finishing write tid 5 to smithi04268048-38 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: oid 38 updating version 0 to 22 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: oid 38 updating version 22 to 27 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: oid 38 updating version 27 to 29 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: oid 38 version 29 is already newer than 28 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 29 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:107: left oid 38 (ObjNum 63 snap 8 seq_num 63) 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:108: finishing write tid 1 to smithi04268048-46 2024-09-20T19:22:02.185 INFO:tasks.rados.rados.0.smithi042.stdout:108: finishing write tid 2 to smithi04268048-46 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: finishing write tid 3 to smithi04268048-46 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: finishing write tid 4 to smithi04268048-46 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: oid 46 updating version 0 to 32 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: oid 46 updating version 32 to 34 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: oid 46 version 34 is already newer than 33 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 34 (ObjNum 64 snap 8 seq_num 64) dirty exists 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: left oid 46 (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:109: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 1415184528) dirty dne 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:104: done (5 left) 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:105: done (4 left) 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:106: done (3 left) 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:107: done (2 left) 2024-09-20T19:22:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:108: done (1 left) 2024-09-20T19:22:02.187 INFO:tasks.rados.rados.0.smithi042.stdout:109: done (0 left) 2024-09-20T19:22:02.187 INFO:tasks.rados.rados.0.smithi042.stdout:110: rollback oid 1 current snap is 8 2024-09-20T19:22:02.187 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 6 2024-09-20T19:22:02.187 INFO:tasks.rados.rados.0.smithi042.stdout:111: write (excl) oid 11 current snap is 8 2024-09-20T19:22:02.187 INFO:tasks.rados.rados.0.smithi042.stdout:111: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-20T19:22:02.198 INFO:tasks.rados.rados.0.smithi042.stdout:111: writing smithi04268048-11 from 692455 to 1314464 tid 1 2024-09-20T19:22:02.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:02.212 INFO:tasks.rados.rados.0.smithi042.stdout:111: writing smithi04268048-11 from 2070506 to 2782513 tid 2 2024-09-20T19:22:02.212 INFO:tasks.rados.rados.0.smithi042.stdout:112: copy_from oid 8 from oid 6 current snap is 8 2024-09-20T19:22:02.212 INFO:tasks.rados.rados.0.smithi042.stdout:110: finishing rollback tid 0 to smithi04268048-1 2024-09-20T19:22:02.212 INFO:tasks.rados.rados.0.smithi042.stdout:110: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:110: finishing rollback tid 2 to smithi04268048-1 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 16 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:110: done (2 left) 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:113: read oid 46 snap -1 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:113: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:114: snap_create 2024-09-20T19:22:02.213 INFO:tasks.rados.rados.0.smithi042.stdout:111: finishing write tid 1 to smithi04268048-11 2024-09-20T19:22:02.216 INFO:tasks.rados.rados.0.smithi042.stdout:113: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.801378+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:00.803428+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.022574+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-20T19:22:02.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.348908+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:02.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:02 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.349015+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:02.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:02.337 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: finishing write tid 2 to smithi04268048-11 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: finishing write tid 3 to smithi04268048-11 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: finishing write tid 4 to smithi04268048-11 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: oid 11 updating version 0 to 43 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: oid 11 version 43 is already newer than 42 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: oid 11 version 43 is already newer than 41 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 43 (ObjNum 65 snap 8 seq_num 65) dirty exists 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:111: left oid 11 (ObjNum 65 snap 8 seq_num 65) 2024-09-20T19:22:02.351 INFO:tasks.rados.rados.0.smithi042.stdout:112: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:22:02.352 INFO:tasks.rados.rados.0.smithi042.stdout:112: finishing copy_from to smithi04268048-8 2024-09-20T19:22:02.352 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-20T19:22:02.383 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:02.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:02.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:02.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:02.690 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:02.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:02.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:02.950 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:02.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:02.997 INFO:tasks.rados.rados.0.smithi042.stdout:111: done (3 left) 2024-09-20T19:22:02.997 INFO:tasks.rados.rados.0.smithi042.stdout:112: done (2 left) 2024-09-20T19:22:02.997 INFO:tasks.rados.rados.0.smithi042.stdout:113: done (1 left) 2024-09-20T19:22:02.997 INFO:tasks.rados.rados.0.smithi042.stdout:114: done (0 left) 2024-09-20T19:22:02.997 INFO:tasks.rados.rados.0.smithi042.stdout:115: snap_create 2024-09-20T19:22:03.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.093 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:03 smithi162 bash[24557]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:03.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.223 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[32075]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.955859+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.957876+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:01.958387+0000 mgr.y (mgr.24491) 157 : cluster [DBG] pgmap v137: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.029226+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.234017+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.e scrub starts 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.235593+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.e scrub ok 2024-09-20T19:22:03.225 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.225 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:03 smithi042 bash[22706]: cluster 2024-09-20T19:22:02.996599+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-20T19:22:03.237 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:03.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:03.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:03.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:03.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:03.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:03.628 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:03.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:03.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:03.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:03.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:03.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:03.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.004 INFO:tasks.rados.rados.0.smithi042.stdout:115: done (0 left) 2024-09-20T19:22:04.004 INFO:tasks.rados.rados.0.smithi042.stdout:116: snap_remove snap 6 2024-09-20T19:22:04.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:04.123 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:04.143 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:04.191 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:22:04.191 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:22:04.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:04 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:04.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.012928+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.015577+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:04 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.002799+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:22:04.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:04.431 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:04.432 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:04.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:04.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:04.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:04.788 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:04.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:04.917 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:04.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:05.009 INFO:tasks.rados.rados.0.smithi042.stdout:116: done (0 left) 2024-09-20T19:22:05.010 INFO:tasks.rados.rados.0.smithi042.stdout:117: write oid 21 current snap is 10 2024-09-20T19:22:05.010 INFO:tasks.rados.rados.0.smithi042.stdout:117: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-20T19:22:05.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:05.033 INFO:tasks.rados.rados.0.smithi042.stdout:117: writing smithi04268048-21 from 709262 to 1406520 tid 1 2024-09-20T19:22:05.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:05.054 INFO:tasks.rados.rados.0.smithi042.stdout:117: writing smithi04268048-21 from 2012635 to 2584653 tid 2 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:117: writing smithi04268048-21 from 3046388 to 3109262 tid 3 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:118: read oid 47 snap -1 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:118: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:119: write (excl) oid 28 current snap is 10 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:117: finishing write tid 1 to smithi04268048-21 2024-09-20T19:22:05.062 INFO:tasks.rados.rados.0.smithi042.stdout:117: finishing write tid 2 to smithi04268048-21 2024-09-20T19:22:05.063 INFO:tasks.rados.rados.0.smithi042.stdout:119: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-20T19:22:05.074 INFO:tasks.rados.rados.0.smithi042.stdout:119: writing smithi04268048-28 from 726069 to 1498576 tid 1 2024-09-20T19:22:05.088 INFO:tasks.rados.rados.0.smithi042.stdout:119: writing smithi04268048-28 from 2071117 to 2786793 tid 2 2024-09-20T19:22:05.089 INFO:tasks.rados.rados.0.smithi042.stdout:120: write (excl) oid 8 current snap is 10 2024-09-20T19:22:05.089 INFO:tasks.rados.rados.0.smithi042.stdout:120: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-20T19:22:05.100 INFO:tasks.rados.rados.0.smithi042.stdout:120: writing smithi04268048-8 from 742876 to 1190632 tid 1 2024-09-20T19:22:05.110 INFO:tasks.rados.rados.0.smithi042.stdout:120: writing smithi04268048-8 from 1729599 to 2305286 tid 2 2024-09-20T19:22:05.117 INFO:tasks.rados.rados.0.smithi042.stdout:120: writing smithi04268048-8 from 3101234 to 3142876 tid 3 2024-09-20T19:22:05.117 INFO:tasks.rados.rados.0.smithi042.stdout:121: delete oid 50 current snap is 10 2024-09-20T19:22:05.117 INFO:tasks.rados.rados.0.smithi042.stdout:117: finishing write tid 3 to smithi04268048-21 2024-09-20T19:22:05.118 INFO:tasks.rados.rados.0.smithi042.stdout:118: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:22:05.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:05.238 INFO:tasks.rados.rados.0.smithi042.stdout:117: finishing write tid 4 to smithi04268048-21 2024-09-20T19:22:05.238 INFO:tasks.rados.rados.0.smithi042.stdout:117: finishing write tid 5 to smithi04268048-21 2024-09-20T19:22:05.238 INFO:tasks.rados.rados.0.smithi042.stdout:117: oid 21 updating version 0 to 27 2024-09-20T19:22:05.238 INFO:tasks.rados.rados.0.smithi042.stdout:117: oid 21 version 27 is already newer than 26 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:117: oid 21 version 27 is already newer than 24 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:117: oid 21 version 27 is already newer than 25 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 27 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:117: left oid 21 (ObjNum 66 snap 10 seq_num 66) 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:119: finishing write tid 1 to smithi04268048-28 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:117: done (4 left) 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:118: done (3 left) 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:121: done (2 left) 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:122: rollback oid 13 current snap is 10 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 8 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:119: finishing write tid 2 to smithi04268048-28 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:119: finishing write tid 3 to smithi04268048-28 2024-09-20T19:22:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:119: finishing write tid 4 to smithi04268048-28 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:119: oid 28 updating version 0 to 9 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:119: oid 28 version 9 is already newer than 8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:119: oid 28 updating version 9 to 10 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 10 (ObjNum 67 snap 10 seq_num 67) dirty exists 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:119: left oid 28 (ObjNum 67 snap 10 seq_num 67) 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: finishing write tid 1 to smithi04268048-8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: finishing write tid 2 to smithi04268048-8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: finishing write tid 3 to smithi04268048-8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: finishing write tid 4 to smithi04268048-8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: finishing write tid 5 to smithi04268048-8 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: oid 8 updating version 0 to 25 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: oid 8 updating version 25 to 27 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: oid 8 version 27 is already newer than 20 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:120: oid 8 version 27 is already newer than 26 2024-09-20T19:22:05.240 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 27 (ObjNum 68 snap 10 seq_num 68) dirty exists 2024-09-20T19:22:05.241 INFO:tasks.rados.rados.0.smithi042.stdout:120: left oid 8 (ObjNum 68 snap 10 seq_num 68) 2024-09-20T19:22:05.241 INFO:tasks.rados.rados.0.smithi042.stdout:119: done (2 left) 2024-09-20T19:22:05.241 INFO:tasks.rados.rados.0.smithi042.stdout:120: done (1 left) 2024-09-20T19:22:05.241 INFO:tasks.rados.rados.0.smithi042.stdout:123: snap_remove snap 9 2024-09-20T19:22:05.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:05.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.292 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:05 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.959008+0000 mgr.y (mgr.24491) 158 : cluster [DBG] pgmap v140: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.6 MiB/s wr, 29 op/s 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.978905+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:03.981088+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:05 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.008732+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:22:05.307 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:05.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:05.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:05.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:22:05.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:05.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:05.706 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:05.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:05.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:05.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:05.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:05.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:06.049 INFO:tasks.rados.rados.0.smithi042.stdout:123: done (1 left) 2024-09-20T19:22:06.049 INFO:tasks.rados.rados.0.smithi042.stdout:124: read oid 11 snap -1 2024-09-20T19:22:06.049 INFO:tasks.rados.rados.0.smithi042.stdout:124: expect (ObjNum 65 snap 8 seq_num 65) 2024-09-20T19:22:06.049 INFO:tasks.rados.rados.0.smithi042.stdout:125: read oid 2 snap 8 2024-09-20T19:22:06.049 INFO:tasks.rados.rados.0.smithi042.stdout:125: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:126: read oid 4 snap 8 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:126: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:127: read oid 12 snap -1 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:127: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:128: rollback oid 31 current snap is 10 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 8 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:122: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:122: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:122: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 24 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:122: done (5 left) 2024-09-20T19:22:06.050 INFO:tasks.rados.rados.0.smithi042.stdout:129: snap_create 2024-09-20T19:22:06.064 INFO:tasks.rados.rados.0.smithi042.stdout:127: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:06.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:06.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:06 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:06 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:06 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:06 smithi162 bash[24557]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:06.163 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 517598 2024-09-20T19:22:06.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:06.203 INFO:tasks.rados.rados.0.smithi042.stdout:128: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:22:06.203 INFO:tasks.rados.rados.0.smithi042.stdout:125: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[32075]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.211 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.233239+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:06.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[22706]: cluster 2024-09-20T19:22:04.233344+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:06.213 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1911980 2024-09-20T19:22:06.225 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1715058 2024-09-20T19:22:06.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:06.251 INFO:tasks.rados.rados.0.smithi042.stdout:128: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:22:06.251 INFO:tasks.rados.rados.0.smithi042.stdout:128: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:22:06.252 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 31 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T19:22:06.252 INFO:tasks.rados.rados.0.smithi042.stdout:124: expect (ObjNum 65 snap 8 seq_num 65) 2024-09-20T19:22:06.328 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:06.338 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 635035 2024-09-20T19:22:06.339 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:06.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:06.378 INFO:tasks.rados.rados.0.smithi042.stdout:126: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:06.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:06.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:06.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:06.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:06.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:06.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:06.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:06.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:06.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:07.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:07.061 INFO:tasks.rados.rados.0.smithi042.stdout:124: done (5 left) 2024-09-20T19:22:07.061 INFO:tasks.rados.rados.0.smithi042.stdout:125: done (4 left) 2024-09-20T19:22:07.061 INFO:tasks.rados.rados.0.smithi042.stdout:126: done (3 left) 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:127: done (2 left) 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:128: done (1 left) 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:129: done (0 left) 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:130: rollback oid 38 current snap is 11 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 10 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:131: write oid 33 current snap is 11 2024-09-20T19:22:07.062 INFO:tasks.rados.rados.0.smithi042.stdout:131: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-20T19:22:07.075 INFO:tasks.rados.rados.0.smithi042.stdout:131: writing smithi04268048-33 from 776490 to 1491097 tid 1 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:131: writing smithi04268048-33 from 2246563 to 2825919 tid 2 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:132: read oid 34 snap 10 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:132: expect deleted 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:133: snap_create 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:130: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:130: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:130: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:22:07.091 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 37 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-20T19:22:07.093 INFO:tasks.rados.rados.0.smithi042.stdout:131: finishing write tid 1 to smithi04268048-33 2024-09-20T19:22:07.094 INFO:tasks.rados.rados.0.smithi042.stdout:131: finishing write tid 2 to smithi04268048-33 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: finishing write tid 3 to smithi04268048-33 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: finishing write tid 4 to smithi04268048-33 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: oid 33 updating version 0 to 28 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: oid 33 updating version 28 to 29 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: oid 33 version 29 is already newer than 27 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 29 (ObjNum 70 snap 11 seq_num 70) dirty exists 2024-09-20T19:22:07.095 INFO:tasks.rados.rados.0.smithi042.stdout:131: left oid 33 (ObjNum 70 snap 11 seq_num 70) 2024-09-20T19:22:07.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.116 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:07 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:07.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:07.262 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:07.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:07.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.485933+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.488027+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:05.959499+0000 mgr.y (mgr.24491) 159 : cluster [DBG] pgmap v143: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 122 MiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.459 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:07 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.045096+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-20T19:22:07.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:07.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:07.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:07.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:07.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:07.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:07.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:08.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:08.013 INFO:tasks.rados.rados.0.smithi042.stdout:130: done (3 left) 2024-09-20T19:22:08.013 INFO:tasks.rados.rados.0.smithi042.stdout:131: done (2 left) 2024-09-20T19:22:08.013 INFO:tasks.rados.rados.0.smithi042.stdout:132: done (1 left) 2024-09-20T19:22:08.014 INFO:tasks.rados.rados.0.smithi042.stdout:133: done (0 left) 2024-09-20T19:22:08.014 INFO:tasks.rados.rados.0.smithi042.stdout:134: write (excl) oid 40 current snap is 12 2024-09-20T19:22:08.014 INFO:tasks.rados.rados.0.smithi042.stdout:134: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-20T19:22:08.024 INFO:tasks.rados.rados.0.smithi042.stdout:134: writing smithi04268048-40 from 793297 to 1583153 tid 1 2024-09-20T19:22:08.037 INFO:tasks.rados.rados.0.smithi042.stdout:134: writing smithi04268048-40 from 2305045 to 3028059 tid 2 2024-09-20T19:22:08.038 INFO:tasks.rados.rados.0.smithi042.stdout:135: write (excl) oid 26 current snap is 12 2024-09-20T19:22:08.038 INFO:tasks.rados.rados.0.smithi042.stdout:134: finishing write tid 1 to smithi04268048-40 2024-09-20T19:22:08.038 INFO:tasks.rados.rados.0.smithi042.stdout:135: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-20T19:22:08.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:08.044 INFO:tasks.rados.rados.0.smithi042.stdout:135: writing smithi04268048-26 from 410104 to 875209 tid 1 2024-09-20T19:22:08.054 INFO:tasks.rados.rados.0.smithi042.stdout:135: writing smithi04268048-26 from 1563527 to 2146552 tid 2 2024-09-20T19:22:08.062 INFO:tasks.rados.rados.0.smithi042.stdout:135: writing smithi04268048-26 from 2810926 to 3210104 tid 3 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:136: delete oid 39 current snap is 12 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: finishing write tid 2 to smithi04268048-40 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: finishing write tid 3 to smithi04268048-40 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: finishing write tid 4 to smithi04268048-40 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: oid 40 updating version 0 to 42 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: oid 40 updating version 42 to 43 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: oid 40 version 43 is already newer than 41 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 43 (ObjNum 71 snap 12 seq_num 71) dirty exists 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:134: left oid 40 (ObjNum 71 snap 12 seq_num 71) 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:135: finishing write tid 1 to smithi04268048-26 2024-09-20T19:22:08.063 INFO:tasks.rados.rados.0.smithi042.stdout:135: finishing write tid 2 to smithi04268048-26 2024-09-20T19:22:08.066 INFO:tasks.rados.rados.0.smithi042.stdout:134: done (2 left) 2024-09-20T19:22:08.066 INFO:tasks.rados.rados.0.smithi042.stdout:136: done (1 left) 2024-09-20T19:22:08.066 INFO:tasks.rados.rados.0.smithi042.stdout:137: write (excl) oid 24 current snap is 12 2024-09-20T19:22:08.066 INFO:tasks.rados.rados.0.smithi042.stdout:137: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-20T19:22:08.074 INFO:tasks.rados.rados.0.smithi042.stdout:137: writing smithi04268048-24 from 443718 to 1059321 tid 1 2024-09-20T19:22:08.081 INFO:tasks.rados.rados.0.smithi042.stdout:137: writing smithi04268048-24 from 1564138 to 2034479 tid 2 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:137: writing smithi04268048-24 from 2633066 to 3243718 tid 3 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:138: snap_create 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: finishing write tid 3 to smithi04268048-26 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: finishing write tid 4 to smithi04268048-26 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: finishing write tid 5 to smithi04268048-26 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: oid 26 updating version 0 to 15 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: oid 26 version 15 is already newer than 14 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: oid 26 version 15 is already newer than 13 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: oid 26 version 15 is already newer than 12 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 15 (ObjNum 72 snap 12 seq_num 72) dirty exists 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:135: left oid 26 (ObjNum 72 snap 12 seq_num 72) 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:137: finishing write tid 1 to smithi04268048-24 2024-09-20T19:22:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:137: finishing write tid 2 to smithi04268048-24 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: finishing write tid 3 to smithi04268048-24 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: finishing write tid 4 to smithi04268048-24 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: finishing write tid 5 to smithi04268048-24 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: oid 24 updating version 0 to 11 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: oid 24 updating version 11 to 13 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: oid 24 version 13 is already newer than 12 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:137: oid 24 updating version 13 to 14 2024-09-20T19:22:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 14 (ObjNum 74 snap 12 seq_num 74) dirty exists 2024-09-20T19:22:08.093 INFO:tasks.rados.rados.0.smithi042.stdout:137: left oid 24 (ObjNum 74 snap 12 seq_num 74) 2024-09-20T19:22:08.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:08.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:08.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:08 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:08.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.924302+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:06.926316+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.058889+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.937125+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.939733+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:08 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.003916+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-20T19:22:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:08.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:08.567 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:08.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:08.698 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:08.801 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:08.825 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:08.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:08.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:08.963 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:22:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:22:08.982 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:22:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:22:09.000 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:22:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:22:09.017 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:22:09.017 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:22:09.040 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:22:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:22:09.059 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:22:09.059 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:22:09.059 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:09.080 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:09.085 INFO:tasks.rados.rados.0.smithi042.stdout:135: done (2 left) 2024-09-20T19:22:09.085 INFO:tasks.rados.rados.0.smithi042.stdout:137: done (1 left) 2024-09-20T19:22:09.085 INFO:tasks.rados.rados.0.smithi042.stdout:138: done (0 left) 2024-09-20T19:22:09.085 INFO:tasks.rados.rados.0.smithi042.stdout:139: snap_create 2024-09-20T19:22:09.086 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:22:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:22:09.117 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:22:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:22:09.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:09.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:09 smithi162 bash[24557]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:09.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:09.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[22706]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:07.960235+0000 mgr.y (mgr.24491) 160 : cluster [DBG] pgmap v146: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.022201+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.024749+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:09 smithi042 bash[32075]: audit 2024-09-20T19:22:08.941851+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.42:0/2338537775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:09.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:09.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:09.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:09.596 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:09.699 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:09.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:09.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:09.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:09.985 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:09.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:10.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:10.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:10 smithi162 bash[24557]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.104 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T19:22:10.105 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:22:10.106 INFO:tasks.rados.rados.0.smithi042.stdout:139: done (0 left) 2024-09-20T19:22:10.106 INFO:tasks.rados.rados.0.smithi042.stdout:140: write (excl) oid 5 current snap is 14 2024-09-20T19:22:10.106 INFO:tasks.rados.rados.0.smithi042.stdout:140: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-20T19:22:10.114 INFO:tasks.rados.rados.0.smithi042.stdout:140: writing smithi04268048-5 from 460525 to 1151377 tid 1 2024-09-20T19:22:10.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:10.124 INFO:tasks.rados.rados.0.smithi042.stdout:140: writing smithi04268048-5 from 1622620 to 2352972 tid 2 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:140: writing smithi04268048-5 from 3060489 to 3260525 tid 3 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:141: read oid 22 snap -1 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:141: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:142: snap_create 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:140: finishing write tid 1 to smithi04268048-5 2024-09-20T19:22:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:140: finishing write tid 2 to smithi04268048-5 2024-09-20T19:22:10.134 INFO:tasks.rados.rados.0.smithi042.stdout:140: finishing write tid 3 to smithi04268048-5 2024-09-20T19:22:10.135 INFO:tasks.rados.rados.0.smithi042.stdout:141: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T19:22:10.135 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:10.201 INFO:tasks.rados.rados.0.smithi042.stdout:140: finishing write tid 4 to smithi04268048-5 2024-09-20T19:22:10.201 INFO:tasks.rados.rados.0.smithi042.stdout:140: finishing write tid 5 to smithi04268048-5 2024-09-20T19:22:10.201 INFO:tasks.rados.rados.0.smithi042.stdout:140: oid 5 updating version 0 to 14 2024-09-20T19:22:10.201 INFO:tasks.rados.rados.0.smithi042.stdout:140: oid 5 version 14 is already newer than 12 2024-09-20T19:22:10.202 INFO:tasks.rados.rados.0.smithi042.stdout:140: oid 5 version 14 is already newer than 13 2024-09-20T19:22:10.202 INFO:tasks.rados.rados.0.smithi042.stdout:140: oid 5 version 14 is already newer than 11 2024-09-20T19:22:10.202 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 14 (ObjNum 75 snap 14 seq_num 75) dirty exists 2024-09-20T19:22:10.202 INFO:tasks.rados.rados.0.smithi042.stdout:140: left oid 5 (ObjNum 75 snap 14 seq_num 75) 2024-09-20T19:22:10.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:10.252 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:10.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:10.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[32075]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.789087+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.792232+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.907743+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.909824+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.937327+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:08.939838+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.019808+0000 mgr.y (mgr.24491) 161 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: cluster 2024-09-20T19:22:09.084831+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.333433+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.42:0/3967734878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.333766+0000 mon.a (mon.0) 1037 : audit [INF] from='client.24791 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.530237+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.42:0/1205869802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:10 smithi042 bash[22706]: audit 2024-09-20T19:22:09.530537+0000 mon.a (mon.0) 1038 : audit [INF] from='client.24797 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:10.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:10.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:10.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:10.747 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:10.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:10.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:10.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:10.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:11.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:11.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:11.106 INFO:tasks.rados.rados.0.smithi042.stdout:140: done (2 left) 2024-09-20T19:22:11.106 INFO:tasks.rados.rados.0.smithi042.stdout:141: done (1 left) 2024-09-20T19:22:11.106 INFO:tasks.rados.rados.0.smithi042.stdout:142: done (0 left) 2024-09-20T19:22:11.107 INFO:tasks.rados.rados.0.smithi042.stdout:143: snap_create 2024-09-20T19:22:11.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.162 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:11.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T19:22:11.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.278 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.278 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.278 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.278 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:11 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:11.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:09.960992+0000 mgr.y (mgr.24491) 162 : cluster [DBG] pgmap v149: 148 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 142 active+clean; 120 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 29 op/s 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.083108+0000 mon.a (mon.0) 1039 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.083174+0000 mon.a (mon.0) 1040 : cluster [INF] Cluster is now healthy 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: audit 2024-09-20T19:22:10.095174+0000 mon.a (mon.0) 1041 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: audit 2024-09-20T19:22:10.095400+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:11 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.103875+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-20T19:22:11.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:11.524 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:11.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:11.675 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:11.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:11.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:11.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:12.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.070 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:12.105 INFO:tasks.rados.rados.0.smithi042.stdout:143: done (0 left) 2024-09-20T19:22:12.105 INFO:tasks.rados.rados.0.smithi042.stdout:144: write (excl) oid 42 current snap is 16 2024-09-20T19:22:12.105 INFO:tasks.rados.rados.0.smithi042.stdout:144: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-20T19:22:12.112 INFO:tasks.rados.rados.0.smithi042.stdout:144: writing smithi04268048-42 from 477332 to 1243433 tid 1 2024-09-20T19:22:12.117 INFO:tasks.rados.rados.0.smithi042.stdout:144: writing smithi04268048-42 from 1681102 to 2155112 tid 2 2024-09-20T19:22:12.124 INFO:tasks.rados.rados.0.smithi042.stdout:144: writing smithi04268048-42 from 2687912 to 3277332 tid 3 2024-09-20T19:22:12.124 INFO:tasks.rados.rados.0.smithi042.stdout:145: copy_from oid 6 from oid 31 current snap is 16 2024-09-20T19:22:12.124 INFO:tasks.rados.rados.0.smithi042.stdout:146: snap_remove snap 10 2024-09-20T19:22:12.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:22:12.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:12.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:12 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:12.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:12.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:12.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.314735+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:10.314846+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.103913+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.288104+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.c scrub starts 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.289873+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.c scrub ok 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: audit 2024-09-20T19:22:11.784892+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: audit 2024-09-20T19:22:11.787031+0000 mon.a (mon.0) 1046 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.911952+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.451 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:12 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.914216+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:22:12.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:12.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:12.593 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:12.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:12.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:12.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:12.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:12.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:13.004 INFO:tasks.rados.rados.0.smithi042.stdout:146: done (2 left) 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:147: copy_from oid 15 from oid 7 current snap is 16 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: finishing write tid 1 to smithi04268048-42 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: finishing write tid 2 to smithi04268048-42 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: finishing write tid 3 to smithi04268048-42 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: finishing write tid 4 to smithi04268048-42 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: finishing write tid 5 to smithi04268048-42 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: oid 42 updating version 0 to 57 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: oid 42 version 57 is already newer than 55 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: oid 42 version 57 is already newer than 56 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: oid 42 version 57 is already newer than 54 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 57 (ObjNum 76 snap 16 seq_num 76) dirty exists 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:144: left oid 42 (ObjNum 76 snap 16 seq_num 76) 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:145: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:22:13.005 INFO:tasks.rados.rados.0.smithi042.stdout:145: finishing copy_from to smithi04268048-6 2024-09-20T19:22:13.006 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 31 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T19:22:13.006 INFO:tasks.rados.rados.0.smithi042.stdout:144: done (2 left) 2024-09-20T19:22:13.006 INFO:tasks.rados.rados.0.smithi042.stdout:145: done (1 left) 2024-09-20T19:22:13.006 INFO:tasks.rados.rados.0.smithi042.stdout:148: delete oid 39 current snap is 16 2024-09-20T19:22:13.027 INFO:tasks.rados.rados.0.smithi042.stdout:148: done (1 left) 2024-09-20T19:22:13.027 INFO:tasks.rados.rados.0.smithi042.stdout:149: rollback oid 19 current snap is 16 2024-09-20T19:22:13.027 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 12 2024-09-20T19:22:13.028 INFO:tasks.rados.rados.0.smithi042.stdout:150: delete oid 11 current snap is 16 2024-09-20T19:22:13.028 INFO:tasks.rados.rados.0.smithi042.stdout:147: finishing copy_from to smithi04268048-15 2024-09-20T19:22:13.028 INFO:tasks.rados.rados.0.smithi042.stdout:147: got expected ENOENT (src dne) 2024-09-20T19:22:13.028 INFO:tasks.rados.rados.0.smithi042.stdout:147: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:149: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:149: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:147: done (2 left) 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:150: done (1 left) 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:151: snap_create 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:149: finishing rollback tid 2 to smithi04268048-19 2024-09-20T19:22:13.033 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 16 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-20T19:22:13.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:13.113 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:13.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:22:13.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.221 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:13 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.242 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:22:13.267 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:22:13.267 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:22:13.267 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:22:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:22:13.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:13.365 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:13.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:13.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.896336+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.898246+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:11.961961+0000 mgr.y (mgr.24491) 163 : cluster [DBG] pgmap v152: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:12.103220+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:13 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.001572+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-20T19:22:13.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:13.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:13.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:13.871 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:13.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:13.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:14.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:14.019 INFO:tasks.rados.rados.0.smithi042.stdout:149: done (1 left) 2024-09-20T19:22:14.019 INFO:tasks.rados.rados.0.smithi042.stdout:151: done (0 left) 2024-09-20T19:22:14.019 INFO:tasks.rados.rados.0.smithi042.stdout:152: write (excl) oid 12 current snap is 17 2024-09-20T19:22:14.019 INFO:tasks.rados.rados.0.smithi042.stdout:152: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-20T19:22:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:152: writing smithi04268048-12 from 527753 to 1235954 tid 1 2024-09-20T19:22:14.042 INFO:tasks.rados.rados.0.smithi042.stdout:152: writing smithi04268048-12 from 1856548 to 2594238 tid 2 2024-09-20T19:22:14.049 INFO:tasks.rados.rados.0.smithi042.stdout:152: writing smithi04268048-12 from 3286534 to 3327753 tid 3 2024-09-20T19:22:14.049 INFO:tasks.rados.rados.0.smithi042.stdout:153: snap_create 2024-09-20T19:22:14.050 INFO:tasks.rados.rados.0.smithi042.stdout:152: finishing write tid 1 to smithi04268048-12 2024-09-20T19:22:14.050 INFO:tasks.rados.rados.0.smithi042.stdout:152: finishing write tid 2 to smithi04268048-12 2024-09-20T19:22:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:152: finishing write tid 3 to smithi04268048-12 2024-09-20T19:22:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:152: finishing write tid 4 to smithi04268048-12 2024-09-20T19:22:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:152: finishing write tid 5 to smithi04268048-12 2024-09-20T19:22:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:152: oid 12 updating version 0 to 20 2024-09-20T19:22:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:152: oid 12 version 20 is already newer than 17 2024-09-20T19:22:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:152: oid 12 version 20 is already newer than 19 2024-09-20T19:22:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:152: oid 12 version 20 is already newer than 18 2024-09-20T19:22:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 20 (ObjNum 79 snap 17 seq_num 79) dirty exists 2024-09-20T19:22:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:152: left oid 12 (ObjNum 79 snap 17 seq_num 79) 2024-09-20T19:22:14.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:14.131 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:14.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:22:14.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:14 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:14.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:14.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.101987+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.105677+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:14 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.017608+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-20T19:22:14.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:14.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:14.622 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:14.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:14.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:14.877 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:14.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:15.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:15.022 INFO:tasks.rados.rados.0.smithi042.stdout:152: done (1 left) 2024-09-20T19:22:15.022 INFO:tasks.rados.rados.0.smithi042.stdout:153: done (0 left) 2024-09-20T19:22:15.022 INFO:tasks.rados.rados.0.smithi042.stdout:154: delete oid 7 current snap is 18 2024-09-20T19:22:15.026 INFO:tasks.rados.rados.0.smithi042.stdout:154: done (0 left) 2024-09-20T19:22:15.026 INFO:tasks.rados.rados.0.smithi042.stdout:155: snap_create 2024-09-20T19:22:15.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:15.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:15.146 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.147 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[32075]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:15 smithi042 bash[22706]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.161 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:15.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:15.289 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:13.962661+0000 mgr.y (mgr.24491) 164 : cluster [DBG] pgmap v155: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.221439+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.222966+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.868617+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.871127+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:22:15.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:15 smithi162 bash[24557]: cluster 2024-09-20T19:22:15.021109+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-20T19:22:15.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:15.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:15.549 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:15.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:15.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:15.746 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:15.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:15.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:15.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:15.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:155: done (0 left) 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:156: read oid 32 snap 14 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:156: expect (ObjNum 60 snap 7 seq_num 60) 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:157: read oid 10 snap 18 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:157: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:16.025 INFO:tasks.rados.rados.0.smithi042.stdout:158: snap_remove snap 11 2024-09-20T19:22:16.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:16.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:16.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:16.299 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:16.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:16 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.843430+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.846665+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.896028+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:14.898488+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:22:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:16 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.022599+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-20T19:22:16.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:16.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:16.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:16.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:16.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:16.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:16.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:16.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:16.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:17.037 INFO:tasks.rados.rados.0.smithi042.stdout:158: done (2 left) 2024-09-20T19:22:17.037 INFO:tasks.rados.rados.0.smithi042.stdout:159: read oid 12 snap 14 2024-09-20T19:22:17.037 INFO:tasks.rados.rados.0.smithi042.stdout:159: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:17.037 INFO:tasks.rados.rados.0.smithi042.stdout:160: read oid 33 snap -1 2024-09-20T19:22:17.038 INFO:tasks.rados.rados.0.smithi042.stdout:160: expect (ObjNum 70 snap 11 seq_num 70) 2024-09-20T19:22:17.038 INFO:tasks.rados.rados.0.smithi042.stdout:161: write (excl) oid 20 current snap is 19 2024-09-20T19:22:17.038 INFO:tasks.rados.rados.0.smithi042.stdout:161: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-20T19:22:17.042 INFO:tasks.rados.rados.0.smithi042.stdout:161: writing smithi04268048-20 from 561367 to 1020066 tid 1 2024-09-20T19:22:17.057 INFO:tasks.rados.rados.0.smithi042.stdout:161: writing smithi04268048-20 from 1573512 to 2314871 tid 2 2024-09-20T19:22:17.067 INFO:tasks.rados.rados.0.smithi042.stdout:161: writing smithi04268048-20 from 2941380 to 3361367 tid 3 2024-09-20T19:22:17.067 INFO:tasks.rados.rados.0.smithi042.stdout:162: read oid 6 snap -1 2024-09-20T19:22:17.067 INFO:tasks.rados.rados.0.smithi042.stdout:162: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:22:17.067 INFO:tasks.rados.rados.0.smithi042.stdout:163: copy_from oid 48 from oid 13 current snap is 19 2024-09-20T19:22:17.067 INFO:tasks.rados.rados.0.smithi042.stdout:156: expect (ObjNum 60 snap 7 seq_num 60) 2024-09-20T19:22:17.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:17.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:17.133 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:17.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:17.215 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:17 smithi162 bash[24557]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:17 smithi162 bash[24557]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:17 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:17 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:157: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:17.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:17.252 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:17.275 INFO:tasks.rados.rados.0.smithi042.stdout:160: expect (ObjNum 70 snap 11 seq_num 70) 2024-09-20T19:22:17.322 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3157979 2024-09-20T19:22:17.327 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:17.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[22706]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[22706]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[32075]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[32075]: cluster 2024-09-20T19:22:15.963433+0000 mgr.y (mgr.24491) 165 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 126 MiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:17 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.024551+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-20T19:22:17.437 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3082763 2024-09-20T19:22:17.443 INFO:tasks.rados.rados.0.smithi042.stdout:161: finishing write tid 1 to smithi04268048-20 2024-09-20T19:22:17.443 INFO:tasks.rados.rados.0.smithi042.stdout:161: finishing write tid 2 to smithi04268048-20 2024-09-20T19:22:17.443 INFO:tasks.rados.rados.0.smithi042.stdout:159: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-20T19:22:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1915991 2024-09-20T19:22:17.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:17.481 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 877669 2024-09-20T19:22:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:17.535 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2087819 2024-09-20T19:22:17.547 INFO:tasks.rados.rados.0.smithi042.stdout:161: finishing write tid 3 to smithi04268048-20 2024-09-20T19:22:17.547 INFO:tasks.rados.rados.0.smithi042.stdout:161: finishing write tid 4 to smithi04268048-20 2024-09-20T19:22:17.547 INFO:tasks.rados.rados.0.smithi042.stdout:162: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:22:17.547 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:17.592 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:17.600 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:17.616 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:17.627 INFO:tasks.rados.rados.0.smithi042.stdout:161: finishing write tid 5 to smithi04268048-20 2024-09-20T19:22:17.627 INFO:tasks.rados.rados.0.smithi042.stdout:161: oid 20 updating version 0 to 6 2024-09-20T19:22:17.627 INFO:tasks.rados.rados.0.smithi042.stdout:161: oid 20 updating version 6 to 8 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:161: oid 20 version 8 is already newer than 7 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:161: oid 20 version 8 is already newer than 5 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 8 (ObjNum 81 snap 19 seq_num 81) dirty exists 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:161: left oid 20 (ObjNum 81 snap 19 seq_num 81) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:163: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:163: finishing copy_from to smithi04268048-48 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 24 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:156: done (6 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:157: done (5 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:159: done (4 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:160: done (3 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:161: done (2 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:162: done (1 left) 2024-09-20T19:22:17.628 INFO:tasks.rados.rados.0.smithi042.stdout:163: done (0 left) 2024-09-20T19:22:17.629 INFO:tasks.rados.rados.0.smithi042.stdout:164: snap_create 2024-09-20T19:22:17.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:17.749 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:17.787 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:17.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:17.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:17.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:17.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:18.008 INFO:tasks.rados.rados.0.smithi042.stdout:164: done (0 left) 2024-09-20T19:22:18.008 INFO:tasks.rados.rados.0.smithi042.stdout:165: snap_create 2024-09-20T19:22:18.014 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:18.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.145 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:18.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:18.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:18.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.389 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.390 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:18 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:18.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:18.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.544735+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:22:18.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:16.548564+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:22:18.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:18 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.007913+0000 mon.a (mon.0) 1053 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-20T19:22:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:18.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:18.810 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:18.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.833 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:18.936 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:18.959 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:18.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:19.009 INFO:tasks.rados.rados.0.smithi042.stdout:165: done (0 left) 2024-09-20T19:22:19.009 INFO:tasks.rados.rados.0.smithi042.stdout:166: write oid 27 current snap is 21 2024-09-20T19:22:19.009 INFO:tasks.rados.rados.0.smithi042.stdout:166: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-20T19:22:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T19:22:19.025 INFO:tasks.rados.rados.0.smithi042.stdout:166: writing smithi04268048-27 from 578174 to 1112122 tid 1 2024-09-20T19:22:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:166: writing smithi04268048-27 from 1515641 to 2000658 tid 2 2024-09-20T19:22:19.060 INFO:tasks.rados.rados.0.smithi042.stdout:166: writing smithi04268048-27 from 2736097 to 3238402 tid 3 2024-09-20T19:22:19.060 INFO:tasks.rados.rados.0.smithi042.stdout:167: write (excl) oid 20 current snap is 21 2024-09-20T19:22:19.060 INFO:tasks.rados.rados.0.smithi042.stdout:166: finishing write tid 1 to smithi04268048-27 2024-09-20T19:22:19.061 INFO:tasks.rados.rados.0.smithi042.stdout:166: finishing write tid 2 to smithi04268048-27 2024-09-20T19:22:19.061 INFO:tasks.rados.rados.0.smithi042.stdout:167: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-20T19:22:19.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:19.075 INFO:tasks.rados.rados.0.smithi042.stdout:167: writing smithi04268048-20 from 594981 to 1204178 tid 1 2024-09-20T19:22:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:19.094 INFO:tasks.rados.rados.0.smithi042.stdout:167: writing smithi04268048-20 from 1974123 to 2719151 tid 2 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:167: writing smithi04268048-20 from 3279873 to 3394981 tid 3 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:168: write (excl) oid 5 current snap is 21 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: finishing write tid 3 to smithi04268048-27 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: finishing write tid 4 to smithi04268048-27 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: finishing write tid 5 to smithi04268048-27 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: oid 27 updating version 0 to 65 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: oid 27 version 65 is already newer than 64 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: oid 27 version 65 is already newer than 63 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:166: oid 27 updating version 65 to 66 2024-09-20T19:22:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 66 (ObjNum 82 snap 21 seq_num 82) dirty exists 2024-09-20T19:22:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:166: left oid 27 (ObjNum 82 snap 21 seq_num 82) 2024-09-20T19:22:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:167: finishing write tid 1 to smithi04268048-20 2024-09-20T19:22:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:168: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-20T19:22:19.112 INFO:tasks.rados.rados.0.smithi042.stdout:168: writing smithi04268048-5 from 611788 to 1012587 tid 1 2024-09-20T19:22:19.125 INFO:tasks.rados.rados.0.smithi042.stdout:168: writing smithi04268048-5 from 1748958 to 2237644 tid 2 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:168: writing smithi04268048-5 from 2907296 to 3411788 tid 3 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:166: done (2 left) 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:169: delete oid 7 current snap is 21 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: finishing write tid 2 to smithi04268048-20 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: finishing write tid 3 to smithi04268048-20 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: finishing write tid 4 to smithi04268048-20 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: finishing write tid 5 to smithi04268048-20 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: oid 20 updating version 0 to 13 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: oid 20 version 13 is already newer than 11 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: oid 20 version 13 is already newer than 10 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: oid 20 version 13 is already newer than 12 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 13 (ObjNum 83 snap 21 seq_num 83) dirty exists 2024-09-20T19:22:19.137 INFO:tasks.rados.rados.0.smithi042.stdout:167: left oid 20 (ObjNum 83 snap 21 seq_num 83) 2024-09-20T19:22:19.138 INFO:tasks.rados.rados.0.smithi042.stdout:168: finishing write tid 1 to smithi04268048-5 2024-09-20T19:22:19.138 INFO:tasks.rados.rados.0.smithi042.stdout:168: finishing write tid 2 to smithi04268048-5 2024-09-20T19:22:19.139 INFO:tasks.rados.rados.0.smithi042.stdout:167: done (2 left) 2024-09-20T19:22:19.139 INFO:tasks.rados.rados.0.smithi042.stdout:169: done (1 left) 2024-09-20T19:22:19.139 INFO:tasks.rados.rados.0.smithi042.stdout:170: snap_remove snap 12 2024-09-20T19:22:19.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:19.215 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.217 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[22706]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.218 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.219 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.220 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:19 smithi042 bash[32075]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:19.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.878007+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.881671+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:17.964154+0000 mgr.y (mgr.24491) 166 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.257141+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.260037+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:18.950852+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.42:0/1379642910' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:18.951485+0000 mon.a (mon.0) 1054 : audit [INF] from='client.24803 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: audit 2024-09-20T19:22:19.001882+0000 mon.a (mon.0) 1055 : audit [INF] from='client.24803 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:19 smithi162 bash[24557]: cluster 2024-09-20T19:22:19.005402+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-20T19:22:19.321 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:19.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:19.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:19.579 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:19.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:19.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:19.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:19.868 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:19.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:19.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:19.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:20.023 INFO:tasks.rados.rados.0.smithi042.stdout:170: done (1 left) 2024-09-20T19:22:20.023 INFO:tasks.rados.rados.0.smithi042.stdout:171: snap_remove snap 18 2024-09-20T19:22:20.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:20.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:20.226 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:20.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:20.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:20 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.621789+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.623483+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.769172+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.771167+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.501 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.868419+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:18.870259+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: audit 2024-09-20T19:22:19.030728+0000 mgr.y (mgr.24491) 167 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:20 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.018963+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-20T19:22:20.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:20.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.646 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:20.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.776 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:20.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:20.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:20.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:21.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:21.016 INFO:tasks.rados.rados.0.smithi042.stdout:171: done (1 left) 2024-09-20T19:22:21.016 INFO:tasks.rados.rados.0.smithi042.stdout:172: rollback oid 19 current snap is 21 2024-09-20T19:22:21.016 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 20 2024-09-20T19:22:21.016 INFO:tasks.rados.rados.0.smithi042.stdout:168: finishing write tid 3 to smithi04268048-5 2024-09-20T19:22:21.016 INFO:tasks.rados.rados.0.smithi042.stdout:168: finishing write tid 4 to smithi04268048-5 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: finishing write tid 5 to smithi04268048-5 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: oid 5 updating version 0 to 23 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: oid 5 version 23 is already newer than 22 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: oid 5 version 23 is already newer than 20 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: oid 5 version 23 is already newer than 21 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 23 (ObjNum 84 snap 21 seq_num 84) dirty exists 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: left oid 5 (ObjNum 84 snap 21 seq_num 84) 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:168: done (1 left) 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:173: read oid 40 snap -1 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:173: expect (ObjNum 71 snap 12 seq_num 71) 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:174: copy_from oid 36 from oid 5 current snap is 21 2024-09-20T19:22:21.017 INFO:tasks.rados.rados.0.smithi042.stdout:175: snap_create 2024-09-20T19:22:21.042 INFO:tasks.rados.rados.0.smithi042.stdout:172: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:22:21.043 INFO:tasks.rados.rados.0.smithi042.stdout:172: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:22:21.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:21.043 INFO:tasks.rados.rados.0.smithi042.stdout:172: finishing rollback tid 2 to smithi04268048-19 2024-09-20T19:22:21.043 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 24 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-20T19:22:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:21.050 INFO:tasks.rados.rados.0.smithi042.stdout:173: expect (ObjNum 71 snap 12 seq_num 71) 2024-09-20T19:22:21.072 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1794117 2024-09-20T19:22:21.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:21.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2547665 2024-09-20T19:22:21.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:21.221 INFO:tasks.rados.rados.0.smithi042.stdout:174: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:22:21.221 INFO:tasks.rados.rados.0.smithi042.stdout:174: finishing copy_from to smithi04268048-36 2024-09-20T19:22:21.221 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 23 (ObjNum 84 snap 21 seq_num 84) dirty exists 2024-09-20T19:22:21.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:21.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:21.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.416 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:21 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:19.964893+0000 mgr.y (mgr.24491) 168 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 134 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.011021+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.013375+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:22:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:21 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.015018+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-20T19:22:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:21.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:21.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:21.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:21.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.692 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:21.795 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:21.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.954 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:21.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:172: done (3 left) 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:173: done (2 left) 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:174: done (1 left) 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:175: done (0 left) 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:176: write oid 24 current snap is 22 2024-09-20T19:22:22.013 INFO:tasks.rados.rados.0.smithi042.stdout:176: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-20T19:22:22.019 INFO:tasks.rados.rados.0.smithi042.stdout:176: writing smithi04268048-24 from 645402 to 1196699 tid 1 2024-09-20T19:22:22.026 INFO:tasks.rados.rados.0.smithi042.stdout:176: writing smithi04268048-24 from 1749569 to 2241924 tid 2 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:176: writing smithi04268048-24 from 2845789 to 3445402 tid 3 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:177: read oid 34 snap -1 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:177: expect deleted 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:178: snap_create 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:176: finishing write tid 1 to smithi04268048-24 2024-09-20T19:22:22.034 INFO:tasks.rados.rados.0.smithi042.stdout:176: finishing write tid 2 to smithi04268048-24 2024-09-20T19:22:22.041 INFO:tasks.rados.rados.0.smithi042.stdout:176: finishing write tid 3 to smithi04268048-24 2024-09-20T19:22:22.041 INFO:tasks.rados.rados.0.smithi042.stdout:176: finishing write tid 4 to smithi04268048-24 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: finishing write tid 5 to smithi04268048-24 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: oid 24 updating version 0 to 22 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: oid 24 version 22 is already newer than 21 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: oid 24 updating version 22 to 23 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: oid 24 version 23 is already newer than 20 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 23 (ObjNum 86 snap 22 seq_num 86) dirty exists 2024-09-20T19:22:22.042 INFO:tasks.rados.rados.0.smithi042.stdout:176: left oid 24 (ObjNum 86 snap 22 seq_num 86) 2024-09-20T19:22:22.057 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:22.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:22.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:22.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:22.237 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:22:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:22:22.255 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:22:22.255 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:22:22.272 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:22:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:22:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:22:22.289 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:22:22.310 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:22:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:22:22.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.322 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:22 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.332 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:22:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:22:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:22.354 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:22:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:22:22.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T19:22:22.376 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:22:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:22:22.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:22.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.669509+0000 osd.0 (osd.0) 91 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:22:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.671192+0000 osd.0 (osd.0) 92 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.889849+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:20.892093+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.051867+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.053891+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.055386+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.c scrub starts 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.057976+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:22 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.012546+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-20T19:22:22.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:22.586 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:22.689 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:22.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:22.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:22.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:22.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:22.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:23.000 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:22:23.006 INFO:tasks.rados.rados.0.smithi042.stdout:176: done (2 left) 2024-09-20T19:22:23.006 INFO:tasks.rados.rados.0.smithi042.stdout:177: done (1 left) 2024-09-20T19:22:23.006 INFO:tasks.rados.rados.0.smithi042.stdout:178: done (0 left) 2024-09-20T19:22:23.006 INFO:tasks.rados.rados.0.smithi042.stdout:179: snap_remove snap 20 2024-09-20T19:22:23.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:23.085 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:23.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:23.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:23.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:23.327 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:22:23.327 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:22:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:23.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:23 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.638505+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.641587+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.756351+0000 osd.6 (osd.6) 63 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.758374+0000 osd.6 (osd.6) 64 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.886311+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.888688+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:21.965785+0000 mgr.y (mgr.24491) 169 : cluster [DBG] pgmap v167: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.058316+0000 osd.5 (osd.5) 87 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:22:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.063861+0000 osd.5 (osd.5) 88 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:22.218524+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? v1:172.21.15.42:0/1948858682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:22.888517+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.999267+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: audit 2024-09-20T19:22:23.000801+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? v1:172.21.15.42:0/2290971246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:23 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.004517+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-20T19:22:23.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:23.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:23.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:23.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:23.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:23.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:23.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:24.021 INFO:tasks.rados.rados.0.smithi042.stdout:179: done (0 left) 2024-09-20T19:22:24.021 INFO:tasks.rados.rados.0.smithi042.stdout:180: write oid 20 current snap is 23 2024-09-20T19:22:24.021 INFO:tasks.rados.rados.0.smithi042.stdout:180: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-20T19:22:24.034 INFO:tasks.rados.rados.0.smithi042.stdout:180: writing smithi04268048-20 from 662209 to 1288755 tid 1 2024-09-20T19:22:24.050 INFO:tasks.rados.rados.0.smithi042.stdout:180: writing smithi04268048-20 from 1808051 to 2444064 tid 2 2024-09-20T19:22:24.065 INFO:tasks.rados.rados.0.smithi042.stdout:180: writing smithi04268048-20 from 2873212 to 3462209 tid 3 2024-09-20T19:22:24.065 INFO:tasks.rados.rados.0.smithi042.stdout:181: write oid 32 current snap is 23 2024-09-20T19:22:24.065 INFO:tasks.rados.rados.0.smithi042.stdout:181: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-20T19:22:24.079 INFO:tasks.rados.rados.0.smithi042.stdout:181: writing smithi04268048-32 from 679016 to 1380811 tid 1 2024-09-20T19:22:24.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:24.089 INFO:tasks.rados.rados.0.smithi042.stdout:181: writing smithi04268048-32 from 1866533 to 2362557 tid 2 2024-09-20T19:22:24.097 INFO:tasks.rados.rados.0.smithi042.stdout:181: writing smithi04268048-32 from 2900635 to 3303278 tid 3 2024-09-20T19:22:24.097 INFO:tasks.rados.rados.0.smithi042.stdout:182: delete oid 17 current snap is 23 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: finishing write tid 1 to smithi04268048-20 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: finishing write tid 2 to smithi04268048-20 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: finishing write tid 3 to smithi04268048-20 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: finishing write tid 4 to smithi04268048-20 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: finishing write tid 5 to smithi04268048-20 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: oid 20 updating version 0 to 23 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: oid 20 updating version 23 to 24 2024-09-20T19:22:24.098 INFO:tasks.rados.rados.0.smithi042.stdout:180: oid 20 version 24 is already newer than 22 2024-09-20T19:22:24.099 INFO:tasks.rados.rados.0.smithi042.stdout:180: oid 20 version 24 is already newer than 21 2024-09-20T19:22:24.099 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 24 (ObjNum 87 snap 23 seq_num 87) dirty exists 2024-09-20T19:22:24.099 INFO:tasks.rados.rados.0.smithi042.stdout:180: left oid 20 (ObjNum 87 snap 23 seq_num 87) 2024-09-20T19:22:24.099 INFO:tasks.rados.rados.0.smithi042.stdout:181: finishing write tid 1 to smithi04268048-32 2024-09-20T19:22:24.099 INFO:tasks.rados.rados.0.smithi042.stdout:181: finishing write tid 2 to smithi04268048-32 2024-09-20T19:22:24.100 INFO:tasks.rados.rados.0.smithi042.stdout:180: done (2 left) 2024-09-20T19:22:24.101 INFO:tasks.rados.rados.0.smithi042.stdout:182: done (1 left) 2024-09-20T19:22:24.101 INFO:tasks.rados.rados.0.smithi042.stdout:183: write (excl) oid 29 current snap is 23 2024-09-20T19:22:24.101 INFO:tasks.rados.rados.0.smithi042.stdout:183: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-20T19:22:24.107 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:24.108 INFO:tasks.rados.rados.0.smithi042.stdout:183: writing smithi04268048-29 from 712630 to 1164923 tid 1 2024-09-20T19:22:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:24.114 INFO:tasks.rados.rados.0.smithi042.stdout:183: writing smithi04268048-29 from 1583497 to 2083190 tid 2 2024-09-20T19:22:24.120 INFO:tasks.rados.rados.0.smithi042.stdout:183: writing smithi04268048-29 from 2555481 to 2980668 tid 3 2024-09-20T19:22:24.120 INFO:tasks.rados.rados.0.smithi042.stdout:184: copy_from oid 34 from oid 7 current snap is 23 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: finishing write tid 3 to smithi04268048-32 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: finishing write tid 4 to smithi04268048-32 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: finishing write tid 5 to smithi04268048-32 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: oid 32 updating version 0 to 44 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: oid 32 version 44 is already newer than 43 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: oid 32 version 44 is already newer than 42 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: oid 32 updating version 44 to 45 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 45 (ObjNum 88 snap 23 seq_num 88) dirty exists 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: left oid 32 (ObjNum 88 snap 23 seq_num 88) 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:183: finishing write tid 1 to smithi04268048-29 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:183: finishing write tid 2 to smithi04268048-29 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:181: done (2 left) 2024-09-20T19:22:24.121 INFO:tasks.rados.rados.0.smithi042.stdout:185: delete oid 48 current snap is 23 2024-09-20T19:22:24.122 INFO:tasks.rados.rados.0.smithi042.stdout:184: finishing copy_from to smithi04268048-34 2024-09-20T19:22:24.122 INFO:tasks.rados.rados.0.smithi042.stdout:184: got expected ENOENT (src dne) 2024-09-20T19:22:24.122 INFO:tasks.rados.rados.0.smithi042.stdout:184: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:22:24.123 INFO:tasks.rados.rados.0.smithi042.stdout:184: done (2 left) 2024-09-20T19:22:24.123 INFO:tasks.rados.rados.0.smithi042.stdout:185: done (1 left) 2024-09-20T19:22:24.123 INFO:tasks.rados.rados.0.smithi042.stdout:186: snap_remove snap 19 2024-09-20T19:22:24.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:24.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:24.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[32075]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:24 smithi042 bash[22706]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:24.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:24.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.594434+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.594593+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.663310+0000 osd.0 (osd.0) 95 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.664800+0000 osd.0 (osd.0) 96 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.764915+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:22.768685+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.977985+0000 osd.4 (osd.4) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.978136+0000 osd.4 (osd.4) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:24 smithi162 bash[24557]: cluster 2024-09-20T19:22:24.010785+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-20T19:22:24.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:24.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:24.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:24.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:24.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:24.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:24.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:24.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:24.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:186: done (1 left) 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:187: rollback oid 26 current snap is 23 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 26 to 14 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: finishing write tid 3 to smithi04268048-29 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: finishing write tid 4 to smithi04268048-29 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: finishing write tid 5 to smithi04268048-29 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: oid 29 updating version 0 to 51 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: oid 29 updating version 51 to 53 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: oid 29 version 53 is already newer than 52 2024-09-20T19:22:25.025 INFO:tasks.rados.rados.0.smithi042.stdout:183: oid 29 updating version 53 to 54 2024-09-20T19:22:25.026 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 54 (ObjNum 90 snap 23 seq_num 90) dirty exists 2024-09-20T19:22:25.026 INFO:tasks.rados.rados.0.smithi042.stdout:183: left oid 29 (ObjNum 90 snap 23 seq_num 90) 2024-09-20T19:22:25.026 INFO:tasks.rados.rados.0.smithi042.stdout:183: done (1 left) 2024-09-20T19:22:25.026 INFO:tasks.rados.rados.0.smithi042.stdout:188: snap_remove snap 16 2024-09-20T19:22:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:25.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:25.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:25.221 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:22:25.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:25.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[22706]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[22706]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[22706]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.271 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.272 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[32075]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.272 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[32075]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.272 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:25 smithi042 bash[32075]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:25.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:25.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:25 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:25 smithi162 bash[24557]: cluster 2024-09-20T19:22:23.966503+0000 mgr.y (mgr.24491) 170 : cluster [DBG] pgmap v170: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T19:22:25.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:25 smithi162 bash[24557]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:25 smithi162 bash[24557]: cluster 2024-09-20T19:22:25.016022+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-20T19:22:25.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:25.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:25.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:25.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:25.683 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:25.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:22:25.787 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:25.857 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:25.961 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:25.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:26.012 INFO:tasks.rados.rados.0.smithi042.stdout:188: done (1 left) 2024-09-20T19:22:26.012 INFO:tasks.rados.rados.0.smithi042.stdout:189: read oid 49 snap -1 2024-09-20T19:22:26.012 INFO:tasks.rados.rados.0.smithi042.stdout:189: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-20T19:22:26.012 INFO:tasks.rados.rados.0.smithi042.stdout:190: write (excl) oid 33 current snap is 23 2024-09-20T19:22:26.012 INFO:tasks.rados.rados.0.smithi042.stdout:190: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-20T19:22:26.023 INFO:tasks.rados.rados.0.smithi042.stdout:190: writing smithi04268048-33 from 746244 to 1465388 tid 1 2024-09-20T19:22:26.030 INFO:tasks.rados.rados.0.smithi042.stdout:190: writing smithi04268048-33 from 2100461 to 2603823 tid 2 2024-09-20T19:22:26.036 INFO:tasks.rados.rados.0.smithi042.stdout:190: writing smithi04268048-33 from 3126680 to 3546244 tid 3 2024-09-20T19:22:26.036 INFO:tasks.rados.rados.0.smithi042.stdout:191: write (excl) oid 38 current snap is 23 2024-09-20T19:22:26.036 INFO:tasks.rados.rados.0.smithi042.stdout:187: finishing rollback tid 0 to smithi04268048-26 2024-09-20T19:22:26.037 INFO:tasks.rados.rados.0.smithi042.stdout:191: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-20T19:22:26.045 INFO:tasks.rados.rados.0.smithi042.stdout:191: writing smithi04268048-38 from 763051 to 1557444 tid 1 2024-09-20T19:22:26.052 INFO:tasks.rados.rados.0.smithi042.stdout:191: writing smithi04268048-38 from 2158943 to 2805963 tid 2 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:191: writing smithi04268048-38 from 3437750 to 3563051 tid 3 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:192: copy_from oid 21 from oid 7 current snap is 23 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:187: finishing rollback tid 1 to smithi04268048-26 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:187: finishing rollback tid 2 to smithi04268048-26 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 25 (ObjNum 72 snap 12 seq_num 72) dirty exists 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:189: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-20T19:22:26.057 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:26.087 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 560400 2024-09-20T19:22:26.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:26.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:26.124 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2071107 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: finishing write tid 1 to smithi04268048-33 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: finishing write tid 2 to smithi04268048-33 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: finishing write tid 3 to smithi04268048-33 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: finishing write tid 4 to smithi04268048-33 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: finishing write tid 5 to smithi04268048-33 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: oid 33 updating version 0 to 40 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: oid 33 updating version 40 to 41 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: oid 33 updating version 41 to 43 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: oid 33 version 43 is already newer than 42 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 43 (ObjNum 92 snap 23 seq_num 92) dirty exists 2024-09-20T19:22:26.131 INFO:tasks.rados.rados.0.smithi042.stdout:190: left oid 33 (ObjNum 92 snap 23 seq_num 92) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: finishing write tid 1 to smithi04268048-38 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:192: finishing copy_from to smithi04268048-21 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:192: got expected ENOENT (src dne) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:192: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: finishing write tid 2 to smithi04268048-38 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: finishing write tid 3 to smithi04268048-38 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: finishing write tid 4 to smithi04268048-38 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:187: done (4 left) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:189: done (3 left) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:190: done (2 left) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:192: done (1 left) 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:193: write (excl) oid 37 current snap is 23 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: finishing write tid 5 to smithi04268048-38 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: oid 38 updating version 0 to 59 2024-09-20T19:22:26.132 INFO:tasks.rados.rados.0.smithi042.stdout:191: oid 38 updating version 59 to 60 2024-09-20T19:22:26.133 INFO:tasks.rados.rados.0.smithi042.stdout:191: oid 38 updating version 60 to 61 2024-09-20T19:22:26.133 INFO:tasks.rados.rados.0.smithi042.stdout:191: oid 38 version 61 is already newer than 58 2024-09-20T19:22:26.133 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 61 (ObjNum 93 snap 23 seq_num 93) dirty exists 2024-09-20T19:22:26.133 INFO:tasks.rados.rados.0.smithi042.stdout:191: left oid 38 (ObjNum 93 snap 23 seq_num 93) 2024-09-20T19:22:26.133 INFO:tasks.rados.rados.0.smithi042.stdout:193: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-20T19:22:26.140 INFO:tasks.rados.rados.0.smithi042.stdout:193: writing smithi04268048-37 from 779858 to 1249500 tid 1 2024-09-20T19:22:26.148 INFO:tasks.rados.rados.0.smithi042.stdout:193: writing smithi04268048-37 from 1817425 to 2324456 tid 2 2024-09-20T19:22:26.156 INFO:tasks.rados.rados.0.smithi042.stdout:193: writing smithi04268048-37 from 2781526 to 3368154 tid 3 2024-09-20T19:22:26.156 INFO:tasks.rados.rados.0.smithi042.stdout:191: done (1 left) 2024-09-20T19:22:26.156 INFO:tasks.rados.rados.0.smithi042.stdout:194: read oid 15 snap -1 2024-09-20T19:22:26.157 INFO:tasks.rados.rados.0.smithi042.stdout:194: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-20T19:22:26.157 INFO:tasks.rados.rados.0.smithi042.stdout:195: read oid 22 snap -1 2024-09-20T19:22:26.157 INFO:tasks.rados.rados.0.smithi042.stdout:195: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T19:22:26.157 INFO:tasks.rados.rados.0.smithi042.stdout:196: snap_remove snap 17 2024-09-20T19:22:26.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:26.242 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:26.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:26.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:26.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:26.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:26.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:26.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:26.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:26.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:26.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:26.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:26.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:26.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:26.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:27.022 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:27.028 INFO:tasks.rados.rados.0.smithi042.stdout:196: done (3 left) 2024-09-20T19:22:27.028 INFO:tasks.rados.rados.0.smithi042.stdout:197: delete oid 17 current snap is 23 2024-09-20T19:22:27.028 INFO:tasks.rados.rados.0.smithi042.stdout:193: finishing write tid 1 to smithi04268048-37 2024-09-20T19:22:27.028 INFO:tasks.rados.rados.0.smithi042.stdout:193: finishing write tid 2 to smithi04268048-37 2024-09-20T19:22:27.028 INFO:tasks.rados.rados.0.smithi042.stdout:193: finishing write tid 3 to smithi04268048-37 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: finishing write tid 4 to smithi04268048-37 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: finishing write tid 5 to smithi04268048-37 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: oid 37 updating version 0 to 37 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: oid 37 version 37 is already newer than 36 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: oid 37 updating version 37 to 38 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: oid 37 version 38 is already newer than 35 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 38 (ObjNum 94 snap 23 seq_num 94) dirty exists 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:193: left oid 37 (ObjNum 94 snap 23 seq_num 94) 2024-09-20T19:22:27.029 INFO:tasks.rados.rados.0.smithi042.stdout:194: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-20T19:22:27.035 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1178520 2024-09-20T19:22:27.051 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1911232 2024-09-20T19:22:27.086 INFO:tasks.rados.rados.0.smithi042.stdout:195: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T19:22:27.112 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:27.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:27.137 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 570290 2024-09-20T19:22:27.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:27.159 INFO:tasks.rados.rados.0.smithi042.stdout:193: done (3 left) 2024-09-20T19:22:27.159 INFO:tasks.rados.rados.0.smithi042.stdout:194: done (2 left) 2024-09-20T19:22:27.159 INFO:tasks.rados.rados.0.smithi042.stdout:195: done (1 left) 2024-09-20T19:22:27.159 INFO:tasks.rados.rados.0.smithi042.stdout:197: done (0 left) 2024-09-20T19:22:27.159 INFO:tasks.rados.rados.0.smithi042.stdout:198: delete oid 20 current snap is 23 2024-09-20T19:22:27.168 INFO:tasks.rados.rados.0.smithi042.stdout:198: done (0 left) 2024-09-20T19:22:27.168 INFO:tasks.rados.rados.0.smithi042.stdout:199: copy_from oid 48 from oid 47 current snap is 23 2024-09-20T19:22:27.168 INFO:tasks.rados.rados.0.smithi042.stdout:200: copy_from oid 34 from oid 25 current snap is 23 2024-09-20T19:22:27.168 INFO:tasks.rados.rados.0.smithi042.stdout:201: copy_from oid 19 from oid 10 current snap is 23 2024-09-20T19:22:27.168 INFO:tasks.rados.rados.0.smithi042.stdout:202: read oid 9 snap -1 2024-09-20T19:22:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:202: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T19:22:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:203: read oid 1 snap 21 2024-09-20T19:22:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:203: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T19:22:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:204: write (excl) oid 45 current snap is 23 2024-09-20T19:22:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:204: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-20T19:22:27.174 INFO:tasks.rados.rados.0.smithi042.stdout:204: writing smithi04268048-45 from 430279 to 1125668 tid 1 2024-09-20T19:22:27.183 INFO:tasks.rados.rados.0.smithi042.stdout:204: writing smithi04268048-45 from 1876518 to 2530876 tid 2 2024-09-20T19:22:27.189 INFO:tasks.rados.rados.0.smithi042.stdout:204: writing smithi04268048-45 from 3031089 to 3567886 tid 3 2024-09-20T19:22:27.189 INFO:tasks.rados.rados.0.smithi042.stdout:205: write oid 24 current snap is 23 2024-09-20T19:22:27.189 INFO:tasks.rados.rados.0.smithi042.stdout:205: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-20T19:22:27.197 INFO:tasks.rados.rados.0.smithi042.stdout:205: writing smithi04268048-24 from 447086 to 1217724 tid 1 2024-09-20T19:22:27.207 INFO:tasks.rados.rados.0.smithi042.stdout:205: writing smithi04268048-24 from 1935000 to 2733016 tid 2 2024-09-20T19:22:27.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:22:27.212 INFO:tasks.rados.rados.0.smithi042.stdout:205: writing smithi04268048-24 from 3458512 to 3647086 tid 3 2024-09-20T19:22:27.212 INFO:tasks.rados.rados.0.smithi042.stdout:206: delete oid 29 current snap is 23 2024-09-20T19:22:27.212 INFO:tasks.rados.rados.0.smithi042.stdout:203: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T19:22:27.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:27.256 INFO:tasks.rados.rados.0.smithi042.stdout:202: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T19:22:27.263 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[32075]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:27 smithi042 bash[22706]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.274 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: cluster 2024-09-20T19:22:25.967012+0000 mgr.y (mgr.24491) 171 : cluster [DBG] pgmap v173: 148 pgs: 4 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 143 active+clean; 146 MiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: cluster 2024-09-20T19:22:26.011542+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: audit 2024-09-20T19:22:26.811154+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:27 smithi162 bash[24557]: audit 2024-09-20T19:22:26.812844+0000 mon.a (mon.0) 1068 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:27.308 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:204: finishing write tid 1 to smithi04268048-45 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:201: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:202: done (7 left) 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:203: done (6 left) 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:206: done (5 left) 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:207: write oid 44 current snap is 23 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:200: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:204: finishing write tid 2 to smithi04268048-45 2024-09-20T19:22:27.341 INFO:tasks.rados.rados.0.smithi042.stdout:207: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-20T19:22:27.343 INFO:tasks.rados.rados.0.smithi042.stdout:207: writing smithi04268048-44 from 480700 to 1118189 tid 1 2024-09-20T19:22:27.349 INFO:tasks.rados.rados.0.smithi042.stdout:207: writing smithi04268048-44 from 1651964 to 2053649 tid 2 2024-09-20T19:22:27.357 INFO:tasks.rados.rados.0.smithi042.stdout:207: writing smithi04268048-44 from 2713358 to 3483971 tid 3 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:208: write oid 9 current snap is 23 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:201: finishing copy_from to smithi04268048-19 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 16 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:199: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:204: finishing write tid 3 to smithi04268048-45 2024-09-20T19:22:27.358 INFO:tasks.rados.rados.0.smithi042.stdout:208: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-20T19:22:27.364 INFO:tasks.rados.rados.0.smithi042.stdout:208: writing smithi04268048-9 from 497507 to 1210245 tid 1 2024-09-20T19:22:27.371 INFO:tasks.rados.rados.0.smithi042.stdout:208: writing smithi04268048-9 from 1710446 to 2372142 tid 2 2024-09-20T19:22:27.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:208: writing smithi04268048-9 from 3140781 to 3697507 tid 3 2024-09-20T19:22:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:201: done (6 left) 2024-09-20T19:22:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:209: read oid 19 snap -1 2024-09-20T19:22:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:209: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:210: snap_remove snap 15 2024-09-20T19:22:27.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:27.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:27.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:27.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:27.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:27.787 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:27.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:27.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:27.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:210: done (7 left) 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:211: read oid 15 snap -1 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:211: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:212: copy_from oid 11 from oid 33 current snap is 23 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:213: read oid 42 snap -1 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:213: expect (ObjNum 76 snap 16 seq_num 76) 2024-09-20T19:22:28.016 INFO:tasks.rados.rados.0.smithi042.stdout:214: read oid 38 snap -1 2024-09-20T19:22:28.017 INFO:tasks.rados.rados.0.smithi042.stdout:214: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-20T19:22:28.017 INFO:tasks.rados.rados.0.smithi042.stdout:215: write oid 2 current snap is 23 2024-09-20T19:22:28.017 INFO:tasks.rados.rados.0.smithi042.stdout:215: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-20T19:22:28.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:28.022 INFO:tasks.rados.rados.0.smithi042.stdout:215: writing smithi04268048-2 from 514314 to 1302301 tid 1 2024-09-20T19:22:28.029 INFO:tasks.rados.rados.0.smithi042.stdout:215: writing smithi04268048-2 from 1768928 to 2174282 tid 2 2024-09-20T19:22:28.037 INFO:tasks.rados.rados.0.smithi042.stdout:215: writing smithi04268048-2 from 2768204 to 3277714 tid 3 2024-09-20T19:22:28.037 INFO:tasks.rados.rados.0.smithi042.stdout:216: write oid 43 current snap is 23 2024-09-20T19:22:28.037 INFO:tasks.rados.rados.0.smithi042.stdout:216: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-20T19:22:28.043 INFO:tasks.rados.rados.0.smithi042.stdout:216: writing smithi04268048-43 from 531121 to 994357 tid 1 2024-09-20T19:22:28.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:28.052 INFO:tasks.rados.rados.0.smithi042.stdout:216: writing smithi04268048-43 from 1711057 to 2376422 tid 2 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:216: writing smithi04268048-43 from 3079274 to 3731121 tid 3 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:217: snap_create 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:199: finishing copy_from to smithi04268048-48 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 25 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:204: finishing write tid 4 to smithi04268048-45 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:204: finishing write tid 5 to smithi04268048-45 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:204: oid 45 updating version 0 to 55 2024-09-20T19:22:28.062 INFO:tasks.rados.rados.0.smithi042.stdout:204: oid 45 version 55 is already newer than 54 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:204: oid 45 version 55 is already newer than 53 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:204: oid 45 updating version 55 to 56 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 56 (ObjNum 97 snap 23 seq_num 97) dirty exists 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:204: left oid 45 (ObjNum 97 snap 23 seq_num 97) 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:200: finishing copy_from to smithi04268048-34 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: finishing write tid 1 to smithi04268048-24 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: finishing write tid 2 to smithi04268048-24 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: finishing write tid 3 to smithi04268048-24 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: finishing write tid 4 to smithi04268048-24 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: finishing write tid 5 to smithi04268048-24 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: oid 24 updating version 0 to 33 2024-09-20T19:22:28.063 INFO:tasks.rados.rados.0.smithi042.stdout:205: oid 24 updating version 33 to 35 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:205: oid 24 version 35 is already newer than 34 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:205: oid 24 updating version 35 to 36 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 36 (ObjNum 98 snap 23 seq_num 98) dirty exists 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:205: left oid 24 (ObjNum 98 snap 23 seq_num 98) 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: finishing write tid 1 to smithi04268048-44 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: finishing write tid 2 to smithi04268048-44 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: finishing write tid 3 to smithi04268048-44 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: finishing write tid 4 to smithi04268048-44 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: finishing write tid 5 to smithi04268048-44 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: oid 44 updating version 0 to 61 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: oid 44 version 61 is already newer than 60 2024-09-20T19:22:28.064 INFO:tasks.rados.rados.0.smithi042.stdout:207: oid 44 version 61 is already newer than 59 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:207: oid 44 version 61 is already newer than 58 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 61 (ObjNum 100 snap 23 seq_num 100) dirty exists 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:207: left oid 44 (ObjNum 100 snap 23 seq_num 100) 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:208: finishing write tid 1 to smithi04268048-9 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:208: finishing write tid 2 to smithi04268048-9 2024-09-20T19:22:28.065 INFO:tasks.rados.rados.0.smithi042.stdout:209: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-20T19:22:28.119 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 78320 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: finishing write tid 3 to smithi04268048-9 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: finishing write tid 4 to smithi04268048-9 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: finishing write tid 5 to smithi04268048-9 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: oid 9 updating version 0 to 78 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: oid 9 updating version 78 to 80 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: oid 9 updating version 80 to 81 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: oid 9 version 81 is already newer than 79 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 81 (ObjNum 101 snap 23 seq_num 101) dirty exists 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:208: left oid 9 (ObjNum 101 snap 23 seq_num 101) 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:211: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-20T19:22:28.149 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:28.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:28.217 INFO:tasks.rados.rados.0.smithi042.stdout:213: expect (ObjNum 76 snap 16 seq_num 76) 2024-09-20T19:22:28.276 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:28.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[22706]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[22706]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[22706]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[22706]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[32075]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.288 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[32075]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[32075]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.289 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:28 smithi042 bash[32075]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:28 smithi162 bash[24557]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:28 smithi162 bash[24557]: cluster 2024-09-20T19:22:27.026969+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-20T19:22:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:28 smithi162 bash[24557]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:28 smithi162 bash[24557]: cluster 2024-09-20T19:22:28.010731+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-20T19:22:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: finishing write tid 1 to smithi04268048-2 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:212: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: finishing write tid 2 to smithi04268048-2 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: finishing write tid 3 to smithi04268048-2 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: finishing write tid 4 to smithi04268048-2 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: finishing write tid 5 to smithi04268048-2 2024-09-20T19:22:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:215: oid 2 updating version 0 to 48 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:215: oid 2 updating version 48 to 50 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:215: oid 2 version 50 is already newer than 49 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:215: oid 2 version 50 is already newer than 47 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 50 (ObjNum 102 snap 23 seq_num 102) dirty exists 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:215: left oid 2 (ObjNum 102 snap 23 seq_num 102) 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:214: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-20T19:22:28.311 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:28.378 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:28.405 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:28.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:28.447 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: finishing write tid 1 to smithi04268048-43 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:212: finishing copy_from to smithi04268048-11 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 43 (ObjNum 92 snap 23 seq_num 92) dirty exists 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: finishing write tid 2 to smithi04268048-43 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: finishing write tid 3 to smithi04268048-43 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: finishing write tid 4 to smithi04268048-43 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: finishing write tid 5 to smithi04268048-43 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: oid 43 updating version 0 to 31 2024-09-20T19:22:28.515 INFO:tasks.rados.rados.0.smithi042.stdout:216: oid 43 updating version 31 to 33 2024-09-20T19:22:28.516 INFO:tasks.rados.rados.0.smithi042.stdout:216: oid 43 version 33 is already newer than 30 2024-09-20T19:22:28.516 INFO:tasks.rados.rados.0.smithi042.stdout:216: oid 43 version 33 is already newer than 32 2024-09-20T19:22:28.516 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 33 (ObjNum 103 snap 23 seq_num 103) dirty exists 2024-09-20T19:22:28.516 INFO:tasks.rados.rados.0.smithi042.stdout:216: left oid 43 (ObjNum 103 snap 23 seq_num 103) 2024-09-20T19:22:28.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:28.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:28.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:28.792 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:28.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:28.948 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:28.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:29.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:29.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:29 smithi042 bash[22706]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:29 smithi042 bash[22706]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:199: done (13 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:200: done (12 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:204: done (11 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:205: done (10 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:207: done (9 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:208: done (8 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:209: done (7 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:211: done (6 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:212: done (5 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:213: done (4 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:214: done (3 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:215: done (2 left) 2024-09-20T19:22:29.074 INFO:tasks.rados.rados.0.smithi042.stdout:216: done (1 left) 2024-09-20T19:22:29.075 INFO:tasks.rados.rados.0.smithi042.stdout:217: done (0 left) 2024-09-20T19:22:29.075 INFO:tasks.rados.rados.0.smithi042.stdout:218: snap_remove snap 13 2024-09-20T19:22:29.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:29.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:29 smithi042 bash[32075]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:29 smithi042 bash[32075]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:29.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:29.192 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:29 smithi162 bash[24557]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.192 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:29 smithi162 bash[24557]: cluster 2024-09-20T19:22:27.967840+0000 mgr.y (mgr.24491) 172 : cluster [DBG] pgmap v176: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:29.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:29.310 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:29.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:29.464 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:29.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.595 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:29.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:29.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:29.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:29.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:29.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:30.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.083 INFO:tasks.rados.rados.0.smithi042.stdout:218: done (0 left) 2024-09-20T19:22:30.083 INFO:tasks.rados.rados.0.smithi042.stdout:219: copy_from oid 12 from oid 49 current snap is 24 2024-09-20T19:22:30.083 INFO:tasks.rados.rados.0.smithi042.stdout:220: delete oid 18 current snap is 24 2024-09-20T19:22:30.097 INFO:tasks.rados.rados.0.smithi042.stdout:220: done (1 left) 2024-09-20T19:22:30.098 INFO:tasks.rados.rados.0.smithi042.stdout:221: delete oid 45 current snap is 24 2024-09-20T19:22:30.103 INFO:tasks.rados.rados.0.smithi042.stdout:221: done (1 left) 2024-09-20T19:22:30.103 INFO:tasks.rados.rados.0.smithi042.stdout:222: rollback oid 43 current snap is 24 2024-09-20T19:22:30.103 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 22 2024-09-20T19:22:30.103 INFO:tasks.rados.rados.0.smithi042.stdout:223: write (excl) oid 37 current snap is 24 2024-09-20T19:22:30.103 INFO:tasks.rados.rados.0.smithi042.stdout:223: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-20T19:22:30.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.117 INFO:tasks.rados.rados.0.smithi042.stdout:223: writing smithi04268048-37 from 581542 to 1270525 tid 1 2024-09-20T19:22:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:30.125 INFO:tasks.rados.rados.0.smithi042.stdout:223: writing smithi04268048-37 from 1886503 to 2299195 tid 2 2024-09-20T19:22:30.135 INFO:tasks.rados.rados.0.smithi042.stdout:223: writing smithi04268048-37 from 2761543 to 3432494 tid 3 2024-09-20T19:22:30.135 INFO:tasks.rados.rados.0.smithi042.stdout:224: read oid 36 snap 14 2024-09-20T19:22:30.135 INFO:tasks.rados.rados.0.smithi042.stdout:224: expect (ObjNum 59 snap 7 seq_num 59) 2024-09-20T19:22:30.135 INFO:tasks.rados.rados.0.smithi042.stdout:225: snap_remove snap 8 2024-09-20T19:22:30.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[32075]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[32075]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[32075]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[32075]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[22706]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[22706]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.243 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[22706]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.244 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:30 smithi042 bash[22706]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:30.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:30.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:30.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:30.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:30 smithi162 bash[24557]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:30 smithi162 bash[24557]: audit 2024-09-20T19:22:29.040503+0000 mgr.y (mgr.24491) 173 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:30.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:30 smithi162 bash[24557]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:30 smithi162 bash[24557]: cluster 2024-09-20T19:22:29.073192+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-20T19:22:30.506 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:30.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:30.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:30.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:30.870 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:30.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:31.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:225: done (4 left) 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:226: copy_from oid 42 from oid 18 current snap is 24 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:227: snap_create 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:219: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:219: finishing copy_from to smithi04268048-12 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T19:22:31.088 INFO:tasks.rados.rados.0.smithi042.stdout:222: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:222: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:222: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 43 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: finishing write tid 1 to smithi04268048-37 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: finishing write tid 2 to smithi04268048-37 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: finishing write tid 3 to smithi04268048-37 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: finishing write tid 4 to smithi04268048-37 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: finishing write tid 5 to smithi04268048-37 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: oid 37 updating version 0 to 54 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: oid 37 version 54 is already newer than 52 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: oid 37 version 54 is already newer than 53 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: oid 37 updating version 54 to 55 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 55 (ObjNum 106 snap 24 seq_num 106) dirty exists 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:223: left oid 37 (ObjNum 106 snap 24 seq_num 106) 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:224: expect (ObjNum 59 snap 7 seq_num 59) 2024-09-20T19:22:31.089 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:31.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:31.146 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:31.159 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:31.256 INFO:tasks.rados.rados.0.smithi042.stdout:226: finishing copy_from to smithi04268048-42 2024-09-20T19:22:31.256 INFO:tasks.rados.rados.0.smithi042.stdout:226: got expected ENOENT (src dne) 2024-09-20T19:22:31.256 INFO:tasks.rados.rados.0.smithi042.stdout:226: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:22:31.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[32075]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[32075]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[32075]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[32075]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[22706]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[22706]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[22706]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:31 smithi042 bash[22706]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:31.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:31.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:31 smithi162 bash[24557]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:31 smithi162 bash[24557]: cluster 2024-09-20T19:22:29.968629+0000 mgr.y (mgr.24491) 174 : cluster [DBG] pgmap v179: 148 pgs: 5 active+clean+snaptrim, 143 active+clean; 157 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-20T19:22:31.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:31 smithi162 bash[24557]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:31 smithi162 bash[24557]: cluster 2024-09-20T19:22:30.082786+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-20T19:22:31.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:31.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:31.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:31.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:31.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:31.892 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:31.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:31.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:32.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:32.040 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T19:22:32.094 INFO:tasks.rados.rados.0.smithi042.stdout:219: done (5 left) 2024-09-20T19:22:32.094 INFO:tasks.rados.rados.0.smithi042.stdout:222: done (4 left) 2024-09-20T19:22:32.094 INFO:tasks.rados.rados.0.smithi042.stdout:223: done (3 left) 2024-09-20T19:22:32.094 INFO:tasks.rados.rados.0.smithi042.stdout:224: done (2 left) 2024-09-20T19:22:32.094 INFO:tasks.rados.rados.0.smithi042.stdout:226: done (1 left) 2024-09-20T19:22:32.095 INFO:tasks.rados.rados.0.smithi042.stdout:227: done (0 left) 2024-09-20T19:22:32.095 INFO:tasks.rados.rados.0.smithi042.stdout:228: delete oid 50 current snap is 25 2024-09-20T19:22:32.103 INFO:tasks.rados.rados.0.smithi042.stdout:228: done (0 left) 2024-09-20T19:22:32.104 INFO:tasks.rados.rados.0.smithi042.stdout:229: write (excl) oid 28 current snap is 25 2024-09-20T19:22:32.104 INFO:tasks.rados.rados.0.smithi042.stdout:229: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-20T19:22:32.110 INFO:tasks.rados.rados.0.smithi042.stdout:229: writing smithi04268048-28 from 615156 to 1170990 tid 1 2024-09-20T19:22:32.115 INFO:tasks.rados.rados.0.smithi042.stdout:229: writing smithi04268048-28 from 1603467 to 2019828 tid 2 2024-09-20T19:22:32.124 INFO:tasks.rados.rados.0.smithi042.stdout:229: writing smithi04268048-28 from 2816389 to 3509884 tid 3 2024-09-20T19:22:32.124 INFO:tasks.rados.rados.0.smithi042.stdout:230: delete oid 26 current snap is 25 2024-09-20T19:22:32.124 INFO:tasks.rados.rados.0.smithi042.stdout:229: finishing write tid 1 to smithi04268048-28 2024-09-20T19:22:32.124 INFO:tasks.rados.rados.0.smithi042.stdout:229: finishing write tid 2 to smithi04268048-28 2024-09-20T19:22:32.127 INFO:tasks.rados.rados.0.smithi042.stdout:230: done (1 left) 2024-09-20T19:22:32.127 INFO:tasks.rados.rados.0.smithi042.stdout:231: snap_remove snap 21 2024-09-20T19:22:32.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:32.168 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:32.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:32.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:32.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:32.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:32 smithi162 bash[24557]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:32 smithi162 bash[24557]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:32 smithi162 bash[24557]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:32 smithi162 bash[24557]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[22706]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[22706]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[22706]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[22706]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[32075]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[32075]: cluster 2024-09-20T19:22:31.086064+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-20T19:22:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[32075]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:32 smithi042 bash[32075]: audit 2024-09-20T19:22:31.958365+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:22:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:32.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.558 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:32.661 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:32.685 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:32.787 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:32.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.945 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:32.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:33.033 INFO:tasks.rados.rados.0.smithi042.stdout:231: done (1 left) 2024-09-20T19:22:33.033 INFO:tasks.rados.rados.0.smithi042.stdout:232: read oid 20 snap 14 2024-09-20T19:22:33.033 INFO:tasks.rados.rados.0.smithi042.stdout:232: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-20T19:22:33.033 INFO:tasks.rados.rados.0.smithi042.stdout:233: write (excl) oid 23 current snap is 25 2024-09-20T19:22:33.034 INFO:tasks.rados.rados.0.smithi042.stdout:233: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-20T19:22:33.035 INFO:tasks.rados.rados.0.smithi042.stdout:233: writing smithi04268048-23 from 648770 to 1355102 tid 1 2024-09-20T19:22:33.043 INFO:tasks.rados.rados.0.smithi042.stdout:233: writing smithi04268048-23 from 2120431 to 2540461 tid 2 2024-09-20T19:22:33.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:233: writing smithi04268048-23 from 3271235 to 3703627 tid 3 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:234: snap_create 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: finishing write tid 3 to smithi04268048-28 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: finishing write tid 4 to smithi04268048-28 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: finishing write tid 5 to smithi04268048-28 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: oid 28 updating version 0 to 42 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: oid 28 updating version 42 to 43 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: oid 28 version 43 is already newer than 40 2024-09-20T19:22:33.052 INFO:tasks.rados.rados.0.smithi042.stdout:229: oid 28 version 43 is already newer than 41 2024-09-20T19:22:33.053 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 43 (ObjNum 108 snap 25 seq_num 108) dirty exists 2024-09-20T19:22:33.053 INFO:tasks.rados.rados.0.smithi042.stdout:229: left oid 28 (ObjNum 108 snap 25 seq_num 108) 2024-09-20T19:22:33.053 INFO:tasks.rados.rados.0.smithi042.stdout:233: finishing write tid 1 to smithi04268048-23 2024-09-20T19:22:33.053 INFO:tasks.rados.rados.0.smithi042.stdout:232: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-20T19:22:33.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:33.094 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 563275 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: finishing write tid 2 to smithi04268048-23 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: finishing write tid 3 to smithi04268048-23 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: finishing write tid 4 to smithi04268048-23 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: finishing write tid 5 to smithi04268048-23 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: oid 23 updating version 0 to 37 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: oid 23 updating version 37 to 42 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: oid 23 version 42 is already newer than 41 2024-09-20T19:22:33.116 INFO:tasks.rados.rados.0.smithi042.stdout:233: oid 23 version 42 is already newer than 40 2024-09-20T19:22:33.117 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 42 (ObjNum 110 snap 25 seq_num 110) dirty exists 2024-09-20T19:22:33.117 INFO:tasks.rados.rados.0.smithi042.stdout:233: left oid 23 (ObjNum 110 snap 25 seq_num 110) 2024-09-20T19:22:33.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:33.198 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:33.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:33.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[32075]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:33 smithi042 bash[22706]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.325 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:33.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:33.456 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:31.970193+0000 mgr.y (mgr.24491) 175 : cluster [DBG] pgmap v182: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: audit 2024-09-20T19:22:32.089599+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.42:0/2431138947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:32.094006+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-20T19:22:33.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:33 smithi162 bash[24557]: cluster 2024-09-20T19:22:33.015993+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-20T19:22:33.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:33.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:33.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:33.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:33.845 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:33.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:33.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:33.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:34.043 INFO:tasks.rados.rados.0.smithi042.stdout:229: done (3 left) 2024-09-20T19:22:34.043 INFO:tasks.rados.rados.0.smithi042.stdout:232: done (2 left) 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:233: done (1 left) 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:234: done (0 left) 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:235: read oid 34 snap -1 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:235: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:236: write (excl) oid 37 current snap is 26 2024-09-20T19:22:34.044 INFO:tasks.rados.rados.0.smithi042.stdout:236: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-20T19:22:34.059 INFO:tasks.rados.rados.0.smithi042.stdout:236: writing smithi04268048-37 from 665577 to 1447158 tid 1 2024-09-20T19:22:34.071 INFO:tasks.rados.rados.0.smithi042.stdout:236: writing smithi04268048-37 from 2062560 to 2626248 tid 2 2024-09-20T19:22:34.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.083 INFO:tasks.rados.rados.0.smithi042.stdout:236: writing smithi04268048-37 from 3182305 to 3825969 tid 3 2024-09-20T19:22:34.083 INFO:tasks.rados.rados.0.smithi042.stdout:237: read oid 11 snap -1 2024-09-20T19:22:34.083 INFO:tasks.rados.rados.0.smithi042.stdout:237: expect (ObjNum 92 snap 23 seq_num 92) 2024-09-20T19:22:34.083 INFO:tasks.rados.rados.0.smithi042.stdout:238: delete oid 25 current snap is 26 2024-09-20T19:22:34.083 INFO:tasks.rados.rados.0.smithi042.stdout:235: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T19:22:34.107 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:34.186 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:34.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:34.226 INFO:tasks.rados.rados.0.smithi042.stdout:237: expect (ObjNum 92 snap 23 seq_num 92) 2024-09-20T19:22:34.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:34.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:34.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: finishing write tid 1 to smithi04268048-37 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: finishing write tid 2 to smithi04268048-37 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: finishing write tid 3 to smithi04268048-37 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: finishing write tid 4 to smithi04268048-37 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: finishing write tid 5 to smithi04268048-37 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: oid 37 updating version 0 to 64 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: oid 37 updating version 64 to 66 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: oid 37 version 66 is already newer than 63 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: oid 37 version 66 is already newer than 65 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 66 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-09-20T19:22:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:236: left oid 37 (ObjNum 111 snap 26 seq_num 111) 2024-09-20T19:22:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:235: done (3 left) 2024-09-20T19:22:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:236: done (2 left) 2024-09-20T19:22:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:237: done (1 left) 2024-09-20T19:22:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:238: done (0 left) 2024-09-20T19:22:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:239: snap_create 2024-09-20T19:22:34.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:34.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T19:22:34.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:34.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:34.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:34.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:34.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:34.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:34.888 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:22:34.888 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:22:34.907 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:22:34.907 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:22:34.924 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:22:34.924 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:22:34.943 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:22:34.943 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:22:34.964 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:22:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:22:34.985 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:22:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:22:34.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.005 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:22:35.005 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:22:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:35.023 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:22:35.023 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:22:35.058 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:22:35.058 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:22:35.058 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:22:35.058 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:22:35.097 INFO:tasks.rados.rados.0.smithi042.stdout:239: done (0 left) 2024-09-20T19:22:35.098 INFO:tasks.rados.rados.0.smithi042.stdout:240: snap_remove snap 22 2024-09-20T19:22:35.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:35.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:35 smithi162 bash[24557]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.334 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[32075]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: cluster 2024-09-20T19:22:33.970878+0000 mgr.y (mgr.24491) 176 : cluster [DBG] pgmap v185: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: cluster 2024-09-20T19:22:34.040889+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:35 smithi042 bash[22706]: audit 2024-09-20T19:22:34.864691+0000 mon.c (mon.2) 39 : audit [DBG] from='client.? v1:172.21.15.42:0/2296718118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:35.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:35.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:35.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:35.523 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:35.626 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:35.653 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:35.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:35.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:35.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:35.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:36.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:22:36.107 INFO:tasks.rados.rados.0.smithi042.stdout:240: done (0 left) 2024-09-20T19:22:36.107 INFO:tasks.rados.rados.0.smithi042.stdout:241: delete oid 20 current snap is 27 2024-09-20T19:22:36.118 INFO:tasks.rados.rados.0.smithi042.stdout:241: done (0 left) 2024-09-20T19:22:36.118 INFO:tasks.rados.rados.0.smithi042.stdout:242: snap_remove snap 25 2024-09-20T19:22:36.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:36.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:36.280 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:36 smithi162 bash[24557]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.280 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:36 smithi162 bash[24557]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.280 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:36 smithi162 bash[24557]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.280 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:36 smithi162 bash[24557]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:36.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.410 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:36.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[32075]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[32075]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[32075]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[32075]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[22706]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[22706]: cluster 2024-09-20T19:22:35.098028+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-20T19:22:36.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[22706]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:36 smithi042 bash[22706]: audit 2024-09-20T19:22:35.377889+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:22:36.427 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:36.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:36.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:36.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:22:36.662 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:36.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.833 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:36.936 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:36.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:37.064 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:37.131 INFO:tasks.rados.rados.0.smithi042.stdout:242: done (0 left) 2024-09-20T19:22:37.131 INFO:tasks.rados.rados.0.smithi042.stdout:243: rollback oid 31 current snap is 27 2024-09-20T19:22:37.132 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 14 2024-09-20T19:22:37.132 INFO:tasks.rados.rados.0.smithi042.stdout:244: rollback oid 19 current snap is 27 2024-09-20T19:22:37.132 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 26 2024-09-20T19:22:37.132 INFO:tasks.rados.rados.0.smithi042.stdout:245: write (excl) oid 26 current snap is 27 2024-09-20T19:22:37.132 INFO:tasks.rados.rados.0.smithi042.stdout:245: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-20T19:22:37.139 INFO:tasks.rados.rados.0.smithi042.stdout:245: writing smithi04268048-26 from 715998 to 1323326 tid 1 2024-09-20T19:22:37.145 INFO:tasks.rados.rados.0.smithi042.stdout:245: writing smithi04268048-26 from 1838006 to 2265374 tid 2 2024-09-20T19:22:37.151 INFO:tasks.rados.rados.0.smithi042.stdout:245: writing smithi04268048-26 from 2864574 to 3342054 tid 3 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:245: writing smithi04268048-26 from 3863835 to 3915998 tid 4 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:244: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:244: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:244: finishing rollback tid 2 to smithi04268048-19 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 55 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 1 to smithi04268048-26 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:243: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:243: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:246: read oid 4 snap -1 2024-09-20T19:22:37.155 INFO:tasks.rados.rados.0.smithi042.stdout:246: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:244: done (3 left) 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:247: copy_from oid 7 from oid 46 current snap is 27 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:243: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 55 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 2 to smithi04268048-26 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 3 to smithi04268048-26 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:243: done (3 left) 2024-09-20T19:22:37.156 INFO:tasks.rados.rados.0.smithi042.stdout:248: delete oid 14 current snap is 27 2024-09-20T19:22:37.157 INFO:tasks.rados.rados.0.smithi042.stdout:248: done (3 left) 2024-09-20T19:22:37.157 INFO:tasks.rados.rados.0.smithi042.stdout:249: read oid 32 snap -1 2024-09-20T19:22:37.157 INFO:tasks.rados.rados.0.smithi042.stdout:249: expect (ObjNum 88 snap 23 seq_num 88) 2024-09-20T19:22:37.157 INFO:tasks.rados.rados.0.smithi042.stdout:250: snap_remove snap 26 2024-09-20T19:22:37.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:37.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[32075]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.205 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.206 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:37 smithi042 bash[22706]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.217 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:37.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.347 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:37.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.478 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:35.971608+0000 mgr.y (mgr.24491) 177 : cluster [DBG] pgmap v188: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 178 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:36.095487+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: audit 2024-09-20T19:22:36.097290+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? v1:172.21.15.42:0/2603951206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:37 smithi162 bash[24557]: cluster 2024-09-20T19:22:36.105423+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-20T19:22:37.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:37.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:37.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:37.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:37.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:37.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:38.017 INFO:tasks.rados.rados.0.smithi042.stdout:250: done (4 left) 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:251: read oid 23 snap 23 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:251: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:252: rollback oid 40 current snap is 27 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 24 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 4 to smithi04268048-26 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 5 to smithi04268048-26 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: finishing write tid 6 to smithi04268048-26 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: oid 26 updating version 0 to 44 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: oid 26 version 44 is already newer than 42 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: oid 26 updating version 44 to 46 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: oid 26 version 46 is already newer than 45 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:245: oid 26 version 46 is already newer than 43 2024-09-20T19:22:38.018 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 46 (ObjNum 114 snap 27 seq_num 114) dirty exists 2024-09-20T19:22:38.019 INFO:tasks.rados.rados.0.smithi042.stdout:245: left oid 26 (ObjNum 114 snap 27 seq_num 114) 2024-09-20T19:22:38.019 INFO:tasks.rados.rados.0.smithi042.stdout:246: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:38.019 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:38.089 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1803585 2024-09-20T19:22:38.092 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:38.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.121 INFO:tasks.rados.rados.0.smithi042.stdout:249: expect (ObjNum 88 snap 23 seq_num 88) 2024-09-20T19:22:38.121 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:38.126 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:38.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:38 smithi162 bash[24557]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:38 smithi162 bash[24557]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:38 smithi162 bash[24557]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:38 smithi162 bash[24557]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.257 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:38.286 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1834308 2024-09-20T19:22:38.322 INFO:tasks.rados.rados.0.smithi042.stdout:247: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:22:38.322 INFO:tasks.rados.rados.0.smithi042.stdout:247: finishing copy_from to smithi04268048-7 2024-09-20T19:22:38.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 34 (ObjNum 64 snap 8 seq_num 64) dirty exists 2024-09-20T19:22:38.323 INFO:tasks.rados.rados.0.smithi042.stdout:251: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T19:22:38.340 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:38.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.365 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:38.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:38.391 INFO:tasks.rados.rados.0.smithi042.stdout:252: finishing rollback tid 0 to smithi04268048-40 2024-09-20T19:22:38.391 INFO:tasks.rados.rados.0.smithi042.stdout:252: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:22:38.391 INFO:tasks.rados.rados.0.smithi042.stdout:252: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 84 (ObjNum 71 snap 12 seq_num 71) dirty exists 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:245: done (5 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:246: done (4 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:247: done (3 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:249: done (2 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:251: done (1 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:252: done (0 left) 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:253: write oid 25 current snap is 27 2024-09-20T19:22:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:253: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-20T19:22:38.411 INFO:tasks.rados.rados.0.smithi042.stdout:253: writing smithi04268048-25 from 749612 to 1223791 tid 1 2024-09-20T19:22:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[22706]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[22706]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[22706]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[22706]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[32075]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[32075]: cluster 2024-09-20T19:22:37.128893+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-20T19:22:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[32075]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:38 smithi042 bash[32075]: cluster 2024-09-20T19:22:38.016143+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-20T19:22:38.430 INFO:tasks.rados.rados.0.smithi042.stdout:253: writing smithi04268048-25 from 1954970 to 2386007 tid 2 2024-09-20T19:22:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:253: writing smithi04268048-25 from 2919420 to 3535797 tid 3 2024-09-20T19:22:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:254: snap_remove snap 23 2024-09-20T19:22:38.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:38.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:38.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:22:38.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:38.645 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:38.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:38.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:38.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:38.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:39.150 INFO:tasks.rados.rados.0.smithi042.stdout:254: done (1 left) 2024-09-20T19:22:39.150 INFO:tasks.rados.rados.0.smithi042.stdout:255: read oid 2 snap 14 2024-09-20T19:22:39.150 INFO:tasks.rados.rados.0.smithi042.stdout:255: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:39.150 INFO:tasks.rados.rados.0.smithi042.stdout:256: snap_remove snap 24 2024-09-20T19:22:39.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:39.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:22:39.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:39.317 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:39.420 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:39 smithi042 bash[22706]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:39 smithi042 bash[22706]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:39 smithi042 bash[32075]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:39 smithi042 bash[32075]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:39.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:39.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:39 smithi162 bash[24557]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:39 smithi162 bash[24557]: cluster 2024-09-20T19:22:37.972373+0000 mgr.y (mgr.24491) 178 : cluster [DBG] pgmap v191: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:39.573 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:39.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:39.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:39.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:39.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:39.936 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:39.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:40.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:40.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:40.153 INFO:tasks.rados.rados.0.smithi042.stdout:256: done (2 left) 2024-09-20T19:22:40.153 INFO:tasks.rados.rados.0.smithi042.stdout:257: delete oid 41 current snap is 27 2024-09-20T19:22:40.153 INFO:tasks.rados.rados.0.smithi042.stdout:253: finishing write tid 1 to smithi04268048-25 2024-09-20T19:22:40.153 INFO:tasks.rados.rados.0.smithi042.stdout:253: finishing write tid 2 to smithi04268048-25 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: finishing write tid 3 to smithi04268048-25 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: finishing write tid 4 to smithi04268048-25 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: finishing write tid 5 to smithi04268048-25 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: oid 25 updating version 0 to 73 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: oid 25 version 73 is already newer than 72 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: oid 25 updating version 73 to 74 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: oid 25 version 74 is already newer than 71 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 74 (ObjNum 116 snap 27 seq_num 116) dirty exists 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:253: left oid 25 (ObjNum 116 snap 27 seq_num 116) 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:255: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:40.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.204 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.205 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:40 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.212 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:40.217 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:40.267 INFO:tasks.rados.rados.0.smithi042.stdout:253: done (2 left) 2024-09-20T19:22:40.267 INFO:tasks.rados.rados.0.smithi042.stdout:255: done (1 left) 2024-09-20T19:22:40.267 INFO:tasks.rados.rados.0.smithi042.stdout:257: done (0 left) 2024-09-20T19:22:40.267 INFO:tasks.rados.rados.0.smithi042.stdout:258: snap_create 2024-09-20T19:22:40.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:40.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: audit 2024-09-20T19:22:39.051625+0000 mgr.y (mgr.24491) 179 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:40.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: cluster 2024-09-20T19:22:39.141889+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-20T19:22:40.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:40 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.143995+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-20T19:22:40.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:40.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:40.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:40.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:40.711 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:40.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:40.790 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:40.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:40.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:40.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:41.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:41.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:41.193 INFO:tasks.rados.rados.0.smithi042.stdout:258: done (0 left) 2024-09-20T19:22:41.193 INFO:tasks.rados.rados.0.smithi042.stdout:259: read oid 12 snap 27 2024-09-20T19:22:41.193 INFO:tasks.rados.rados.0.smithi042.stdout:259: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-20T19:22:41.194 INFO:tasks.rados.rados.0.smithi042.stdout:260: write (excl) oid 36 current snap is 28 2024-09-20T19:22:41.194 INFO:tasks.rados.rados.0.smithi042.stdout:260: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-20T19:22:41.216 INFO:tasks.rados.rados.0.smithi042.stdout:260: writing smithi04268048-36 from 783226 to 1407903 tid 1 2024-09-20T19:22:41.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:41.234 INFO:tasks.rados.rados.0.smithi042.stdout:260: writing smithi04268048-36 from 2071934 to 2506640 tid 2 2024-09-20T19:22:41.238 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:41 smithi042 bash[32075]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.239 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:41 smithi042 bash[32075]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.239 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:41 smithi042 bash[22706]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.239 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:41 smithi042 bash[22706]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.250 INFO:tasks.rados.rados.0.smithi042.stdout:260: writing smithi04268048-36 from 2974266 to 3613187 tid 3 2024-09-20T19:22:41.250 INFO:tasks.rados.rados.0.smithi042.stdout:261: copy_from oid 47 from oid 19 current snap is 28 2024-09-20T19:22:41.251 INFO:tasks.rados.rados.0.smithi042.stdout:259: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-20T19:22:41.251 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:41.280 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.311 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.341 INFO:tasks.rados.rados.0.smithi042.stdout:260: finishing write tid 1 to smithi04268048-36 2024-09-20T19:22:41.341 INFO:tasks.rados.rados.0.smithi042.stdout:260: finishing write tid 2 to smithi04268048-36 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:259: done (2 left) 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:262: read oid 23 snap -1 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:262: expect (ObjNum 110 snap 25 seq_num 110) 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:263: read oid 4 snap -1 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:263: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:264: read oid 8 snap 14 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:264: expect (ObjNum 68 snap 10 seq_num 68) 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:265: write oid 38 current snap is 28 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:260: finishing write tid 3 to smithi04268048-36 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:260: finishing write tid 4 to smithi04268048-36 2024-09-20T19:22:41.342 INFO:tasks.rados.rados.0.smithi042.stdout:265: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-20T19:22:41.347 INFO:tasks.rados.rados.0.smithi042.stdout:265: writing smithi04268048-38 from 400033 to 1099959 tid 1 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:265: writing smithi04268048-38 from 1614063 to 2000033 tid 2 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:266: write (excl) oid 5 current snap is 28 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: finishing write tid 5 to smithi04268048-36 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: oid 36 updating version 0 to 46 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: oid 36 updating version 46 to 48 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: oid 36 updating version 48 to 49 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: oid 36 version 49 is already newer than 47 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 49 (ObjNum 118 snap 28 seq_num 118) dirty exists 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:260: left oid 36 (ObjNum 118 snap 28 seq_num 118) 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:261: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:22:41.353 INFO:tasks.rados.rados.0.smithi042.stdout:261: finishing copy_from to smithi04268048-47 2024-09-20T19:22:41.354 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 55 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T19:22:41.354 INFO:tasks.rados.rados.0.smithi042.stdout:266: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-20T19:22:41.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:41.363 INFO:tasks.rados.rados.0.smithi042.stdout:266: writing smithi04268048-5 from 416840 to 1192015 tid 1 2024-09-20T19:22:41.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:41 smithi162 bash[24557]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:41 smithi162 bash[24557]: cluster 2024-09-20T19:22:39.973070+0000 mgr.y (mgr.24491) 180 : cluster [DBG] pgmap v194: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 169 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:266: writing smithi04268048-5 from 1672545 to 2016840 tid 2 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:260: done (6 left) 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:261: done (5 left) 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:267: read oid 1 snap 27 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:267: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:268: write oid 34 current snap is 28 2024-09-20T19:22:41.371 INFO:tasks.rados.rados.0.smithi042.stdout:268: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-20T19:22:41.378 INFO:tasks.rados.rados.0.smithi042.stdout:268: writing smithi04268048-34 from 433647 to 884071 tid 1 2024-09-20T19:22:41.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:41.387 INFO:tasks.rados.rados.0.smithi042.stdout:268: writing smithi04268048-34 from 1331027 to 1913060 tid 2 2024-09-20T19:22:41.387 INFO:tasks.rados.rados.0.smithi042.stdout:269: read oid 7 snap 27 2024-09-20T19:22:41.387 INFO:tasks.rados.rados.0.smithi042.stdout:269: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:270: rollback oid 50 current snap is 28 2024-09-20T19:22:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 27 2024-09-20T19:22:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:271: write oid 2 current snap is 28 2024-09-20T19:22:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:271: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-20T19:22:41.395 INFO:tasks.rados.rados.0.smithi042.stdout:271: writing smithi04268048-2 from 450454 to 1092480 tid 1 2024-09-20T19:22:41.402 INFO:tasks.rados.rados.0.smithi042.stdout:271: writing smithi04268048-2 from 1505862 to 2050454 tid 2 2024-09-20T19:22:41.402 INFO:tasks.rados.rados.0.smithi042.stdout:272: copy_from oid 21 from oid 41 current snap is 28 2024-09-20T19:22:41.403 INFO:tasks.rados.rados.0.smithi042.stdout:263: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:41.403 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.446 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.469 INFO:tasks.rados.rados.0.smithi042.stdout:264: expect (ObjNum 68 snap 10 seq_num 68) 2024-09-20T19:22:41.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:41.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:41.556 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.604 INFO:tasks.rados.rados.0.smithi042.stdout:265: finishing write tid 1 to smithi04268048-38 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: finishing write tid 2 to smithi04268048-38 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: finishing write tid 3 to smithi04268048-38 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: finishing write tid 4 to smithi04268048-38 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: oid 38 updating version 0 to 91 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: oid 38 version 91 is already newer than 90 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: oid 38 updating version 91 to 92 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 92 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:265: left oid 38 (ObjNum 119 snap 28 seq_num 119) 2024-09-20T19:22:41.605 INFO:tasks.rados.rados.0.smithi042.stdout:262: expect (ObjNum 110 snap 25 seq_num 110) 2024-09-20T19:22:41.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:41.639 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3686475 2024-09-20T19:22:41.641 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:41.677 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3048425 2024-09-20T19:22:41.694 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:41.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:41.756 INFO:tasks.rados.rados.0.smithi042.stdout:266: finishing write tid 1 to smithi04268048-5 2024-09-20T19:22:41.756 INFO:tasks.rados.rados.0.smithi042.stdout:266: finishing write tid 2 to smithi04268048-5 2024-09-20T19:22:41.756 INFO:tasks.rados.rados.0.smithi042.stdout:266: finishing write tid 3 to smithi04268048-5 2024-09-20T19:22:41.756 INFO:tasks.rados.rados.0.smithi042.stdout:267: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T19:22:41.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:266: finishing write tid 4 to smithi04268048-5 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:266: oid 5 updating version 0 to 85 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:266: oid 5 version 85 is already newer than 84 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:266: oid 5 updating version 85 to 86 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 86 (ObjNum 120 snap 28 seq_num 120) dirty exists 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:266: left oid 5 (ObjNum 120 snap 28 seq_num 120) 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:268: finishing write tid 1 to smithi04268048-34 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:262: done (10 left) 2024-09-20T19:22:41.812 INFO:tasks.rados.rados.0.smithi042.stdout:263: done (9 left) 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:264: done (8 left) 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:265: done (7 left) 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:266: done (6 left) 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:267: done (5 left) 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:273: snap_create 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:270: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 0 (ObjNum 33 snap 0 seq_num 1414828832) dirty dne 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: finishing write tid 2 to smithi04268048-34 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: finishing write tid 3 to smithi04268048-34 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: finishing write tid 4 to smithi04268048-34 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: oid 34 updating version 0 to 23 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: oid 34 updating version 23 to 24 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:268: oid 34 updating version 24 to 25 2024-09-20T19:22:41.813 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 25 (ObjNum 121 snap 28 seq_num 121) dirty exists 2024-09-20T19:22:41.814 INFO:tasks.rados.rados.0.smithi042.stdout:268: left oid 34 (ObjNum 121 snap 28 seq_num 121) 2024-09-20T19:22:41.814 INFO:tasks.rados.rados.0.smithi042.stdout:269: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:41.819 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1303498 2024-09-20T19:22:41.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:41.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:41.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: finishing write tid 1 to smithi04268048-2 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:272: finishing copy_from to smithi04268048-21 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:272: got expected ENOENT (src dne) 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:272: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: finishing write tid 2 to smithi04268048-2 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: finishing write tid 3 to smithi04268048-2 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: finishing write tid 4 to smithi04268048-2 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: oid 2 updating version 0 to 69 2024-09-20T19:22:41.943 INFO:tasks.rados.rados.0.smithi042.stdout:271: oid 2 version 69 is already newer than 68 2024-09-20T19:22:41.944 INFO:tasks.rados.rados.0.smithi042.stdout:271: oid 2 version 69 is already newer than 67 2024-09-20T19:22:41.944 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 69 (ObjNum 122 snap 28 seq_num 122) dirty exists 2024-09-20T19:22:41.944 INFO:tasks.rados.rados.0.smithi042.stdout:271: left oid 2 (ObjNum 122 snap 28 seq_num 122) 2024-09-20T19:22:41.998 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:42.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:268: done (5 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:269: done (4 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:270: done (3 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:271: done (2 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:272: done (1 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:273: done (0 left) 2024-09-20T19:22:42.199 INFO:tasks.rados.rados.0.smithi042.stdout:274: delete oid 45 current snap is 29 2024-09-20T19:22:42.201 INFO:tasks.rados.rados.0.smithi042.stdout:274: done (0 left) 2024-09-20T19:22:42.201 INFO:tasks.rados.rados.0.smithi042.stdout:275: write oid 19 current snap is 29 2024-09-20T19:22:42.201 INFO:tasks.rados.rados.0.smithi042.stdout:275: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-20T19:22:42.213 INFO:tasks.rados.rados.0.smithi042.stdout:275: writing smithi04268048-19 from 484068 to 1276592 tid 1 2024-09-20T19:22:42.221 INFO:tasks.rados.rados.0.smithi042.stdout:275: writing smithi04268048-19 from 1906473 to 2084068 tid 2 2024-09-20T19:22:42.221 INFO:tasks.rados.rados.0.smithi042.stdout:276: snap_create 2024-09-20T19:22:42.224 INFO:tasks.rados.rados.0.smithi042.stdout:275: finishing write tid 1 to smithi04268048-19 2024-09-20T19:22:42.224 INFO:tasks.rados.rados.0.smithi042.stdout:275: finishing write tid 2 to smithi04268048-19 2024-09-20T19:22:42.224 INFO:tasks.rados.rados.0.smithi042.stdout:275: finishing write tid 3 to smithi04268048-19 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:275: finishing write tid 4 to smithi04268048-19 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:275: oid 19 updating version 0 to 65 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:275: oid 19 version 65 is already newer than 64 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:275: oid 19 version 65 is already newer than 63 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 65 (ObjNum 124 snap 29 seq_num 124) dirty exists 2024-09-20T19:22:42.225 INFO:tasks.rados.rados.0.smithi042.stdout:275: left oid 19 (ObjNum 124 snap 29 seq_num 124) 2024-09-20T19:22:42.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:42.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:42.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:42.413 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[22706]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:42 smithi042 bash[32075]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:42.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.850553+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:42.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:40.850774+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:42.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: cluster 2024-09-20T19:22:41.180958+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-20T19:22:42.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:42 smithi162 bash[24557]: audit 2024-09-20T19:22:41.840370+0000 mon.a (mon.0) 1089 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:42.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:42.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:42.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:42.826 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:42.910 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:42.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:43.014 INFO:tasks.rados.rados.0.smithi042.stdout:275: done (1 left) 2024-09-20T19:22:43.014 INFO:tasks.rados.rados.0.smithi042.stdout:276: done (0 left) 2024-09-20T19:22:43.014 INFO:tasks.rados.rados.0.smithi042.stdout:277: write (excl) oid 46 current snap is 30 2024-09-20T19:22:43.014 INFO:tasks.rados.rados.0.smithi042.stdout:277: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-20T19:22:43.020 INFO:tasks.rados.rados.0.smithi042.stdout:277: writing smithi04268048-46 from 500875 to 968648 tid 1 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:277: writing smithi04268048-46 from 1564955 to 2100875 tid 2 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:278: copy_from oid 40 from oid 14 current snap is 30 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:279: read oid 6 snap -1 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:279: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:280: write (excl) oid 47 current snap is 30 2024-09-20T19:22:43.028 INFO:tasks.rados.rados.0.smithi042.stdout:280: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-20T19:22:43.034 INFO:tasks.rados.rados.0.smithi042.stdout:280: writing smithi04268048-47 from 517682 to 1060704 tid 1 2024-09-20T19:22:43.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:43.040 INFO:tasks.rados.rados.0.smithi042.stdout:280: writing smithi04268048-47 from 1623437 to 2117682 tid 2 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:281: snap_create 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: finishing write tid 1 to smithi04268048-46 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: finishing write tid 2 to smithi04268048-46 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: finishing write tid 3 to smithi04268048-46 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: finishing write tid 4 to smithi04268048-46 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: oid 46 updating version 0 to 99 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: oid 46 updating version 99 to 100 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: oid 46 version 100 is already newer than 98 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 100 (ObjNum 125 snap 30 seq_num 125) dirty exists 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:277: left oid 46 (ObjNum 125 snap 30 seq_num 125) 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:278: finishing copy_from to smithi04268048-40 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:278: got expected ENOENT (src dne) 2024-09-20T19:22:43.041 INFO:tasks.rados.rados.0.smithi042.stdout:278: finishing copy_from racing read to smithi04268048-40 2024-09-20T19:22:43.048 INFO:tasks.rados.rados.0.smithi042.stdout:280: finishing write tid 1 to smithi04268048-47 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: finishing write tid 2 to smithi04268048-47 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: finishing write tid 3 to smithi04268048-47 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: finishing write tid 4 to smithi04268048-47 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: oid 47 updating version 0 to 103 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: oid 47 updating version 103 to 104 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: oid 47 updating version 104 to 105 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 105 (ObjNum 126 snap 30 seq_num 126) dirty exists 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:280: left oid 47 (ObjNum 126 snap 30 seq_num 126) 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:279: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:22:43.052 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:43.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:43.128 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1064221 2024-09-20T19:22:43.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:43.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:43.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:43 smithi162 bash[24557]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:43.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[32075]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:41.974570+0000 mgr.y (mgr.24491) 181 : cluster [DBG] pgmap v197: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.199008+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:43 smithi042 bash[22706]: cluster 2024-09-20T19:22:43.013882+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-20T19:22:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:43.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:43.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:43.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:43.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:43.840 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:43.841 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:43.944 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:43.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:43.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:44.012 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:277: done (4 left) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:278: done (3 left) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:279: done (2 left) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:280: done (1 left) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:281: done (0 left) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:282: read oid 33 snap -1 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:282: expect (ObjNum 92 snap 23 seq_num 92) 2024-09-20T19:22:44.016 INFO:tasks.rados.rados.0.smithi042.stdout:283: snap_create 2024-09-20T19:22:44.036 INFO:tasks.rados.rados.0.smithi042.stdout:282: expect (ObjNum 92 snap 23 seq_num 92) 2024-09-20T19:22:44.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:44.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:44.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:44.207 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.207 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:44.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:44.344 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[22706]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.345 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:44 smithi042 bash[32075]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:44.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.643662+0000 osd.3 (osd.3) 101 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:42.644018+0000 osd.3 (osd.3) 102 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: audit 2024-09-20T19:22:43.992132+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:22:44.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: audit 2024-09-20T19:22:44.012268+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.42:0/694041280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:22:44.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:44 smithi162 bash[24557]: cluster 2024-09-20T19:22:44.015938+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-20T19:22:44.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:44.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:44.721 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:44.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:44.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:44.881 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:44.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:44.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:45.019 INFO:tasks.rados.rados.0.smithi042.stdout:282: done (1 left) 2024-09-20T19:22:45.019 INFO:tasks.rados.rados.0.smithi042.stdout:283: done (0 left) 2024-09-20T19:22:45.019 INFO:tasks.rados.rados.0.smithi042.stdout:284: read oid 3 snap -1 2024-09-20T19:22:45.020 INFO:tasks.rados.rados.0.smithi042.stdout:284: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:45.020 INFO:tasks.rados.rados.0.smithi042.stdout:285: delete oid 26 current snap is 32 2024-09-20T19:22:45.021 INFO:tasks.rados.rados.0.smithi042.stdout:285: done (1 left) 2024-09-20T19:22:45.021 INFO:tasks.rados.rados.0.smithi042.stdout:286: write oid 47 current snap is 32 2024-09-20T19:22:45.021 INFO:tasks.rados.rados.0.smithi042.stdout:286: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-20T19:22:45.028 INFO:tasks.rados.rados.0.smithi042.stdout:286: writing smithi04268048-47 from 551296 to 1244816 tid 1 2024-09-20T19:22:45.034 INFO:tasks.rados.rados.0.smithi042.stdout:286: writing smithi04268048-47 from 2024048 to 2151296 tid 2 2024-09-20T19:22:45.034 INFO:tasks.rados.rados.0.smithi042.stdout:287: snap_remove snap 31 2024-09-20T19:22:45.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:45.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:45.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:45.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[32075]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[32075]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[32075]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[32075]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[22706]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[22706]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[22706]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:45 smithi042 bash[22706]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:45.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:45.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:45.537 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:45 smithi162 bash[24557]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:45 smithi162 bash[24557]: cluster 2024-09-20T19:22:43.975013+0000 mgr.y (mgr.24491) 182 : cluster [DBG] pgmap v200: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 36 op/s 2024-09-20T19:22:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:45 smithi162 bash[24557]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:45 smithi162 bash[24557]: cluster 2024-09-20T19:22:45.018271+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-20T19:22:45.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:45.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:45.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:45.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:45.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:45.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:46.032 INFO:tasks.rados.rados.0.smithi042.stdout:287: done (2 left) 2024-09-20T19:22:46.032 INFO:tasks.rados.rados.0.smithi042.stdout:288: read oid 7 snap -1 2024-09-20T19:22:46.033 INFO:tasks.rados.rados.0.smithi042.stdout:288: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:46.033 INFO:tasks.rados.rados.0.smithi042.stdout:289: delete oid 23 current snap is 32 2024-09-20T19:22:46.033 INFO:tasks.rados.rados.0.smithi042.stdout:284: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:22:46.034 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:46.037 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 908930 2024-09-20T19:22:46.058 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:46.059 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: finishing write tid 1 to smithi04268048-47 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: finishing write tid 2 to smithi04268048-47 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:284: done (3 left) 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:289: done (2 left) 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:290: snap_create 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: finishing write tid 3 to smithi04268048-47 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: finishing write tid 4 to smithi04268048-47 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: oid 47 updating version 0 to 107 2024-09-20T19:22:46.096 INFO:tasks.rados.rados.0.smithi042.stdout:286: oid 47 updating version 107 to 108 2024-09-20T19:22:46.097 INFO:tasks.rados.rados.0.smithi042.stdout:286: oid 47 updating version 108 to 109 2024-09-20T19:22:46.097 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 109 (ObjNum 128 snap 32 seq_num 128) dirty exists 2024-09-20T19:22:46.097 INFO:tasks.rados.rados.0.smithi042.stdout:286: left oid 47 (ObjNum 128 snap 32 seq_num 128) 2024-09-20T19:22:46.097 INFO:tasks.rados.rados.0.smithi042.stdout:288: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:22:46.145 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:46.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:46.188 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:46.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:46.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:46.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:46.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:46.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:46.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:46.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:46.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:46.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:46.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T19:22:46.794 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:22:46.794 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:22:46.794 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:22:46.795 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:22:46.795 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:22:46.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:46.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:46.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:46.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:46.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:47.026 INFO:tasks.rados.rados.0.smithi042.stdout:286: done (2 left) 2024-09-20T19:22:47.026 INFO:tasks.rados.rados.0.smithi042.stdout:288: done (1 left) 2024-09-20T19:22:47.026 INFO:tasks.rados.rados.0.smithi042.stdout:290: done (0 left) 2024-09-20T19:22:47.027 INFO:tasks.rados.rados.0.smithi042.stdout:291: delete oid 30 current snap is 33 2024-09-20T19:22:47.036 INFO:tasks.rados.rados.0.smithi042.stdout:291: done (0 left) 2024-09-20T19:22:47.036 INFO:tasks.rados.rados.0.smithi042.stdout:292: snap_remove snap 29 2024-09-20T19:22:47.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:47.087 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:47.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:47.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:47.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:47 smithi162 bash[24557]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:47 smithi162 bash[24557]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:47 smithi162 bash[24557]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:47 smithi162 bash[24557]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:47.331 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[32075]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[32075]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[32075]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[32075]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[22706]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[22706]: cluster 2024-09-20T19:22:45.975658+0000 mgr.y (mgr.24491) 183 : cluster [DBG] pgmap v203: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 179 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[22706]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.332 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:47 smithi042 bash[22706]: cluster 2024-09-20T19:22:46.024516+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-20T19:22:47.346 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:47.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:47.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:47.579 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:47.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:47.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:47.843 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:47.863 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:47.868 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:22:47.869 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:22:47.887 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:22:47.887 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:22:47.903 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:22:47.903 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:22:47.920 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:22:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:22:47.944 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:22:47.944 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:22:47.965 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:22:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:22:47.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:47.986 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:22:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:22:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:47.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:48.008 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:22:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:22:48.014 INFO:tasks.rados.rados.0.smithi042.stdout:292: done (0 left) 2024-09-20T19:22:48.014 INFO:tasks.rados.rados.0.smithi042.stdout:293: copy_from oid 46 from oid 21 current snap is 33 2024-09-20T19:22:48.014 INFO:tasks.rados.rados.0.smithi042.stdout:294: read oid 29 snap -1 2024-09-20T19:22:48.014 INFO:tasks.rados.rados.0.smithi042.stdout:294: expect deleted 2024-09-20T19:22:48.014 INFO:tasks.rados.rados.0.smithi042.stdout:295: delete oid 40 current snap is 33 2024-09-20T19:22:48.029 INFO:tasks.rados.rados.0.smithi042.stdout:294: done (2 left) 2024-09-20T19:22:48.029 INFO:tasks.rados.rados.0.smithi042.stdout:295: done (1 left) 2024-09-20T19:22:48.029 INFO:tasks.rados.rados.0.smithi042.stdout:296: snap_create 2024-09-20T19:22:48.060 INFO:tasks.rados.rados.0.smithi042.stdout:293: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:22:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:293: finishing copy_from to smithi04268048-46 2024-09-20T19:22:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 27 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:22:48.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:48.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:48.217 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:48.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:48 smithi162 bash[24557]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:48.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:48.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[22706]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: cluster 2024-09-20T19:22:47.026043+0000 mon.a (mon.0) 1097 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: audit 2024-09-20T19:22:47.850714+0000 mon.a (mon.0) 1098 : audit [DBG] from='client.? v1:172.21.15.42:0/1125961287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:48 smithi042 bash[32075]: cluster 2024-09-20T19:22:48.014313+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-20T19:22:48.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:48.507 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:48.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:48.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:48.770 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:48.873 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:48.900 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:48.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:49.003 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:49.032 INFO:tasks.rados.rados.0.smithi042.stdout:293: done (1 left) 2024-09-20T19:22:49.032 INFO:tasks.rados.rados.0.smithi042.stdout:296: done (0 left) 2024-09-20T19:22:49.032 INFO:tasks.rados.rados.0.smithi042.stdout:297: snap_create 2024-09-20T19:22:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:49.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:49.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:49.192 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:22:49.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:49.274 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[22706]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[22706]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[22706]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[22706]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[32075]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[32075]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[32075]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:49 smithi042 bash[32075]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:49.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:49 smithi162 bash[24557]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:49 smithi162 bash[24557]: cluster 2024-09-20T19:22:47.976165+0000 mgr.y (mgr.24491) 184 : cluster [DBG] pgmap v206: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:49 smithi162 bash[24557]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.297 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:49 smithi162 bash[24557]: cluster 2024-09-20T19:22:49.031760+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-20T19:22:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:49.399 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:49.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:49.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:49.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:49.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:49.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:49.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:49.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:49.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:49.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:50.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:22:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:297: done (0 left) 2024-09-20T19:22:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:298: write oid 2 current snap is 35 2024-09-20T19:22:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:298: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-20T19:22:50.046 INFO:tasks.rados.rados.0.smithi042.stdout:298: writing smithi04268048-2 from 618524 to 1329393 tid 1 2024-09-20T19:22:50.052 INFO:tasks.rados.rados.0.smithi042.stdout:298: writing smithi04268048-2 from 1857976 to 2218524 tid 2 2024-09-20T19:22:50.052 INFO:tasks.rados.rados.0.smithi042.stdout:299: copy_from oid 30 from oid 36 current snap is 35 2024-09-20T19:22:50.052 INFO:tasks.rados.rados.0.smithi042.stdout:300: delete oid 10 current snap is 35 2024-09-20T19:22:50.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:50.062 INFO:tasks.rados.rados.0.smithi042.stdout:300: done (2 left) 2024-09-20T19:22:50.062 INFO:tasks.rados.rados.0.smithi042.stdout:301: read oid 29 snap -1 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:301: expect deleted 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:302: rollback oid 12 current snap is 35 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 28 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:303: rollback oid 34 current snap is 35 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 28 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:304: write oid 20 current snap is 35 2024-09-20T19:22:50.063 INFO:tasks.rados.rados.0.smithi042.stdout:304: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-20T19:22:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:50.069 INFO:tasks.rados.rados.0.smithi042.stdout:304: writing smithi04268048-20 from 652138 to 1113505 tid 1 2024-09-20T19:22:50.076 INFO:tasks.rados.rados.0.smithi042.stdout:304: writing smithi04268048-20 from 1574940 to 2252138 tid 2 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:305: read oid 4 snap 30 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:305: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:306: snap_create 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: finishing write tid 1 to smithi04268048-2 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: finishing write tid 2 to smithi04268048-2 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: finishing write tid 3 to smithi04268048-2 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: finishing write tid 4 to smithi04268048-2 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: oid 2 updating version 0 to 73 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: oid 2 version 73 is already newer than 71 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: oid 2 version 73 is already newer than 72 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 73 (ObjNum 132 snap 35 seq_num 132) dirty exists 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:298: left oid 2 (ObjNum 132 snap 35 seq_num 132) 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:303: finishing rollback tid 0 to smithi04268048-34 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:303: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:22:50.077 INFO:tasks.rados.rados.0.smithi042.stdout:303: finishing rollback tid 2 to smithi04268048-34 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 28 (ObjNum 121 snap 28 seq_num 121) dirty exists 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:302: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:302: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:302: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 63 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:299: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:22:50.078 INFO:tasks.rados.rados.0.smithi042.stdout:304: finishing write tid 1 to smithi04268048-20 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:299: finishing copy_from to smithi04268048-30 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 49 (ObjNum 118 snap 28 seq_num 118) dirty exists 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: finishing write tid 2 to smithi04268048-20 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: finishing write tid 3 to smithi04268048-20 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: finishing write tid 4 to smithi04268048-20 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: oid 20 updating version 0 to 45 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: oid 20 version 45 is already newer than 43 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: oid 20 version 45 is already newer than 44 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 45 (ObjNum 134 snap 35 seq_num 134) dirty exists 2024-09-20T19:22:50.080 INFO:tasks.rados.rados.0.smithi042.stdout:304: left oid 20 (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:22:50.084 INFO:tasks.rados.rados.0.smithi042.stdout:305: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-20T19:22:50.114 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 268418 2024-09-20T19:22:50.146 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1742769 2024-09-20T19:22:50.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:50.289 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.299 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.299 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.299 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.300 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:50 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.061026+0000 mgr.y (mgr.24491) 185 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.616322+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.42:0/796641087' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:49.616664+0000 mon.a (mon.0) 1101 : audit [INF] from='client.24851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.024085+0000 mon.a (mon.0) 1102 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: audit 2024-09-20T19:22:50.026191+0000 mon.a (mon.0) 1103 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:22:50.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:50 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.033314+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-20T19:22:50.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:50.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:50.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:50.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:50.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:50.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:50.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:50.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:50.966 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:50.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:298: done (7 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:299: done (6 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:301: done (5 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:302: done (4 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:303: done (3 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:304: done (2 left) 2024-09-20T19:22:51.061 INFO:tasks.rados.rados.0.smithi042.stdout:305: done (1 left) 2024-09-20T19:22:51.062 INFO:tasks.rados.rados.0.smithi042.stdout:306: done (0 left) 2024-09-20T19:22:51.062 INFO:tasks.rados.rados.0.smithi042.stdout:307: write oid 49 current snap is 36 2024-09-20T19:22:51.062 INFO:tasks.rados.rados.0.smithi042.stdout:307: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-20T19:22:51.065 INFO:tasks.rados.rados.0.smithi042.stdout:307: writing smithi04268048-49 from 668945 to 1205561 tid 1 2024-09-20T19:22:51.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.070 INFO:tasks.rados.rados.0.smithi042.stdout:307: writing smithi04268048-49 from 1633422 to 2124785 tid 2 2024-09-20T19:22:51.070 INFO:tasks.rados.rados.0.smithi042.stdout:308: delete oid 18 current snap is 36 2024-09-20T19:22:51.077 INFO:tasks.rados.rados.0.smithi042.stdout:307: finishing write tid 1 to smithi04268048-49 2024-09-20T19:22:51.077 INFO:tasks.rados.rados.0.smithi042.stdout:307: finishing write tid 2 to smithi04268048-49 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: finishing write tid 3 to smithi04268048-49 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: finishing write tid 4 to smithi04268048-49 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: oid 49 updating version 0 to 108 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: oid 49 version 108 is already newer than 107 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: oid 49 version 108 is already newer than 106 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 108 (ObjNum 135 snap 36 seq_num 135) dirty exists 2024-09-20T19:22:51.078 INFO:tasks.rados.rados.0.smithi042.stdout:307: left oid 49 (ObjNum 135 snap 36 seq_num 135) 2024-09-20T19:22:51.079 INFO:tasks.rados.rados.0.smithi042.stdout:307: done (1 left) 2024-09-20T19:22:51.079 INFO:tasks.rados.rados.0.smithi042.stdout:308: done (0 left) 2024-09-20T19:22:51.080 INFO:tasks.rados.rados.0.smithi042.stdout:309: snap_remove snap 27 2024-09-20T19:22:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:51.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:51.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:51 smithi162 bash[24557]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:51 smithi162 bash[24557]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:51.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:51.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:51.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:51 smithi042 bash[22706]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:51 smithi042 bash[22706]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.343 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:51 smithi042 bash[32075]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.343 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:51 smithi042 bash[32075]: cluster 2024-09-20T19:22:49.976653+0000 mgr.y (mgr.24491) 186 : cluster [DBG] pgmap v209: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 183 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 8.8 MiB/s rd, 936 KiB/s wr, 15 op/s 2024-09-20T19:22:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:51.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:51.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:51.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.610 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:51.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:51.736 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:51.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:51.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:51.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.987 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:51.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:52.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:22:52.074 INFO:tasks.rados.rados.0.smithi042.stdout:309: done (0 left) 2024-09-20T19:22:52.074 INFO:tasks.rados.rados.0.smithi042.stdout:310: snap_remove snap 35 2024-09-20T19:22:52.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:52.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:52.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[32075]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:52 smithi042 bash[22706]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:52.241 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:52.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:52.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.635337+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:50.638241+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.055997+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.088876+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.090436+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:51.338867+0000 mon.a (mon.0) 1106 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.030915+0000 mon.a (mon.0) 1107 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:22:52.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.031948+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:22:52.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:52 smithi162 bash[24557]: audit 2024-09-20T19:22:52.038469+0000 mon.a (mon.0) 1109 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:22:52.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:52.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:52.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:52.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:52.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:52.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:52.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:52.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:52.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:22:52.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:53.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:53.025 INFO:tasks.rados.rados.0.smithi042.stdout:310: done (0 left) 2024-09-20T19:22:53.025 INFO:tasks.rados.rados.0.smithi042.stdout:311: write oid 50 current snap is 36 2024-09-20T19:22:53.025 INFO:tasks.rados.rados.0.smithi042.stdout:311: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-20T19:22:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:53.029 INFO:tasks.rados.rados.0.smithi042.stdout:311: writing smithi04268048-50 from 702559 to 1106026 tid 1 2024-09-20T19:22:53.040 INFO:tasks.rados.rados.0.smithi042.stdout:311: writing smithi04268048-50 from 1750386 to 2245418 tid 2 2024-09-20T19:22:53.040 INFO:tasks.rados.rados.0.smithi042.stdout:312: rollback oid 32 current snap is 36 2024-09-20T19:22:53.040 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 34 2024-09-20T19:22:53.040 INFO:tasks.rados.rados.0.smithi042.stdout:313: rollback oid 29 current snap is 36 2024-09-20T19:22:53.040 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 34 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:314: read oid 37 snap 30 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:314: expect (ObjNum 111 snap 26 seq_num 111) 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:315: read oid 44 snap -1 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:315: expect (ObjNum 100 snap 23 seq_num 100) 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:316: read oid 16 snap 28 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:316: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-20T19:22:53.041 INFO:tasks.rados.rados.0.smithi042.stdout:317: snap_create 2024-09-20T19:22:53.045 INFO:tasks.rados.rados.0.smithi042.stdout:311: finishing write tid 1 to smithi04268048-50 2024-09-20T19:22:53.049 INFO:tasks.rados.rados.0.smithi042.stdout:316: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-20T19:22:53.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:53.132 INFO:tasks.rados.rados.0.smithi042.stdout:311: finishing write tid 2 to smithi04268048-50 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: finishing write tid 3 to smithi04268048-50 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: finishing write tid 4 to smithi04268048-50 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: oid 50 updating version 0 to 116 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: oid 50 updating version 116 to 118 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: oid 50 version 118 is already newer than 117 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 118 (ObjNum 137 snap 36 seq_num 137) dirty exists 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:311: left oid 50 (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:312: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:312: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:312: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 118 (ObjNum 88 snap 23 seq_num 88) dirty exists 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:314: expect (ObjNum 111 snap 26 seq_num 111) 2024-09-20T19:22:53.133 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:53.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:53.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:53 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.219 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 416708 2024-09-20T19:22:53.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:53.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:53.301 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:53.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:53.386 INFO:tasks.rados.rados.0.smithi042.stdout:313: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:22:53.386 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 64 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:22:53.386 INFO:tasks.rados.rados.0.smithi042.stdout:315: expect (ObjNum 100 snap 23 seq_num 100) 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.780417+0000 osd.6 (osd.6) 69 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.781676+0000 osd.6 (osd.6) 70 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.808335+0000 osd.0 (osd.0) 97 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:22:53.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.812047+0000 osd.0 (osd.0) 98 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:51.977923+0000 mgr.y (mgr.24491) 187 : cluster [DBG] pgmap v212: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.074143+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:53 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.016839+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-20T19:22:53.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:53.510 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:22:53.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:53.533 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:53.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:53.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:53.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:53.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:53.798 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:53.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:53.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:53.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:54.031 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:311: done (6 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:312: done (5 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:313: done (4 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:314: done (3 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:315: done (2 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:316: done (1 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:317: done (0 left) 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:318: write oid 4 current snap is 37 2024-09-20T19:22:54.037 INFO:tasks.rados.rados.0.smithi042.stdout:318: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-20T19:22:54.046 INFO:tasks.rados.rados.0.smithi042.stdout:318: writing smithi04268048-4 from 719366 to 1198082 tid 1 2024-09-20T19:22:54.056 INFO:tasks.rados.rados.0.smithi042.stdout:318: writing smithi04268048-4 from 1808868 to 2319366 tid 2 2024-09-20T19:22:54.056 INFO:tasks.rados.rados.0.smithi042.stdout:319: snap_create 2024-09-20T19:22:54.057 INFO:tasks.rados.rados.0.smithi042.stdout:318: finishing write tid 1 to smithi04268048-4 2024-09-20T19:22:54.058 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: finishing write tid 2 to smithi04268048-4 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: finishing write tid 3 to smithi04268048-4 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: finishing write tid 4 to smithi04268048-4 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: oid 4 updating version 0 to 122 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: oid 4 version 122 is already newer than 121 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:318: oid 4 version 122 is already newer than 120 2024-09-20T19:22:54.061 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 122 (ObjNum 138 snap 37 seq_num 138) dirty exists 2024-09-20T19:22:54.062 INFO:tasks.rados.rados.0.smithi042.stdout:318: left oid 4 (ObjNum 138 snap 37 seq_num 138) 2024-09-20T19:22:54.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:54 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:54.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.302 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.764755+0000 osd.2 (osd.2) 79 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.303 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.768106+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.771017+0000 osd.2 (osd.2) 80 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:52.771758+0000 osd.6 (osd.6) 72 : cluster [DBG] 7.c scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.161689+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.163551+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.304 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:54 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.034878+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-20T19:22:54.318 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:54.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:54.450 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:54.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:54.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:54.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:54.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:54.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:54.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:54.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:54.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:318: done (1 left) 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:319: done (0 left) 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:320: rollback oid 15 current snap is 38 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 34 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:321: rollback oid 31 current snap is 38 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 32 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:322: read oid 25 snap 28 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:322: expect (ObjNum 116 snap 27 seq_num 116) 2024-09-20T19:22:55.039 INFO:tasks.rados.rados.0.smithi042.stdout:323: delete oid 38 current snap is 38 2024-09-20T19:22:55.048 INFO:tasks.rados.rados.0.smithi042.stdout:320: finishing rollback tid 0 to smithi04268048-15 2024-09-20T19:22:55.048 INFO:tasks.rados.rados.0.smithi042.stdout:322: expect (ObjNum 116 snap 27 seq_num 116) 2024-09-20T19:22:55.061 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2586460 2024-09-20T19:22:55.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:55.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:55.105 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3388710 2024-09-20T19:22:55.117 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1792180 2024-09-20T19:22:55.147 INFO:tasks.rados.rados.0.smithi042.stdout:320: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:22:55.147 INFO:tasks.rados.rados.0.smithi042.stdout:320: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:22:55.147 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 50 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-20T19:22:55.147 INFO:tasks.rados.rados.0.smithi042.stdout:321: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:22:55.147 INFO:tasks.rados.rados.0.smithi042.stdout:321: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:321: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 61 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:320: done (3 left) 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:321: done (2 left) 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:322: done (1 left) 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:323: done (0 left) 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:324: rollback oid 16 current snap is 38 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 36 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:325: read oid 24 snap -1 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:325: expect (ObjNum 98 snap 23 seq_num 98) 2024-09-20T19:22:55.148 INFO:tasks.rados.rados.0.smithi042.stdout:326: delete oid 22 current snap is 38 2024-09-20T19:22:55.149 INFO:tasks.rados.rados.0.smithi042.stdout:326: done (2 left) 2024-09-20T19:22:55.149 INFO:tasks.rados.rados.0.smithi042.stdout:327: delete oid 23 current snap is 38 2024-09-20T19:22:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:324: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:22:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:324: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:327: done (2 left) 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:328: read oid 42 snap 30 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:328: expect (ObjNum 76 snap 16 seq_num 76) 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:329: rollback oid 30 current snap is 38 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 33 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:330: read oid 17 snap -1 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:330: expect deleted 2024-09-20T19:22:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:331: delete oid 34 current snap is 38 2024-09-20T19:22:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:330: done (5 left) 2024-09-20T19:22:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:331: done (4 left) 2024-09-20T19:22:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:332: copy_from oid 18 from oid 12 current snap is 38 2024-09-20T19:22:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:333: copy_from oid 15 from oid 35 current snap is 38 2024-09-20T19:22:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:334: snap_remove snap 37 2024-09-20T19:22:55.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:55.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:55.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:55.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:55 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.657265+0000 osd.3 (osd.3) 105 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.659866+0000 osd.3 (osd.3) 106 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:53.978623+0000 mgr.y (mgr.24491) 188 : cluster [DBG] pgmap v215: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:55 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.036881+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-20T19:22:55.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:55.479 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:55.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:55.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:55.745 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:22:55.745 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:22:55.833 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:55.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:55.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:56.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:56.093 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.100 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22706]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.108 INFO:tasks.rados.rados.0.smithi042.stdout:334: done (6 left) 2024-09-20T19:22:56.108 INFO:tasks.rados.rados.0.smithi042.stdout:335: write (excl) oid 10 current snap is 38 2024-09-20T19:22:56.108 INFO:tasks.rados.rados.0.smithi042.stdout:335: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-20T19:22:56.114 INFO:tasks.rados.rados.0.smithi042.stdout:335: writing smithi04268048-10 from 403401 to 858362 tid 1 2024-09-20T19:22:56.114 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:56.121 INFO:tasks.rados.rados.0.smithi042.stdout:335: writing smithi04268048-10 from 1584925 to 2090964 tid 2 2024-09-20T19:22:56.122 INFO:tasks.rados.rados.0.smithi042.stdout:336: snap_create 2024-09-20T19:22:56.122 INFO:tasks.rados.rados.0.smithi042.stdout:329: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:22:56.122 INFO:tasks.rados.rados.0.smithi042.stdout:329: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:22:56.122 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 91 (ObjNum 129 snap 0 seq_num 1415166224) dirty dne 2024-09-20T19:22:56.122 INFO:tasks.rados.rados.0.smithi042.stdout:328: expect (ObjNum 76 snap 16 seq_num 76) 2024-09-20T19:22:56.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:56.218 INFO:tasks.rados.rados.0.smithi042.stdout:324: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:22:56.218 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 54 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-20T19:22:56.218 INFO:tasks.rados.rados.0.smithi042.stdout:325: expect (ObjNum 98 snap 23 seq_num 98) 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.241 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.242 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.242 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.242 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[32075]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.242 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:56.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:56.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.849451+0000 osd.2 (osd.2) 81 : cluster [DBG] 7.e scrub starts 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.853368+0000 osd.2 (osd.2) 82 : cluster [DBG] 7.e scrub ok 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.920320+0000 osd.5 (osd.5) 89 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: cluster 2024-09-20T19:22:54.923715+0000 osd.5 (osd.5) 90 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.356 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:56 smithi162 bash[24557]: audit 2024-09-20T19:22:55.907821+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:22:56.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:56.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:22:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:22:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:22:56.447 INFO:tasks.rados.rados.0.smithi042.stdout:332: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:22:56.447 INFO:tasks.rados.rados.0.smithi042.stdout:333: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:22:56.447 INFO:tasks.rados.rados.0.smithi042.stdout:332: finishing copy_from to smithi04268048-18 2024-09-20T19:22:56.447 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 63 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:333: finishing copy_from to smithi04268048-15 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: finishing write tid 1 to smithi04268048-10 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: finishing write tid 2 to smithi04268048-10 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: finishing write tid 3 to smithi04268048-10 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: finishing write tid 4 to smithi04268048-10 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: oid 10 updating version 0 to 110 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: oid 10 version 110 is already newer than 108 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: oid 10 version 110 is already newer than 109 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 110 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-09-20T19:22:56.448 INFO:tasks.rados.rados.0.smithi042.stdout:335: left oid 10 (ObjNum 143 snap 38 seq_num 143) 2024-09-20T19:22:56.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:56.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:56.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:56.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:56.699 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:22:56.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:56.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:56.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:57.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.029 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:57.109 INFO:tasks.rados.rados.0.smithi042.stdout:324: done (7 left) 2024-09-20T19:22:57.109 INFO:tasks.rados.rados.0.smithi042.stdout:325: done (6 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:328: done (5 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:329: done (4 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:332: done (3 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:333: done (2 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:335: done (1 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:336: done (0 left) 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:337: write oid 13 current snap is 39 2024-09-20T19:22:57.110 INFO:tasks.rados.rados.0.smithi042.stdout:337: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-20T19:22:57.114 INFO:tasks.rados.rados.0.smithi042.stdout:337: writing smithi04268048-13 from 420208 to 950418 tid 1 2024-09-20T19:22:57.123 INFO:tasks.rados.rados.0.smithi042.stdout:337: writing smithi04268048-13 from 1527054 to 2293104 tid 2 2024-09-20T19:22:57.123 INFO:tasks.rados.rados.0.smithi042.stdout:338: snap_remove snap 28 2024-09-20T19:22:57.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.157 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:57.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:57.271 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.272 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:57 smithi162 bash[24557]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.296 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[32075]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.844714+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.b scrub starts 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.847300+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.b scrub ok 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:55.979105+0000 mgr.y (mgr.24491) 189 : cluster [DBG] pgmap v218: 148 pgs: 148 active+clean; 194 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: audit 2024-09-20T19:22:56.092921+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? v1:172.21.15.42:0/2491605338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:22:57.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.108093+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-20T19:22:57.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:57 smithi042 bash[22706]: audit 2024-09-20T19:22:56.852109+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:22:57.298 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T19:22:57.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:57.520 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:57.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:57.764 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:57.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:22:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:22:57.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:57.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:57.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:58.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:58.041 INFO:tasks.rados.rados.0.smithi042.stdout:338: done (1 left) 2024-09-20T19:22:58.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: write oid 25 current snap is 39 2024-09-20T19:22:58.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-20T19:22:58.044 INFO:tasks.rados.rados.0.smithi042.stdout:339: writing smithi04268048-25 from 437015 to 1158827 tid 1 2024-09-20T19:22:58.051 INFO:tasks.rados.rados.0.smithi042.stdout:339: writing smithi04268048-25 from 1701889 to 2211597 tid 2 2024-09-20T19:22:58.051 INFO:tasks.rados.rados.0.smithi042.stdout:337: finishing write tid 1 to smithi04268048-13 2024-09-20T19:22:58.052 INFO:tasks.rados.rados.0.smithi042.stdout:337: finishing write tid 2 to smithi04268048-13 2024-09-20T19:22:58.052 INFO:tasks.rados.rados.0.smithi042.stdout:337: finishing write tid 3 to smithi04268048-13 2024-09-20T19:22:58.052 INFO:tasks.rados.rados.0.smithi042.stdout:337: finishing write tid 4 to smithi04268048-13 2024-09-20T19:22:58.052 INFO:tasks.rados.rados.0.smithi042.stdout:337: oid 13 updating version 0 to 54 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:337: oid 13 version 54 is already newer than 53 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:337: oid 13 updating version 54 to 55 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 55 (ObjNum 144 snap 39 seq_num 144) dirty exists 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:337: left oid 13 (ObjNum 144 snap 39 seq_num 144) 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:337: done (1 left) 2024-09-20T19:22:58.053 INFO:tasks.rados.rados.0.smithi042.stdout:340: snap_remove snap 33 2024-09-20T19:22:58.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:58.161 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:58.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:58 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:58.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.796023+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.797046+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.838624+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.840392+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.904959+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:56.906250+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.106455+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-20T19:22:58.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.197590+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:22:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.201935+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:22:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:58 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.023576+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-20T19:22:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:58.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:58.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:22:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:22:58.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:58.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:58.779 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:22:58.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:58.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:58.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:58.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:22:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:58.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:22:59.033 INFO:tasks.rados.rados.0.smithi042.stdout:340: done (1 left) 2024-09-20T19:22:59.033 INFO:tasks.rados.rados.0.smithi042.stdout:341: snap_remove snap 38 2024-09-20T19:22:59.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:59.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:22:59.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:22:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:22:59.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.215 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.216 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:22:59 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.231 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:59.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:59.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.492 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:22:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:57.979771+0000 mgr.y (mgr.24491) 190 : cluster [DBG] pgmap v221: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.176233+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.e scrub starts 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.180275+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.e scrub ok 2024-09-20T19:22:59.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:22:59 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.032631+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-20T19:22:59.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:22:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:22:59.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:59.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:22:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:22:59.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:59.881 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:22:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:22:59.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:22:59.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:00.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:341: done (1 left) 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:342: snap_create 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: finishing write tid 1 to smithi04268048-25 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: finishing write tid 2 to smithi04268048-25 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: finishing write tid 3 to smithi04268048-25 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: finishing write tid 4 to smithi04268048-25 2024-09-20T19:23:00.041 INFO:tasks.rados.rados.0.smithi042.stdout:339: oid 25 updating version 0 to 98 2024-09-20T19:23:00.042 INFO:tasks.rados.rados.0.smithi042.stdout:339: oid 25 updating version 98 to 99 2024-09-20T19:23:00.042 INFO:tasks.rados.rados.0.smithi042.stdout:339: oid 25 updating version 99 to 100 2024-09-20T19:23:00.042 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 100 (ObjNum 145 snap 39 seq_num 145) dirty exists 2024-09-20T19:23:00.042 INFO:tasks.rados.rados.0.smithi042.stdout:339: left oid 25 (ObjNum 145 snap 39 seq_num 145) 2024-09-20T19:23:00.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.143 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:00.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.620641+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.622784+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.798442+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:58.800868+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: audit 2024-09-20T19:22:59.072367+0000 mgr.y (mgr.24491) 191 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.192860+0000 osd.4 (osd.4) 103 : cluster [DBG] 2.f scrub starts 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.196222+0000 osd.4 (osd.4) 104 : cluster [DBG] 2.f scrub ok 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.038580+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-20T19:23:00.389 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:23:00.389 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:23:00.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:00.408 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:23:00.409 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:23:00.428 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:23:00.428 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:23:00.445 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:23:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:23:00.467 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:23:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:23:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:23:00.489 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:23:00.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:00.510 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:23:00.510 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:23:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:23:00.530 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:23:00.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:00.758 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:00.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:00.913 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:00.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:01.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:01.038 INFO:tasks.rados.rados.0.smithi042.stdout:339: done (1 left) 2024-09-20T19:23:01.038 INFO:tasks.rados.rados.0.smithi042.stdout:342: done (0 left) 2024-09-20T19:23:01.038 INFO:tasks.rados.rados.0.smithi042.stdout:343: write (excl) oid 36 current snap is 40 2024-09-20T19:23:01.038 INFO:tasks.rados.rados.0.smithi042.stdout:343: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-20T19:23:01.045 INFO:tasks.rados.rados.0.smithi042.stdout:343: writing smithi04268048-36 from 453822 to 1250883 tid 1 2024-09-20T19:23:01.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:01.052 INFO:tasks.rados.rados.0.smithi042.stdout:343: writing smithi04268048-36 from 1760371 to 2413737 tid 2 2024-09-20T19:23:01.053 INFO:tasks.rados.rados.0.smithi042.stdout:344: write (excl) oid 47 current snap is 40 2024-09-20T19:23:01.053 INFO:tasks.rados.rados.0.smithi042.stdout:343: finishing write tid 1 to smithi04268048-36 2024-09-20T19:23:01.053 INFO:tasks.rados.rados.0.smithi042.stdout:344: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-20T19:23:01.058 INFO:tasks.rados.rados.0.smithi042.stdout:344: writing smithi04268048-47 from 470629 to 942939 tid 1 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:344: writing smithi04268048-47 from 1418853 to 1932230 tid 2 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:345: snap_create 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:343: finishing write tid 2 to smithi04268048-36 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:343: finishing write tid 3 to smithi04268048-36 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:343: finishing write tid 4 to smithi04268048-36 2024-09-20T19:23:01.064 INFO:tasks.rados.rados.0.smithi042.stdout:343: oid 36 updating version 0 to 64 2024-09-20T19:23:01.065 INFO:tasks.rados.rados.0.smithi042.stdout:343: oid 36 version 64 is already newer than 63 2024-09-20T19:23:01.065 INFO:tasks.rados.rados.0.smithi042.stdout:343: oid 36 updating version 64 to 65 2024-09-20T19:23:01.065 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 65 (ObjNum 146 snap 40 seq_num 146) dirty exists 2024-09-20T19:23:01.065 INFO:tasks.rados.rados.0.smithi042.stdout:343: left oid 36 (ObjNum 146 snap 40 seq_num 146) 2024-09-20T19:23:01.071 INFO:tasks.rados.rados.0.smithi042.stdout:344: finishing write tid 1 to smithi04268048-47 2024-09-20T19:23:01.071 INFO:tasks.rados.rados.0.smithi042.stdout:344: finishing write tid 2 to smithi04268048-47 2024-09-20T19:23:01.071 INFO:tasks.rados.rados.0.smithi042.stdout:344: finishing write tid 3 to smithi04268048-47 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:344: finishing write tid 4 to smithi04268048-47 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:344: oid 47 updating version 0 to 131 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:344: oid 47 updating version 131 to 132 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:344: oid 47 version 132 is already newer than 130 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 132 (ObjNum 147 snap 40 seq_num 147) dirty exists 2024-09-20T19:23:01.072 INFO:tasks.rados.rados.0.smithi042.stdout:344: left oid 47 (ObjNum 147 snap 40 seq_num 147) 2024-09-20T19:23:01.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:01.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:01.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:01 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:01.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:01.313 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:01.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:01.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: cluster 2024-09-20T19:22:59.980484+0000 mgr.y (mgr.24491) 192 : cluster [DBG] pgmap v224: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 196 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:23:01.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: audit 2024-09-20T19:23:00.369750+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.42:0/1514693809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:01.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:01 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.037918+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-20T19:23:01.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:01.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:01.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:01.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:01.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:01.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:01.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:01.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:01.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:01.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:02.039 INFO:tasks.rados.rados.0.smithi042.stdout:343: done (2 left) 2024-09-20T19:23:02.039 INFO:tasks.rados.rados.0.smithi042.stdout:344: done (1 left) 2024-09-20T19:23:02.039 INFO:tasks.rados.rados.0.smithi042.stdout:345: done (0 left) 2024-09-20T19:23:02.039 INFO:tasks.rados.rados.0.smithi042.stdout:346: write oid 45 current snap is 41 2024-09-20T19:23:02.039 INFO:tasks.rados.rados.0.smithi042.stdout:346: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-20T19:23:02.044 INFO:tasks.rados.rados.0.smithi042.stdout:346: writing smithi04268048-45 from 487436 to 1034995 tid 1 2024-09-20T19:23:02.053 INFO:tasks.rados.rados.0.smithi042.stdout:346: writing smithi04268048-45 from 1760982 to 2418017 tid 2 2024-09-20T19:23:02.053 INFO:tasks.rados.rados.0.smithi042.stdout:347: read oid 2 snap -1 2024-09-20T19:23:02.053 INFO:tasks.rados.rados.0.smithi042.stdout:347: expect (ObjNum 132 snap 35 seq_num 132) 2024-09-20T19:23:02.053 INFO:tasks.rados.rados.0.smithi042.stdout:348: snap_create 2024-09-20T19:23:02.053 INFO:tasks.rados.rados.0.smithi042.stdout:346: finishing write tid 1 to smithi04268048-45 2024-09-20T19:23:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:346: finishing write tid 2 to smithi04268048-45 2024-09-20T19:23:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:346: finishing write tid 3 to smithi04268048-45 2024-09-20T19:23:02.061 INFO:tasks.rados.rados.0.smithi042.stdout:346: finishing write tid 4 to smithi04268048-45 2024-09-20T19:23:02.062 INFO:tasks.rados.rados.0.smithi042.stdout:346: oid 45 updating version 0 to 130 2024-09-20T19:23:02.062 INFO:tasks.rados.rados.0.smithi042.stdout:346: oid 45 version 130 is already newer than 129 2024-09-20T19:23:02.062 INFO:tasks.rados.rados.0.smithi042.stdout:346: oid 45 updating version 130 to 131 2024-09-20T19:23:02.062 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 131 (ObjNum 148 snap 41 seq_num 148) dirty exists 2024-09-20T19:23:02.062 INFO:tasks.rados.rados.0.smithi042.stdout:346: left oid 45 (ObjNum 148 snap 41 seq_num 148) 2024-09-20T19:23:02.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.069 INFO:tasks.rados.rados.0.smithi042.stdout:347: expect (ObjNum 132 snap 35 seq_num 132) 2024-09-20T19:23:02.069 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:02.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:02.124 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:02.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.207 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.207 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.207 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.207 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:02 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:02.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:02.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:02.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.686107+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.b scrub starts 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:00.687654+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.b scrub ok 2024-09-20T19:23:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.186435+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:23:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.187963+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:23:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:02 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.038676+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-20T19:23:02.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:02.560 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:02.561 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:02.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:02.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:02.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:02.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:02.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:02.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.996 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:02.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:03.025 INFO:tasks.rados.rados.0.smithi042.stdout:346: done (2 left) 2024-09-20T19:23:03.025 INFO:tasks.rados.rados.0.smithi042.stdout:347: done (1 left) 2024-09-20T19:23:03.025 INFO:tasks.rados.rados.0.smithi042.stdout:348: done (0 left) 2024-09-20T19:23:03.025 INFO:tasks.rados.rados.0.smithi042.stdout:349: write (excl) oid 43 current snap is 42 2024-09-20T19:23:03.025 INFO:tasks.rados.rados.0.smithi042.stdout:349: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-20T19:23:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:23:03.031 INFO:tasks.rados.rados.0.smithi042.stdout:349: writing smithi04268048-43 from 504243 to 1127051 tid 1 2024-09-20T19:23:03.040 INFO:tasks.rados.rados.0.smithi042.stdout:349: writing smithi04268048-43 from 1819464 to 2336510 tid 2 2024-09-20T19:23:03.040 INFO:tasks.rados.rados.0.smithi042.stdout:350: copy_from oid 9 from oid 5 current snap is 42 2024-09-20T19:23:03.040 INFO:tasks.rados.rados.0.smithi042.stdout:351: write oid 24 current snap is 42 2024-09-20T19:23:03.040 INFO:tasks.rados.rados.0.smithi042.stdout:351: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-20T19:23:03.047 INFO:tasks.rados.rados.0.smithi042.stdout:351: writing smithi04268048-24 from 521050 to 1219107 tid 1 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:351: writing smithi04268048-24 from 1877946 to 2521050 tid 2 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:352: write oid 36 current snap is 42 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: finishing write tid 1 to smithi04268048-43 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: finishing write tid 2 to smithi04268048-43 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: finishing write tid 3 to smithi04268048-43 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: finishing write tid 4 to smithi04268048-43 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: oid 43 updating version 0 to 73 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: oid 43 version 73 is already newer than 72 2024-09-20T19:23:03.056 INFO:tasks.rados.rados.0.smithi042.stdout:349: oid 43 version 73 is already newer than 71 2024-09-20T19:23:03.057 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 73 (ObjNum 149 snap 42 seq_num 149) dirty exists 2024-09-20T19:23:03.057 INFO:tasks.rados.rados.0.smithi042.stdout:349: left oid 43 (ObjNum 149 snap 42 seq_num 149) 2024-09-20T19:23:03.057 INFO:tasks.rados.rados.0.smithi042.stdout:352: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-20T19:23:03.064 INFO:tasks.rados.rados.0.smithi042.stdout:352: writing smithi04268048-36 from 537857 to 1311163 tid 1 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:352: writing smithi04268048-36 from 1936428 to 2457143 tid 2 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:349: done (3 left) 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:353: copy_from oid 32 from oid 43 current snap is 42 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:351: finishing write tid 1 to smithi04268048-24 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:351: finishing write tid 2 to smithi04268048-24 2024-09-20T19:23:03.072 INFO:tasks.rados.rados.0.smithi042.stdout:351: finishing write tid 3 to smithi04268048-24 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:351: finishing write tid 4 to smithi04268048-24 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:351: oid 24 updating version 0 to 70 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:351: oid 24 version 70 is already newer than 69 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:351: oid 24 updating version 70 to 71 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 71 (ObjNum 150 snap 42 seq_num 150) dirty exists 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:351: left oid 24 (ObjNum 150 snap 42 seq_num 150) 2024-09-20T19:23:03.073 INFO:tasks.rados.rados.0.smithi042.stdout:350: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:350: finishing copy_from to smithi04268048-9 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 86 (ObjNum 120 snap 28 seq_num 120) dirty exists 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:352: finishing write tid 1 to smithi04268048-36 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:350: done (3 left) 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:351: done (2 left) 2024-09-20T19:23:03.074 INFO:tasks.rados.rados.0.smithi042.stdout:354: snap_remove snap 41 2024-09-20T19:23:03.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:03.118 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:03.220 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:03.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.233 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.234 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:03 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.246 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:03.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:03.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:03.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.982279+0000 mgr.y (mgr.24491) 193 : cluster [DBG] pgmap v227: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:03.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.993305+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:23:03.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:01.995244+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.236066+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.238433+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:02.356423+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.42:0/1566460675' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:02.356866+0000 mon.a (mon.0) 1124 : audit [INF] from='client.24869 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.016729+0000 mon.a (mon.0) 1125 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.016778+0000 mon.a (mon.0) 1126 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:03.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: audit 2024-09-20T19:23:03.019939+0000 mon.a (mon.0) 1127 : audit [INF] from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:03.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:03 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.024400+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-20T19:23:03.508 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:03.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.637 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:03.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:03.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:03.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:03.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:04.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:354: done (2 left) 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:355: snap_create 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: finishing write tid 2 to smithi04268048-36 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: finishing write tid 3 to smithi04268048-36 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: finishing write tid 4 to smithi04268048-36 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: oid 36 updating version 0 to 68 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: oid 36 version 68 is already newer than 67 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: oid 36 updating version 68 to 69 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 69 (ObjNum 151 snap 42 seq_num 151) dirty exists 2024-09-20T19:23:04.042 INFO:tasks.rados.rados.0.smithi042.stdout:352: left oid 36 (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:23:04.043 INFO:tasks.rados.rados.0.smithi042.stdout:353: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:23:04.043 INFO:tasks.rados.rados.0.smithi042.stdout:353: finishing copy_from to smithi04268048-32 2024-09-20T19:23:04.043 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 73 (ObjNum 149 snap 42 seq_num 149) dirty exists 2024-09-20T19:23:04.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:23:04.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:04.154 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:04.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.287 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:04.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.402 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:04 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.417 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:04.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.638074+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:02.640258+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.223508+0000 osd.4 (osd.4) 107 : cluster [DBG] 5.e scrub starts 2024-09-20T19:23:04.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.226263+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.e scrub ok 2024-09-20T19:23:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:04 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.041208+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-20T19:23:04.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:04.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:04.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.674 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:04.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:04.802 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:23:04.802 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:23:04.803 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:23:04.803 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:23:04.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:04.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:04.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:05.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:352: done (2 left) 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:353: done (1 left) 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:355: done (0 left) 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:356: read oid 20 snap -1 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:356: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:357: write (excl) oid 27 current snap is 43 2024-09-20T19:23:05.040 INFO:tasks.rados.rados.0.smithi042.stdout:357: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-20T19:23:05.045 INFO:tasks.rados.rados.0.smithi042.stdout:357: writing smithi04268048-27 from 554664 to 1003219 tid 1 2024-09-20T19:23:05.052 INFO:tasks.rados.rados.0.smithi042.stdout:357: writing smithi04268048-27 from 1478557 to 2142930 tid 2 2024-09-20T19:23:05.052 INFO:tasks.rados.rados.0.smithi042.stdout:358: snap_remove snap 32 2024-09-20T19:23:05.065 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:05.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:05.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:05 smithi162 bash[24557]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:05.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[32075]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.689850+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.692527+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.942905+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.d scrub starts 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.945420+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.d scrub ok 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:03.983004+0000 mgr.y (mgr.24491) 194 : cluster [DBG] pgmap v230: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.039399+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:05 smithi042 bash[22706]: audit 2024-09-20T19:23:05.040158+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:23:05.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:05.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:05.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:23:05.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:05.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:05.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:05.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:06.038 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T19:23:06.060 INFO:tasks.rados.rados.0.smithi042.stdout:358: done (2 left) 2024-09-20T19:23:06.060 INFO:tasks.rados.rados.0.smithi042.stdout:359: write oid 42 current snap is 43 2024-09-20T19:23:06.060 INFO:tasks.rados.rados.0.smithi042.stdout:359: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-20T19:23:06.067 INFO:tasks.rados.rados.0.smithi042.stdout:359: writing smithi04268048-42 from 571471 to 1211628 tid 1 2024-09-20T19:23:06.073 INFO:tasks.rados.rados.0.smithi042.stdout:359: writing smithi04268048-42 from 1653392 to 2177776 tid 2 2024-09-20T19:23:06.073 INFO:tasks.rados.rados.0.smithi042.stdout:360: snap_remove snap 14 2024-09-20T19:23:06.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:06.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:06.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:06.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:23:06.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:06.363 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.364 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.365 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:06 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:06.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:06.503 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.310571+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.f scrub starts 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.313222+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.f scrub ok 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.900631+0000 osd.5 (osd.5) 97 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:04.906499+0000 osd.5 (osd.5) 98 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.062366+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.062594+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: audit 2024-09-20T19:23:06.038511+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.42:0/2167339495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:06 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.046237+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-20T19:23:06.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:06.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:06.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.892 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:06.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:06.908 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T19:23:06.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:07.022 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:07.046 INFO:tasks.rados.rados.0.smithi042.stdout:360: done (3 left) 2024-09-20T19:23:07.047 INFO:tasks.rados.rados.0.smithi042.stdout:361: copy_from oid 47 from oid 21 current snap is 43 2024-09-20T19:23:07.053 INFO:tasks.rados.rados.0.smithi042.stdout:362: write oid 9 current snap is 43 2024-09-20T19:23:07.053 INFO:tasks.rados.rados.0.smithi042.stdout:362: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-20T19:23:07.058 INFO:tasks.rados.rados.0.smithi042.stdout:362: writing smithi04268048-9 from 588278 to 1303684 tid 1 2024-09-20T19:23:07.077 INFO:tasks.rados.rados.0.smithi042.stdout:362: writing smithi04268048-9 from 1711874 to 2379916 tid 2 2024-09-20T19:23:07.077 INFO:tasks.rados.rados.0.smithi042.stdout:363: rollback oid 48 current snap is 43 2024-09-20T19:23:07.077 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 42 2024-09-20T19:23:07.077 INFO:tasks.rados.rados.0.smithi042.stdout:356: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:07.077 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 466774 2024-09-20T19:23:07.125 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:07.144 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:07.147 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 889037 2024-09-20T19:23:07.161 INFO:tasks.rados.rados.0.smithi042.stdout:357: finishing write tid 1 to smithi04268048-27 2024-09-20T19:23:07.161 INFO:tasks.rados.rados.0.smithi042.stdout:357: finishing write tid 2 to smithi04268048-27 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:356: done (5 left) 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:364: snap_create 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: finishing write tid 3 to smithi04268048-27 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: finishing write tid 4 to smithi04268048-27 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: oid 27 updating version 0 to 139 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: oid 27 version 139 is already newer than 138 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: oid 27 updating version 139 to 140 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 140 (ObjNum 152 snap 43 seq_num 152) dirty exists 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:357: left oid 27 (ObjNum 152 snap 43 seq_num 152) 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:359: finishing write tid 1 to smithi04268048-42 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:359: finishing write tid 2 to smithi04268048-42 2024-09-20T19:23:07.162 INFO:tasks.rados.rados.0.smithi042.stdout:359: finishing write tid 3 to smithi04268048-42 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:359: finishing write tid 4 to smithi04268048-42 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:359: oid 42 updating version 0 to 148 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:359: oid 42 version 148 is already newer than 147 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:359: oid 42 version 148 is already newer than 146 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 148 (ObjNum 153 snap 43 seq_num 153) dirty exists 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:359: left oid 42 (ObjNum 153 snap 43 seq_num 153) 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:361: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:362: finishing write tid 1 to smithi04268048-9 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:361: finishing copy_from to smithi04268048-47 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 27 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:362: finishing write tid 2 to smithi04268048-9 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:362: finishing write tid 3 to smithi04268048-9 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:362: finishing write tid 4 to smithi04268048-9 2024-09-20T19:23:07.163 INFO:tasks.rados.rados.0.smithi042.stdout:362: oid 9 updating version 0 to 153 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:362: oid 9 version 153 is already newer than 150 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:362: oid 9 updating version 153 to 154 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 154 (ObjNum 154 snap 43 seq_num 154) dirty exists 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:362: left oid 9 (ObjNum 154 snap 43 seq_num 154) 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:363: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:363: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:363: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:23:07.164 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 120 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:23:07.247 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:07.273 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:07.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:07 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:07.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.359090+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.360931+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.718406+0000 osd.0 (osd.0) 105 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.723572+0000 osd.0 (osd.0) 106 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:05.983704+0000 mgr.y (mgr.24491) 195 : cluster [DBG] pgmap v233: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 199 MiB data, 703 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:07 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.044204+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-20T19:23:07.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:07.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:07.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:07.759 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:07.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:07.917 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:07.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:08.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:08.034 INFO:tasks.rados.rados.0.smithi042.stdout:357: done (5 left) 2024-09-20T19:23:08.034 INFO:tasks.rados.rados.0.smithi042.stdout:359: done (4 left) 2024-09-20T19:23:08.034 INFO:tasks.rados.rados.0.smithi042.stdout:361: done (3 left) 2024-09-20T19:23:08.034 INFO:tasks.rados.rados.0.smithi042.stdout:362: done (2 left) 2024-09-20T19:23:08.034 INFO:tasks.rados.rados.0.smithi042.stdout:363: done (1 left) 2024-09-20T19:23:08.035 INFO:tasks.rados.rados.0.smithi042.stdout:364: done (0 left) 2024-09-20T19:23:08.035 INFO:tasks.rados.rados.0.smithi042.stdout:365: read oid 48 snap -1 2024-09-20T19:23:08.035 INFO:tasks.rados.rados.0.smithi042.stdout:365: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:23:08.035 INFO:tasks.rados.rados.0.smithi042.stdout:366: snap_remove snap 39 2024-09-20T19:23:08.053 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:08.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:08.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:08.286 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:08 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:08.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.440 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:08.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.671430+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:06.675666+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:08 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.030948+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-20T19:23:08.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:08.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:08.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:08.800 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:08.828 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:08.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:08.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:09.038 INFO:tasks.rados.rados.0.smithi042.stdout:366: done (1 left) 2024-09-20T19:23:09.038 INFO:tasks.rados.rados.0.smithi042.stdout:367: write (excl) oid 27 current snap is 44 2024-09-20T19:23:09.038 INFO:tasks.rados.rados.0.smithi042.stdout:367: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-20T19:23:09.046 INFO:tasks.rados.rados.0.smithi042.stdout:367: writing smithi04268048-27 from 605085 to 1395740 tid 1 2024-09-20T19:23:09.053 INFO:tasks.rados.rados.0.smithi042.stdout:367: writing smithi04268048-27 from 2170356 to 2605085 tid 2 2024-09-20T19:23:09.054 INFO:tasks.rados.rados.0.smithi042.stdout:368: rollback oid 23 current snap is 44 2024-09-20T19:23:09.054 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 34 2024-09-20T19:23:09.054 INFO:tasks.rados.rados.0.smithi042.stdout:365: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:23:09.055 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:09.058 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1411519 2024-09-20T19:23:09.076 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:09.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:09.106 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 49112 2024-09-20T19:23:09.136 INFO:tasks.rados.rados.0.smithi042.stdout:367: finishing write tid 1 to smithi04268048-27 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:368: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 50 (ObjNum 129 snap 0 seq_num 1415223776) dirty dne 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:367: finishing write tid 2 to smithi04268048-27 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:365: done (2 left) 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:368: done (1 left) 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:369: read oid 17 snap -1 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:369: expect deleted 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:370: read oid 40 snap -1 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:370: expect deleted 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:371: snap_create 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:367: finishing write tid 3 to smithi04268048-27 2024-09-20T19:23:09.137 INFO:tasks.rados.rados.0.smithi042.stdout:367: finishing write tid 4 to smithi04268048-27 2024-09-20T19:23:09.138 INFO:tasks.rados.rados.0.smithi042.stdout:367: oid 27 updating version 0 to 162 2024-09-20T19:23:09.138 INFO:tasks.rados.rados.0.smithi042.stdout:367: oid 27 version 162 is already newer than 160 2024-09-20T19:23:09.138 INFO:tasks.rados.rados.0.smithi042.stdout:367: oid 27 version 162 is already newer than 161 2024-09-20T19:23:09.138 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 162 (ObjNum 155 snap 44 seq_num 155) dirty exists 2024-09-20T19:23:09.138 INFO:tasks.rados.rados.0.smithi042.stdout:367: left oid 27 (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:23:09.181 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:09.206 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:09.308 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.320 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.320 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.320 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:09 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:09.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:09.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:09.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.606847+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:23:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.609274+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:23:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:07.984530+0000 mgr.y (mgr.24491) 196 : cluster [DBG] pgmap v236: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:09 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.034554+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-20T19:23:09.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:09.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.796 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:09.899 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:09.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:10.032 INFO:tasks.rados.rados.0.smithi042.stdout:367: done (3 left) 2024-09-20T19:23:10.032 INFO:tasks.rados.rados.0.smithi042.stdout:369: done (2 left) 2024-09-20T19:23:10.032 INFO:tasks.rados.rados.0.smithi042.stdout:370: done (1 left) 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:371: done (0 left) 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:372: read oid 40 snap -1 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:372: expect deleted 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:373: read oid 19 snap -1 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:373: expect (ObjNum 124 snap 29 seq_num 124) 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:374: write oid 6 current snap is 45 2024-09-20T19:23:10.033 INFO:tasks.rados.rados.0.smithi042.stdout:374: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-20T19:23:10.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:10.038 INFO:tasks.rados.rados.0.smithi042.stdout:374: writing smithi04268048-6 from 621892 to 1087796 tid 1 2024-09-20T19:23:10.047 INFO:tasks.rados.rados.0.smithi042.stdout:374: writing smithi04268048-6 from 1712485 to 2384196 tid 2 2024-09-20T19:23:10.047 INFO:tasks.rados.rados.0.smithi042.stdout:373: expect (ObjNum 124 snap 29 seq_num 124) 2024-09-20T19:23:10.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:10.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:10.172 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:374: finishing write tid 1 to smithi04268048-6 2024-09-20T19:23:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:374: finishing write tid 2 to smithi04268048-6 2024-09-20T19:23:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:374: finishing write tid 3 to smithi04268048-6 2024-09-20T19:23:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:374: finishing write tid 4 to smithi04268048-6 2024-09-20T19:23:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:374: oid 6 updating version 0 to 73 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:374: oid 6 updating version 73 to 75 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:374: oid 6 version 75 is already newer than 74 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 75 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:374: left oid 6 (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:375: read oid 32 snap 34 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:375: expect (ObjNum 88 snap 23 seq_num 88) 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:372: done (3 left) 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:373: done (2 left) 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:374: done (1 left) 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:376: write (excl) oid 37 current snap is 45 2024-09-20T19:23:10.209 INFO:tasks.rados.rados.0.smithi042.stdout:376: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-20T19:23:10.215 INFO:tasks.rados.rados.0.smithi042.stdout:376: writing smithi04268048-37 from 638699 to 1179852 tid 1 2024-09-20T19:23:10.222 INFO:tasks.rados.rados.0.smithi042.stdout:376: writing smithi04268048-37 from 1770967 to 2302689 tid 2 2024-09-20T19:23:10.222 INFO:tasks.rados.rados.0.smithi042.stdout:377: read oid 6 snap -1 2024-09-20T19:23:10.222 INFO:tasks.rados.rados.0.smithi042.stdout:377: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:23:10.222 INFO:tasks.rados.rados.0.smithi042.stdout:378: snap_remove snap 30 2024-09-20T19:23:10.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:10.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:10 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:10.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:10.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.303587+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.305705+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.712365+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:08.714579+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: audit 2024-09-20T19:23:09.081970+0000 mgr.y (mgr.24491) 197 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.435947+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.436230+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:10 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.031519+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-20T19:23:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:10.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:10.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:10.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:10.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:10.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:10.851 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:10.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:10.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:10.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:11.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:11.195 INFO:tasks.rados.rados.0.smithi042.stdout:378: done (3 left) 2024-09-20T19:23:11.195 INFO:tasks.rados.rados.0.smithi042.stdout:379: rollback oid 13 current snap is 45 2024-09-20T19:23:11.195 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 42 2024-09-20T19:23:11.195 INFO:tasks.rados.rados.0.smithi042.stdout:380: snap_create 2024-09-20T19:23:11.195 INFO:tasks.rados.rados.0.smithi042.stdout:375: expect (ObjNum 88 snap 23 seq_num 88) 2024-09-20T19:23:11.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:11.223 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1938668 2024-09-20T19:23:11.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:11 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:11 smithi162 bash[24557]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:11.252 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:11.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:11.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:11.390 INFO:tasks.rados.rados.0.smithi042.stdout:376: finishing write tid 1 to smithi04268048-37 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: finishing write tid 2 to smithi04268048-37 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: finishing write tid 3 to smithi04268048-37 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: finishing write tid 4 to smithi04268048-37 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: oid 37 updating version 0 to 98 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: oid 37 version 98 is already newer than 97 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: oid 37 version 98 is already newer than 96 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 98 (ObjNum 157 snap 45 seq_num 157) dirty exists 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:376: left oid 37 (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:377: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:23:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 408426 2024-09-20T19:23:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:11 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:11 smithi042 bash[22706]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:11 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:11 smithi042 bash[32075]: cluster 2024-09-20T19:23:09.985198+0000 mgr.y (mgr.24491) 198 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 212 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2024-09-20T19:23:11.460 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:11.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:11.470 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:11.486 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:11.512 INFO:tasks.rados.rados.0.smithi042.stdout:379: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:23:11.512 INFO:tasks.rados.rados.0.smithi042.stdout:379: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:23:11.512 INFO:tasks.rados.rados.0.smithi042.stdout:379: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:23:11.512 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 85 (ObjNum 144 snap 39 seq_num 144) dirty exists 2024-09-20T19:23:11.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:11.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:11.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:11.846 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:11.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:11.976 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:12.001 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:12.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:12.187 INFO:tasks.rados.rados.0.smithi042.stdout:375: done (4 left) 2024-09-20T19:23:12.187 INFO:tasks.rados.rados.0.smithi042.stdout:376: done (3 left) 2024-09-20T19:23:12.188 INFO:tasks.rados.rados.0.smithi042.stdout:377: done (2 left) 2024-09-20T19:23:12.188 INFO:tasks.rados.rados.0.smithi042.stdout:379: done (1 left) 2024-09-20T19:23:12.188 INFO:tasks.rados.rados.0.smithi042.stdout:380: done (0 left) 2024-09-20T19:23:12.188 INFO:tasks.rados.rados.0.smithi042.stdout:381: copy_from oid 45 from oid 11 current snap is 46 2024-09-20T19:23:12.188 INFO:tasks.rados.rados.0.smithi042.stdout:382: snap_create 2024-09-20T19:23:12.208 INFO:tasks.rados.rados.0.smithi042.stdout:381: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:23:12.219 INFO:tasks.rados.rados.0.smithi042.stdout:381: finishing copy_from to smithi04268048-45 2024-09-20T19:23:12.220 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 43 (ObjNum 92 snap 23 seq_num 92) dirty exists 2024-09-20T19:23:12.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:12.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:12.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:12.391 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:12.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:12.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:12 smithi162 bash[24557]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:12.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:12.550 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:23:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:23:12.568 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:23:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:23:12.585 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:23:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:23:12.602 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:23:12.602 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:23:12.622 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:23:12.622 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:23:12.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[32075]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.329163+0000 osd.1 (osd.1) 115 : cluster [DBG] 7.b scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.333307+0000 osd.1 (osd.1) 116 : cluster [DBG] 7.b scrub ok 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.707038+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:10.711124+0000 osd.0 (osd.0) 110 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.004256+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:23:12.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.005582+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.189975+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: audit 2024-09-20T19:23:11.871157+0000 mon.a (mon.0) 1139 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: audit 2024-09-20T19:23:11.872487+0000 mon.a (mon.0) 1140 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:12 smithi042 bash[22706]: cluster 2024-09-20T19:23:12.186996+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-20T19:23:12.644 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:23:12.644 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:23:12.645 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:12.665 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:23:12.665 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:23:12.686 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:23:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:23:12.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:12.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:12.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:12.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:12.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:13.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:13.036 INFO:tasks.rados.rados.0.smithi042.stdout:381: done (1 left) 2024-09-20T19:23:13.036 INFO:tasks.rados.rados.0.smithi042.stdout:382: done (0 left) 2024-09-20T19:23:13.036 INFO:tasks.rados.rados.0.smithi042.stdout:383: snap_remove snap 34 2024-09-20T19:23:13.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:13.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:13.168 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:13.271 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:13.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:13.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:13.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:13 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:13.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.303094+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.304472+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.697194+0000 osd.0 (osd.0) 111 : cluster [DBG] 7.f scrub starts 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.701891+0000 osd.0 (osd.0) 112 : cluster [DBG] 7.f scrub ok 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:11.986796+0000 mgr.y (mgr.24491) 199 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: audit 2024-09-20T19:23:12.531379+0000 mon.a (mon.0) 1142 : audit [DBG] from='client.? v1:172.21.15.42:0/2228706466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:13 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.033253+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-20T19:23:13.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:13.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:13.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:13.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:13.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:383: done (0 left) 2024-09-20T19:23:14.052 INFO:tasks.rados.rados.0.smithi042.stdout:384: copy_from oid 13 from oid 29 current snap is 47 2024-09-20T19:23:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:385: write oid 20 current snap is 47 2024-09-20T19:23:14.053 INFO:tasks.rados.rados.0.smithi042.stdout:385: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-20T19:23:14.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.074 INFO:tasks.rados.rados.0.smithi042.stdout:385: writing smithi04268048-20 from 655506 to 1271908 tid 1 2024-09-20T19:23:14.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:385: writing smithi04268048-20 from 1829449 to 2504829 tid 2 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:386: rollback oid 41 current snap is 47 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 42 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:384: finishing copy_from to smithi04268048-13 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:384: got expected ENOENT (src dne) 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:387: write oid 17 current snap is 47 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:384: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:385: finishing write tid 1 to smithi04268048-20 2024-09-20T19:23:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:387: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-20T19:23:14.105 INFO:tasks.rados.rados.0.smithi042.stdout:387: writing smithi04268048-17 from 672313 to 1363964 tid 1 2024-09-20T19:23:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:387: writing smithi04268048-17 from 1887931 to 2306969 tid 2 2024-09-20T19:23:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:384: done (3 left) 2024-09-20T19:23:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:388: read oid 43 snap -1 2024-09-20T19:23:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:388: expect (ObjNum 149 snap 42 seq_num 149) 2024-09-20T19:23:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:389: delete oid 16 current snap is 47 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:386: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: finishing write tid 2 to smithi04268048-20 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: finishing write tid 3 to smithi04268048-20 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: finishing write tid 4 to smithi04268048-20 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: oid 20 updating version 0 to 49 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: oid 20 updating version 49 to 50 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: oid 20 updating version 50 to 51 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 51 (ObjNum 158 snap 47 seq_num 158) dirty exists 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:385: left oid 20 (ObjNum 158 snap 47 seq_num 158) 2024-09-20T19:23:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:387: finishing write tid 1 to smithi04268048-17 2024-09-20T19:23:14.117 INFO:tasks.rados.rados.0.smithi042.stdout:385: done (4 left) 2024-09-20T19:23:14.117 INFO:tasks.rados.rados.0.smithi042.stdout:386: done (3 left) 2024-09-20T19:23:14.117 INFO:tasks.rados.rados.0.smithi042.stdout:389: done (2 left) 2024-09-20T19:23:14.117 INFO:tasks.rados.rados.0.smithi042.stdout:390: write (excl) oid 22 current snap is 47 2024-09-20T19:23:14.117 INFO:tasks.rados.rados.0.smithi042.stdout:390: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-20T19:23:14.124 INFO:tasks.rados.rados.0.smithi042.stdout:390: writing smithi04268048-22 from 705927 to 1264429 tid 1 2024-09-20T19:23:14.131 INFO:tasks.rados.rados.0.smithi042.stdout:390: writing smithi04268048-22 from 2004895 to 2427602 tid 2 2024-09-20T19:23:14.132 INFO:tasks.rados.rados.0.smithi042.stdout:391: snap_create 2024-09-20T19:23:14.132 INFO:tasks.rados.rados.0.smithi042.stdout:387: finishing write tid 2 to smithi04268048-17 2024-09-20T19:23:14.132 INFO:tasks.rados.rados.0.smithi042.stdout:387: finishing write tid 3 to smithi04268048-17 2024-09-20T19:23:14.132 INFO:tasks.rados.rados.0.smithi042.stdout:388: expect (ObjNum 149 snap 42 seq_num 149) 2024-09-20T19:23:14.132 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:14.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:14.247 INFO:tasks.rados.rados.0.smithi042.stdout:387: finishing write tid 4 to smithi04268048-17 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:387: oid 17 updating version 0 to 65 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:387: oid 17 updating version 65 to 66 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:387: oid 17 updating version 66 to 67 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 67 (ObjNum 159 snap 47 seq_num 159) dirty exists 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:387: left oid 17 (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: finishing write tid 1 to smithi04268048-22 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: finishing write tid 2 to smithi04268048-22 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: finishing write tid 3 to smithi04268048-22 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: finishing write tid 4 to smithi04268048-22 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: oid 22 updating version 0 to 20 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: oid 22 version 20 is already newer than 18 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:390: oid 22 version 20 is already newer than 19 2024-09-20T19:23:14.248 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 20 (ObjNum 161 snap 47 seq_num 161) dirty exists 2024-09-20T19:23:14.249 INFO:tasks.rados.rados.0.smithi042.stdout:390: left oid 22 (ObjNum 161 snap 47 seq_num 161) 2024-09-20T19:23:14.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:14.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:14.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:14.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:14.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:14.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:14.735 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:14.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:14.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:14.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:14.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:14.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:15.063 INFO:tasks.rados.rados.0.smithi042.stdout:387: done (3 left) 2024-09-20T19:23:15.064 INFO:tasks.rados.rados.0.smithi042.stdout:388: done (2 left) 2024-09-20T19:23:15.064 INFO:tasks.rados.rados.0.smithi042.stdout:390: done (1 left) 2024-09-20T19:23:15.064 INFO:tasks.rados.rados.0.smithi042.stdout:391: done (0 left) 2024-09-20T19:23:15.064 INFO:tasks.rados.rados.0.smithi042.stdout:392: delete oid 24 current snap is 48 2024-09-20T19:23:15.070 INFO:tasks.rados.rados.0.smithi042.stdout:392: done (0 left) 2024-09-20T19:23:15.070 INFO:tasks.rados.rados.0.smithi042.stdout:393: write oid 5 current snap is 48 2024-09-20T19:23:15.070 INFO:tasks.rados.rados.0.smithi042.stdout:393: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-20T19:23:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:393: writing smithi04268048-5 from 739541 to 1448541 tid 1 2024-09-20T19:23:15.093 INFO:tasks.rados.rados.0.smithi042.stdout:393: writing smithi04268048-5 from 2121859 to 2548235 tid 2 2024-09-20T19:23:15.093 INFO:tasks.rados.rados.0.smithi042.stdout:394: read oid 17 snap -1 2024-09-20T19:23:15.094 INFO:tasks.rados.rados.0.smithi042.stdout:394: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:15.094 INFO:tasks.rados.rados.0.smithi042.stdout:395: snap_remove snap 36 2024-09-20T19:23:15.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:15.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:15 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:15.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.318 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.319 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.620925+0000 osd.6 (osd.6) 75 : cluster [DBG] 7.d scrub starts 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.624260+0000 osd.6 (osd.6) 76 : cluster [DBG] 7.d scrub ok 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:13.987343+0000 mgr.y (mgr.24491) 200 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.022061+0000 osd.0 (osd.0) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.022382+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.320 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.034423+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:23:15.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.039325+0000 osd.4 (osd.4) 110 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:23:15.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.321 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:15 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.050350+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-20T19:23:15.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:15.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:15.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:15.651 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:15.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:15.781 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:15.803 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:23:15.803 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:23:15.884 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:15.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:15.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:16.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:16.079 INFO:tasks.rados.rados.0.smithi042.stdout:395: done (2 left) 2024-09-20T19:23:16.079 INFO:tasks.rados.rados.0.smithi042.stdout:396: write (excl) oid 16 current snap is 48 2024-09-20T19:23:16.079 INFO:tasks.rados.rados.0.smithi042.stdout:396: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-20T19:23:16.089 INFO:tasks.rados.rados.0.smithi042.stdout:396: writing smithi04268048-16 from 756348 to 1540597 tid 1 2024-09-20T19:23:16.095 INFO:tasks.rados.rados.0.smithi042.stdout:396: writing smithi04268048-16 from 2063988 to 2750375 tid 2 2024-09-20T19:23:16.095 INFO:tasks.rados.rados.0.smithi042.stdout:397: copy_from oid 48 from oid 14 current snap is 48 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:393: finishing write tid 1 to smithi04268048-5 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:398: read oid 38 snap -1 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:398: expect deleted 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:399: write oid 12 current snap is 48 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:393: finishing write tid 2 to smithi04268048-5 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:393: finishing write tid 3 to smithi04268048-5 2024-09-20T19:23:16.096 INFO:tasks.rados.rados.0.smithi042.stdout:399: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-20T19:23:16.102 INFO:tasks.rados.rados.0.smithi042.stdout:399: writing smithi04268048-12 from 773155 to 1232653 tid 1 2024-09-20T19:23:16.107 INFO:tasks.rados.rados.0.smithi042.stdout:399: writing smithi04268048-12 from 1722470 to 2152515 tid 2 2024-09-20T19:23:16.111 INFO:tasks.rados.rados.0.smithi042.stdout:399: writing smithi04268048-12 from 2648676 to 2773155 tid 3 2024-09-20T19:23:16.111 INFO:tasks.rados.rados.0.smithi042.stdout:400: read oid 34 snap -1 2024-09-20T19:23:16.111 INFO:tasks.rados.rados.0.smithi042.stdout:400: expect deleted 2024-09-20T19:23:16.111 INFO:tasks.rados.rados.0.smithi042.stdout:401: snap_remove snap 47 2024-09-20T19:23:16.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:16.159 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:16.163 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.164 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.165 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.323499+0000 osd.7 (osd.7) 81 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:23:16.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.325770+0000 osd.7 (osd.7) 82 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.864778+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:14.865019+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:16 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.049432+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-20T19:23:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:16.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:16.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:16.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:16.651 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:16.679 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:16.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:16.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:23:16.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:16.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:17.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:17.068 INFO:tasks.rados.rados.0.smithi042.stdout:401: done (7 left) 2024-09-20T19:23:17.068 INFO:tasks.rados.rados.0.smithi042.stdout:402: rollback oid 32 current snap is 48 2024-09-20T19:23:17.068 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 42 2024-09-20T19:23:17.068 INFO:tasks.rados.rados.0.smithi042.stdout:403: write oid 46 current snap is 48 2024-09-20T19:23:17.068 INFO:tasks.rados.rados.0.smithi042.stdout:403: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-20T19:23:17.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T19:23:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:23:17.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:17.077 INFO:tasks.rados.rados.0.smithi042.stdout:403: writing smithi04268048-46 from 789962 to 1324709 tid 1 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:403: writing smithi04268048-46 from 1780952 to 2471008 tid 2 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:404: delete oid 30 current snap is 48 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:393: finishing write tid 4 to smithi04268048-5 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:393: oid 5 updating version 0 to 106 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:393: oid 5 version 106 is already newer than 105 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:393: oid 5 version 106 is already newer than 104 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 106 (ObjNum 163 snap 48 seq_num 163) dirty exists 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:393: left oid 5 (ObjNum 163 snap 48 seq_num 163) 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:394: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:17.085 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:17.119 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:17.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:17.179 INFO:tasks.rados.rados.0.smithi042.stdout:396: finishing write tid 1 to smithi04268048-16 2024-09-20T19:23:17.179 INFO:tasks.rados.rados.0.smithi042.stdout:397: finishing copy_from to smithi04268048-48 2024-09-20T19:23:17.179 INFO:tasks.rados.rados.0.smithi042.stdout:397: got expected ENOENT (src dne) 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:397: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: finishing write tid 2 to smithi04268048-16 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: finishing write tid 3 to smithi04268048-16 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: finishing write tid 4 to smithi04268048-16 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: oid 16 updating version 0 to 112 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: oid 16 version 112 is already newer than 110 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: oid 16 version 112 is already newer than 111 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 112 (ObjNum 164 snap 48 seq_num 164) dirty exists 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: left oid 16 (ObjNum 164 snap 48 seq_num 164) 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:399: finishing write tid 1 to smithi04268048-12 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:393: done (9 left) 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:394: done (8 left) 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:396: done (7 left) 2024-09-20T19:23:17.180 INFO:tasks.rados.rados.0.smithi042.stdout:397: done (6 left) 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:398: done (5 left) 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:404: done (4 left) 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:405: snap_create 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: finishing write tid 2 to smithi04268048-12 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: finishing write tid 3 to smithi04268048-12 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: finishing write tid 4 to smithi04268048-12 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: finishing write tid 5 to smithi04268048-12 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: oid 12 updating version 0 to 96 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: oid 12 updating version 96 to 98 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: oid 12 version 98 is already newer than 95 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: oid 12 version 98 is already newer than 97 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 98 (ObjNum 165 snap 48 seq_num 165) dirty exists 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:399: left oid 12 (ObjNum 165 snap 48 seq_num 165) 2024-09-20T19:23:17.181 INFO:tasks.rados.rados.0.smithi042.stdout:402: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:402: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:402: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 171 (ObjNum 149 snap 42 seq_num 149) dirty exists 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: finishing write tid 1 to smithi04268048-46 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: finishing write tid 2 to smithi04268048-46 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: finishing write tid 3 to smithi04268048-46 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: finishing write tid 4 to smithi04268048-46 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: oid 46 updating version 0 to 175 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: oid 46 version 175 is already newer than 173 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: oid 46 version 175 is already newer than 174 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 175 (ObjNum 166 snap 48 seq_num 166) dirty exists 2024-09-20T19:23:17.182 INFO:tasks.rados.rados.0.smithi042.stdout:403: left oid 46 (ObjNum 166 snap 48 seq_num 166) 2024-09-20T19:23:17.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:17.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:17 smithi162 bash[24557]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.319 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:23:17.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:17.331 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.331 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[32075]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.736123+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.737521+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:15.987893+0000 mgr.y (mgr.24491) 201 : cluster [DBG] pgmap v248: 148 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 140 active+clean; 156 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.054686+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.42:0/596673894' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.060004+0000 mon.a (mon.0) 1146 : audit [INF] from='client.24887 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.069341+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.223898+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.42:0/3012008648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:17 smithi042 bash[22706]: audit 2024-09-20T19:23:16.224624+0000 mon.a (mon.0) 1148 : audit [INF] from='client.24893 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:23:17.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:17.454 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:17.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:17.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:17.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:17.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:17.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:17.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:17.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:17.974 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:17.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:18.032 INFO:tasks.rados.rados.0.smithi042.stdout:399: done (4 left) 2024-09-20T19:23:18.033 INFO:tasks.rados.rados.0.smithi042.stdout:400: done (3 left) 2024-09-20T19:23:18.033 INFO:tasks.rados.rados.0.smithi042.stdout:402: done (2 left) 2024-09-20T19:23:18.033 INFO:tasks.rados.rados.0.smithi042.stdout:403: done (1 left) 2024-09-20T19:23:18.033 INFO:tasks.rados.rados.0.smithi042.stdout:405: done (0 left) 2024-09-20T19:23:18.033 INFO:tasks.rados.rados.0.smithi042.stdout:406: delete oid 11 current snap is 49 2024-09-20T19:23:18.047 INFO:tasks.rados.rados.0.smithi042.stdout:406: done (0 left) 2024-09-20T19:23:18.048 INFO:tasks.rados.rados.0.smithi042.stdout:407: snap_create 2024-09-20T19:23:18.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:18.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:18.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:18.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[22706]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:18 smithi042 bash[32075]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:18.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.458848+0000 osd.2 (osd.2) 89 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:16.460996+0000 osd.2 (osd.2) 90 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.061657+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: audit 2024-09-20T19:23:17.062859+0000 mon.a (mon.0) 1150 : audit [INF] from='client.24887 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: audit 2024-09-20T19:23:17.062959+0000 mon.a (mon.0) 1151 : audit [INF] from='client.24893 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.067813+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:18 smithi162 bash[24557]: cluster 2024-09-20T19:23:18.032153+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-20T19:23:18.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:18.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:18.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T19:23:18.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:18.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:18.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:18.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:18.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:18.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:19.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:19.039 INFO:tasks.rados.rados.0.smithi042.stdout:407: done (0 left) 2024-09-20T19:23:19.040 INFO:tasks.rados.rados.0.smithi042.stdout:408: snap_create 2024-09-20T19:23:19.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:19.131 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:19.131 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:19 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.131 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:19 smithi162 bash[24557]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.131 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:19 smithi162 bash[24557]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.132 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:19 smithi162 bash[24557]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:19.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:19.281 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:19.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:19.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:19.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[32075]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[32075]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[32075]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[22706]: cluster 2024-09-20T19:23:17.988451+0000 mgr.y (mgr.24491) 202 : cluster [DBG] pgmap v251: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[22706]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:19 smithi042 bash[22706]: cluster 2024-09-20T19:23:19.038582+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-20T19:23:19.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:19.543 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:19.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:19.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:19.804 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:19.907 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:19.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:20.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.043 INFO:tasks.rados.rados.0.smithi042.stdout:408: done (0 left) 2024-09-20T19:23:20.043 INFO:tasks.rados.rados.0.smithi042.stdout:409: write (excl) oid 16 current snap is 51 2024-09-20T19:23:20.043 INFO:tasks.rados.rados.0.smithi042.stdout:409: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-20T19:23:20.057 INFO:tasks.rados.rados.0.smithi042.stdout:409: writing smithi04268048-16 from 440383 to 917230 tid 1 2024-09-20T19:23:20.066 INFO:tasks.rados.rados.0.smithi042.stdout:409: writing smithi04268048-16 from 1556398 to 1993781 tid 2 2024-09-20T19:23:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:20.073 INFO:tasks.rados.rados.0.smithi042.stdout:409: writing smithi04268048-16 from 2474721 to 2840383 tid 3 2024-09-20T19:23:20.073 INFO:tasks.rados.rados.0.smithi042.stdout:410: write oid 40 current snap is 51 2024-09-20T19:23:20.073 INFO:tasks.rados.rados.0.smithi042.stdout:409: finishing write tid 1 to smithi04268048-16 2024-09-20T19:23:20.073 INFO:tasks.rados.rados.0.smithi042.stdout:410: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-20T19:23:20.077 INFO:tasks.rados.rados.0.smithi042.stdout:410: writing smithi04268048-40 from 457190 to 1009286 tid 1 2024-09-20T19:23:20.084 INFO:tasks.rados.rados.0.smithi042.stdout:410: writing smithi04268048-40 from 1614880 to 2195921 tid 2 2024-09-20T19:23:20.089 INFO:tasks.rados.rados.0.smithi042.stdout:410: writing smithi04268048-40 from 2785791 to 2857190 tid 3 2024-09-20T19:23:20.090 INFO:tasks.rados.rados.0.smithi042.stdout:411: snap_remove snap 44 2024-09-20T19:23:20.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[32075]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[32075]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[32075]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[32075]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[22706]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[22706]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[22706]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:20 smithi042 bash[22706]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:20.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:20.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.450 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:20.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:20 smithi162 bash[24557]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:20 smithi162 bash[24557]: audit 2024-09-20T19:23:19.092382+0000 mgr.y (mgr.24491) 203 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:20.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:20 smithi162 bash[24557]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:20 smithi162 bash[24557]: cluster 2024-09-20T19:23:20.038717+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-20T19:23:20.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:20.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:23:20.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:20.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:20.838 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:20.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.967 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:20.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:21.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:21.098 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:21.107 INFO:tasks.rados.rados.0.smithi042.stdout:411: done (2 left) 2024-09-20T19:23:21.107 INFO:tasks.rados.rados.0.smithi042.stdout:412: snap_remove snap 48 2024-09-20T19:23:21.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:21.230 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:21.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:21.346 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:21 smithi042 bash[22706]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.346 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:21 smithi042 bash[22706]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:21 smithi042 bash[32075]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:21 smithi042 bash[32075]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.362 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:21.465 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:21.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:21 smithi162 bash[24557]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:21 smithi162 bash[24557]: cluster 2024-09-20T19:23:19.989344+0000 mgr.y (mgr.24491) 204 : cluster [DBG] pgmap v254: 148 pgs: 7 active+clean+snaptrim, 141 active+clean; 139 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 15 op/s 2024-09-20T19:23:21.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:21.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:21.723 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:21.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:21.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:21.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:21.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:21.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:22.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.123 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:22 smithi162 bash[24557]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.123 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:22 smithi162 bash[24557]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.129 INFO:tasks.rados.rados.0.smithi042.stdout:412: done (2 left) 2024-09-20T19:23:22.129 INFO:tasks.rados.rados.0.smithi042.stdout:413: copy_from oid 35 from oid 15 current snap is 51 2024-09-20T19:23:22.129 INFO:tasks.rados.rados.0.smithi042.stdout:414: write oid 49 current snap is 51 2024-09-20T19:23:22.129 INFO:tasks.rados.rados.0.smithi042.stdout:414: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-20T19:23:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:22.140 INFO:tasks.rados.rados.0.smithi042.stdout:414: writing smithi04268048-49 from 473997 to 1101342 tid 1 2024-09-20T19:23:22.154 INFO:tasks.rados.rados.0.smithi042.stdout:414: writing smithi04268048-49 from 1673362 to 2114414 tid 2 2024-09-20T19:23:22.165 INFO:tasks.rados.rados.0.smithi042.stdout:414: writing smithi04268048-49 from 2529567 to 2873997 tid 3 2024-09-20T19:23:22.165 INFO:tasks.rados.rados.0.smithi042.stdout:415: write (excl) oid 23 current snap is 51 2024-09-20T19:23:22.165 INFO:tasks.rados.rados.0.smithi042.stdout:409: finishing write tid 2 to smithi04268048-16 2024-09-20T19:23:22.165 INFO:tasks.rados.rados.0.smithi042.stdout:409: finishing write tid 3 to smithi04268048-16 2024-09-20T19:23:22.165 INFO:tasks.rados.rados.0.smithi042.stdout:409: finishing write tid 4 to smithi04268048-16 2024-09-20T19:23:22.166 INFO:tasks.rados.rados.0.smithi042.stdout:415: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-20T19:23:22.180 INFO:tasks.rados.rados.0.smithi042.stdout:415: writing smithi04268048-23 from 490804 to 1193398 tid 1 2024-09-20T19:23:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:22.189 INFO:tasks.rados.rados.0.smithi042.stdout:415: writing smithi04268048-23 from 1615491 to 2200201 tid 2 2024-09-20T19:23:22.194 INFO:tasks.rados.rados.0.smithi042.stdout:415: writing smithi04268048-23 from 2724284 to 2890804 tid 3 2024-09-20T19:23:22.194 INFO:tasks.rados.rados.0.smithi042.stdout:416: delete oid 38 current snap is 51 2024-09-20T19:23:22.194 INFO:tasks.rados.rados.0.smithi042.stdout:409: finishing write tid 5 to smithi04268048-16 2024-09-20T19:23:22.194 INFO:tasks.rados.rados.0.smithi042.stdout:409: oid 16 updating version 0 to 119 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:409: oid 16 version 119 is already newer than 117 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:409: oid 16 version 119 is already newer than 118 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:409: oid 16 version 119 is already newer than 116 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 119 (ObjNum 169 snap 51 seq_num 169) dirty exists 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:409: left oid 16 (ObjNum 169 snap 51 seq_num 169) 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: finishing write tid 1 to smithi04268048-40 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: finishing write tid 2 to smithi04268048-40 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: finishing write tid 3 to smithi04268048-40 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: finishing write tid 4 to smithi04268048-40 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: finishing write tid 5 to smithi04268048-40 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: oid 40 updating version 0 to 140 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: oid 40 updating version 140 to 142 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: oid 40 version 142 is already newer than 139 2024-09-20T19:23:22.195 INFO:tasks.rados.rados.0.smithi042.stdout:410: oid 40 version 142 is already newer than 141 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 142 (ObjNum 170 snap 51 seq_num 170) dirty exists 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:410: left oid 40 (ObjNum 170 snap 51 seq_num 170) 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: finishing write tid 1 to smithi04268048-49 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:413: finishing copy_from racing read to smithi04268048-35 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: finishing write tid 2 to smithi04268048-49 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:413: finishing copy_from to smithi04268048-35 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: finishing write tid 3 to smithi04268048-49 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: finishing write tid 4 to smithi04268048-49 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: finishing write tid 5 to smithi04268048-49 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: oid 49 updating version 0 to 183 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: oid 49 updating version 183 to 186 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: oid 49 version 186 is already newer than 185 2024-09-20T19:23:22.196 INFO:tasks.rados.rados.0.smithi042.stdout:414: oid 49 version 186 is already newer than 184 2024-09-20T19:23:22.197 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 186 (ObjNum 171 snap 51 seq_num 171) dirty exists 2024-09-20T19:23:22.197 INFO:tasks.rados.rados.0.smithi042.stdout:414: left oid 49 (ObjNum 171 snap 51 seq_num 171) 2024-09-20T19:23:22.197 INFO:tasks.rados.rados.0.smithi042.stdout:415: finishing write tid 1 to smithi04268048-23 2024-09-20T19:23:22.197 INFO:tasks.rados.rados.0.smithi042.stdout:415: finishing write tid 2 to smithi04268048-23 2024-09-20T19:23:22.197 INFO:tasks.rados.rados.0.smithi042.stdout:415: finishing write tid 3 to smithi04268048-23 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: finishing write tid 4 to smithi04268048-23 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: finishing write tid 5 to smithi04268048-23 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: oid 23 updating version 0 to 70 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: oid 23 updating version 70 to 71 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: oid 23 version 71 is already newer than 68 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: oid 23 version 71 is already newer than 69 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 71 (ObjNum 172 snap 51 seq_num 172) dirty exists 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: left oid 23 (ObjNum 172 snap 51 seq_num 172) 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:409: done (5 left) 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:410: done (4 left) 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:413: done (3 left) 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:414: done (2 left) 2024-09-20T19:23:22.198 INFO:tasks.rados.rados.0.smithi042.stdout:415: done (1 left) 2024-09-20T19:23:22.199 INFO:tasks.rados.rados.0.smithi042.stdout:416: done (0 left) 2024-09-20T19:23:22.199 INFO:tasks.rados.rados.0.smithi042.stdout:417: delete oid 44 current snap is 51 2024-09-20T19:23:22.199 INFO:tasks.rados.rados.0.smithi042.stdout:417: done (0 left) 2024-09-20T19:23:22.199 INFO:tasks.rados.rados.0.smithi042.stdout:418: copy_from oid 28 from oid 37 current snap is 51 2024-09-20T19:23:22.199 INFO:tasks.rados.rados.0.smithi042.stdout:419: snap_create 2024-09-20T19:23:22.209 INFO:tasks.rados.rados.0.smithi042.stdout:418: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:23:22.211 INFO:tasks.rados.rados.0.smithi042.stdout:418: finishing copy_from to smithi04268048-28 2024-09-20T19:23:22.211 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 98 (ObjNum 157 snap 45 seq_num 157) dirty exists 2024-09-20T19:23:22.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:22.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:22.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:22.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:22 smithi042 bash[32075]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.405 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:22 smithi042 bash[32075]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:22 smithi042 bash[22706]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:22 smithi042 bash[22706]: cluster 2024-09-20T19:23:21.099666+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-20T19:23:22.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:22.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:22.626 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:22.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:22.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:22.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:22.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:23.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:23.043 INFO:tasks.rados.rados.0.smithi042.stdout:418: done (1 left) 2024-09-20T19:23:23.043 INFO:tasks.rados.rados.0.smithi042.stdout:419: done (0 left) 2024-09-20T19:23:23.043 INFO:tasks.rados.rados.0.smithi042.stdout:420: write (excl) oid 26 current snap is 52 2024-09-20T19:23:23.044 INFO:tasks.rados.rados.0.smithi042.stdout:420: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-20T19:23:23.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:23.050 INFO:tasks.rados.rados.0.smithi042.stdout:420: writing smithi04268048-26 from 541225 to 1185919 tid 1 2024-09-20T19:23:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:23.057 INFO:tasks.rados.rados.0.smithi042.stdout:420: writing smithi04268048-26 from 1907290 to 2355680 tid 2 2024-09-20T19:23:23.058 INFO:tasks.rados.rados.0.smithi042.stdout:421: read oid 36 snap -1 2024-09-20T19:23:23.058 INFO:tasks.rados.rados.0.smithi042.stdout:421: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:23:23.058 INFO:tasks.rados.rados.0.smithi042.stdout:422: delete oid 1 current snap is 52 2024-09-20T19:23:23.058 INFO:tasks.rados.rados.0.smithi042.stdout:420: finishing write tid 1 to smithi04268048-26 2024-09-20T19:23:23.063 INFO:tasks.rados.rados.0.smithi042.stdout:421: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:23:23.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[22706]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:23 smithi042 bash[32075]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:23.197 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: finishing write tid 2 to smithi04268048-26 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: finishing write tid 3 to smithi04268048-26 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: finishing write tid 4 to smithi04268048-26 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: oid 26 updating version 0 to 66 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: oid 26 updating version 66 to 68 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: oid 26 version 68 is already newer than 67 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 68 (ObjNum 175 snap 52 seq_num 175) dirty exists 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: left oid 26 (ObjNum 175 snap 52 seq_num 175) 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:420: done (2 left) 2024-09-20T19:23:23.269 INFO:tasks.rados.rados.0.smithi042.stdout:421: done (1 left) 2024-09-20T19:23:23.270 INFO:tasks.rados.rados.0.smithi042.stdout:422: done (0 left) 2024-09-20T19:23:23.270 INFO:tasks.rados.rados.0.smithi042.stdout:423: write oid 24 current snap is 52 2024-09-20T19:23:23.270 INFO:tasks.rados.rados.0.smithi042.stdout:423: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-20T19:23:23.276 INFO:tasks.rados.rados.0.smithi042.stdout:423: writing smithi04268048-24 from 574839 to 1370031 tid 1 2024-09-20T19:23:23.282 INFO:tasks.rados.rados.0.smithi042.stdout:423: writing smithi04268048-24 from 1907901 to 2359960 tid 2 2024-09-20T19:23:23.282 INFO:tasks.rados.rados.0.smithi042.stdout:424: copy_from oid 18 from oid 2 current snap is 52 2024-09-20T19:23:23.282 INFO:tasks.rados.rados.0.smithi042.stdout:425: copy_from oid 50 from oid 42 current snap is 52 2024-09-20T19:23:23.282 INFO:tasks.rados.rados.0.smithi042.stdout:426: snap_remove snap 51 2024-09-20T19:23:23.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:23.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:21.991116+0000 mgr.y (mgr.24491) 205 : cluster [DBG] pgmap v257: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:22.123723+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:23 smithi162 bash[24557]: cluster 2024-09-20T19:23:23.043422+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-20T19:23:23.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:23.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:23.573 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:23.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:23.704 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:23.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:23.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:23.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:23.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:24.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:426: done (3 left) 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:427: rollback oid 45 current snap is 52 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 49 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:428: snap_create 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:423: finishing write tid 1 to smithi04268048-24 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:423: finishing write tid 2 to smithi04268048-24 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:423: finishing write tid 3 to smithi04268048-24 2024-09-20T19:23:24.139 INFO:tasks.rados.rados.0.smithi042.stdout:423: finishing write tid 4 to smithi04268048-24 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:423: oid 24 updating version 0 to 127 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:423: oid 24 updating version 127 to 128 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:423: oid 24 version 128 is already newer than 126 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 128 (ObjNum 177 snap 52 seq_num 177) dirty exists 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:423: left oid 24 (ObjNum 177 snap 52 seq_num 177) 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:424: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:425: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:425: finishing copy_from to smithi04268048-50 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 148 (ObjNum 153 snap 43 seq_num 153) dirty exists 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:424: finishing copy_from to smithi04268048-18 2024-09-20T19:23:24.140 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 73 (ObjNum 132 snap 35 seq_num 132) dirty exists 2024-09-20T19:23:24.145 INFO:tasks.rados.rados.0.smithi042.stdout:427: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:23:24.145 INFO:tasks.rados.rados.0.smithi042.stdout:427: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:23:24.145 INFO:tasks.rados.rados.0.smithi042.stdout:427: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:23:24.146 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 192 (ObjNum 92 snap 23 seq_num 92) dirty exists 2024-09-20T19:23:24.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:24.326 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:24.456 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:24.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:24.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:24.747 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:24.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:24.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:24.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:25.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:423: done (4 left) 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:424: done (3 left) 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:425: done (2 left) 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:427: done (1 left) 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:428: done (0 left) 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:429: write (excl) oid 12 current snap is 53 2024-09-20T19:23:25.133 INFO:tasks.rados.rados.0.smithi042.stdout:429: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-20T19:23:25.139 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:25.139 INFO:tasks.rados.rados.0.smithi042.stdout:429: writing smithi04268048-12 from 591646 to 1062087 tid 1 2024-09-20T19:23:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:25.146 INFO:tasks.rados.rados.0.smithi042.stdout:429: writing smithi04268048-12 from 1566383 to 2162100 tid 2 2024-09-20T19:23:25.151 INFO:tasks.rados.rados.0.smithi042.stdout:429: writing smithi04268048-12 from 2888822 to 2991646 tid 3 2024-09-20T19:23:25.152 INFO:tasks.rados.rados.0.smithi042.stdout:430: read oid 9 snap -1 2024-09-20T19:23:25.152 INFO:tasks.rados.rados.0.smithi042.stdout:430: expect (ObjNum 154 snap 43 seq_num 154) 2024-09-20T19:23:25.152 INFO:tasks.rados.rados.0.smithi042.stdout:431: delete oid 4 current snap is 53 2024-09-20T19:23:25.152 INFO:tasks.rados.rados.0.smithi042.stdout:429: finishing write tid 1 to smithi04268048-12 2024-09-20T19:23:25.152 INFO:tasks.rados.rados.0.smithi042.stdout:429: finishing write tid 2 to smithi04268048-12 2024-09-20T19:23:25.153 INFO:tasks.rados.rados.0.smithi042.stdout:429: finishing write tid 3 to smithi04268048-12 2024-09-20T19:23:25.154 INFO:tasks.rados.rados.0.smithi042.stdout:429: finishing write tid 4 to smithi04268048-12 2024-09-20T19:23:25.154 INFO:tasks.rados.rados.0.smithi042.stdout:429: finishing write tid 5 to smithi04268048-12 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:429: oid 12 updating version 0 to 107 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:429: oid 12 version 107 is already newer than 104 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:429: oid 12 version 107 is already newer than 105 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:429: oid 12 version 107 is already newer than 106 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 107 (ObjNum 178 snap 53 seq_num 178) dirty exists 2024-09-20T19:23:25.155 INFO:tasks.rados.rados.0.smithi042.stdout:429: left oid 12 (ObjNum 178 snap 53 seq_num 178) 2024-09-20T19:23:25.165 INFO:tasks.rados.rados.0.smithi042.stdout:430: expect (ObjNum 154 snap 43 seq_num 154) 2024-09-20T19:23:25.165 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:25.215 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:25.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:25.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[22706]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[22706]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[22706]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[22706]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.251 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[32075]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.251 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[32075]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.251 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[32075]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.251 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:25 smithi042 bash[32075]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.264 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:25.297 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2151572 2024-09-20T19:23:25.302 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:23:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:23:25.306 INFO:tasks.rados.rados.0.smithi042.stdout:429: done (2 left) 2024-09-20T19:23:25.306 INFO:tasks.rados.rados.0.smithi042.stdout:430: done (1 left) 2024-09-20T19:23:25.307 INFO:tasks.rados.rados.0.smithi042.stdout:431: done (0 left) 2024-09-20T19:23:25.307 INFO:tasks.rados.rados.0.smithi042.stdout:432: snap_create 2024-09-20T19:23:25.321 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:23:25.321 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:23:25.339 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:23:25.340 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:23:25.352 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:23:25.352 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:23:25.360 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:23:25.360 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:23:25.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:25.378 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:25 smithi162 bash[24557]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:25 smithi162 bash[24557]: cluster 2024-09-20T19:23:23.991778+0000 mgr.y (mgr.24491) 206 : cluster [DBG] pgmap v260: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 906 KiB/s wr, 6 op/s 2024-09-20T19:23:25.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:25 smithi162 bash[24557]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:25 smithi162 bash[24557]: cluster 2024-09-20T19:23:24.133544+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-20T19:23:25.386 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:23:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:23:25.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:25.406 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:23:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:23:25.422 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:23:25.423 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:23:25.442 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:23:25.443 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:23:25.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:25.516 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:25.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:25.644 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:25.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:25.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:25.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:25.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:25.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:25.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:26.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:26.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:26.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:26.159 INFO:tasks.rados.rados.0.smithi042.stdout:432: done (0 left) 2024-09-20T19:23:26.159 INFO:tasks.rados.rados.0.smithi042.stdout:433: snap_remove snap 43 2024-09-20T19:23:26.159 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T19:23:26.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:26.265 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[22706]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.277 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:26 smithi042 bash[32075]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:26.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: cluster 2024-09-20T19:23:25.133013+0000 mon.a (mon.0) 1160 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.283616+0000 mon.c (mon.2) 45 : audit [DBG] from='client.? v1:172.21.15.42:0/2676342256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.567397+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.42:0/2396403397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:26 smithi162 bash[24557]: audit 2024-09-20T19:23:25.567838+0000 mon.a (mon.0) 1161 : audit [INF] from='client.24905 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:23:26.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:26.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:26.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:26.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:26.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:26.786 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:26.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:26.942 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:26.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:27.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:27.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:433: done (0 left) 2024-09-20T19:23:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:434: copy_from oid 19 from oid 38 current snap is 54 2024-09-20T19:23:27.169 INFO:tasks.rados.rados.0.smithi042.stdout:435: snap_create 2024-09-20T19:23:27.173 INFO:tasks.rados.rados.0.smithi042.stdout:434: finishing copy_from to smithi04268048-19 2024-09-20T19:23:27.173 INFO:tasks.rados.rados.0.smithi042.stdout:434: got expected ENOENT (src dne) 2024-09-20T19:23:27.174 INFO:tasks.rados.rados.0.smithi042.stdout:434: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:23:27.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:27.199 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:27.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:27.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[22706]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:27 smithi042 bash[32075]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:27.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: cluster 2024-09-20T19:23:25.992485+0000 mgr.y (mgr.24491) 207 : cluster [DBG] pgmap v263: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 141 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.150599+0000 mon.a (mon.0) 1162 : audit [INF] from='client.24905 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: cluster 2024-09-20T19:23:26.157418+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.892941+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:27 smithi162 bash[24557]: audit 2024-09-20T19:23:26.894974+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:27.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:27.565 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:27.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:27.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:27.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:27.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:27.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:28.039 INFO:tasks.rados.rados.0.smithi042.stdout:434: done (1 left) 2024-09-20T19:23:28.039 INFO:tasks.rados.rados.0.smithi042.stdout:435: done (0 left) 2024-09-20T19:23:28.039 INFO:tasks.rados.rados.0.smithi042.stdout:436: snap_remove snap 40 2024-09-20T19:23:28.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:28.114 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:28.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:28.245 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:28.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:28.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:28.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[22706]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[22706]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[22706]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[22706]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[32075]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[32075]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[32075]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:28 smithi042 bash[32075]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:28.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:28.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:28 smithi162 bash[24557]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:28 smithi162 bash[24557]: cluster 2024-09-20T19:23:27.160371+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-20T19:23:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:28 smithi162 bash[24557]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:28 smithi162 bash[24557]: cluster 2024-09-20T19:23:28.037679+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-20T19:23:28.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:28.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:28.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:28.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:28.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:28.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:28.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:23:28.996 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T19:23:28.996 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:23:28.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:436: done (0 left) 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:437: read oid 7 snap 45 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:437: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:438: rollback oid 21 current snap is 55 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 46 2024-09-20T19:23:29.041 INFO:tasks.rados.rados.0.smithi042.stdout:439: delete oid 43 current snap is 55 2024-09-20T19:23:29.048 INFO:tasks.rados.rados.0.smithi042.stdout:439: done (2 left) 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:440: rollback oid 10 current snap is 55 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 42 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:441: rollback oid 24 current snap is 55 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 49 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:442: read oid 31 snap 42 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:442: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:23:29.049 INFO:tasks.rados.rados.0.smithi042.stdout:443: delete oid 22 current snap is 55 2024-09-20T19:23:29.052 INFO:tasks.rados.rados.0.smithi042.stdout:441: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:23:29.053 INFO:tasks.rados.rados.0.smithi042.stdout:441: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:23:29.053 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 139 (ObjNum 33 snap 0 seq_num 6) dirty dne 2024-09-20T19:23:29.061 INFO:tasks.rados.rados.0.smithi042.stdout:442: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-20T19:23:29.122 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.128 INFO:tasks.rados.rados.0.smithi042.stdout:437: expect (ObjNum 64 snap 8 seq_num 64) 2024-09-20T19:23:29.137 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 967985 2024-09-20T19:23:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:29.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:29.221 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:29.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:438: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:438: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:438: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 149 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:440: finishing rollback tid 0 to smithi04268048-10 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:440: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:440: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 147 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:437: done (5 left) 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:438: done (4 left) 2024-09-20T19:23:29.271 INFO:tasks.rados.rados.0.smithi042.stdout:440: done (3 left) 2024-09-20T19:23:29.272 INFO:tasks.rados.rados.0.smithi042.stdout:441: done (2 left) 2024-09-20T19:23:29.272 INFO:tasks.rados.rados.0.smithi042.stdout:442: done (1 left) 2024-09-20T19:23:29.272 INFO:tasks.rados.rados.0.smithi042.stdout:443: done (0 left) 2024-09-20T19:23:29.272 INFO:tasks.rados.rados.0.smithi042.stdout:444: snap_create 2024-09-20T19:23:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:29.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[22706]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[22706]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[22706]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[22706]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[32075]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[32075]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[32075]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:29 smithi042 bash[32075]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:29.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:29.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:29 smithi162 bash[24557]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:29 smithi162 bash[24557]: cluster 2024-09-20T19:23:27.992979+0000 mgr.y (mgr.24491) 208 : cluster [DBG] pgmap v266: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:29.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:29 smithi162 bash[24557]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:29 smithi162 bash[24557]: cluster 2024-09-20T19:23:29.040108+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-20T19:23:29.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:29.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:29.762 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:29.790 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:29.893 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:29.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:30.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:30.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:30.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:30.182 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:23:30.281 INFO:tasks.rados.rados.0.smithi042.stdout:444: done (0 left) 2024-09-20T19:23:30.281 INFO:tasks.rados.rados.0.smithi042.stdout:445: snap_remove snap 45 2024-09-20T19:23:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:30.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:30.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:30 smithi162 bash[24557]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:30 smithi162 bash[24557]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:30 smithi162 bash[24557]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:30 smithi162 bash[24557]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:30.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:30.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[32075]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[32075]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[32075]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[32075]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[22706]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[22706]: audit 2024-09-20T19:23:29.100324+0000 mgr.y (mgr.24491) 209 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:30.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[22706]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:30 smithi042 bash[22706]: audit 2024-09-20T19:23:29.481080+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:30.541 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:30.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:30.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:30.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:30.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:30.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:30.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:30.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:30.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:23:31.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:31.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.200 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:31 smithi162 bash[24557]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.206 INFO:tasks.rados.rados.0.smithi042.stdout:445: done (0 left) 2024-09-20T19:23:31.206 INFO:tasks.rados.rados.0.smithi042.stdout:446: write (excl) oid 28 current snap is 56 2024-09-20T19:23:31.206 INFO:tasks.rados.rados.0.smithi042.stdout:446: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-20T19:23:31.214 INFO:tasks.rados.rados.0.smithi042.stdout:446: writing smithi04268048-28 from 658874 to 1430311 tid 1 2024-09-20T19:23:31.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:31.221 INFO:tasks.rados.rados.0.smithi042.stdout:446: writing smithi04268048-28 from 2083958 to 2687013 tid 2 2024-09-20T19:23:31.221 INFO:tasks.rados.rados.0.smithi042.stdout:447: delete oid 25 current snap is 56 2024-09-20T19:23:31.221 INFO:tasks.rados.rados.0.smithi042.stdout:446: finishing write tid 1 to smithi04268048-28 2024-09-20T19:23:31.223 INFO:tasks.rados.rados.0.smithi042.stdout:447: done (1 left) 2024-09-20T19:23:31.224 INFO:tasks.rados.rados.0.smithi042.stdout:448: snap_remove snap 50 2024-09-20T19:23:31.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.329 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[32075]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:29.993577+0000 mgr.y (mgr.24491) 210 : cluster [DBG] pgmap v269: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 161 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 16 op/s 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:30.175239+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: audit 2024-09-20T19:23:30.182755+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.42:0/958052395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.330 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:31 smithi042 bash[22706]: cluster 2024-09-20T19:23:30.189997+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-20T19:23:31.345 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:31.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:31.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:31.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:31.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:31.731 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:31.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:31.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:31.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:31.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:32.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:32.215 INFO:tasks.rados.rados.0.smithi042.stdout:448: done (1 left) 2024-09-20T19:23:32.215 INFO:tasks.rados.rados.0.smithi042.stdout:449: write oid 11 current snap is 56 2024-09-20T19:23:32.215 INFO:tasks.rados.rados.0.smithi042.stdout:449: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-20T19:23:32.225 INFO:tasks.rados.rados.0.smithi042.stdout:449: writing smithi04268048-11 from 692488 to 1330776 tid 1 2024-09-20T19:23:32.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:32.233 INFO:tasks.rados.rados.0.smithi042.stdout:449: writing smithi04268048-11 from 1917275 to 2523999 tid 2 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:450: rollback oid 38 current snap is 56 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 55 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: finishing write tid 2 to smithi04268048-28 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: finishing write tid 3 to smithi04268048-28 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: finishing write tid 4 to smithi04268048-28 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: oid 28 updating version 0 to 99 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: oid 28 updating version 99 to 101 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: oid 28 version 101 is already newer than 100 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 101 (ObjNum 182 snap 56 seq_num 182) dirty exists 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: left oid 28 (ObjNum 182 snap 56 seq_num 182) 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:449: finishing write tid 1 to smithi04268048-11 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:446: done (2 left) 2024-09-20T19:23:32.235 INFO:tasks.rados.rados.0.smithi042.stdout:451: copy_from oid 23 from oid 27 current snap is 56 2024-09-20T19:23:32.236 INFO:tasks.rados.rados.0.smithi042.stdout:452: write oid 1 current snap is 56 2024-09-20T19:23:32.236 INFO:tasks.rados.rados.0.smithi042.stdout:452: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-20T19:23:32.244 INFO:tasks.rados.rados.0.smithi042.stdout:452: writing smithi04268048-1 from 709295 to 1422832 tid 1 2024-09-20T19:23:32.252 INFO:tasks.rados.rados.0.smithi042.stdout:452: writing smithi04268048-1 from 1859404 to 2609786 tid 2 2024-09-20T19:23:32.252 INFO:tasks.rados.rados.0.smithi042.stdout:453: write (excl) oid 41 current snap is 56 2024-09-20T19:23:32.252 INFO:tasks.rados.rados.0.smithi042.stdout:450: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:23:32.252 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: finishing write tid 2 to smithi04268048-11 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: finishing write tid 3 to smithi04268048-11 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: finishing write tid 4 to smithi04268048-11 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: oid 11 updating version 0 to 208 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: oid 11 version 208 is already newer than 207 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: oid 11 updating version 208 to 209 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 209 (ObjNum 184 snap 56 seq_num 184) dirty exists 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:449: left oid 11 (ObjNum 184 snap 56 seq_num 184) 2024-09-20T19:23:32.253 INFO:tasks.rados.rados.0.smithi042.stdout:453: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-20T19:23:32.253 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:32.261 INFO:tasks.rados.rados.0.smithi042.stdout:453: writing smithi04268048-41 from 726102 to 1514888 tid 1 2024-09-20T19:23:32.267 INFO:tasks.rados.rados.0.smithi042.stdout:453: writing smithi04268048-41 from 1917886 to 2528279 tid 2 2024-09-20T19:23:32.271 INFO:tasks.rados.rados.0.smithi042.stdout:453: writing smithi04268048-41 from 3108206 to 3126102 tid 3 2024-09-20T19:23:32.271 INFO:tasks.rados.rados.0.smithi042.stdout:449: done (4 left) 2024-09-20T19:23:32.271 INFO:tasks.rados.rados.0.smithi042.stdout:450: done (3 left) 2024-09-20T19:23:32.271 INFO:tasks.rados.rados.0.smithi042.stdout:454: snap_remove snap 52 2024-09-20T19:23:32.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:32 smithi162 bash[24557]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:32 smithi162 bash[24557]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:32.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:32.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:32 smithi042 bash[32075]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:32 smithi042 bash[32075]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:32 smithi042 bash[22706]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:32 smithi042 bash[22706]: cluster 2024-09-20T19:23:31.197089+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-20T19:23:32.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:32.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:32.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:32.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:32.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:32.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:32.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:32.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:33.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:33.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:33.057 INFO:tasks.rados.rados.0.smithi042.stdout:454: done (3 left) 2024-09-20T19:23:33.057 INFO:tasks.rados.rados.0.smithi042.stdout:455: write (excl) oid 33 current snap is 56 2024-09-20T19:23:33.057 INFO:tasks.rados.rados.0.smithi042.stdout:455: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-20T19:23:33.059 INFO:tasks.rados.rados.0.smithi042.stdout:455: writing smithi04268048-33 from 742909 to 1206944 tid 1 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:455: writing smithi04268048-33 from 1976368 to 2730419 tid 2 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:451: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:452: finishing write tid 1 to smithi04268048-1 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:452: finishing write tid 2 to smithi04268048-1 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:451: finishing copy_from to smithi04268048-23 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 162 (ObjNum 155 snap 44 seq_num 155) dirty exists 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:452: finishing write tid 3 to smithi04268048-1 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:452: finishing write tid 4 to smithi04268048-1 2024-09-20T19:23:33.069 INFO:tasks.rados.rados.0.smithi042.stdout:452: oid 1 updating version 0 to 105 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:452: oid 1 updating version 105 to 106 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:452: oid 1 version 106 is already newer than 104 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 106 (ObjNum 185 snap 56 seq_num 185) dirty exists 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:452: left oid 1 (ObjNum 185 snap 56 seq_num 185) 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: finishing write tid 1 to smithi04268048-41 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: finishing write tid 2 to smithi04268048-41 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: finishing write tid 3 to smithi04268048-41 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: finishing write tid 4 to smithi04268048-41 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: finishing write tid 5 to smithi04268048-41 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: oid 41 updating version 0 to 91 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: oid 41 updating version 91 to 94 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: oid 41 version 94 is already newer than 92 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:453: oid 41 version 94 is already newer than 93 2024-09-20T19:23:33.070 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 94 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:453: left oid 41 (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:451: done (3 left) 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:452: done (2 left) 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:453: done (1 left) 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:456: copy_from oid 2 from oid 10 current snap is 56 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:457: write oid 31 current snap is 56 2024-09-20T19:23:33.071 INFO:tasks.rados.rados.0.smithi042.stdout:457: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-20T19:23:33.076 INFO:tasks.rados.rados.0.smithi042.stdout:457: writing smithi04268048-31 from 759716 to 1299000 tid 1 2024-09-20T19:23:33.085 INFO:tasks.rados.rados.0.smithi042.stdout:457: writing smithi04268048-31 from 2034850 to 2648912 tid 2 2024-09-20T19:23:33.086 INFO:tasks.rados.rados.0.smithi042.stdout:458: snap_remove snap 42 2024-09-20T19:23:33.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:23:33.174 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:33.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.288 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:33 smithi162 bash[24557]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:33.405 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.413 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[32075]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:31.995161+0000 mgr.y (mgr.24491) 211 : cluster [DBG] pgmap v272: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:32.214027+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-20T19:23:33.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.415 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:33 smithi042 bash[22706]: cluster 2024-09-20T19:23:33.042858+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-20T19:23:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:33.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:33.551 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:33.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:33.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:33.778 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:23:33.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:33.809 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:33.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.940 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:33.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:34.043 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:34.064 INFO:tasks.rados.rados.0.smithi042.stdout:458: done (3 left) 2024-09-20T19:23:34.064 INFO:tasks.rados.rados.0.smithi042.stdout:459: snap_remove snap 54 2024-09-20T19:23:34.075 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:34.178 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:34.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:34.338 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:34.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:34.468 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:34.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:34.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:34.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:34.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:34.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:34.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:34.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:459: done (3 left) 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:460: rollback oid 50 current snap is 56 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 49 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:461: read oid 37 snap 46 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:461: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:35.077 INFO:tasks.rados.rados.0.smithi042.stdout:462: snap_remove snap 53 2024-09-20T19:23:35.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:35.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[22706]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[22706]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[22706]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[22706]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[32075]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[32075]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[32075]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:35 smithi042 bash[32075]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:35.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:35 smithi162 bash[24557]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:35 smithi162 bash[24557]: cluster 2024-09-20T19:23:33.995765+0000 mgr.y (mgr.24491) 212 : cluster [DBG] pgmap v275: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s rd, 336 KiB/s wr, 9 op/s 2024-09-20T19:23:35.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:35 smithi162 bash[24557]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:35 smithi162 bash[24557]: cluster 2024-09-20T19:23:34.062840+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-20T19:23:35.241 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:35.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:35.363 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:23:35.363 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:23:35.363 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:23:35.363 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:35.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:35.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:35.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:35.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:35.725 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:35.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:35.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:35.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:35.888 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:35.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:35.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:462: done (5 left) 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:463: read oid 3 snap -1 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:463: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:464: write (excl) oid 40 current snap is 56 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:455: finishing write tid 1 to smithi04268048-33 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:455: finishing write tid 2 to smithi04268048-33 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:455: finishing write tid 3 to smithi04268048-33 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:455: finishing write tid 4 to smithi04268048-33 2024-09-20T19:23:36.081 INFO:tasks.rados.rados.0.smithi042.stdout:455: oid 33 updating version 0 to 98 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:455: oid 33 version 98 is already newer than 97 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:455: oid 33 version 98 is already newer than 96 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 98 (ObjNum 187 snap 56 seq_num 187) dirty exists 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:455: left oid 33 (ObjNum 187 snap 56 seq_num 187) 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:457: finishing write tid 1 to smithi04268048-31 2024-09-20T19:23:36.082 INFO:tasks.rados.rados.0.smithi042.stdout:464: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-20T19:23:36.089 INFO:tasks.rados.rados.0.smithi042.stdout:464: writing smithi04268048-40 from 776523 to 1391056 tid 1 2024-09-20T19:23:36.098 INFO:tasks.rados.rados.0.smithi042.stdout:464: writing smithi04268048-40 from 1976979 to 2734699 tid 2 2024-09-20T19:23:36.098 INFO:tasks.rados.rados.0.smithi042.stdout:455: done (6 left) 2024-09-20T19:23:36.098 INFO:tasks.rados.rados.0.smithi042.stdout:465: copy_from oid 19 from oid 41 current snap is 56 2024-09-20T19:23:36.098 INFO:tasks.rados.rados.0.smithi042.stdout:456: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: finishing write tid 2 to smithi04268048-31 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: finishing write tid 3 to smithi04268048-31 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: finishing write tid 4 to smithi04268048-31 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: oid 31 updating version 0 to 110 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: oid 31 version 110 is already newer than 108 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: oid 31 version 110 is already newer than 109 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 110 (ObjNum 188 snap 56 seq_num 188) dirty exists 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:457: left oid 31 (ObjNum 188 snap 56 seq_num 188) 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:456: finishing copy_from to smithi04268048-2 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 147 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-09-20T19:23:36.099 INFO:tasks.rados.rados.0.smithi042.stdout:461: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:36.121 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:36.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T19:23:36.131 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:36 smithi042 bash[32075]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.131 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:36 smithi042 bash[32075]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:36 smithi042 bash[22706]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:36 smithi042 bash[22706]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.144 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:36.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:460: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:460: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:460: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 217 (ObjNum 137 snap 36 seq_num 137) dirty exists 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:463: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-20T19:23:36.270 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.303 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1698205 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: finishing write tid 1 to smithi04268048-40 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: finishing write tid 2 to smithi04268048-40 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: finishing write tid 3 to smithi04268048-40 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: finishing write tid 4 to smithi04268048-40 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: oid 40 updating version 0 to 156 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: oid 40 updating version 156 to 157 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:464: oid 40 version 157 is already newer than 155 2024-09-20T19:23:36.324 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 157 (ObjNum 189 snap 56 seq_num 189) dirty exists 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:464: left oid 40 (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:465: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:465: finishing copy_from to smithi04268048-19 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 94 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:456: done (6 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:457: done (5 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:460: done (4 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:461: done (3 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:463: done (2 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:464: done (1 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:465: done (0 left) 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:466: read oid 49 snap -1 2024-09-20T19:23:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:466: expect (ObjNum 171 snap 51 seq_num 171) 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:467: read oid 17 snap -1 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:467: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:468: copy_from oid 32 from oid 10 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:469: copy_from oid 15 from oid 12 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:470: rollback oid 16 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 49 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:471: copy_from oid 4 from oid 39 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:472: copy_from oid 45 from oid 47 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:473: read oid 27 snap -1 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:473: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:474: rollback oid 37 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 37 to 55 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:475: rollback oid 21 current snap is 56 2024-09-20T19:23:36.326 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 46 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:476: rollback oid 43 current snap is 56 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 46 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:477: copy_from oid 18 from oid 23 current snap is 56 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:478: rollback oid 34 current snap is 56 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 55 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:479: rollback oid 7 current snap is 56 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 49 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:480: read oid 13 snap 46 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:480: expect (ObjNum 144 snap 39 seq_num 144) 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:481: read oid 41 snap -1 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout:481: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:36.327 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout:478: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout:478: done (15 left) 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout:482: read oid 36 snap 55 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout:482: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:23:36.328 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:23:36.335 INFO:tasks.rados.rados.0.smithi042.stdout:480: expect (ObjNum 144 snap 39 seq_num 144) 2024-09-20T19:23:36.336 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:36.377 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.378 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:36 smithi162 bash[24557]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:36 smithi162 bash[24557]: cluster 2024-09-20T19:23:35.075387+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-20T19:23:36.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:470: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:23:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:466: expect (ObjNum 171 snap 51 seq_num 171) 2024-09-20T19:23:36.462 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:36.492 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:36.505 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 135536 2024-09-20T19:23:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:36.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:36.593 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2476593 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:470: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:470: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 159 (ObjNum 164 snap 48 seq_num 164) dirty exists 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:476: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:476: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:23:36.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 129 (ObjNum 149 snap 42 seq_num 149) dirty exists 2024-09-20T19:23:36.602 INFO:tasks.rados.rados.0.smithi042.stdout:482: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:23:36.602 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:36.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:36.733 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:36.798 INFO:tasks.rados.rados.0.smithi042.stdout:479: finishing rollback tid 0 to smithi04268048-7 2024-09-20T19:23:36.798 INFO:tasks.rados.rados.0.smithi042.stdout:474: finishing rollback tid 0 to smithi04268048-37 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:475: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:479: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:466: done (15 left) 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:470: done (14 left) 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:476: done (13 left) 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:480: done (12 left) 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:482: done (11 left) 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:483: read oid 29 snap 49 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:483: expect deleted 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:484: snap_create 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:474: finishing rollback tid 1 to smithi04268048-37 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:475: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:475: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:23:36.799 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 189 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:474: finishing rollback tid 2 to smithi04268048-37 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 134 (ObjNum 157 snap 45 seq_num 157) dirty exists 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:479: finishing rollback tid 2 to smithi04268048-7 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 72 (ObjNum 64 snap 8 seq_num 64) dirty exists 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:469: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:473: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:23:36.800 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:36.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:36.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:36.935 INFO:tasks.rados.rados.0.smithi042.stdout:471: finishing copy_from to smithi04268048-4 2024-09-20T19:23:36.935 INFO:tasks.rados.rados.0.smithi042.stdout:471: got expected ENOENT (src dne) 2024-09-20T19:23:36.935 INFO:tasks.rados.rados.0.smithi042.stdout:471: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:23:36.935 INFO:tasks.rados.rados.0.smithi042.stdout:467: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:36.936 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:36.967 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 702493 2024-09-20T19:23:36.987 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:37.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:37.015 INFO:tasks.rados.rados.0.smithi042.stdout:469: finishing copy_from to smithi04268048-15 2024-09-20T19:23:37.015 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 107 (ObjNum 178 snap 53 seq_num 178) dirty exists 2024-09-20T19:23:37.015 INFO:tasks.rados.rados.0.smithi042.stdout:481: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:37.028 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2272509 2024-09-20T19:23:37.030 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:37.055 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:477: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:472: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:468: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:468: finishing copy_from to smithi04268048-32 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 147 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-09-20T19:23:37.093 INFO:tasks.rados.rados.0.smithi042.stdout:477: finishing copy_from to smithi04268048-18 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 162 (ObjNum 155 snap 44 seq_num 155) dirty exists 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:472: finishing copy_from to smithi04268048-45 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 27 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:467: done (12 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:468: done (11 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:469: done (10 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:471: done (9 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:472: done (8 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:473: done (7 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:474: done (6 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:475: done (5 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:477: done (4 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:479: done (3 left) 2024-09-20T19:23:37.094 INFO:tasks.rados.rados.0.smithi042.stdout:481: done (2 left) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:483: done (1 left) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:484: done (0 left) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:485: rollback oid 20 current snap is 57 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 20 to 46 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:486: copy_from oid 5 from oid 34 current snap is 57 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:487: read oid 26 snap -1 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:487: expect (ObjNum 175 snap 52 seq_num 175) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:488: read oid 2 snap -1 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:488: expect (ObjNum 143 snap 38 seq_num 143) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:489: read oid 37 snap -1 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:489: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:490: copy_from oid 12 from oid 41 current snap is 57 2024-09-20T19:23:37.095 INFO:tasks.rados.rados.0.smithi042.stdout:491: write oid 9 current snap is 57 2024-09-20T19:23:37.096 INFO:tasks.rados.rados.0.smithi042.stdout:491: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-20T19:23:37.124 INFO:tasks.rados.rados.0.smithi042.stdout:491: writing smithi04268048-9 from 793330 to 1483112 tid 1 2024-09-20T19:23:37.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:37.142 INFO:tasks.rados.rados.0.smithi042.stdout:491: writing smithi04268048-9 from 2035461 to 2653192 tid 2 2024-09-20T19:23:37.144 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:37 smithi162 bash[24557]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.144 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:37 smithi162 bash[24557]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.144 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:37 smithi162 bash[24557]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.144 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:37 smithi162 bash[24557]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:491: writing smithi04268048-9 from 3101545 to 3193330 tid 3 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:492: copy_from oid 19 from oid 22 current snap is 57 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:485: finishing rollback tid 0 to smithi04268048-20 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:485: finishing rollback tid 1 to smithi04268048-20 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:485: finishing rollback tid 2 to smithi04268048-20 2024-09-20T19:23:37.150 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 67 (ObjNum 134 snap 35 seq_num 134) dirty exists 2024-09-20T19:23:37.151 INFO:tasks.rados.rados.0.smithi042.stdout:487: expect (ObjNum 175 snap 52 seq_num 175) 2024-09-20T19:23:37.151 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:37.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:37.229 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:37.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:37.263 INFO:tasks.rados.rados.0.smithi042.stdout:489: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:37.263 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:37.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:37.337 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1607354 2024-09-20T19:23:37.375 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 565657 2024-09-20T19:23:37.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[22706]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[22706]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[22706]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[22706]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[32075]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[32075]: cluster 2024-09-20T19:23:35.996305+0000 mgr.y (mgr.24491) 213 : cluster [DBG] pgmap v278: 148 pgs: 6 active+clean+snaptrim, 142 active+clean; 154 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[32075]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:37 smithi042 bash[32075]: cluster 2024-09-20T19:23:36.080370+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-20T19:23:37.419 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:37.431 INFO:tasks.rados.rados.0.smithi042.stdout:486: finishing copy_from to smithi04268048-5 2024-09-20T19:23:37.431 INFO:tasks.rados.rados.0.smithi042.stdout:486: got expected ENOENT (src dne) 2024-09-20T19:23:37.431 INFO:tasks.rados.rados.0.smithi042.stdout:486: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:23:37.431 INFO:tasks.rados.rados.0.smithi042.stdout:488: expect (ObjNum 143 snap 38 seq_num 143) 2024-09-20T19:23:37.478 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:23:37.478 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:23:37.484 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1372495 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:490: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:490: finishing copy_from to smithi04268048-12 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 94 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:491: finishing write tid 1 to smithi04268048-9 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:491: finishing write tid 2 to smithi04268048-9 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:485: done (7 left) 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:486: done (6 left) 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:487: done (5 left) 2024-09-20T19:23:37.501 INFO:tasks.rados.rados.0.smithi042.stdout:488: done (4 left) 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:489: done (3 left) 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:490: done (2 left) 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:493: delete oid 38 current snap is 57 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:492: finishing copy_from to smithi04268048-19 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:492: got expected ENOENT (src dne) 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:492: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: finishing write tid 3 to smithi04268048-9 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: finishing write tid 4 to smithi04268048-9 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: finishing write tid 5 to smithi04268048-9 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: oid 9 updating version 0 to 225 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: oid 9 version 225 is already newer than 223 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: oid 9 updating version 225 to 226 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:491: oid 9 version 226 is already newer than 224 2024-09-20T19:23:37.502 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 226 (ObjNum 190 snap 57 seq_num 190) dirty exists 2024-09-20T19:23:37.503 INFO:tasks.rados.rados.0.smithi042.stdout:491: left oid 9 (ObjNum 190 snap 57 seq_num 190) 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:491: done (2 left) 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:492: done (1 left) 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:493: done (0 left) 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:494: rollback oid 28 current snap is 57 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 46 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:495: read oid 21 snap 46 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:495: expect (ObjNum 66 snap 10 seq_num 66) 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:496: write (excl) oid 48 current snap is 57 2024-09-20T19:23:37.504 INFO:tasks.rados.rados.0.smithi042.stdout:496: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-20T19:23:37.509 INFO:tasks.rados.rados.0.smithi042.stdout:496: writing smithi04268048-48 from 426944 to 983577 tid 1 2024-09-20T19:23:37.516 INFO:tasks.rados.rados.0.smithi042.stdout:496: writing smithi04268048-48 from 1468778 to 2090178 tid 2 2024-09-20T19:23:37.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:37.523 INFO:tasks.rados.rados.0.smithi042.stdout:496: writing smithi04268048-48 from 2872744 to 3226944 tid 3 2024-09-20T19:23:37.523 INFO:tasks.rados.rados.0.smithi042.stdout:497: snap_remove snap 55 2024-09-20T19:23:37.541 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:37.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:37.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:37.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:37.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:37.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:37.928 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:37.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:38.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:38.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:38.061 INFO:tasks.rados.rados.0.smithi042.stdout:497: done (3 left) 2024-09-20T19:23:38.061 INFO:tasks.rados.rados.0.smithi042.stdout:498: delete oid 46 current snap is 57 2024-09-20T19:23:38.062 INFO:tasks.rados.rados.0.smithi042.stdout:494: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:23:38.062 INFO:tasks.rados.rados.0.smithi042.stdout:494: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:23:38.062 INFO:tasks.rados.rados.0.smithi042.stdout:494: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:23:38.062 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 104 (ObjNum 108 snap 25 seq_num 108) dirty exists 2024-09-20T19:23:38.062 INFO:tasks.rados.rados.0.smithi042.stdout:495: expect (ObjNum 66 snap 10 seq_num 66) 2024-09-20T19:23:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:38.146 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2654220 2024-09-20T19:23:38.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[32075]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.175 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:38 smithi042 bash[22706]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.186 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: finishing write tid 1 to smithi04268048-48 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: finishing write tid 2 to smithi04268048-48 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:494: done (3 left) 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:495: done (2 left) 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:498: done (1 left) 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:499: write oid 1 current snap is 57 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: finishing write tid 3 to smithi04268048-48 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: finishing write tid 4 to smithi04268048-48 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: finishing write tid 5 to smithi04268048-48 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: oid 48 updating version 0 to 194 2024-09-20T19:23:38.225 INFO:tasks.rados.rados.0.smithi042.stdout:496: oid 48 updating version 194 to 195 2024-09-20T19:23:38.226 INFO:tasks.rados.rados.0.smithi042.stdout:496: oid 48 updating version 195 to 196 2024-09-20T19:23:38.226 INFO:tasks.rados.rados.0.smithi042.stdout:496: oid 48 version 196 is already newer than 193 2024-09-20T19:23:38.226 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 196 (ObjNum 192 snap 57 seq_num 192) dirty exists 2024-09-20T19:23:38.226 INFO:tasks.rados.rados.0.smithi042.stdout:496: left oid 48 (ObjNum 192 snap 57 seq_num 192) 2024-09-20T19:23:38.226 INFO:tasks.rados.rados.0.smithi042.stdout:499: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-20T19:23:38.231 INFO:tasks.rados.rados.0.smithi042.stdout:499: writing smithi04268048-1 from 460558 to 1167689 tid 1 2024-09-20T19:23:38.238 INFO:tasks.rados.rados.0.smithi042.stdout:499: writing smithi04268048-1 from 1869389 to 2378105 tid 2 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:499: writing smithi04268048-1 from 3094884 to 3260558 tid 3 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:496: done (1 left) 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:500: read oid 29 snap -1 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:500: expect deleted 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:501: read oid 4 snap -1 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:501: expect deleted 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:499: finishing write tid 1 to smithi04268048-1 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:499: finishing write tid 2 to smithi04268048-1 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:502: copy_from oid 26 from oid 28 current snap is 57 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:503: copy_from oid 16 from oid 24 current snap is 57 2024-09-20T19:23:38.244 INFO:tasks.rados.rados.0.smithi042.stdout:504: snap_remove snap 49 2024-09-20T19:23:38.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: cluster 2024-09-20T19:23:37.080068+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: audit 2024-09-20T19:23:37.469565+0000 mgr.y (mgr.24491) 214 : audit [DBG] from='client.15060 v1:172.21.15.42:0/491368887' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:38 smithi162 bash[24557]: cluster 2024-09-20T19:23:38.045692+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-20T19:23:38.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:38.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:38.446 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:38.549 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:38.578 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:38.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:38.708 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:38.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:38.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:38.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:38.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:39.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:39.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:39.121 INFO:tasks.rados.rados.0.smithi042.stdout:504: done (5 left) 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:505: snap_create 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:503: finishing copy_from to smithi04268048-16 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:503: got expected ENOENT (src dne) 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:503: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: finishing write tid 3 to smithi04268048-1 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: finishing write tid 4 to smithi04268048-1 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: finishing write tid 5 to smithi04268048-1 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: oid 1 updating version 0 to 123 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: oid 1 version 123 is already newer than 120 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: oid 1 version 123 is already newer than 122 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: oid 1 version 123 is already newer than 121 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 123 (ObjNum 194 snap 57 seq_num 194) dirty exists 2024-09-20T19:23:39.122 INFO:tasks.rados.rados.0.smithi042.stdout:499: left oid 1 (ObjNum 194 snap 57 seq_num 194) 2024-09-20T19:23:39.123 INFO:tasks.rados.rados.0.smithi042.stdout:502: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:23:39.123 INFO:tasks.rados.rados.0.smithi042.stdout:502: finishing copy_from to smithi04268048-26 2024-09-20T19:23:39.123 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 104 (ObjNum 108 snap 25 seq_num 108) dirty exists 2024-09-20T19:23:39.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:39.229 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:39.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:39.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:39 smithi162 bash[24557]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:39 smithi162 bash[24557]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.359 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:39 smithi042 bash[32075]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:39 smithi042 bash[32075]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:39 smithi042 bash[22706]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:39 smithi042 bash[22706]: cluster 2024-09-20T19:23:37.996938+0000 mgr.y (mgr.24491) 215 : cluster [DBG] pgmap v281: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:39.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:39.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:39.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:39.721 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:39.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.869 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:39.890 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:23:39.890 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:23:39.908 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:23:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:23:39.925 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:23:39.925 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:23:39.942 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:23:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:23:39.963 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:23:39.963 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:23:39.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.985 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:23:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:23:39.993 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:39.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:40.007 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:23:40.007 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:23:40.028 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:23:40.028 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:23:40.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:499: done (5 left) 2024-09-20T19:23:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:500: done (4 left) 2024-09-20T19:23:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:501: done (3 left) 2024-09-20T19:23:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:502: done (2 left) 2024-09-20T19:23:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:503: done (1 left) 2024-09-20T19:23:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:505: done (0 left) 2024-09-20T19:23:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:506: delete oid 39 current snap is 58 2024-09-20T19:23:40.125 INFO:tasks.rados.rados.0.smithi042.stdout:506: done (0 left) 2024-09-20T19:23:40.125 INFO:tasks.rados.rados.0.smithi042.stdout:507: snap_create 2024-09-20T19:23:40.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:40.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[22706]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:40 smithi042 bash[32075]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.361 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:40.440 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-20T19:23:37.077513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":184485496,"num_objects":286,"num_object_clones":43,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":5,"num_read":2184,"num_read_kb":694602,"num_write":1312,"num_write_kb":280980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2516,"ondisk_log_size":2516,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":689240,"kb_used_data":414776,"kb_used_omap":12,"kb_used_meta":274291,"kb_avail":749271976,"statfs":{"total":767960285184,"available":767254503424,"internally_reserved":0,"allocated":424730624,"data_stored":506109938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":280874544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22504917,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":-3,"num_read":91,"num_read_kb":72672,"num_write":43,"num_write_kb":26490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.041160"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089774+0000","last_change":"2024-09-20T19:21:59.826969+0000","last_active":"2024-09-20T19:23:36.089774+0000","last_peered":"2024-09-20T19:23:36.089774+0000","last_clean":"2024-09-20T19:23:36.089774+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:23:36.089774+0000","last_undegraded":"2024-09-20T19:23:36.089774+0000","last_fullsized":"2024-09-20T19:23:36.089774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:59.826916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:59.826916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135572+0000","last_change":"2024-09-20T19:22:18.623572+0000","last_active":"2024-09-20T19:23:36.135572+0000","last_peered":"2024-09-20T19:23:36.135572+0000","last_clean":"2024-09-20T19:23:36.135572+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:23:36.135572+0000","last_undegraded":"2024-09-20T19:23:36.135572+0000","last_fullsized":"2024-09-20T19:23:36.135572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.623500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.623500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":371,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090463+0000","last_change":"2024-09-20T19:22:14.898580+0000","last_active":"2024-09-20T19:23:36.090463+0000","last_peered":"2024-09-20T19:23:36.090463+0000","last_clean":"2024-09-20T19:23:36.090463+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:23:36.090463+0000","last_undegraded":"2024-09-20T19:23:36.090463+0000","last_fullsized":"2024-09-20T19:23:36.090463+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:14.898494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.898494+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.088737+0000","last_change":"2024-09-20T19:20:33.800940+0000","last_active":"2024-09-20T19:23:36.088737+0000","last_peered":"2024-09-20T19:23:36.088737+0000","last_clean":"2024-09-20T19:23:36.088737+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:23:36.088737+0000","last_undegraded":"2024-09-20T19:23:36.088737+0000","last_fullsized":"2024-09-20T19:23:36.088737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.800794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.800794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.141852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090930+0000","last_change":"2024-09-20T19:22:58.622973+0000","last_active":"2024-09-20T19:23:36.090930+0000","last_peered":"2024-09-20T19:23:36.090930+0000","last_clean":"2024-09-20T19:23:36.090930+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:23:36.090930+0000","last_undegraded":"2024-09-20T19:23:36.090930+0000","last_fullsized":"2024-09-20T19:23:36.090930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:58.622796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.622796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.819628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.088941+0000","last_change":"2024-09-20T19:20:34.776570+0000","last_active":"2024-09-20T19:23:36.088941+0000","last_peered":"2024-09-20T19:23:36.088941+0000","last_clean":"2024-09-20T19:23:36.088941+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:23:36.088941+0000","last_undegraded":"2024-09-20T19:23:36.088941+0000","last_fullsized":"2024-09-20T19:23:36.088941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:34.776387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:34.776387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:45.153030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.078649+0000","last_change":"2024-09-20T19:22:03.981228+0000","last_active":"2024-09-20T19:23:36.078649+0000","last_peered":"2024-09-20T19:23:36.078649+0000","last_clean":"2024-09-20T19:23:36.078649+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:23:36.078649+0000","last_undegraded":"2024-09-20T19:23:36.078649+0000","last_fullsized":"2024-09-20T19:23:36.078649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.981101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.981101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.455385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089520+0000","last_change":"2024-09-20T19:22:14.871281+0000","last_active":"2024-09-20T19:23:36.089520+0000","last_peered":"2024-09-20T19:23:36.089520+0000","last_clean":"2024-09-20T19:23:36.089520+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:23:36.089520+0000","last_undegraded":"2024-09-20T19:23:36.089520+0000","last_fullsized":"2024-09-20T19:23:36.089520+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:14.871139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.871139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.971798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136131+0000","last_change":"2024-09-20T19:23:08.305837+0000","last_active":"2024-09-20T19:23:36.136131+0000","last_peered":"2024-09-20T19:23:36.136131+0000","last_clean":"2024-09-20T19:23:36.136131+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:23:36.136131+0000","last_undegraded":"2024-09-20T19:23:36.136131+0000","last_fullsized":"2024-09-20T19:23:36.136131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.305714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.305714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:26.456491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090583+0000","last_change":"2024-09-20T19:23:02.238503+0000","last_active":"2024-09-20T19:23:36.090583+0000","last_peered":"2024-09-20T19:23:36.090583+0000","last_clean":"2024-09-20T19:23:36.090583+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:23:36.090583+0000","last_undegraded":"2024-09-20T19:23:36.090583+0000","last_fullsized":"2024-09-20T19:23:36.090583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:23:02.238438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.238438+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:03.780590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134935+0000","last_change":"2024-09-20T19:22:08.909960+0000","last_active":"2024-09-20T19:23:36.134935+0000","last_peered":"2024-09-20T19:23:36.134935+0000","last_clean":"2024-09-20T19:23:36.134935+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:23:36.134935+0000","last_undegraded":"2024-09-20T19:23:36.134935+0000","last_fullsized":"2024-09-20T19:23:36.134935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:08.909835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.909835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":337,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101237+0000","last_change":"2024-09-20T19:22:20.013446+0000","last_active":"2024-09-20T19:23:36.101237+0000","last_peered":"2024-09-20T19:23:36.101237+0000","last_clean":"2024-09-20T19:23:36.101237+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:23:36.101237+0000","last_undegraded":"2024-09-20T19:23:36.101237+0000","last_fullsized":"2024-09-20T19:23:36.101237+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.013380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.013380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:43.342445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136750+0000","last_change":"2024-09-20T19:22:20.892258+0000","last_active":"2024-09-20T19:23:36.136750+0000","last_peered":"2024-09-20T19:23:36.136750+0000","last_clean":"2024-09-20T19:23:36.136750+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:23:36.136750+0000","last_undegraded":"2024-09-20T19:23:36.136750+0000","last_fullsized":"2024-09-20T19:23:36.136750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.892102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.892102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:45.137642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090025+0000","last_change":"2024-09-20T19:22:07.939814+0000","last_active":"2024-09-20T19:23:36.090025+0000","last_peered":"2024-09-20T19:23:36.090025+0000","last_clean":"2024-09-20T19:23:36.090025+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:23:36.090025+0000","last_undegraded":"2024-09-20T19:23:36.090025+0000","last_fullsized":"2024-09-20T19:23:36.090025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:07.939740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:07.939740+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101673+0000","last_change":"2024-09-20T19:21:07.802088+0000","last_active":"2024-09-20T19:23:36.101673+0000","last_peered":"2024-09-20T19:23:36.101673+0000","last_clean":"2024-09-20T19:23:36.101673+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:23:36.101673+0000","last_undegraded":"2024-09-20T19:23:36.101673+0000","last_fullsized":"2024-09-20T19:23:36.101673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.801948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.801948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.133106+0000","last_change":"2024-09-20T19:21:08.846790+0000","last_active":"2024-09-20T19:23:36.133106+0000","last_peered":"2024-09-20T19:23:36.133106+0000","last_clean":"2024-09-20T19:23:36.133106+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:23:36.133106+0000","last_undegraded":"2024-09-20T19:23:36.133106+0000","last_fullsized":"2024-09-20T19:23:36.133106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.846665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.846665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.808843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"225'63","reported_seq":642,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.080751+0000","last_change":"2024-09-20T19:23:36.081478+0000","last_active":"2024-09-20T19:23:36.080751+0000","last_peered":"2024-09-20T19:23:36.080751+0000","last_clean":"2024-09-20T19:23:36.080751+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:23:36.080751+0000","last_undegraded":"2024-09-20T19:23:36.080751+0000","last_fullsized":"2024-09-20T19:23:36.080751+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'45","last_scrub_stamp":"2024-09-20T19:22:54.853379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.853379+0000","objects_scrubbed":2,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.954856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012364100000000001,"stat_sum":{"num_bytes":3694597,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":11858,"num_write":28,"num_write_kb":8005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.d","version":"77'17","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091505+0000","last_change":"2024-09-20T19:21:27.280916+0000","last_active":"2024-09-20T19:23:36.091505+0000","last_peered":"2024-09-20T19:23:36.091505+0000","last_clean":"2024-09-20T19:23:36.091505+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:23:36.091505+0000","last_undegraded":"2024-09-20T19:23:36.091505+0000","last_fullsized":"2024-09-20T19:23:36.091505+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:21:27.280627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:27.280627+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.122506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136274+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:23:36.136274+0000","last_peered":"2024-09-20T19:23:36.136274+0000","last_clean":"2024-09-20T19:23:36.136274+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:23:36.136274+0000","last_undegraded":"2024-09-20T19:23:36.136274+0000","last_fullsized":"2024-09-20T19:23:36.136274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083294+0000","last_change":"2024-09-20T19:21:25.836294+0000","last_active":"2024-09-20T19:23:36.083294+0000","last_peered":"2024-09-20T19:23:36.083294+0000","last_clean":"2024-09-20T19:23:36.083294+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:23:36.083294+0000","last_undegraded":"2024-09-20T19:23:36.083294+0000","last_fullsized":"2024-09-20T19:23:36.083294+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.836217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.836217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:42.982780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":330,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.080370+0000","last_change":"2024-09-20T19:21:25.309366+0000","last_active":"2024-09-20T19:23:36.080370+0000","last_peered":"2024-09-20T19:23:36.080370+0000","last_clean":"2024-09-20T19:23:36.080370+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:23:36.080370+0000","last_undegraded":"2024-09-20T19:23:36.080370+0000","last_fullsized":"2024-09-20T19:23:36.080370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.309306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.309306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:41.514892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"227'134","reported_seq":707,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.338951+0000","last_change":"2024-09-20T19:23:36.136672+0000","last_active":"2024-09-20T19:23:36.338951+0000","last_peered":"2024-09-20T19:23:36.338951+0000","last_clean":"2024-09-20T19:23:36.338951+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:23:36.338951+0000","last_undegraded":"2024-09-20T19:23:36.338951+0000","last_fullsized":"2024-09-20T19:23:36.338951+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'100","last_scrub_stamp":"2024-09-20T19:23:11.701907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.701907+0000","objects_scrubbed":3,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.561678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018191780000000001,"stat_sum":{"num_bytes":6087240,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":55,"num_read_kb":34602,"num_write":52,"num_write_kb":14692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.c","version":"77'10","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091146+0000","last_change":"2024-09-20T19:22:21.058046+0000","last_active":"2024-09-20T19:23:36.091146+0000","last_peered":"2024-09-20T19:23:36.091146+0000","last_clean":"2024-09-20T19:23:36.091146+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:23:36.091146+0000","last_undegraded":"2024-09-20T19:23:36.091146+0000","last_fullsized":"2024-09-20T19:23:36.091146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:22:21.057981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.057981+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:39.479461+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.080188+0000","last_change":"2024-09-20T19:21:16.692876+0000","last_active":"2024-09-20T19:23:36.080188+0000","last_peered":"2024-09-20T19:23:36.080188+0000","last_clean":"2024-09-20T19:23:36.080188+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:23:36.080188+0000","last_undegraded":"2024-09-20T19:23:36.080188+0000","last_fullsized":"2024-09-20T19:23:36.080188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:16.692759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.692759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.508852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":413,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079823+0000","last_change":"2024-09-20T19:21:19.204377+0000","last_active":"2024-09-20T19:23:36.079823+0000","last_peered":"2024-09-20T19:23:36.079823+0000","last_clean":"2024-09-20T19:23:36.079823+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:23:36.079823+0000","last_undegraded":"2024-09-20T19:23:36.079823+0000","last_fullsized":"2024-09-20T19:23:36.079823+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:19.204313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:19.204313+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091403+0000","last_change":"2024-09-20T19:21:00.850362+0000","last_active":"2024-09-20T19:23:36.091403+0000","last_peered":"2024-09-20T19:23:36.091403+0000","last_clean":"2024-09-20T19:23:36.091403+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:23:36.091403+0000","last_undegraded":"2024-09-20T19:23:36.091403+0000","last_fullsized":"2024-09-20T19:23:36.091403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.850236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.850236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.099845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"226'116","reported_seq":681,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.095241+0000","last_change":"2024-09-20T19:23:35.224344+0000","last_active":"2024-09-20T19:23:36.095241+0000","last_peered":"2024-09-20T19:23:36.095241+0000","last_clean":"2024-09-20T19:23:36.095241+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:23:36.095241+0000","last_undegraded":"2024-09-20T19:23:36.095241+0000","last_fullsized":"2024-09-20T19:23:36.095241+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'57","last_scrub_stamp":"2024-09-20T19:22:52.771766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771766+0000","objects_scrubbed":6,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.047941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.13963045599999999,"stat_sum":{"num_bytes":16196845,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":71,"num_read_kb":48905,"num_write":44,"num_write_kb":12953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"},{"start":"34","length":"1"}]},{"pgid":"4.f","version":"77'15","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083970+0000","last_change":"2024-09-20T19:23:04.313290+0000","last_active":"2024-09-20T19:23:36.083970+0000","last_peered":"2024-09-20T19:23:36.083970+0000","last_clean":"2024-09-20T19:23:36.083970+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:23:36.083970+0000","last_undegraded":"2024-09-20T19:23:36.083970+0000","last_fullsized":"2024-09-20T19:23:36.083970+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.291138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.078809+0000","last_change":"2024-09-20T19:20:58.724906+0000","last_active":"2024-09-20T19:23:36.078809+0000","last_peered":"2024-09-20T19:23:36.078809+0000","last_clean":"2024-09-20T19:23:36.078809+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:23:36.078809+0000","last_undegraded":"2024-09-20T19:23:36.078809+0000","last_fullsized":"2024-09-20T19:23:36.078809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.724842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.724842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:41.996987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":419,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091874+0000","last_change":"2024-09-20T19:23:03.226408+0000","last_active":"2024-09-20T19:23:36.091874+0000","last_peered":"2024-09-20T19:23:36.091874+0000","last_clean":"2024-09-20T19:23:36.091874+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:23:36.091874+0000","last_undegraded":"2024-09-20T19:23:36.091874+0000","last_fullsized":"2024-09-20T19:23:36.091874+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.212037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":337,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102152+0000","last_change":"2024-09-20T19:23:03.945590+0000","last_active":"2024-09-20T19:23:36.102152+0000","last_peered":"2024-09-20T19:23:36.102152+0000","last_clean":"2024-09-20T19:23:36.102152+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:23:36.102152+0000","last_undegraded":"2024-09-20T19:23:36.102152+0000","last_fullsized":"2024-09-20T19:23:36.102152+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:03.945432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.945432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.503088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"224'75","reported_seq":655,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:23:36.333977+0000","last_change":"2024-09-20T19:23:35.224246+0000","last_active":"2024-09-20T19:23:36.333977+0000","last_peered":"2024-09-20T19:23:36.333977+0000","last_clean":"2024-09-20T19:23:36.333977+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:23:36.333977+0000","last_undegraded":"2024-09-20T19:23:36.333977+0000","last_fullsized":"2024-09-20T19:23:36.333977+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'64","last_scrub_stamp":"2024-09-20T19:23:13.624271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:13.624271+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.332434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.14004755999999999,"stat_sum":{"num_bytes":10336272,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":79,"num_read_kb":60151,"num_write":34,"num_write_kb":14539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"4.e","version":"77'11","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.092067+0000","last_change":"2024-09-20T19:22:58.180428+0000","last_active":"2024-09-20T19:23:36.092067+0000","last_peered":"2024-09-20T19:23:36.092067+0000","last_clean":"2024-09-20T19:23:36.092067+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:23:36.092067+0000","last_undegraded":"2024-09-20T19:23:36.092067+0000","last_fullsized":"2024-09-20T19:23:36.092067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:58.180287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.180287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.237767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":218,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.076897+0000","last_change":"2024-09-20T19:21:53.043331+0000","last_active":"2024-09-20T19:23:36.076897+0000","last_peered":"2024-09-20T19:23:36.076897+0000","last_clean":"2024-09-20T19:23:36.076897+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:23:36.076897+0000","last_undegraded":"2024-09-20T19:23:36.076897+0000","last_fullsized":"2024-09-20T19:23:36.076897+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:24.063388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.063388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102937+0000","last_change":"2024-09-20T19:20:58.711594+0000","last_active":"2024-09-20T19:23:36.102937+0000","last_peered":"2024-09-20T19:23:36.102937+0000","last_clean":"2024-09-20T19:23:36.102937+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:23:36.102937+0000","last_undegraded":"2024-09-20T19:23:36.102937+0000","last_fullsized":"2024-09-20T19:23:36.102937+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.711472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.711472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":322,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090160+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:23:36.090160+0000","last_peered":"2024-09-20T19:23:36.090160+0000","last_clean":"2024-09-20T19:23:36.090160+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:23:36.090160+0000","last_undegraded":"2024-09-20T19:23:36.090160+0000","last_fullsized":"2024-09-20T19:23:36.090160+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.744413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"224'26","reported_seq":581,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:23:36.090062+0000","last_change":"2024-09-20T19:23:35.217796+0000","last_active":"2024-09-20T19:23:36.090062+0000","last_peered":"2024-09-20T19:23:36.090062+0000","last_clean":"2024-09-20T19:23:36.090062+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:23:36.090062+0000","last_undegraded":"2024-09-20T19:23:36.090062+0000","last_fullsized":"2024-09-20T19:23:36.090062+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13697645899999999,"stat_sum":{"num_bytes":2705927,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":21,"num_read_kb":14840,"num_write":10,"num_write_kb":1940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"}]},{"pgid":"4.1","version":"77'14","reported_seq":352,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089222+0000","last_change":"2024-09-20T19:19:47.026064+0000","last_active":"2024-09-20T19:23:36.089222+0000","last_peered":"2024-09-20T19:23:36.089222+0000","last_clean":"2024-09-20T19:23:36.089222+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:23:36.089222+0000","last_undegraded":"2024-09-20T19:23:36.089222+0000","last_fullsized":"2024-09-20T19:23:36.089222+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:19:47.025912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:19:47.025912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:43.164438+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.096969+0000","last_change":"2024-09-20T19:22:01.958010+0000","last_active":"2024-09-20T19:23:36.096969+0000","last_peered":"2024-09-20T19:23:36.096969+0000","last_clean":"2024-09-20T19:23:36.096969+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:23:36.096969+0000","last_undegraded":"2024-09-20T19:23:36.096969+0000","last_fullsized":"2024-09-20T19:23:36.096969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:01.957885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:01.957885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":410,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089503+0000","last_change":"2024-09-20T19:21:07.002029+0000","last_active":"2024-09-20T19:23:36.089503+0000","last_peered":"2024-09-20T19:23:36.089503+0000","last_clean":"2024-09-20T19:23:36.089503+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:23:36.089503+0000","last_undegraded":"2024-09-20T19:23:36.089503+0000","last_fullsized":"2024-09-20T19:23:36.089503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:07.001951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.001951+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.724883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":321,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134910+0000","last_change":"2024-09-20T19:20:06.283382+0000","last_active":"2024-09-20T19:23:36.134910+0000","last_peered":"2024-09-20T19:23:36.134910+0000","last_clean":"2024-09-20T19:23:36.134910+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:23:36.134910+0000","last_undegraded":"2024-09-20T19:23:36.134910+0000","last_fullsized":"2024-09-20T19:23:36.134910+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:06.283250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:06.283250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:43.959864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"227'129","reported_seq":679,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.336576+0000","last_change":"2024-09-20T19:23:35.217699+0000","last_active":"2024-09-20T19:23:36.336576+0000","last_peered":"2024-09-20T19:23:36.336576+0000","last_clean":"2024-09-20T19:23:36.336576+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:23:36.336576+0000","last_undegraded":"2024-09-20T19:23:36.336576+0000","last_fullsized":"2024-09-20T19:23:36.336576+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13440909300000001,"stat_sum":{"num_bytes":10097086,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":60,"num_read_kb":52652,"num_write":44,"num_write_kb":12689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"},{"start":"34","length":"1"}]},{"pgid":"4.0","version":"77'18","reported_seq":378,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089714+0000","last_change":"2024-09-20T19:22:00.803489+0000","last_active":"2024-09-20T19:23:36.089714+0000","last_peered":"2024-09-20T19:23:36.089714+0000","last_clean":"2024-09-20T19:23:36.089714+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:23:36.089714+0000","last_undegraded":"2024-09-20T19:23:36.089714+0000","last_fullsized":"2024-09-20T19:23:36.089714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:22:00.803432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:00.803432+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":344,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083813+0000","last_change":"2024-09-20T19:21:01.687454+0000","last_active":"2024-09-20T19:23:36.083813+0000","last_peered":"2024-09-20T19:23:36.083813+0000","last_clean":"2024-09-20T19:23:36.083813+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:23:36.083813+0000","last_undegraded":"2024-09-20T19:23:36.083813+0000","last_fullsized":"2024-09-20T19:23:36.083813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:21:01.687305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.687305+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:45.058000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.088923+0000","last_change":"2024-09-20T19:22:03.015704+0000","last_active":"2024-09-20T19:23:36.088923+0000","last_peered":"2024-09-20T19:23:36.088923+0000","last_clean":"2024-09-20T19:23:36.088923+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:23:36.088923+0000","last_undegraded":"2024-09-20T19:23:36.088923+0000","last_fullsized":"2024-09-20T19:23:36.088923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.015585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.015585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.008629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090013+0000","last_change":"2024-09-20T19:21:15.032164+0000","last_active":"2024-09-20T19:23:36.090013+0000","last_peered":"2024-09-20T19:23:36.090013+0000","last_clean":"2024-09-20T19:23:36.090013+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:23:36.090013+0000","last_undegraded":"2024-09-20T19:23:36.090013+0000","last_fullsized":"2024-09-20T19:23:36.090013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:15.032044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:15.032044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:46.133662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"227'157","reported_seq":721,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.112159+0000","last_change":"2024-09-20T19:23:35.218828+0000","last_active":"2024-09-20T19:23:36.112159+0000","last_peered":"2024-09-20T19:23:36.112159+0000","last_clean":"2024-09-20T19:23:36.112159+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:23:36.112159+0000","last_undegraded":"2024-09-20T19:23:36.112159+0000","last_fullsized":"2024-09-20T19:23:36.112159+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'118","last_scrub_stamp":"2024-09-20T19:23:04.906510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.906510+0000","objects_scrubbed":10,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.125625+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.076437305999999997,"stat_sum":{"num_bytes":6952177,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":79,"num_read_kb":56407,"num_write":66,"num_write_kb":14623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"},{"start":"34","length":"1"}]},{"pgid":"4.3","version":"77'19","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135079+0000","last_change":"2024-09-20T19:21:07.855976+0000","last_active":"2024-09-20T19:23:36.135079+0000","last_peered":"2024-09-20T19:23:36.135079+0000","last_clean":"2024-09-20T19:23:36.135079+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:23:36.135079+0000","last_undegraded":"2024-09-20T19:23:36.135079+0000","last_fullsized":"2024-09-20T19:23:36.135079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:21:07.855828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.855828+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":287,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079501+0000","last_change":"2024-09-20T19:23:16.461133+0000","last_active":"2024-09-20T19:23:36.079501+0000","last_peered":"2024-09-20T19:23:36.079501+0000","last_clean":"2024-09-20T19:23:36.079501+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:23:36.079501+0000","last_undegraded":"2024-09-20T19:23:36.079501+0000","last_fullsized":"2024-09-20T19:23:36.079501+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:16.461005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:16.461005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.435064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":340,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.094689+0000","last_change":"2024-09-20T19:22:21.758505+0000","last_active":"2024-09-20T19:23:36.094689+0000","last_peered":"2024-09-20T19:23:36.094689+0000","last_clean":"2024-09-20T19:23:36.094689+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:23:36.094689+0000","last_undegraded":"2024-09-20T19:23:36.094689+0000","last_fullsized":"2024-09-20T19:23:36.094689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.758384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.758384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":323,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.078583+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:23:36.078583+0000","last_peered":"2024-09-20T19:23:36.078583+0000","last_clean":"2024-09-20T19:23:36.078583+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:23:36.078583+0000","last_undegraded":"2024-09-20T19:23:36.078583+0000","last_fullsized":"2024-09-20T19:23:36.078583+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.495223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"227'161","reported_seq":732,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.359784+0000","last_change":"2024-09-20T19:23:36.097916+0000","last_active":"2024-09-20T19:23:36.359784+0000","last_peered":"2024-09-20T19:23:36.359784+0000","last_clean":"2024-09-20T19:23:36.359784+0000","last_became_active":"2024-09-20T19:21:30.244233+0000","last_became_peered":"2024-09-20T19:21:30.244233+0000","last_unstale":"2024-09-20T19:23:36.359784+0000","last_undegraded":"2024-09-20T19:23:36.359784+0000","last_fullsized":"2024-09-20T19:23:36.359784+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'93","last_scrub_stamp":"2024-09-20T19:23:14.039336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.039336+0000","objects_scrubbed":6,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.666770+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0082535079999999997,"stat_sum":{"num_bytes":15854055,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":52,"num_read_kb":20353,"num_write":56,"num_write_kb":18633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.2","version":"77'10","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079939+0000","last_change":"2024-09-20T19:22:17.881823+0000","last_active":"2024-09-20T19:23:36.079939+0000","last_peered":"2024-09-20T19:23:36.079939+0000","last_clean":"2024-09-20T19:23:36.079939+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:23:36.079939+0000","last_undegraded":"2024-09-20T19:23:36.079939+0000","last_fullsized":"2024-09-20T19:23:36.079939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:22:17.881680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:17.881680+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:42.798407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083993+0000","last_change":"2024-09-20T19:22:21.888862+0000","last_active":"2024-09-20T19:23:36.083993+0000","last_peered":"2024-09-20T19:23:36.083993+0000","last_clean":"2024-09-20T19:23:36.083993+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:23:36.083993+0000","last_undegraded":"2024-09-20T19:23:36.083993+0000","last_fullsized":"2024-09-20T19:23:36.083993+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":411,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134465+0000","last_change":"2024-09-20T19:21:10.853855+0000","last_active":"2024-09-20T19:23:36.134465+0000","last_peered":"2024-09-20T19:23:36.134465+0000","last_clean":"2024-09-20T19:23:36.134465+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:23:36.134465+0000","last_undegraded":"2024-09-20T19:23:36.134465+0000","last_fullsized":"2024-09-20T19:23:36.134465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:10.853722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.853722+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:39.775734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.132982+0000","last_change":"2024-09-20T19:22:56.797105+0000","last_active":"2024-09-20T19:23:36.132982+0000","last_peered":"2024-09-20T19:23:36.132982+0000","last_clean":"2024-09-20T19:23:36.132982+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:23:36.132982+0000","last_undegraded":"2024-09-20T19:23:36.132982+0000","last_fullsized":"2024-09-20T19:23:36.132982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.054135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"225'104","reported_seq":676,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.332045+0000","last_change":"2024-09-20T19:23:35.074918+0000","last_active":"2024-09-20T19:23:36.332045+0000","last_peered":"2024-09-20T19:23:36.332045+0000","last_clean":"2024-09-20T19:23:36.332045+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:23:36.332045+0000","last_undegraded":"2024-09-20T19:23:36.332045+0000","last_fullsized":"2024-09-20T19:23:36.332045+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17218535099999999,"stat_sum":{"num_bytes":12512357,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":55326,"num_write":55,"num_write_kb":19268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"4.4","version":"77'30","reported_seq":463,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083990+0000","last_change":"2024-09-20T19:22:14.846750+0000","last_active":"2024-09-20T19:23:36.083990+0000","last_peered":"2024-09-20T19:23:36.083990+0000","last_clean":"2024-09-20T19:23:36.083990+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:23:36.083990+0000","last_undegraded":"2024-09-20T19:23:36.083990+0000","last_fullsized":"2024-09-20T19:23:36.083990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:22:14.846671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.846671+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.459830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101920+0000","last_change":"2024-09-20T19:23:01.995350+0000","last_active":"2024-09-20T19:23:36.101920+0000","last_peered":"2024-09-20T19:23:36.101920+0000","last_clean":"2024-09-20T19:23:36.101920+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:23:36.101920+0000","last_undegraded":"2024-09-20T19:23:36.101920+0000","last_fullsized":"2024-09-20T19:23:36.101920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:01.995251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.995251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.855295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135912+0000","last_change":"2024-09-20T19:22:22.664865+0000","last_active":"2024-09-20T19:23:36.135912+0000","last_peered":"2024-09-20T19:23:36.135912+0000","last_clean":"2024-09-20T19:23:36.135912+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:23:36.135912+0000","last_undegraded":"2024-09-20T19:23:36.135912+0000","last_fullsized":"2024-09-20T19:23:36.135912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:22.664804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.664804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:39.512682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090748+0000","last_change":"2024-09-20T19:23:08.714718+0000","last_active":"2024-09-20T19:23:36.090748+0000","last_peered":"2024-09-20T19:23:36.090748+0000","last_clean":"2024-09-20T19:23:36.090748+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:23:36.090748+0000","last_undegraded":"2024-09-20T19:23:36.090748+0000","last_fullsized":"2024-09-20T19:23:36.090748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.714593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.714593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.378300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"227'191","reported_seq":765,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.370260+0000","last_change":"2024-09-20T19:23:36.139767+0000","last_active":"2024-09-20T19:23:36.370260+0000","last_peered":"2024-09-20T19:23:36.370260+0000","last_clean":"2024-09-20T19:23:36.370260+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:23:36.370260+0000","last_undegraded":"2024-09-20T19:23:36.370260+0000","last_fullsized":"2024-09-20T19:23:36.370260+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'86","last_scrub_stamp":"2024-09-20T19:22:51.812052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.812052+0000","objects_scrubbed":9,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:10.984385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0055746370000000003,"stat_sum":{"num_bytes":15932937,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":66,"num_read_kb":36861,"num_write":62,"num_write_kb":28128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.7","version":"77'13","reported_seq":283,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.080165+0000","last_change":"2024-09-20T19:21:22.193770+0000","last_active":"2024-09-20T19:23:36.080165+0000","last_peered":"2024-09-20T19:23:36.080165+0000","last_clean":"2024-09-20T19:23:36.080165+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:23:36.080165+0000","last_undegraded":"2024-09-20T19:23:36.080165+0000","last_fullsized":"2024-09-20T19:23:36.080165+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:21:22.193573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:22.193573+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:45.624081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":664,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.078412+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:23:36.078412+0000","last_peered":"2024-09-20T19:23:36.078412+0000","last_clean":"2024-09-20T19:23:36.078412+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:23:36.078412+0000","last_undegraded":"2024-09-20T19:23:36.078412+0000","last_fullsized":"2024-09-20T19:23:36.078412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.476879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":342,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.077436+0000","last_change":"2024-09-20T19:21:10.216852+0000","last_active":"2024-09-20T19:23:36.077436+0000","last_peered":"2024-09-20T19:23:36.077436+0000","last_clean":"2024-09-20T19:23:36.077436+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:23:36.077436+0000","last_undegraded":"2024-09-20T19:23:36.077436+0000","last_fullsized":"2024-09-20T19:23:36.077436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:10.216700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.216700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.081619+0000","last_change":"2024-09-20T19:20:30.146960+0000","last_active":"2024-09-20T19:23:36.081619+0000","last_peered":"2024-09-20T19:23:36.081619+0000","last_clean":"2024-09-20T19:23:36.081619+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:23:36.081619+0000","last_undegraded":"2024-09-20T19:23:36.081619+0000","last_fullsized":"2024-09-20T19:23:36.081619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:30.146806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:30.146806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:40.541944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":337,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135811+0000","last_change":"2024-09-20T19:23:11.304539+0000","last_active":"2024-09-20T19:23:36.135811+0000","last_peered":"2024-09-20T19:23:36.135811+0000","last_clean":"2024-09-20T19:23:36.135811+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:23:36.135811+0000","last_undegraded":"2024-09-20T19:23:36.135811+0000","last_fullsized":"2024-09-20T19:23:36.135811+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.304478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.304478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.382250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"226'221","reported_seq":789,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.340784+0000","last_change":"2024-09-20T19:23:32.125719+0000","last_active":"2024-09-20T19:23:36.340784+0000","last_peered":"2024-09-20T19:23:36.340784+0000","last_clean":"2024-09-20T19:23:36.340784+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:23:36.340784+0000","last_undegraded":"2024-09-20T19:23:36.340784+0000","last_fullsized":"2024-09-20T19:23:36.340784+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.914243099,"stat_sum":{"num_bytes":29899557,"num_objects":13,"num_object_clones":6,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":152,"num_read_kb":98341,"num_write":103,"num_write_kb":43525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"}]},{"pgid":"4.6","version":"77'12","reported_seq":365,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135589+0000","last_change":"2024-09-20T19:22:16.548672+0000","last_active":"2024-09-20T19:23:36.135589+0000","last_peered":"2024-09-20T19:23:36.135589+0000","last_clean":"2024-09-20T19:23:36.135589+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:23:36.135589+0000","last_undegraded":"2024-09-20T19:23:36.135589+0000","last_fullsized":"2024-09-20T19:23:36.135589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:16.548570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:16.548570+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.801626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134004+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:23:36.134004+0000","last_peered":"2024-09-20T19:23:36.134004+0000","last_clean":"2024-09-20T19:23:36.134004+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:23:36.134004+0000","last_undegraded":"2024-09-20T19:23:36.134004+0000","last_fullsized":"2024-09-20T19:23:36.134004+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:36.737498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136473+0000","last_change":"2024-09-20T19:22:18.870386+0000","last_active":"2024-09-20T19:23:36.136473+0000","last_peered":"2024-09-20T19:23:36.136473+0000","last_clean":"2024-09-20T19:23:36.136473+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:23:36.136473+0000","last_undegraded":"2024-09-20T19:23:36.136473+0000","last_fullsized":"2024-09-20T19:23:36.136473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.870268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.870268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.666304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.103173+0000","last_change":"2024-09-20T19:22:56.906325+0000","last_active":"2024-09-20T19:23:36.103173+0000","last_peered":"2024-09-20T19:23:36.103173+0000","last_clean":"2024-09-20T19:23:36.103173+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:23:36.103173+0000","last_undegraded":"2024-09-20T19:23:36.103173+0000","last_fullsized":"2024-09-20T19:23:36.103173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.906255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.906255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.054482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083178+0000","last_change":"2024-09-20T19:22:06.926452+0000","last_active":"2024-09-20T19:23:36.083178+0000","last_peered":"2024-09-20T19:23:36.083178+0000","last_clean":"2024-09-20T19:23:36.083178+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:23:36.083178+0000","last_undegraded":"2024-09-20T19:23:36.083178+0000","last_fullsized":"2024-09-20T19:23:36.083178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:06.926325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:06.926325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.522917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"227'72","reported_seq":650,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.338919+0000","last_change":"2024-09-20T19:23:36.137019+0000","last_active":"2024-09-20T19:23:36.338919+0000","last_peered":"2024-09-20T19:23:36.338919+0000","last_clean":"2024-09-20T19:23:36.338919+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:23:36.338919+0000","last_undegraded":"2024-09-20T19:23:36.338919+0000","last_fullsized":"2024-09-20T19:23:36.338919+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'61","last_scrub_stamp":"2024-09-20T19:23:06.675677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:06.675677+0000","objects_scrubbed":4,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.544802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002023197,"stat_sum":{"num_bytes":6016873,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":33516,"num_write":31,"num_write_kb":8909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.5","version":"77'16","reported_seq":424,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.095800+0000","last_change":"2024-09-20T19:21:12.740077+0000","last_active":"2024-09-20T19:23:36.095800+0000","last_peered":"2024-09-20T19:23:36.095800+0000","last_clean":"2024-09-20T19:23:36.095800+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:23:36.095800+0000","last_undegraded":"2024-09-20T19:23:36.095800+0000","last_fullsized":"2024-09-20T19:23:36.095800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:21:12.739915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:12.739915+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:40.787907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090414+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:23:36.090414+0000","last_peered":"2024-09-20T19:23:36.090414+0000","last_clean":"2024-09-20T19:23:36.090414+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:23:36.090414+0000","last_undegraded":"2024-09-20T19:23:36.090414+0000","last_fullsized":"2024-09-20T19:23:36.090414+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:37.629192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102168+0000","last_change":"2024-09-20T19:20:52.723249+0000","last_active":"2024-09-20T19:23:36.102168+0000","last_peered":"2024-09-20T19:23:36.102168+0000","last_clean":"2024-09-20T19:23:36.102168+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:23:36.102168+0000","last_undegraded":"2024-09-20T19:23:36.102168+0000","last_fullsized":"2024-09-20T19:23:36.102168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.723121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.723121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136974+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:23:36.136974+0000","last_peered":"2024-09-20T19:23:36.136974+0000","last_clean":"2024-09-20T19:23:36.136974+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:23:36.136974+0000","last_undegraded":"2024-09-20T19:23:36.136974+0000","last_fullsized":"2024-09-20T19:23:36.136974+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.023904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136520+0000","last_change":"2024-09-20T19:20:53.868159+0000","last_active":"2024-09-20T19:23:36.136520+0000","last_peered":"2024-09-20T19:23:36.136520+0000","last_clean":"2024-09-20T19:23:36.136520+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:23:36.136520+0000","last_undegraded":"2024-09-20T19:23:36.136520+0000","last_fullsized":"2024-09-20T19:23:36.136520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.868034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.868034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.287685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":329,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.103637+0000","last_change":"2024-09-20T19:20:39.803967+0000","last_active":"2024-09-20T19:23:36.103637+0000","last_peered":"2024-09-20T19:23:36.103637+0000","last_clean":"2024-09-20T19:23:36.103637+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:23:36.103637+0000","last_undegraded":"2024-09-20T19:23:36.103637+0000","last_fullsized":"2024-09-20T19:23:36.103637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:39.803847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.803847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.047616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"225'107","reported_seq":686,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.332258+0000","last_change":"2024-09-20T19:23:36.134475+0000","last_active":"2024-09-20T19:23:36.332258+0000","last_peered":"2024-09-20T19:23:36.332258+0000","last_clean":"2024-09-20T19:23:36.332258+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:23:36.332258+0000","last_undegraded":"2024-09-20T19:23:36.332258+0000","last_fullsized":"2024-09-20T19:23:36.332258+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'71","last_scrub_stamp":"2024-09-20T19:23:05.723582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.723582+0000","objects_scrubbed":6,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:06.160266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001062002,"stat_sum":{"num_bytes":4958065,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":43704,"num_write":40,"num_write_kb":13904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.a","version":"77'19","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.094986+0000","last_change":"2024-09-20T19:20:52.718858+0000","last_active":"2024-09-20T19:23:36.094986+0000","last_peered":"2024-09-20T19:23:36.094986+0000","last_clean":"2024-09-20T19:23:36.094986+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:23:36.094986+0000","last_undegraded":"2024-09-20T19:23:36.094986+0000","last_fullsized":"2024-09-20T19:23:36.094986+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:20:52.718711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.718711+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.617546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079625+0000","last_change":"2024-09-20T19:22:11.289926+0000","last_active":"2024-09-20T19:23:36.079625+0000","last_peered":"2024-09-20T19:23:36.079625+0000","last_clean":"2024-09-20T19:23:36.079625+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:23:36.079625+0000","last_undegraded":"2024-09-20T19:23:36.079625+0000","last_fullsized":"2024-09-20T19:23:36.079625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.289877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.289877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079162+0000","last_change":"2024-09-20T19:21:06.345504+0000","last_active":"2024-09-20T19:23:36.079162+0000","last_peered":"2024-09-20T19:23:36.079162+0000","last_clean":"2024-09-20T19:23:36.079162+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:23:36.079162+0000","last_undegraded":"2024-09-20T19:23:36.079162+0000","last_fullsized":"2024-09-20T19:23:36.079162+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.345373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.345373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":321,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136703+0000","last_change":"2024-09-20T19:19:51.341908+0000","last_active":"2024-09-20T19:23:36.136703+0000","last_peered":"2024-09-20T19:23:36.136703+0000","last_clean":"2024-09-20T19:23:36.136703+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:23:36.136703+0000","last_undegraded":"2024-09-20T19:23:36.136703+0000","last_fullsized":"2024-09-20T19:23:36.136703+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:51.341777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:51.341777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:41.940173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"227'240","reported_seq":815,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.364000+0000","last_change":"2024-09-20T19:23:36.077895+0000","last_active":"2024-09-20T19:23:36.364000+0000","last_peered":"2024-09-20T19:23:36.364000+0000","last_clean":"2024-09-20T19:23:36.364000+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:23:36.364000+0000","last_undegraded":"2024-09-20T19:23:36.364000+0000","last_fullsized":"2024-09-20T19:23:36.364000+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'112","last_scrub_stamp":"2024-09-20T19:22:52.771031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771031+0000","objects_scrubbed":11,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:54.828177+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086657300000000004,"stat_sum":{"num_bytes":23520309,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":102,"num_read_kb":74848,"num_write":85,"num_write_kb":40408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"2"},{"start":"2f","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"3"}]},{"pgid":"4.b","version":"77'9","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134732+0000","last_change":"2024-09-20T19:22:55.847371+0000","last_active":"2024-09-20T19:23:36.134732+0000","last_peered":"2024-09-20T19:23:36.134732+0000","last_clean":"2024-09-20T19:23:36.134732+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:23:36.134732+0000","last_undegraded":"2024-09-20T19:23:36.134732+0000","last_fullsized":"2024-09-20T19:23:36.134732+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:55.847304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:55.847304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:04.129113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083773+0000","last_change":"2024-09-20T19:21:05.740388+0000","last_active":"2024-09-20T19:23:36.083773+0000","last_peered":"2024-09-20T19:23:36.083773+0000","last_clean":"2024-09-20T19:23:36.083773+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:23:36.083773+0000","last_undegraded":"2024-09-20T19:23:36.083773+0000","last_fullsized":"2024-09-20T19:23:36.083773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.740278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.740278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.483483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.077909+0000","last_change":"2024-09-20T19:21:17.226835+0000","last_active":"2024-09-20T19:23:36.077909+0000","last_peered":"2024-09-20T19:23:36.077909+0000","last_clean":"2024-09-20T19:23:36.077909+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:23:36.077909+0000","last_undegraded":"2024-09-20T19:23:36.077909+0000","last_fullsized":"2024-09-20T19:23:36.077909+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:17.226714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:17.226714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.520900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":340,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.133854+0000","last_change":"2024-09-20T19:22:21.641730+0000","last_active":"2024-09-20T19:23:36.133854+0000","last_peered":"2024-09-20T19:23:36.133854+0000","last_clean":"2024-09-20T19:23:36.133854+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:23:36.133854+0000","last_undegraded":"2024-09-20T19:23:36.133854+0000","last_fullsized":"2024-09-20T19:23:36.133854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:22:21.641600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.641600+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"226'53","reported_seq":616,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.079737+0000","last_change":"2024-09-20T19:23:35.213256+0000","last_active":"2024-09-20T19:23:36.079737+0000","last_peered":"2024-09-20T19:23:36.079737+0000","last_clean":"2024-09-20T19:23:36.079737+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:23:36.079737+0000","last_undegraded":"2024-09-20T19:23:36.079737+0000","last_fullsized":"2024-09-20T19:23:36.079737+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'35","last_scrub_stamp":"2024-09-20T19:23:10.333319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.333319+0000","objects_scrubbed":4,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:17.565953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.123173849,"stat_sum":{"num_bytes":5714314,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21280,"num_write":19,"num_write_kb":11601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"},{"start":"34","length":"1"}]},{"pgid":"4.8","version":"77'15","reported_seq":370,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101118+0000","last_change":"2024-09-20T19:22:22.063998+0000","last_active":"2024-09-20T19:23:36.101118+0000","last_peered":"2024-09-20T19:23:36.101118+0000","last_clean":"2024-09-20T19:23:36.101118+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:23:36.101118+0000","last_undegraded":"2024-09-20T19:23:36.101118+0000","last_fullsized":"2024-09-20T19:23:36.101118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:22.063871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.063871+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.143016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.080956+0000","last_change":"2024-09-20T19:22:02.235649+0000","last_active":"2024-09-20T19:23:36.080956+0000","last_peered":"2024-09-20T19:23:36.080956+0000","last_clean":"2024-09-20T19:23:36.080956+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:23:36.080956+0000","last_undegraded":"2024-09-20T19:23:36.080956+0000","last_fullsized":"2024-09-20T19:23:36.080956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:02.235598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:02.235598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.519095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":417,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134718+0000","last_change":"2024-09-20T19:23:01.188032+0000","last_active":"2024-09-20T19:23:36.134718+0000","last_peered":"2024-09-20T19:23:36.134718+0000","last_clean":"2024-09-20T19:23:36.134718+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:23:36.134718+0000","last_undegraded":"2024-09-20T19:23:36.134718+0000","last_fullsized":"2024-09-20T19:23:36.134718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:01.187968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.187968+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.789452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":329,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101119+0000","last_change":"2024-09-20T19:21:01.739612+0000","last_active":"2024-09-20T19:23:36.101119+0000","last_peered":"2024-09-20T19:23:36.101119+0000","last_clean":"2024-09-20T19:23:36.101119+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:23:36.101119+0000","last_undegraded":"2024-09-20T19:23:36.101119+0000","last_fullsized":"2024-09-20T19:23:36.101119+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:01.739492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.739492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.006870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"227'89","reported_seq":638,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:36.117938+0000","last_change":"2024-09-20T19:23:35.214523+0000","last_active":"2024-09-20T19:23:36.117938+0000","last_peered":"2024-09-20T19:23:36.117938+0000","last_clean":"2024-09-20T19:23:36.117938+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:23:36.117938+0000","last_undegraded":"2024-09-20T19:23:36.117938+0000","last_fullsized":"2024-09-20T19:23:36.117938+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.162038823,"stat_sum":{"num_bytes":13410378,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":50,"num_read_kb":29925,"num_write":34,"num_write_kb":15490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2d","length":"1"},{"start":"2f","length":"1"}]},{"pgid":"4.9","version":"77'12","reported_seq":389,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089208+0000","last_change":"2024-09-20T19:22:57.202081+0000","last_active":"2024-09-20T19:23:36.089208+0000","last_peered":"2024-09-20T19:23:36.089208+0000","last_clean":"2024-09-20T19:23:36.089208+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:23:36.089208+0000","last_undegraded":"2024-09-20T19:23:36.089208+0000","last_fullsized":"2024-09-20T19:23:36.089208+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:57.201945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:57.201945+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.217023+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.093262+0000","last_change":"2024-09-20T19:22:59.196366+0000","last_active":"2024-09-20T19:23:36.093262+0000","last_peered":"2024-09-20T19:23:36.093262+0000","last_clean":"2024-09-20T19:23:36.093262+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:23:36.093262+0000","last_undegraded":"2024-09-20T19:23:36.093262+0000","last_fullsized":"2024-09-20T19:23:36.093262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:22:59.196235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:59.196235+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.646992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":342,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.077349+0000","last_change":"2024-09-20T19:22:18.260176+0000","last_active":"2024-09-20T19:23:36.077349+0000","last_peered":"2024-09-20T19:23:36.077349+0000","last_clean":"2024-09-20T19:23:36.077349+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:23:36.077349+0000","last_undegraded":"2024-09-20T19:23:36.077349+0000","last_fullsized":"2024-09-20T19:23:36.077349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.260046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.260046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.501604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":330,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089948+0000","last_change":"2024-09-20T19:23:00.687733+0000","last_active":"2024-09-20T19:23:36.089948+0000","last_peered":"2024-09-20T19:23:36.089948+0000","last_clean":"2024-09-20T19:23:36.089948+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:23:36.089948+0000","last_undegraded":"2024-09-20T19:23:36.089948+0000","last_fullsized":"2024-09-20T19:23:36.089948+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:00.687658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:00.687658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.540716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135495+0000","last_change":"2024-09-20T19:22:58.800944+0000","last_active":"2024-09-20T19:23:36.135495+0000","last_peered":"2024-09-20T19:23:36.135495+0000","last_clean":"2024-09-20T19:23:36.135495+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:23:36.135495+0000","last_undegraded":"2024-09-20T19:23:36.135495+0000","last_fullsized":"2024-09-20T19:23:36.135495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:58.800874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.800874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:05.321600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.077538+0000","last_change":"2024-09-20T19:23:07.609407+0000","last_active":"2024-09-20T19:23:36.077538+0000","last_peered":"2024-09-20T19:23:36.077538+0000","last_clean":"2024-09-20T19:23:36.077538+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:23:36.077538+0000","last_undegraded":"2024-09-20T19:23:36.077538+0000","last_fullsized":"2024-09-20T19:23:36.077538+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:07.609283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:07.609283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:17.108303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091253+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:23:36.091253+0000","last_peered":"2024-09-20T19:23:36.091253+0000","last_clean":"2024-09-20T19:23:36.091253+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:23:36.091253+0000","last_undegraded":"2024-09-20T19:23:36.091253+0000","last_fullsized":"2024-09-20T19:23:36.091253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:35.796722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079125+0000","last_change":"2024-09-20T19:22:56.840530+0000","last_active":"2024-09-20T19:23:36.079125+0000","last_peered":"2024-09-20T19:23:36.079125+0000","last_clean":"2024-09-20T19:23:36.079125+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:23:36.079125+0000","last_undegraded":"2024-09-20T19:23:36.079125+0000","last_fullsized":"2024-09-20T19:23:36.079125+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.840403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.840403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.237174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.096075+0000","last_change":"2024-09-20T19:23:02.640412+0000","last_active":"2024-09-20T19:23:36.096075+0000","last_peered":"2024-09-20T19:23:36.096075+0000","last_clean":"2024-09-20T19:23:36.096075+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:23:36.096075+0000","last_undegraded":"2024-09-20T19:23:36.096075+0000","last_fullsized":"2024-09-20T19:23:36.096075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:02.640270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.640270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.542638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091408+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:23:36.091408+0000","last_peered":"2024-09-20T19:23:36.091408+0000","last_clean":"2024-09-20T19:23:36.091408+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:23:36.091408+0000","last_undegraded":"2024-09-20T19:23:36.091408+0000","last_fullsized":"2024-09-20T19:23:36.091408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:27.710208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102800+0000","last_change":"2024-09-20T19:21:28.981452+0000","last_active":"2024-09-20T19:23:36.102800+0000","last_peered":"2024-09-20T19:23:36.102800+0000","last_clean":"2024-09-20T19:23:36.102800+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:23:36.102800+0000","last_undegraded":"2024-09-20T19:23:36.102800+0000","last_fullsized":"2024-09-20T19:23:36.102800+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.981320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.981320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:46.013292+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":329,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135267+0000","last_change":"2024-09-20T19:21:29.100795+0000","last_active":"2024-09-20T19:23:36.135267+0000","last_peered":"2024-09-20T19:23:36.135267+0000","last_clean":"2024-09-20T19:23:36.135267+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:23:36.135267+0000","last_undegraded":"2024-09-20T19:23:36.135267+0000","last_fullsized":"2024-09-20T19:23:36.135267+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.100718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.100718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:23.284037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":353,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090810+0000","last_change":"2024-09-20T19:20:32.059785+0000","last_active":"2024-09-20T19:23:36.090810+0000","last_peered":"2024-09-20T19:23:36.090810+0000","last_clean":"2024-09-20T19:23:36.090810+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:23:36.090810+0000","last_undegraded":"2024-09-20T19:23:36.090810+0000","last_fullsized":"2024-09-20T19:23:36.090810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:32.059603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.059603+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.628487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.101686+0000","last_change":"2024-09-20T19:22:21.054003+0000","last_active":"2024-09-20T19:23:36.101686+0000","last_peered":"2024-09-20T19:23:36.101686+0000","last_clean":"2024-09-20T19:23:36.101686+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:23:36.101686+0000","last_undegraded":"2024-09-20T19:23:36.101686+0000","last_fullsized":"2024-09-20T19:23:36.101686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.053899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.053899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":417,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102368+0000","last_change":"2024-09-20T19:22:13.105842+0000","last_active":"2024-09-20T19:23:36.102368+0000","last_peered":"2024-09-20T19:23:36.102368+0000","last_clean":"2024-09-20T19:23:36.102368+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:23:36.102368+0000","last_undegraded":"2024-09-20T19:23:36.102368+0000","last_fullsized":"2024-09-20T19:23:36.102368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:13.105692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:13.105692+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134519+0000","last_change":"2024-09-20T19:23:15.737592+0000","last_active":"2024-09-20T19:23:36.134519+0000","last_peered":"2024-09-20T19:23:36.134519+0000","last_clean":"2024-09-20T19:23:36.134519+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:23:36.134519+0000","last_undegraded":"2024-09-20T19:23:36.134519+0000","last_fullsized":"2024-09-20T19:23:36.134519+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:15.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:15.737537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.525979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":361,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102399+0000","last_change":"2024-09-20T19:22:54.923862+0000","last_active":"2024-09-20T19:23:36.102399+0000","last_peered":"2024-09-20T19:23:36.102399+0000","last_clean":"2024-09-20T19:23:36.102399+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:23:36.102399+0000","last_undegraded":"2024-09-20T19:23:36.102399+0000","last_fullsized":"2024-09-20T19:23:36.102399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:54.923725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.923725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:02.018467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135517+0000","last_change":"2024-09-20T19:22:20.671257+0000","last_active":"2024-09-20T19:23:36.135517+0000","last_peered":"2024-09-20T19:23:36.135517+0000","last_clean":"2024-09-20T19:23:36.135517+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:23:36.135517+0000","last_undegraded":"2024-09-20T19:23:36.135517+0000","last_fullsized":"2024-09-20T19:23:36.135517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.671196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.671196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:40.398530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":410,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090383+0000","last_change":"2024-09-20T19:22:53.660006+0000","last_active":"2024-09-20T19:23:36.090383+0000","last_peered":"2024-09-20T19:23:36.090383+0000","last_clean":"2024-09-20T19:23:36.090383+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:23:36.090383+0000","last_undegraded":"2024-09-20T19:23:36.090383+0000","last_fullsized":"2024-09-20T19:23:36.090383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:53.659878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.659878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:06.807822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.092272+0000","last_change":"2024-09-20T19:22:11.914350+0000","last_active":"2024-09-20T19:23:36.092272+0000","last_peered":"2024-09-20T19:23:36.092272+0000","last_clean":"2024-09-20T19:23:36.092272+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:23:36.092272+0000","last_undegraded":"2024-09-20T19:23:36.092272+0000","last_fullsized":"2024-09-20T19:23:36.092272+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.914224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.914224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.081852+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:23:36.081852+0000","last_peered":"2024-09-20T19:23:36.081852+0000","last_clean":"2024-09-20T19:23:36.081852+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:23:36.081852+0000","last_undegraded":"2024-09-20T19:23:36.081852+0000","last_fullsized":"2024-09-20T19:23:36.081852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:36.994421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":340,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.095416+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:23:36.095416+0000","last_peered":"2024-09-20T19:23:36.095416+0000","last_clean":"2024-09-20T19:23:36.095416+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:23:36.095416+0000","last_undegraded":"2024-09-20T19:23:36.095416+0000","last_fullsized":"2024-09-20T19:23:36.095416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:37.755439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090089+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:23:36.090089+0000","last_peered":"2024-09-20T19:23:36.090089+0000","last_clean":"2024-09-20T19:23:36.090089+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:23:36.090089+0000","last_undegraded":"2024-09-20T19:23:36.090089+0000","last_fullsized":"2024-09-20T19:23:36.090089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.132924+0000","last_change":"2024-09-20T19:21:21.676309+0000","last_active":"2024-09-20T19:23:36.132924+0000","last_peered":"2024-09-20T19:23:36.132924+0000","last_clean":"2024-09-20T19:23:36.132924+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:23:36.132924+0000","last_undegraded":"2024-09-20T19:23:36.132924+0000","last_fullsized":"2024-09-20T19:23:36.132924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:21.676157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.676157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:41.743838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090601+0000","last_change":"2024-09-20T19:20:52.914253+0000","last_active":"2024-09-20T19:23:36.090601+0000","last_peered":"2024-09-20T19:23:36.090601+0000","last_clean":"2024-09-20T19:23:36.090601+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:23:36.090601+0000","last_undegraded":"2024-09-20T19:23:36.090601+0000","last_fullsized":"2024-09-20T19:23:36.090601+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:52.914121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.914121+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.086622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083731+0000","last_change":"2024-09-20T19:22:53.163608+0000","last_active":"2024-09-20T19:23:36.083731+0000","last_peered":"2024-09-20T19:23:36.083731+0000","last_clean":"2024-09-20T19:23:36.083731+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:23:36.083731+0000","last_undegraded":"2024-09-20T19:23:36.083731+0000","last_fullsized":"2024-09-20T19:23:36.083731+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:53.163555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.163555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:09.815764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079632+0000","last_change":"2024-09-20T19:21:06.774687+0000","last_active":"2024-09-20T19:23:36.079632+0000","last_peered":"2024-09-20T19:23:36.079632+0000","last_clean":"2024-09-20T19:23:36.079632+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:23:36.079632+0000","last_undegraded":"2024-09-20T19:23:36.079632+0000","last_fullsized":"2024-09-20T19:23:36.079632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.774547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.774547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:39.061173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":322,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089628+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:23:36.089628+0000","last_peered":"2024-09-20T19:23:36.089628+0000","last_clean":"2024-09-20T19:23:36.089628+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:23:36.089628+0000","last_undegraded":"2024-09-20T19:23:36.089628+0000","last_fullsized":"2024-09-20T19:23:36.089628+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:43.598526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":352,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089668+0000","last_change":"2024-09-20T19:22:50.638425+0000","last_active":"2024-09-20T19:23:36.089668+0000","last_peered":"2024-09-20T19:23:36.089668+0000","last_clean":"2024-09-20T19:23:36.089668+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:23:36.089668+0000","last_undegraded":"2024-09-20T19:23:36.089668+0000","last_fullsized":"2024-09-20T19:23:36.089668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:22:50.638257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:50.638257+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.137027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.103027+0000","last_change":"2024-09-20T19:21:25.913956+0000","last_active":"2024-09-20T19:23:36.103027+0000","last_peered":"2024-09-20T19:23:36.103027+0000","last_clean":"2024-09-20T19:23:36.103027+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:23:36.103027+0000","last_undegraded":"2024-09-20T19:23:36.103027+0000","last_fullsized":"2024-09-20T19:23:36.103027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.913821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.913821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:38.011162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":332,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.095043+0000","last_change":"2024-09-20T19:20:38.745217+0000","last_active":"2024-09-20T19:23:36.095043+0000","last_peered":"2024-09-20T19:23:36.095043+0000","last_clean":"2024-09-20T19:23:36.095043+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:23:36.095043+0000","last_undegraded":"2024-09-20T19:23:36.095043+0000","last_fullsized":"2024-09-20T19:23:36.095043+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.745091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.745091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.791875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":321,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134780+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:23:36.134780+0000","last_peered":"2024-09-20T19:23:36.134780+0000","last_clean":"2024-09-20T19:23:36.134780+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:23:36.134780+0000","last_undegraded":"2024-09-20T19:23:36.134780+0000","last_fullsized":"2024-09-20T19:23:36.134780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:39.947846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":365,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.089892+0000","last_change":"2024-09-20T19:22:08.939934+0000","last_active":"2024-09-20T19:23:36.089892+0000","last_peered":"2024-09-20T19:23:36.089892+0000","last_clean":"2024-09-20T19:23:36.089892+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:23:36.089892+0000","last_undegraded":"2024-09-20T19:23:36.089892+0000","last_fullsized":"2024-09-20T19:23:36.089892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.939842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.939842+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.096758+0000","last_change":"2024-09-20T19:22:51.781731+0000","last_active":"2024-09-20T19:23:36.096758+0000","last_peered":"2024-09-20T19:23:36.096758+0000","last_clean":"2024-09-20T19:23:36.096758+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:23:36.096758+0000","last_undegraded":"2024-09-20T19:23:36.096758+0000","last_fullsized":"2024-09-20T19:23:36.096758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.781680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.781680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:00.226292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.134565+0000","last_change":"2024-09-20T19:23:14.325927+0000","last_active":"2024-09-20T19:23:36.134565+0000","last_peered":"2024-09-20T19:23:36.134565+0000","last_clean":"2024-09-20T19:23:36.134565+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:23:36.134565+0000","last_undegraded":"2024-09-20T19:23:36.134565+0000","last_fullsized":"2024-09-20T19:23:36.134565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:14.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:20.929643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090595+0000","last_change":"2024-09-20T19:22:11.898382+0000","last_active":"2024-09-20T19:23:36.090595+0000","last_peered":"2024-09-20T19:23:36.090595+0000","last_clean":"2024-09-20T19:23:36.090595+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:23:36.090595+0000","last_undegraded":"2024-09-20T19:23:36.090595+0000","last_fullsized":"2024-09-20T19:23:36.090595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.898255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.898255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:37.746423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":323,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.136289+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:23:36.136289+0000","last_peered":"2024-09-20T19:23:36.136289+0000","last_clean":"2024-09-20T19:23:36.136289+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:23:36.136289+0000","last_undegraded":"2024-09-20T19:23:36.136289+0000","last_fullsized":"2024-09-20T19:23:36.136289+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:46.997157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.102348+0000","last_change":"2024-09-20T19:23:11.005669+0000","last_active":"2024-09-20T19:23:36.102348+0000","last_peered":"2024-09-20T19:23:36.102348+0000","last_clean":"2024-09-20T19:23:36.102348+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:23:36.102348+0000","last_undegraded":"2024-09-20T19:23:36.102348+0000","last_fullsized":"2024-09-20T19:23:36.102348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.005587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.005587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.106114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135398+0000","last_change":"2024-09-20T19:23:10.711312+0000","last_active":"2024-09-20T19:23:36.135398+0000","last_peered":"2024-09-20T19:23:36.135398+0000","last_clean":"2024-09-20T19:23:36.135398+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:23:36.135398+0000","last_undegraded":"2024-09-20T19:23:36.135398+0000","last_fullsized":"2024-09-20T19:23:36.135398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:10.711138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.711138+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.536408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":419,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.096436+0000","last_change":"2024-09-20T19:22:08.792421+0000","last_active":"2024-09-20T19:23:36.096436+0000","last_peered":"2024-09-20T19:23:36.096436+0000","last_clean":"2024-09-20T19:23:36.096436+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:23:36.096436+0000","last_undegraded":"2024-09-20T19:23:36.096436+0000","last_fullsized":"2024-09-20T19:23:36.096436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.792243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.792243+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083305+0000","last_change":"2024-09-20T19:23:05.361065+0000","last_active":"2024-09-20T19:23:36.083305+0000","last_peered":"2024-09-20T19:23:36.083305+0000","last_clean":"2024-09-20T19:23:36.083305+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:23:36.083305+0000","last_undegraded":"2024-09-20T19:23:36.083305+0000","last_fullsized":"2024-09-20T19:23:36.083305+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:05.360941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.360941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.569089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090206+0000","last_change":"2024-09-20T19:23:03.692672+0000","last_active":"2024-09-20T19:23:36.090206+0000","last_peered":"2024-09-20T19:23:36.090206+0000","last_clean":"2024-09-20T19:23:36.090206+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:23:36.090206+0000","last_undegraded":"2024-09-20T19:23:36.090206+0000","last_fullsized":"2024-09-20T19:23:36.090206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:03.692537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.692537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:10.678096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.096692+0000","last_change":"2024-09-20T19:22:22.768858+0000","last_active":"2024-09-20T19:23:36.096692+0000","last_peered":"2024-09-20T19:23:36.096692+0000","last_clean":"2024-09-20T19:23:36.096692+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:23:36.096692+0000","last_undegraded":"2024-09-20T19:23:36.096692+0000","last_fullsized":"2024-09-20T19:23:36.096692+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:22.768697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.768697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:36.167630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.135308+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:23:36.135308+0000","last_peered":"2024-09-20T19:23:36.135308+0000","last_clean":"2024-09-20T19:23:36.135308+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:23:36.135308+0000","last_undegraded":"2024-09-20T19:23:36.135308+0000","last_fullsized":"2024-09-20T19:23:36.135308+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:48.836697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091805+0000","last_change":"2024-09-20T19:20:59.816653+0000","last_active":"2024-09-20T19:23:36.091805+0000","last_peered":"2024-09-20T19:23:36.091805+0000","last_clean":"2024-09-20T19:23:36.091805+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:23:36.091805+0000","last_undegraded":"2024-09-20T19:23:36.091805+0000","last_fullsized":"2024-09-20T19:23:36.091805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:59.816510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:59.816510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.141868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.095164+0000","last_change":"2024-09-20T19:22:18.771300+0000","last_active":"2024-09-20T19:23:36.095164+0000","last_peered":"2024-09-20T19:23:36.095164+0000","last_clean":"2024-09-20T19:23:36.095164+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:23:36.095164+0000","last_undegraded":"2024-09-20T19:23:36.095164+0000","last_fullsized":"2024-09-20T19:23:36.095164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.771181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.771181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.079345+0000","last_change":"2024-09-20T19:21:21.225432+0000","last_active":"2024-09-20T19:23:36.079345+0000","last_peered":"2024-09-20T19:23:36.079345+0000","last_clean":"2024-09-20T19:23:36.079345+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:23:36.079345+0000","last_undegraded":"2024-09-20T19:23:36.079345+0000","last_fullsized":"2024-09-20T19:23:36.079345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:21:21.225282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.225282+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.083536+0000","last_change":"2024-09-20T19:22:51.090500+0000","last_active":"2024-09-20T19:23:36.083536+0000","last_peered":"2024-09-20T19:23:36.083536+0000","last_clean":"2024-09-20T19:23:36.083536+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:23:36.083536+0000","last_undegraded":"2024-09-20T19:23:36.083536+0000","last_fullsized":"2024-09-20T19:23:36.083536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.090439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.090439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:52.055207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":322,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091242+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:23:36.091242+0000","last_peered":"2024-09-20T19:23:36.091242+0000","last_clean":"2024-09-20T19:23:36.091242+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:23:36.091242+0000","last_undegraded":"2024-09-20T19:23:36.091242+0000","last_fullsized":"2024-09-20T19:23:36.091242+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:41.627136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":342,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090952+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:23:36.090952+0000","last_peered":"2024-09-20T19:23:36.090952+0000","last_clean":"2024-09-20T19:23:36.090952+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:23:36.090952+0000","last_undegraded":"2024-09-20T19:23:36.090952+0000","last_fullsized":"2024-09-20T19:23:36.090952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.730587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":361,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.090326+0000","last_change":"2024-09-20T19:21:24.033642+0000","last_active":"2024-09-20T19:23:36.090326+0000","last_peered":"2024-09-20T19:23:36.090326+0000","last_clean":"2024-09-20T19:23:36.090326+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:23:36.090326+0000","last_undegraded":"2024-09-20T19:23:36.090326+0000","last_fullsized":"2024-09-20T19:23:36.090326+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:21:24.033570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.033570+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.742580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-20T19:23:36.091090+0000","last_change":"2024-09-20T19:21:08.918562+0000","last_active":"2024-09-20T19:23:36.091090+0000","last_peered":"2024-09-20T19:23:36.091090+0000","last_clean":"2024-09-20T19:23:36.091090+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:23:36.091090+0000","last_undegraded":"2024-09-20T19:23:36.091090+0000","last_fullsized":"2024-09-20T19:23:36.091090+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.918443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.918443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.121645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":183888989,"num_objects":87,"num_object_clones":43,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":5,"num_read":1008,"num_read_kb":693569,"num_write":763,"num_write_kb":279307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":422498304,"data_stored":520993443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1938,"ondisk_log_size":1938,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907019,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90204,"kb_used_data":59016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93654948,"statfs":{"total":95995035648,"available":95902666752,"internally_reserved":0,"allocated":60432384,"data_stored":72816124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234070,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77828,"kb_used_data":46644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93667324,"statfs":{"total":95995035648,"available":95915339776,"internally_reserved":0,"allocated":47763456,"data_stored":51989037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561120,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56636,"kb_used_data":25448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688516,"statfs":{"total":95995035648,"available":95937040384,"internally_reserved":0,"allocated":26058752,"data_stored":28293424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855468,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83020,"kb_used_data":47664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93662132,"statfs":{"total":95995035648,"available":95910023168,"internally_reserved":0,"allocated":48807936,"data_stored":62726731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084406,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99348,"kb_used_data":64000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93645804,"statfs":{"total":95995035648,"available":95893303296,"internally_reserved":0,"allocated":65536000,"data_stored":82961234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378756,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106204,"kb_used_data":66696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93638948,"statfs":{"total":95995035648,"available":95886282752,"internally_reserved":0,"allocated":68296704,"data_stored":84300740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67892,"kb_used_data":36708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677260,"statfs":{"total":95995035648,"available":95925514240,"internally_reserved":0,"allocated":37588992,"data_stored":41834969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738524,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108108,"kb_used_data":68600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93637044,"statfs":{"total":95995035648,"available":95884333056,"internally_reserved":0,"allocated":70246400,"data_stored":81187679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67514368,"data_stored":79415696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35442688,"data_stored":40652712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66162688,"data_stored":83119110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63381504,"data_stored":81779684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57794560,"data_stored":77342994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":23932928,"data_stored":27111648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45047808,"data_stored":50217984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":63221760,"data_stored":81353615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:23:40.446 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:23:40.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: audit 2024-09-20T19:23:39.108419+0000 mgr.y (mgr.24491) 216 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: cluster 2024-09-20T19:23:39.118572+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: audit 2024-09-20T19:23:39.872964+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.42:0/1969255628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:40 smithi162 bash[24557]: cluster 2024-09-20T19:23:40.119832+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-20T19:23:40.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:40.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.638 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:40.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:40.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:40.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:40.903 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:23:41.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:41.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:41.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:507: done (0 left) 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:508: copy_from oid 1 from oid 20 current snap is 59 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:509: read oid 48 snap 56 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:509: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:510: write oid 32 current snap is 59 2024-09-20T19:23:41.138 INFO:tasks.rados.rados.0.smithi042.stdout:510: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-20T19:23:41.144 INFO:tasks.rados.rados.0.smithi042.stdout:510: writing smithi04268048-32 from 494172 to 951801 tid 1 2024-09-20T19:23:41.152 INFO:tasks.rados.rados.0.smithi042.stdout:510: writing smithi04268048-32 from 1586353 to 2098738 tid 2 2024-09-20T19:23:41.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:41.159 INFO:tasks.rados.rados.0.smithi042.stdout:510: writing smithi04268048-32 from 2749730 to 3294172 tid 3 2024-09-20T19:23:41.159 INFO:tasks.rados.rados.0.smithi042.stdout:511: copy_from oid 4 from oid 18 current snap is 59 2024-09-20T19:23:41.159 INFO:tasks.rados.rados.0.smithi042.stdout:509: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:23:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:41.191 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2616991 2024-09-20T19:23:41.194 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:41.247 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 952278 2024-09-20T19:23:41.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:41.267 INFO:tasks.rados.rados.0.smithi042.stdout:508: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:23:41.267 INFO:tasks.rados.rados.0.smithi042.stdout:510: finishing write tid 1 to smithi04268048-32 2024-09-20T19:23:41.267 INFO:tasks.rados.rados.0.smithi042.stdout:510: finishing write tid 2 to smithi04268048-32 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:508: finishing copy_from to smithi04268048-1 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 67 (ObjNum 134 snap 35 seq_num 134) dirty exists 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: finishing write tid 3 to smithi04268048-32 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: finishing write tid 4 to smithi04268048-32 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: finishing write tid 5 to smithi04268048-32 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: oid 32 updating version 0 to 259 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: oid 32 updating version 259 to 260 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: oid 32 updating version 260 to 261 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: oid 32 version 261 is already newer than 258 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 261 (ObjNum 196 snap 59 seq_num 196) dirty exists 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:510: left oid 32 (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:511: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:511: finishing copy_from to smithi04268048-4 2024-09-20T19:23:41.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 162 (ObjNum 155 snap 44 seq_num 155) dirty exists 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:508: done (3 left) 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:509: done (2 left) 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:510: done (1 left) 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:511: done (0 left) 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:512: read oid 13 snap -1 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:512: expect (ObjNum 144 snap 39 seq_num 144) 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:513: write oid 45 current snap is 59 2024-09-20T19:23:41.269 INFO:tasks.rados.rados.0.smithi042.stdout:513: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-20T19:23:41.280 INFO:tasks.rados.rados.0.smithi042.stdout:513: writing smithi04268048-45 from 510979 to 1043857 tid 1 2024-09-20T19:23:41.288 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:41.292 INFO:tasks.rados.rados.0.smithi042.stdout:513: writing smithi04268048-45 from 1528482 to 2300878 tid 2 2024-09-20T19:23:41.300 INFO:tasks.rados.rados.0.smithi042.stdout:513: writing smithi04268048-45 from 3060800 to 3310979 tid 3 2024-09-20T19:23:41.300 INFO:tasks.rados.rados.0.smithi042.stdout:514: write oid 11 current snap is 59 2024-09-20T19:23:41.300 INFO:tasks.rados.rados.0.smithi042.stdout:514: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-20T19:23:41.310 INFO:tasks.rados.rados.0.smithi042.stdout:514: writing smithi04268048-11 from 527786 to 1252266 tid 1 2024-09-20T19:23:41.318 INFO:tasks.rados.rados.0.smithi042.stdout:514: writing smithi04268048-11 from 1703317 to 2219371 tid 2 2024-09-20T19:23:41.327 INFO:tasks.rados.rados.0.smithi042.stdout:514: writing smithi04268048-11 from 2804576 to 3239611 tid 3 2024-09-20T19:23:41.327 INFO:tasks.rados.rados.0.smithi042.stdout:515: read oid 40 snap -1 2024-09-20T19:23:41.327 INFO:tasks.rados.rados.0.smithi042.stdout:515: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:41.327 INFO:tasks.rados.rados.0.smithi042.stdout:516: delete oid 20 current snap is 59 2024-09-20T19:23:41.327 INFO:tasks.rados.rados.0.smithi042.stdout:512: expect (ObjNum 144 snap 39 seq_num 144) 2024-09-20T19:23:41.362 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:41.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:41 smithi162 bash[24557]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:41 smithi162 bash[24557]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:41 smithi162 bash[24557]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:41 smithi162 bash[24557]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:41.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[22706]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[22706]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[22706]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[22706]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[32075]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[32075]: cluster 2024-09-20T19:23:39.997331+0000 mgr.y (mgr.24491) 217 : cluster [DBG] pgmap v284: 148 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 176 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 6.5 MiB/s wr, 33 op/s 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[32075]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:41 smithi042 bash[32075]: cluster 2024-09-20T19:23:41.136371+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: finishing write tid 1 to smithi04268048-45 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: finishing write tid 2 to smithi04268048-45 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: finishing write tid 3 to smithi04268048-45 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: finishing write tid 4 to smithi04268048-45 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: finishing write tid 5 to smithi04268048-45 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: oid 45 updating version 0 to 263 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: oid 45 updating version 263 to 266 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: oid 45 version 266 is already newer than 265 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:513: oid 45 version 266 is already newer than 264 2024-09-20T19:23:41.437 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 266 (ObjNum 197 snap 59 seq_num 197) dirty exists 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:513: left oid 45 (ObjNum 197 snap 59 seq_num 197) 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: finishing write tid 1 to smithi04268048-11 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: finishing write tid 2 to smithi04268048-11 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: finishing write tid 3 to smithi04268048-11 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: finishing write tid 4 to smithi04268048-11 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: finishing write tid 5 to smithi04268048-11 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: oid 11 updating version 0 to 240 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: oid 11 version 240 is already newer than 237 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: oid 11 version 240 is already newer than 238 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: oid 11 version 240 is already newer than 239 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 240 (ObjNum 198 snap 59 seq_num 198) dirty exists 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:514: left oid 11 (ObjNum 198 snap 59 seq_num 198) 2024-09-20T19:23:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:515: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:41.492 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1563808 2024-09-20T19:23:41.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:41.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:41.574 INFO:tasks.rados.rados.0.smithi042.stdout:512: done (4 left) 2024-09-20T19:23:41.574 INFO:tasks.rados.rados.0.smithi042.stdout:513: done (3 left) 2024-09-20T19:23:41.574 INFO:tasks.rados.rados.0.smithi042.stdout:514: done (2 left) 2024-09-20T19:23:41.574 INFO:tasks.rados.rados.0.smithi042.stdout:515: done (1 left) 2024-09-20T19:23:41.575 INFO:tasks.rados.rados.0.smithi042.stdout:516: done (0 left) 2024-09-20T19:23:41.575 INFO:tasks.rados.rados.0.smithi042.stdout:517: write oid 48 current snap is 59 2024-09-20T19:23:41.575 INFO:tasks.rados.rados.0.smithi042.stdout:517: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-20T19:23:41.580 INFO:tasks.rados.rados.0.smithi042.stdout:517: writing smithi04268048-48 from 561400 to 1036378 tid 1 2024-09-20T19:23:41.587 INFO:tasks.rados.rados.0.smithi042.stdout:517: writing smithi04268048-48 from 1820281 to 2340004 tid 2 2024-09-20T19:23:41.593 INFO:tasks.rados.rados.0.smithi042.stdout:517: writing smithi04268048-48 from 2859422 to 3317001 tid 3 2024-09-20T19:23:41.593 INFO:tasks.rados.rados.0.smithi042.stdout:518: delete oid 42 current snap is 59 2024-09-20T19:23:41.593 INFO:tasks.rados.rados.0.smithi042.stdout:517: finishing write tid 1 to smithi04268048-48 2024-09-20T19:23:41.593 INFO:tasks.rados.rados.0.smithi042.stdout:517: finishing write tid 2 to smithi04268048-48 2024-09-20T19:23:41.596 INFO:tasks.rados.rados.0.smithi042.stdout:518: done (1 left) 2024-09-20T19:23:41.596 INFO:tasks.rados.rados.0.smithi042.stdout:519: snap_remove snap 46 2024-09-20T19:23:41.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:41.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:41.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:41.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:41.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:42.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:42.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:23:42.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:42.161 INFO:tasks.rados.rados.0.smithi042.stdout:519: done (1 left) 2024-09-20T19:23:42.161 INFO:tasks.rados.rados.0.smithi042.stdout:520: write oid 19 current snap is 59 2024-09-20T19:23:42.162 INFO:tasks.rados.rados.0.smithi042.stdout:517: finishing write tid 3 to smithi04268048-48 2024-09-20T19:23:42.162 INFO:tasks.rados.rados.0.smithi042.stdout:520: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-20T19:23:42.169 INFO:tasks.rados.rados.0.smithi042.stdout:520: writing smithi04268048-19 from 595014 to 1220490 tid 1 2024-09-20T19:23:42.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:42 smithi162 bash[24557]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:42 smithi162 bash[24557]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:42 smithi162 bash[24557]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:42 smithi162 bash[24557]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.176 INFO:tasks.rados.rados.0.smithi042.stdout:520: writing smithi04268048-19 from 1820892 to 2344284 tid 2 2024-09-20T19:23:42.183 INFO:tasks.rados.rados.0.smithi042.stdout:520: writing smithi04268048-19 from 2797915 to 3394391 tid 3 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:521: copy_from oid 13 from oid 41 current snap is 59 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: finishing write tid 4 to smithi04268048-48 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: finishing write tid 5 to smithi04268048-48 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: oid 48 updating version 0 to 214 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: oid 48 updating version 214 to 217 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: oid 48 version 217 is already newer than 215 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: oid 48 version 217 is already newer than 216 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 217 (ObjNum 200 snap 59 seq_num 200) dirty exists 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: left oid 48 (ObjNum 200 snap 59 seq_num 200) 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:520: finishing write tid 1 to smithi04268048-19 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:520: finishing write tid 2 to smithi04268048-19 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:517: done (2 left) 2024-09-20T19:23:42.184 INFO:tasks.rados.rados.0.smithi042.stdout:522: read oid 12 snap -1 2024-09-20T19:23:42.185 INFO:tasks.rados.rados.0.smithi042.stdout:522: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:42.185 INFO:tasks.rados.rados.0.smithi042.stdout:523: rollback oid 48 current snap is 59 2024-09-20T19:23:42.185 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 56 2024-09-20T19:23:42.185 INFO:tasks.rados.rados.0.smithi042.stdout:524: snap_create 2024-09-20T19:23:42.185 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:42.190 INFO:tasks.rados.rados.0.smithi042.stdout:523: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:23:42.190 INFO:tasks.rados.rados.0.smithi042.stdout:523: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:23:42.190 INFO:tasks.rados.rados.0.smithi042.stdout:523: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:23:42.190 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 228 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:23:42.190 INFO:tasks.rados.rados.0.smithi042.stdout:520: finishing write tid 3 to smithi04268048-19 2024-09-20T19:23:42.191 INFO:tasks.rados.rados.0.smithi042.stdout:522: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:42.191 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:42.226 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:42.286 INFO:tasks.rados.rados.0.smithi042.stdout:520: finishing write tid 4 to smithi04268048-19 2024-09-20T19:23:42.286 INFO:tasks.rados.rados.0.smithi042.stdout:520: finishing write tid 5 to smithi04268048-19 2024-09-20T19:23:42.286 INFO:tasks.rados.rados.0.smithi042.stdout:520: oid 19 updating version 0 to 105 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:520: oid 19 updating version 105 to 108 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:520: oid 19 version 108 is already newer than 106 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:520: oid 19 version 108 is already newer than 107 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 108 (ObjNum 202 snap 59 seq_num 202) dirty exists 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:520: left oid 19 (ObjNum 202 snap 59 seq_num 202) 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:521: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:521: finishing copy_from to smithi04268048-13 2024-09-20T19:23:42.287 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 94 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:42.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:42.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:42.377 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:23:42.377 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:23:42.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[22706]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[22706]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[22706]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[22706]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[32075]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[32075]: audit 2024-09-20T19:23:41.926515+0000 mon.a (mon.0) 1184 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[32075]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:42 smithi042 bash[32075]: audit 2024-09-20T19:23:42.042513+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:23:42.432 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:42.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:42.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:42.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:42.796 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:42.822 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:42.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:42.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:42.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:520: done (4 left) 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:521: done (3 left) 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:522: done (2 left) 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:523: done (1 left) 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:524: done (0 left) 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:525: read oid 10 snap 56 2024-09-20T19:23:43.050 INFO:tasks.rados.rados.0.smithi042.stdout:525: expect (ObjNum 143 snap 38 seq_num 143) 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:526: copy_from oid 5 from oid 35 current snap is 60 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:527: rollback oid 50 current snap is 60 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 59 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:528: read oid 11 snap -1 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:528: expect (ObjNum 198 snap 59 seq_num 198) 2024-09-20T19:23:43.051 INFO:tasks.rados.rados.0.smithi042.stdout:529: snap_create 2024-09-20T19:23:43.055 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:43.073 INFO:tasks.rados.rados.0.smithi042.stdout:525: expect (ObjNum 143 snap 38 seq_num 143) 2024-09-20T19:23:43.082 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:43.117 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:43.140 INFO:tasks.rados.rados.0.smithi042.stdout:528: expect (ObjNum 198 snap 59 seq_num 198) 2024-09-20T19:23:43.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:43.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:43.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.242 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.243 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:43 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T19:23:43.269 INFO:tasks.rados.rados.0.smithi042.stdout:526: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:23:43.269 INFO:tasks.rados.rados.0.smithi042.stdout:527: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:23:43.270 INFO:tasks.rados.rados.0.smithi042.stdout:527: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:23:43.270 INFO:tasks.rados.rados.0.smithi042.stdout:527: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:23:43.270 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 250 (ObjNum 137 snap 36 seq_num 137) dirty exists 2024-09-20T19:23:43.270 INFO:tasks.rados.rados.0.smithi042.stdout:526: finishing copy_from to smithi04268048-5 2024-09-20T19:23:43.270 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:23:43.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:41.998942+0000 mgr.y (mgr.24491) 218 : cluster [DBG] pgmap v287: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.141786+0000 mon.a (mon.0) 1186 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: audit 2024-09-20T19:23:42.149329+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? v1:172.21.15.42:0/304538389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.160844+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.251612+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.c scrub starts 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.253357+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.c scrub ok 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.677375+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.677741+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:43 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.047870+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-20T19:23:43.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:43.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:43.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:43.586 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:43.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:23:43.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:43.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:43.845 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:43.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:43.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:525: done (4 left) 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:526: done (3 left) 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:527: done (2 left) 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:528: done (1 left) 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:529: done (0 left) 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:530: write (excl) oid 31 current snap is 61 2024-09-20T19:23:44.055 INFO:tasks.rados.rados.0.smithi042.stdout:530: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-20T19:23:44.076 INFO:tasks.rados.rados.0.smithi042.stdout:530: writing smithi04268048-31 from 611821 to 1312546 tid 1 2024-09-20T19:23:44.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:44.099 INFO:tasks.rados.rados.0.smithi042.stdout:530: writing smithi04268048-31 from 1879374 to 2662777 tid 2 2024-09-20T19:23:44.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:44.109 INFO:tasks.rados.rados.0.smithi042.stdout:530: writing smithi04268048-31 from 3341691 to 3411821 tid 3 2024-09-20T19:23:44.109 INFO:tasks.rados.rados.0.smithi042.stdout:531: copy_from oid 21 from oid 25 current snap is 61 2024-09-20T19:23:44.109 INFO:tasks.rados.rados.0.smithi042.stdout:530: finishing write tid 1 to smithi04268048-31 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:530: finishing write tid 2 to smithi04268048-31 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:532: read oid 15 snap -1 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:532: expect (ObjNum 178 snap 53 seq_num 178) 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:533: read oid 40 snap -1 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:533: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:44.110 INFO:tasks.rados.rados.0.smithi042.stdout:534: snap_remove snap 56 2024-09-20T19:23:44.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:44.220 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.221 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:44 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:44.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:44.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.245311+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.246401+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.755751+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:23:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:42.757929+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.285583+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.288796+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:44 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.051126+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-20T19:23:44.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:44.601 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:44.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:44.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:44.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:44.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:44.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:44.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.022 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:45.051 INFO:tasks.rados.rados.0.smithi042.stdout:534: done (4 left) 2024-09-20T19:23:45.051 INFO:tasks.rados.rados.0.smithi042.stdout:535: read oid 37 snap -1 2024-09-20T19:23:45.051 INFO:tasks.rados.rados.0.smithi042.stdout:535: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:45.052 INFO:tasks.rados.rados.0.smithi042.stdout:536: write (excl) oid 38 current snap is 61 2024-09-20T19:23:45.052 INFO:tasks.rados.rados.0.smithi042.stdout:531: finishing copy_from to smithi04268048-21 2024-09-20T19:23:45.052 INFO:tasks.rados.rados.0.smithi042.stdout:531: got expected ENOENT (src dne) 2024-09-20T19:23:45.052 INFO:tasks.rados.rados.0.smithi042.stdout:536: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-20T19:23:45.059 INFO:tasks.rados.rados.0.smithi042.stdout:536: writing smithi04268048-38 from 628628 to 1404602 tid 1 2024-09-20T19:23:45.059 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:23:45.066 INFO:tasks.rados.rados.0.smithi042.stdout:536: writing smithi04268048-38 from 1937856 to 2464917 tid 2 2024-09-20T19:23:45.072 INFO:tasks.rados.rados.0.smithi042.stdout:536: writing smithi04268048-38 from 3252761 to 3428628 tid 3 2024-09-20T19:23:45.072 INFO:tasks.rados.rados.0.smithi042.stdout:531: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: finishing write tid 3 to smithi04268048-31 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: finishing write tid 4 to smithi04268048-31 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: finishing write tid 5 to smithi04268048-31 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: oid 31 updating version 0 to 136 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: oid 31 updating version 136 to 137 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: oid 31 updating version 137 to 138 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: oid 31 version 138 is already newer than 135 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 138 (ObjNum 203 snap 61 seq_num 203) dirty exists 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:530: left oid 31 (ObjNum 203 snap 61 seq_num 203) 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:533: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:45.073 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:45.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:45.138 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1922668 2024-09-20T19:23:45.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 163755 2024-09-20T19:23:45.210 INFO:tasks.rados.rados.0.smithi042.stdout:532: expect (ObjNum 178 snap 53 seq_num 178) 2024-09-20T19:23:45.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.276 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 751393 2024-09-20T19:23:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:45.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:45.379 INFO:tasks.rados.rados.0.smithi042.stdout:535: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:45.379 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:45.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:45 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.495 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:45.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:45.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.750878+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.752725+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:43.999640+0000 mgr.y (mgr.24491) 219 : cluster [DBG] pgmap v290: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:45 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.050396+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-20T19:23:45.530 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: finishing write tid 1 to smithi04268048-38 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: finishing write tid 2 to smithi04268048-38 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: finishing write tid 3 to smithi04268048-38 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: finishing write tid 4 to smithi04268048-38 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: finishing write tid 5 to smithi04268048-38 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: oid 38 updating version 0 to 168 2024-09-20T19:23:45.553 INFO:tasks.rados.rados.0.smithi042.stdout:536: oid 38 updating version 168 to 170 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:536: oid 38 version 170 is already newer than 167 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:536: oid 38 version 170 is already newer than 169 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 170 (ObjNum 204 snap 61 seq_num 204) dirty exists 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:536: left oid 38 (ObjNum 204 snap 61 seq_num 204) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:530: done (5 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:531: done (4 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:532: done (3 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:533: done (2 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:535: done (1 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:536: done (0 left) 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:537: read oid 42 snap -1 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:537: expect deleted 2024-09-20T19:23:45.554 INFO:tasks.rados.rados.0.smithi042.stdout:538: copy_from oid 33 from oid 13 current snap is 61 2024-09-20T19:23:45.555 INFO:tasks.rados.rados.0.smithi042.stdout:539: write (excl) oid 28 current snap is 61 2024-09-20T19:23:45.555 INFO:tasks.rados.rados.0.smithi042.stdout:539: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-20T19:23:45.573 INFO:tasks.rados.rados.0.smithi042.stdout:539: writing smithi04268048-28 from 645435 to 1096658 tid 1 2024-09-20T19:23:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:45.589 INFO:tasks.rados.rados.0.smithi042.stdout:539: writing smithi04268048-28 from 1879985 to 2667057 tid 2 2024-09-20T19:23:45.594 INFO:tasks.rados.rados.0.smithi042.stdout:539: writing smithi04268048-28 from 3280184 to 3445435 tid 3 2024-09-20T19:23:45.594 INFO:tasks.rados.rados.0.smithi042.stdout:540: snap_create 2024-09-20T19:23:45.594 INFO:tasks.rados.rados.0.smithi042.stdout:538: finishing copy_from racing read to smithi04268048-33 2024-09-20T19:23:45.595 INFO:tasks.rados.rados.0.smithi042.stdout:538: finishing copy_from to smithi04268048-33 2024-09-20T19:23:45.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 94 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:45.595 INFO:tasks.rados.rados.0.smithi042.stdout:539: finishing write tid 1 to smithi04268048-28 2024-09-20T19:23:45.595 INFO:tasks.rados.rados.0.smithi042.stdout:539: finishing write tid 2 to smithi04268048-28 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: finishing write tid 3 to smithi04268048-28 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: finishing write tid 4 to smithi04268048-28 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: finishing write tid 5 to smithi04268048-28 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: oid 28 updating version 0 to 114 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: oid 28 updating version 114 to 115 2024-09-20T19:23:45.596 INFO:tasks.rados.rados.0.smithi042.stdout:539: oid 28 version 115 is already newer than 113 2024-09-20T19:23:45.597 INFO:tasks.rados.rados.0.smithi042.stdout:539: oid 28 version 115 is already newer than 112 2024-09-20T19:23:45.597 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 115 (ObjNum 205 snap 61 seq_num 205) dirty exists 2024-09-20T19:23:45.597 INFO:tasks.rados.rados.0.smithi042.stdout:539: left oid 28 (ObjNum 205 snap 61 seq_num 205) 2024-09-20T19:23:45.633 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:45.651 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:45.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:45.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:45.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:45.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:46.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:46.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:46.137 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.163 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:537: done (3 left) 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:538: done (2 left) 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:539: done (1 left) 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:540: done (0 left) 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:541: write (excl) oid 24 current snap is 62 2024-09-20T19:23:46.172 INFO:tasks.rados.rados.0.smithi042.stdout:541: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-20T19:23:46.179 INFO:tasks.rados.rados.0.smithi042.stdout:541: writing smithi04268048-24 from 662242 to 1305067 tid 1 2024-09-20T19:23:46.186 INFO:tasks.rados.rados.0.smithi042.stdout:541: writing smithi04268048-24 from 2054820 to 2585550 tid 2 2024-09-20T19:23:46.191 INFO:tasks.rados.rados.0.smithi042.stdout:541: writing smithi04268048-24 from 3307607 to 3462242 tid 3 2024-09-20T19:23:46.191 INFO:tasks.rados.rados.0.smithi042.stdout:542: snap_remove snap 57 2024-09-20T19:23:46.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:46.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:46.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:46 smithi162 bash[24557]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[32075]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.570093+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.572397+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.718432+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:46 smithi042 bash[22706]: cluster 2024-09-20T19:23:44.721428+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:23:46.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:46.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:46.688 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:46.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:46.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:46.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:46.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:47.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:47.079 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:47.175 INFO:tasks.rados.rados.0.smithi042.stdout:542: done (1 left) 2024-09-20T19:23:47.176 INFO:tasks.rados.rados.0.smithi042.stdout:543: write oid 29 current snap is 62 2024-09-20T19:23:47.176 INFO:tasks.rados.rados.0.smithi042.stdout:543: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-20T19:23:47.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:47.184 INFO:tasks.rados.rados.0.smithi042.stdout:543: writing smithi04268048-29 from 679049 to 1397123 tid 1 2024-09-20T19:23:47.192 INFO:tasks.rados.rados.0.smithi042.stdout:543: writing smithi04268048-29 from 2113302 to 2787690 tid 2 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:543: writing smithi04268048-29 from 3335030 to 3479049 tid 3 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:544: delete oid 16 current snap is 62 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: finishing write tid 1 to smithi04268048-24 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: finishing write tid 2 to smithi04268048-24 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: finishing write tid 3 to smithi04268048-24 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: finishing write tid 4 to smithi04268048-24 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: finishing write tid 5 to smithi04268048-24 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: oid 24 updating version 0 to 177 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: oid 24 version 177 is already newer than 176 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: oid 24 updating version 177 to 179 2024-09-20T19:23:47.197 INFO:tasks.rados.rados.0.smithi042.stdout:541: oid 24 version 179 is already newer than 178 2024-09-20T19:23:47.198 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 179 (ObjNum 206 snap 62 seq_num 206) dirty exists 2024-09-20T19:23:47.198 INFO:tasks.rados.rados.0.smithi042.stdout:541: left oid 24 (ObjNum 206 snap 62 seq_num 206) 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:541: done (2 left) 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:544: done (1 left) 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:545: rollback oid 32 current snap is 62 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 60 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:546: write oid 21 current snap is 62 2024-09-20T19:23:47.201 INFO:tasks.rados.rados.0.smithi042.stdout:546: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-20T19:23:47.208 INFO:tasks.rados.rados.0.smithi042.stdout:546: writing smithi04268048-21 from 712663 to 1181235 tid 1 2024-09-20T19:23:47.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:47.215 INFO:tasks.rados.rados.0.smithi042.stdout:546: writing smithi04268048-21 from 1713913 to 2391970 tid 2 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:546: writing smithi04268048-21 from 2873523 to 3465256 tid 3 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:547: snap_create 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: finishing write tid 1 to smithi04268048-29 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: finishing write tid 2 to smithi04268048-29 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: finishing write tid 3 to smithi04268048-29 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: finishing write tid 4 to smithi04268048-29 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: finishing write tid 5 to smithi04268048-29 2024-09-20T19:23:47.222 INFO:tasks.rados.rados.0.smithi042.stdout:543: oid 29 updating version 0 to 174 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:543: oid 29 version 174 is already newer than 173 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:543: oid 29 version 174 is already newer than 171 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:543: oid 29 version 174 is already newer than 172 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 174 (ObjNum 207 snap 62 seq_num 207) dirty exists 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:543: left oid 29 (ObjNum 207 snap 62 seq_num 207) 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:545: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:545: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:545: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 286 (ObjNum 196 snap 59 seq_num 196) dirty exists 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:546: finishing write tid 1 to smithi04268048-21 2024-09-20T19:23:47.223 INFO:tasks.rados.rados.0.smithi042.stdout:546: finishing write tid 2 to smithi04268048-21 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: finishing write tid 3 to smithi04268048-21 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: finishing write tid 4 to smithi04268048-21 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: finishing write tid 5 to smithi04268048-21 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: oid 21 updating version 0 to 234 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: oid 21 updating version 234 to 236 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: oid 21 updating version 236 to 237 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: oid 21 version 237 is already newer than 235 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 237 (ObjNum 209 snap 62 seq_num 209) dirty exists 2024-09-20T19:23:47.225 INFO:tasks.rados.rados.0.smithi042.stdout:546: left oid 21 (ObjNum 209 snap 62 seq_num 209) 2024-09-20T19:23:47.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:47.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:47.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:47 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:47.466 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:47.509 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:23:47.511 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.540311+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.542575+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.929203+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:45.929538+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.000332+0000 mgr.y (mgr.24491) 220 : cluster [DBG] pgmap v293: 148 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 141 active+clean; 187 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:47 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.172628+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-20T19:23:47.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:47.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:47.614 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:47.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:47.716 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:47.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:47.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:47.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:47.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:47.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:48.058 INFO:tasks.rados.rados.0.smithi042.stdout:543: done (3 left) 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:545: done (2 left) 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:546: done (1 left) 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:547: done (0 left) 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:548: read oid 1 snap -1 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:548: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:549: rollback oid 6 current snap is 63 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 62 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:550: copy_from oid 5 from oid 42 current snap is 63 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:551: write oid 15 current snap is 63 2024-09-20T19:23:48.059 INFO:tasks.rados.rados.0.smithi042.stdout:551: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-20T19:23:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:23:48.068 INFO:tasks.rados.rados.0.smithi042.stdout:551: writing smithi04268048-15 from 729470 to 1273291 tid 1 2024-09-20T19:23:48.077 INFO:tasks.rados.rados.0.smithi042.stdout:551: writing smithi04268048-15 from 1772395 to 2310463 tid 2 2024-09-20T19:23:48.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:48.084 INFO:tasks.rados.rados.0.smithi042.stdout:551: writing smithi04268048-15 from 2900946 to 3303951 tid 3 2024-09-20T19:23:48.084 INFO:tasks.rados.rados.0.smithi042.stdout:552: copy_from oid 25 from oid 10 current snap is 63 2024-09-20T19:23:48.084 INFO:tasks.rados.rados.0.smithi042.stdout:550: finishing copy_from to smithi04268048-5 2024-09-20T19:23:48.084 INFO:tasks.rados.rados.0.smithi042.stdout:550: got expected ENOENT (src dne) 2024-09-20T19:23:48.084 INFO:tasks.rados.rados.0.smithi042.stdout:550: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:23:48.085 INFO:tasks.rados.rados.0.smithi042.stdout:548: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:48.103 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:48.106 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:48.130 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 553888 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:549: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:549: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:549: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 150 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: finishing write tid 1 to smithi04268048-15 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: finishing write tid 2 to smithi04268048-15 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: finishing write tid 3 to smithi04268048-15 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: finishing write tid 4 to smithi04268048-15 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: finishing write tid 5 to smithi04268048-15 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: oid 15 updating version 0 to 184 2024-09-20T19:23:48.151 INFO:tasks.rados.rados.0.smithi042.stdout:551: oid 15 updating version 184 to 185 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:551: oid 15 updating version 185 to 186 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:551: oid 15 version 186 is already newer than 183 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 186 (ObjNum 210 snap 63 seq_num 210) dirty exists 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:551: left oid 15 (ObjNum 210 snap 63 seq_num 210) 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:552: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:548: done (4 left) 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:549: done (3 left) 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:550: done (2 left) 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:551: done (1 left) 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:553: write (excl) oid 6 current snap is 63 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:552: finishing copy_from to smithi04268048-25 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 147 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-09-20T19:23:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:553: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-20T19:23:48.158 INFO:tasks.rados.rados.0.smithi042.stdout:553: writing smithi04268048-6 from 746277 to 1365347 tid 1 2024-09-20T19:23:48.166 INFO:tasks.rados.rados.0.smithi042.stdout:553: writing smithi04268048-6 from 1830877 to 2512603 tid 2 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:553: writing smithi04268048-6 from 2928369 to 3546277 tid 3 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:552: done (1 left) 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:554: read oid 50 snap 62 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:554: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:555: write (excl) oid 18 current snap is 63 2024-09-20T19:23:48.171 INFO:tasks.rados.rados.0.smithi042.stdout:555: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-20T19:23:48.179 INFO:tasks.rados.rados.0.smithi042.stdout:555: writing smithi04268048-18 from 763084 to 1457403 tid 1 2024-09-20T19:23:48.185 INFO:tasks.rados.rados.0.smithi042.stdout:555: writing smithi04268048-18 from 1889359 to 2431096 tid 2 2024-09-20T19:23:48.192 INFO:tasks.rados.rados.0.smithi042.stdout:555: writing smithi04268048-18 from 2955792 to 3497694 tid 3 2024-09-20T19:23:48.192 INFO:tasks.rados.rados.0.smithi042.stdout:556: delete oid 46 current snap is 63 2024-09-20T19:23:48.192 INFO:tasks.rados.rados.0.smithi042.stdout:553: finishing write tid 1 to smithi04268048-6 2024-09-20T19:23:48.192 INFO:tasks.rados.rados.0.smithi042.stdout:553: finishing write tid 2 to smithi04268048-6 2024-09-20T19:23:48.192 INFO:tasks.rados.rados.0.smithi042.stdout:553: finishing write tid 3 to smithi04268048-6 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: finishing write tid 4 to smithi04268048-6 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: finishing write tid 5 to smithi04268048-6 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: oid 6 updating version 0 to 152 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: oid 6 version 152 is already newer than 151 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: oid 6 updating version 152 to 154 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: oid 6 version 154 is already newer than 153 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 154 (ObjNum 211 snap 63 seq_num 211) dirty exists 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:553: left oid 6 (ObjNum 211 snap 63 seq_num 211) 2024-09-20T19:23:48.193 INFO:tasks.rados.rados.0.smithi042.stdout:554: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:23:48.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.218 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.219 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:48 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.225 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1987261 2024-09-20T19:23:48.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:48.259 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: finishing write tid 1 to smithi04268048-18 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: finishing write tid 2 to smithi04268048-18 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: finishing write tid 3 to smithi04268048-18 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: finishing write tid 4 to smithi04268048-18 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: finishing write tid 5 to smithi04268048-18 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: oid 18 updating version 0 to 241 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: oid 18 updating version 241 to 243 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: oid 18 version 243 is already newer than 240 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: oid 18 version 243 is already newer than 242 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 243 (ObjNum 212 snap 63 seq_num 212) dirty exists 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:555: left oid 18 (ObjNum 212 snap 63 seq_num 212) 2024-09-20T19:23:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:553: done (3 left) 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:554: done (2 left) 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:555: done (1 left) 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:556: done (0 left) 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:557: read oid 46 snap -1 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:557: expect deleted 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:558: copy_from oid 13 from oid 32 current snap is 63 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:559: write oid 29 current snap is 63 2024-09-20T19:23:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:559: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-20T19:23:48.297 INFO:tasks.rados.rados.0.smithi042.stdout:559: writing smithi04268048-29 from 796698 to 1357868 tid 1 2024-09-20T19:23:48.307 INFO:tasks.rados.rados.0.smithi042.stdout:559: writing smithi04268048-29 from 2006323 to 2551729 tid 2 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:559: writing smithi04268048-29 from 3126991 to 3596698 tid 3 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:560: copy_from oid 39 from oid 8 current snap is 63 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:558: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:557: done (3 left) 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:561: snap_create 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:558: finishing copy_from to smithi04268048-13 2024-09-20T19:23:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 286 (ObjNum 196 snap 59 seq_num 196) dirty exists 2024-09-20T19:23:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:559: finishing write tid 1 to smithi04268048-29 2024-09-20T19:23:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:559: finishing write tid 2 to smithi04268048-29 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: finishing write tid 3 to smithi04268048-29 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: finishing write tid 4 to smithi04268048-29 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: finishing write tid 5 to smithi04268048-29 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: oid 29 updating version 0 to 178 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: oid 29 updating version 178 to 179 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: oid 29 version 179 is already newer than 176 2024-09-20T19:23:48.319 INFO:tasks.rados.rados.0.smithi042.stdout:559: oid 29 version 179 is already newer than 177 2024-09-20T19:23:48.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 179 (ObjNum 214 snap 63 seq_num 214) dirty exists 2024-09-20T19:23:48.320 INFO:tasks.rados.rados.0.smithi042.stdout:559: left oid 29 (ObjNum 214 snap 63 seq_num 214) 2024-09-20T19:23:48.328 INFO:tasks.rados.rados.0.smithi042.stdout:560: finishing copy_from racing read to smithi04268048-39 2024-09-20T19:23:48.334 INFO:tasks.rados.rados.0.smithi042.stdout:560: finishing copy_from to smithi04268048-39 2024-09-20T19:23:48.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 27 (ObjNum 68 snap 10 seq_num 68) dirty exists 2024-09-20T19:23:48.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:48.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:48.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.298941+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:23:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.300360+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.631989+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:46.636878+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.001237+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.003592+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.174264+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.196161+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:23:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.200167+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:23:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: audit 2024-09-20T19:23:47.499827+0000 mgr.y (mgr.24491) 221 : audit [DBG] from='client.15078 v1:172.21.15.42:0/3418906645' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:48 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.050219+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-20T19:23:48.465 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:48.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:48.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:48.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:48.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:48.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:48.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:48.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:48.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:48.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:49.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:49.086 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-20T19:23:47.170881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154479896,"num_objects":259,"num_object_clones":19,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":2,"num_read":2335,"num_read_kb":810453,"num_write":1390,"num_write_kb":319015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2796,"ondisk_log_size":2796,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":659468,"kb_used_data":380824,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":749301748,"statfs":{"total":767960285184,"available":767284989952,"internally_reserved":0,"allocated":389963776,"data_stored":412749695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-41785393,"num_objects":-28,"num_object_clones":-26,"num_object_copies":-84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":-4,"num_read":69,"num_read_kb":57830,"num_write":32,"num_write_kb":16239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.129438"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":358,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184073+0000","last_change":"2024-09-20T19:21:59.826969+0000","last_active":"2024-09-20T19:23:46.184073+0000","last_peered":"2024-09-20T19:23:46.184073+0000","last_clean":"2024-09-20T19:23:46.184073+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:23:46.184073+0000","last_undegraded":"2024-09-20T19:23:46.184073+0000","last_fullsized":"2024-09-20T19:23:46.184073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:59.826916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:59.826916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":371,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171546+0000","last_change":"2024-09-20T19:22:18.623572+0000","last_active":"2024-09-20T19:23:46.171546+0000","last_peered":"2024-09-20T19:23:46.171546+0000","last_clean":"2024-09-20T19:23:46.171546+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:23:46.171546+0000","last_undegraded":"2024-09-20T19:23:46.171546+0000","last_fullsized":"2024-09-20T19:23:46.171546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.623500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.623500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":391,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184181+0000","last_change":"2024-09-20T19:22:14.898580+0000","last_active":"2024-09-20T19:23:46.184181+0000","last_peered":"2024-09-20T19:23:46.184181+0000","last_clean":"2024-09-20T19:23:46.184181+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:23:46.184181+0000","last_undegraded":"2024-09-20T19:23:46.184181+0000","last_fullsized":"2024-09-20T19:23:46.184181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:14.898494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.898494+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":355,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.177301+0000","last_change":"2024-09-20T19:20:33.800940+0000","last_active":"2024-09-20T19:23:46.177301+0000","last_peered":"2024-09-20T19:23:46.177301+0000","last_clean":"2024-09-20T19:23:46.177301+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:23:46.177301+0000","last_undegraded":"2024-09-20T19:23:46.177301+0000","last_fullsized":"2024-09-20T19:23:46.177301+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:33.800794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:33.800794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.141852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":370,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.185145+0000","last_change":"2024-09-20T19:22:58.622973+0000","last_active":"2024-09-20T19:23:46.185145+0000","last_peered":"2024-09-20T19:23:46.185145+0000","last_clean":"2024-09-20T19:23:46.185145+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:23:46.185145+0000","last_undegraded":"2024-09-20T19:23:46.185145+0000","last_fullsized":"2024-09-20T19:23:46.185145+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:58.622796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.622796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.819628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":375,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.177600+0000","last_change":"2024-09-20T19:20:34.776570+0000","last_active":"2024-09-20T19:23:46.177600+0000","last_peered":"2024-09-20T19:23:46.177600+0000","last_clean":"2024-09-20T19:23:46.177600+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:23:46.177600+0000","last_undegraded":"2024-09-20T19:23:46.177600+0000","last_fullsized":"2024-09-20T19:23:46.177600+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:34.776387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:34.776387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239044+0000","last_change":"2024-09-20T19:22:03.981228+0000","last_active":"2024-09-20T19:23:46.239044+0000","last_peered":"2024-09-20T19:23:46.239044+0000","last_clean":"2024-09-20T19:23:46.239044+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:23:46.239044+0000","last_undegraded":"2024-09-20T19:23:46.239044+0000","last_fullsized":"2024-09-20T19:23:46.239044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.981101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.981101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.455385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178038+0000","last_change":"2024-09-20T19:22:14.871281+0000","last_active":"2024-09-20T19:23:46.178038+0000","last_peered":"2024-09-20T19:23:46.178038+0000","last_clean":"2024-09-20T19:23:46.178038+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:23:46.178038+0000","last_undegraded":"2024-09-20T19:23:46.178038+0000","last_fullsized":"2024-09-20T19:23:46.178038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:14.871139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.871139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.971798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.249184+0000","last_change":"2024-09-20T19:23:08.305837+0000","last_active":"2024-09-20T19:23:46.249184+0000","last_peered":"2024-09-20T19:23:46.249184+0000","last_clean":"2024-09-20T19:23:46.249184+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:23:46.249184+0000","last_undegraded":"2024-09-20T19:23:46.249184+0000","last_fullsized":"2024-09-20T19:23:46.249184+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.305714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.305714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:26.456491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":377,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.177935+0000","last_change":"2024-09-20T19:23:02.238503+0000","last_active":"2024-09-20T19:23:46.177935+0000","last_peered":"2024-09-20T19:23:46.177935+0000","last_clean":"2024-09-20T19:23:46.177935+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:23:46.177935+0000","last_undegraded":"2024-09-20T19:23:46.177935+0000","last_fullsized":"2024-09-20T19:23:46.177935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:23:02.238438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.238438+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:03.780590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245239+0000","last_change":"2024-09-20T19:22:08.909960+0000","last_active":"2024-09-20T19:23:46.245239+0000","last_peered":"2024-09-20T19:23:46.245239+0000","last_clean":"2024-09-20T19:23:46.245239+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:23:46.245239+0000","last_undegraded":"2024-09-20T19:23:46.245239+0000","last_fullsized":"2024-09-20T19:23:46.245239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:08.909835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.909835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":357,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245493+0000","last_change":"2024-09-20T19:22:20.013446+0000","last_active":"2024-09-20T19:23:46.245493+0000","last_peered":"2024-09-20T19:23:46.245493+0000","last_clean":"2024-09-20T19:23:46.245493+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:23:46.245493+0000","last_undegraded":"2024-09-20T19:23:46.245493+0000","last_fullsized":"2024-09-20T19:23:46.245493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.013380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.013380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246686+0000","last_change":"2024-09-20T19:22:20.892258+0000","last_active":"2024-09-20T19:23:46.246686+0000","last_peered":"2024-09-20T19:23:46.246686+0000","last_clean":"2024-09-20T19:23:46.246686+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:23:46.246686+0000","last_undegraded":"2024-09-20T19:23:46.246686+0000","last_fullsized":"2024-09-20T19:23:46.246686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.892102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.892102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":383,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178644+0000","last_change":"2024-09-20T19:22:07.939814+0000","last_active":"2024-09-20T19:23:46.178644+0000","last_peered":"2024-09-20T19:23:46.178644+0000","last_clean":"2024-09-20T19:23:46.178644+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:23:46.178644+0000","last_undegraded":"2024-09-20T19:23:46.178644+0000","last_fullsized":"2024-09-20T19:23:46.178644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:07.939740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:07.939740+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.138214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":353,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.247820+0000","last_change":"2024-09-20T19:21:07.802088+0000","last_active":"2024-09-20T19:23:46.247820+0000","last_peered":"2024-09-20T19:23:46.247820+0000","last_clean":"2024-09-20T19:23:46.247820+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:23:46.247820+0000","last_undegraded":"2024-09-20T19:23:46.247820+0000","last_fullsized":"2024-09-20T19:23:46.247820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.801948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.801948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.780869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":351,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.174493+0000","last_change":"2024-09-20T19:21:08.846790+0000","last_active":"2024-09-20T19:23:46.174493+0000","last_peered":"2024-09-20T19:23:46.174493+0000","last_clean":"2024-09-20T19:23:46.174493+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:23:46.174493+0000","last_undegraded":"2024-09-20T19:23:46.174493+0000","last_fullsized":"2024-09-20T19:23:46.174493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.846665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.846665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.808843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"236'77","reported_seq":702,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171931+0000","last_change":"2024-09-20T19:23:45.252001+0000","last_active":"2024-09-20T19:23:46.171931+0000","last_peered":"2024-09-20T19:23:46.171931+0000","last_clean":"2024-09-20T19:23:46.171931+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:23:46.171931+0000","last_undegraded":"2024-09-20T19:23:46.171931+0000","last_fullsized":"2024-09-20T19:23:46.171931+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'45","last_scrub_stamp":"2024-09-20T19:22:54.853379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.853379+0000","objects_scrubbed":2,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:09.597498+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0065001520000000004,"stat_sum":{"num_bytes":2252138,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":11858,"num_write":31,"num_write_kb":8005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.d","version":"77'17","reported_seq":387,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179212+0000","last_change":"2024-09-20T19:21:27.280916+0000","last_active":"2024-09-20T19:23:46.179212+0000","last_peered":"2024-09-20T19:23:46.179212+0000","last_clean":"2024-09-20T19:23:46.179212+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:23:46.179212+0000","last_undegraded":"2024-09-20T19:23:46.179212+0000","last_fullsized":"2024-09-20T19:23:46.179212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:21:27.280627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:27.280627+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.122506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246430+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:23:46.246430+0000","last_peered":"2024-09-20T19:23:46.246430+0000","last_clean":"2024-09-20T19:23:46.246430+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:23:46.246430+0000","last_undegraded":"2024-09-20T19:23:46.246430+0000","last_fullsized":"2024-09-20T19:23:46.246430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":355,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239302+0000","last_change":"2024-09-20T19:21:25.836294+0000","last_active":"2024-09-20T19:23:46.239302+0000","last_peered":"2024-09-20T19:23:46.239302+0000","last_clean":"2024-09-20T19:23:46.239302+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:23:46.239302+0000","last_undegraded":"2024-09-20T19:23:46.239302+0000","last_fullsized":"2024-09-20T19:23:46.239302+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.836217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.836217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172233+0000","last_change":"2024-09-20T19:21:25.309366+0000","last_active":"2024-09-20T19:23:46.172233+0000","last_peered":"2024-09-20T19:23:46.172233+0000","last_clean":"2024-09-20T19:23:46.172233+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:23:46.172233+0000","last_undegraded":"2024-09-20T19:23:46.172233+0000","last_fullsized":"2024-09-20T19:23:46.172233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.309306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.309306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"236'146","reported_seq":772,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171306+0000","last_change":"2024-09-20T19:23:45.065319+0000","last_active":"2024-09-20T19:23:46.171306+0000","last_peered":"2024-09-20T19:23:46.171306+0000","last_clean":"2024-09-20T19:23:46.171306+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:23:46.171306+0000","last_undegraded":"2024-09-20T19:23:46.171306+0000","last_fullsized":"2024-09-20T19:23:46.171306+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'100","last_scrub_stamp":"2024-09-20T19:23:11.701907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.701907+0000","objects_scrubbed":3,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.011856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.01051244,"stat_sum":{"num_bytes":8235397,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":57819,"num_write":53,"num_write_kb":17483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.c","version":"77'10","reported_seq":390,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179375+0000","last_change":"2024-09-20T19:23:42.253431+0000","last_active":"2024-09-20T19:23:46.179375+0000","last_peered":"2024-09-20T19:23:46.179375+0000","last_clean":"2024-09-20T19:23:46.179375+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:23:46.179375+0000","last_undegraded":"2024-09-20T19:23:46.179375+0000","last_fullsized":"2024-09-20T19:23:46.179375+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:42.253363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.253363+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.814248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239640+0000","last_change":"2024-09-20T19:21:16.692876+0000","last_active":"2024-09-20T19:23:46.239640+0000","last_peered":"2024-09-20T19:23:46.239640+0000","last_clean":"2024-09-20T19:23:46.239640+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:23:46.239640+0000","last_undegraded":"2024-09-20T19:23:46.239640+0000","last_fullsized":"2024-09-20T19:23:46.239640+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:16.692759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.692759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.508852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":435,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171828+0000","last_change":"2024-09-20T19:21:19.204377+0000","last_active":"2024-09-20T19:23:46.171828+0000","last_peered":"2024-09-20T19:23:46.171828+0000","last_clean":"2024-09-20T19:23:46.171828+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:23:46.171828+0000","last_undegraded":"2024-09-20T19:23:46.171828+0000","last_fullsized":"2024-09-20T19:23:46.171828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:19.204313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:19.204313+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":351,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179010+0000","last_change":"2024-09-20T19:21:00.850362+0000","last_active":"2024-09-20T19:23:46.179010+0000","last_peered":"2024-09-20T19:23:46.179010+0000","last_clean":"2024-09-20T19:23:46.179010+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:23:46.179010+0000","last_undegraded":"2024-09-20T19:23:46.179010+0000","last_fullsized":"2024-09-20T19:23:46.179010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.850236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.850236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.099845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"236'142","reported_seq":763,"reported_epoch":237,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:46.192392+0000","last_change":"2024-09-20T19:23:45.562672+0000","last_active":"2024-09-20T19:23:46.192392+0000","last_peered":"2024-09-20T19:23:46.192392+0000","last_clean":"2024-09-20T19:23:46.192392+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:23:46.192392+0000","last_undegraded":"2024-09-20T19:23:46.192392+0000","last_fullsized":"2024-09-20T19:23:46.192392+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'57","last_scrub_stamp":"2024-09-20T19:22:52.771766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771766+0000","objects_scrubbed":6,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:05.937335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0303917389999999,"stat_sum":{"num_bytes":16173413,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":48907,"num_write":55,"num_write_kb":18023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"4"}]},{"pgid":"4.f","version":"77'15","reported_seq":392,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.240339+0000","last_change":"2024-09-20T19:23:04.313290+0000","last_active":"2024-09-20T19:23:46.240339+0000","last_peered":"2024-09-20T19:23:46.240339+0000","last_clean":"2024-09-20T19:23:46.240339+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:23:46.240339+0000","last_undegraded":"2024-09-20T19:23:46.240339+0000","last_fullsized":"2024-09-20T19:23:46.240339+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.291138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":371,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239512+0000","last_change":"2024-09-20T19:23:45.542703+0000","last_active":"2024-09-20T19:23:46.239512+0000","last_peered":"2024-09-20T19:23:46.239512+0000","last_clean":"2024-09-20T19:23:46.239512+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:23:46.239512+0000","last_undegraded":"2024-09-20T19:23:46.239512+0000","last_fullsized":"2024-09-20T19:23:46.239512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:45.542589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:45.542589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.881406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":441,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179635+0000","last_change":"2024-09-20T19:23:03.226408+0000","last_active":"2024-09-20T19:23:46.179635+0000","last_peered":"2024-09-20T19:23:46.179635+0000","last_clean":"2024-09-20T19:23:46.179635+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:23:46.179635+0000","last_undegraded":"2024-09-20T19:23:46.179635+0000","last_fullsized":"2024-09-20T19:23:46.179635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.212037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":357,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.248046+0000","last_change":"2024-09-20T19:23:03.945590+0000","last_active":"2024-09-20T19:23:46.248046+0000","last_peered":"2024-09-20T19:23:46.248046+0000","last_clean":"2024-09-20T19:23:46.248046+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:23:46.248046+0000","last_undegraded":"2024-09-20T19:23:46.248046+0000","last_fullsized":"2024-09-20T19:23:46.248046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:03.945432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.945432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.503088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"232'79","reported_seq":713,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191295+0000","last_change":"2024-09-20T19:23:45.563109+0000","last_active":"2024-09-20T19:23:46.191295+0000","last_peered":"2024-09-20T19:23:46.191295+0000","last_clean":"2024-09-20T19:23:46.191295+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:23:46.191295+0000","last_undegraded":"2024-09-20T19:23:46.191295+0000","last_fullsized":"2024-09-20T19:23:46.191295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'64","last_scrub_stamp":"2024-09-20T19:23:13.624271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:13.624271+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.085077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00067029700000000004,"stat_sum":{"num_bytes":7445468,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":60151,"num_write":34,"num_write_kb":14539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.e","version":"77'11","reported_seq":386,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179894+0000","last_change":"2024-09-20T19:22:58.180428+0000","last_active":"2024-09-20T19:23:46.179894+0000","last_peered":"2024-09-20T19:23:46.179894+0000","last_clean":"2024-09-20T19:23:46.179894+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:23:46.179894+0000","last_undegraded":"2024-09-20T19:23:46.179894+0000","last_fullsized":"2024-09-20T19:23:46.179894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:58.180287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.180287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.237767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":238,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.170532+0000","last_change":"2024-09-20T19:21:53.043331+0000","last_active":"2024-09-20T19:23:46.170532+0000","last_peered":"2024-09-20T19:23:46.170532+0000","last_clean":"2024-09-20T19:23:46.170532+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:23:46.170532+0000","last_undegraded":"2024-09-20T19:23:46.170532+0000","last_fullsized":"2024-09-20T19:23:46.170532+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:24.063388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.063388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":353,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.250730+0000","last_change":"2024-09-20T19:20:58.711594+0000","last_active":"2024-09-20T19:23:46.250730+0000","last_peered":"2024-09-20T19:23:46.250730+0000","last_clean":"2024-09-20T19:23:46.250730+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:23:46.250730+0000","last_undegraded":"2024-09-20T19:23:46.250730+0000","last_fullsized":"2024-09-20T19:23:46.250730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.711472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.711472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.810411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":342,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184307+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:23:46.184307+0000","last_peered":"2024-09-20T19:23:46.184307+0000","last_clean":"2024-09-20T19:23:46.184307+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:23:46.184307+0000","last_undegraded":"2024-09-20T19:23:46.184307+0000","last_fullsized":"2024-09-20T19:23:46.184307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.744413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"230'30","reported_seq":642,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184201+0000","last_change":"2024-09-20T19:23:45.238041+0000","last_active":"2024-09-20T19:23:46.184201+0000","last_peered":"2024-09-20T19:23:46.184201+0000","last_clean":"2024-09-20T19:23:46.184201+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:23:46.184201+0000","last_undegraded":"2024-09-20T19:23:46.184201+0000","last_fullsized":"2024-09-20T19:23:46.184201+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.188704178,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":14840,"num_write":10,"num_write_kb":1940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.1","version":"77'14","reported_seq":380,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178492+0000","last_change":"2024-09-20T19:23:43.288937+0000","last_active":"2024-09-20T19:23:46.178492+0000","last_peered":"2024-09-20T19:23:46.178492+0000","last_clean":"2024-09-20T19:23:46.178492+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:23:46.178492+0000","last_undegraded":"2024-09-20T19:23:46.178492+0000","last_fullsized":"2024-09-20T19:23:46.178492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:23:43.288805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.288805+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:54.121381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":368,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.192557+0000","last_change":"2024-09-20T19:22:01.958010+0000","last_active":"2024-09-20T19:23:46.192557+0000","last_peered":"2024-09-20T19:23:46.192557+0000","last_clean":"2024-09-20T19:23:46.192557+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:23:46.192557+0000","last_undegraded":"2024-09-20T19:23:46.192557+0000","last_fullsized":"2024-09-20T19:23:46.192557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:01.957885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:01.957885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":432,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.183868+0000","last_change":"2024-09-20T19:21:07.002029+0000","last_active":"2024-09-20T19:23:46.183868+0000","last_peered":"2024-09-20T19:23:46.183868+0000","last_clean":"2024-09-20T19:23:46.183868+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:23:46.183868+0000","last_undegraded":"2024-09-20T19:23:46.183868+0000","last_fullsized":"2024-09-20T19:23:46.183868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:07.001951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.001951+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.724883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.300434+0000","last_change":"2024-09-20T19:23:46.300434+0000","last_active":"2024-09-20T19:23:46.300434+0000","last_peered":"2024-09-20T19:23:46.300434+0000","last_clean":"2024-09-20T19:23:46.300434+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:23:46.300434+0000","last_undegraded":"2024-09-20T19:23:46.300434+0000","last_fullsized":"2024-09-20T19:23:46.300434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:46.300364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.300364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.891484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"236'143","reported_seq":752,"reported_epoch":237,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:46.185348+0000","last_change":"2024-09-20T19:23:45.240566+0000","last_active":"2024-09-20T19:23:46.185348+0000","last_peered":"2024-09-20T19:23:46.185348+0000","last_clean":"2024-09-20T19:23:46.185348+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:23:46.185348+0000","last_undegraded":"2024-09-20T19:23:46.185348+0000","last_fullsized":"2024-09-20T19:23:46.185348+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.190648869,"stat_sum":{"num_bytes":5630345,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":66535,"num_write":45,"num_write_kb":15742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"4"}]},{"pgid":"4.0","version":"77'18","reported_seq":398,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184172+0000","last_change":"2024-09-20T19:22:00.803489+0000","last_active":"2024-09-20T19:23:46.184172+0000","last_peered":"2024-09-20T19:23:46.184172+0000","last_clean":"2024-09-20T19:23:46.184172+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:23:46.184172+0000","last_undegraded":"2024-09-20T19:23:46.184172+0000","last_fullsized":"2024-09-20T19:23:46.184172+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:22:00.803432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:00.803432+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":364,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.240054+0000","last_change":"2024-09-20T19:21:01.687454+0000","last_active":"2024-09-20T19:23:46.240054+0000","last_peered":"2024-09-20T19:23:46.240054+0000","last_clean":"2024-09-20T19:23:46.240054+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:23:46.240054+0000","last_undegraded":"2024-09-20T19:23:46.240054+0000","last_fullsized":"2024-09-20T19:23:46.240054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:21:01.687305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.687305+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.177584+0000","last_change":"2024-09-20T19:22:03.015704+0000","last_active":"2024-09-20T19:23:46.177584+0000","last_peered":"2024-09-20T19:23:46.177584+0000","last_clean":"2024-09-20T19:23:46.177584+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:23:46.177584+0000","last_undegraded":"2024-09-20T19:23:46.177584+0000","last_fullsized":"2024-09-20T19:23:46.177584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.015585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.015585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.008629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":351,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178509+0000","last_change":"2024-09-20T19:21:15.032164+0000","last_active":"2024-09-20T19:23:46.178509+0000","last_peered":"2024-09-20T19:23:46.178509+0000","last_clean":"2024-09-20T19:23:46.178509+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:23:46.178509+0000","last_undegraded":"2024-09-20T19:23:46.178509+0000","last_fullsized":"2024-09-20T19:23:46.178509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:15.032044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:15.032044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"236'170","reported_seq":804,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245075+0000","last_change":"2024-09-20T19:23:45.239021+0000","last_active":"2024-09-20T19:23:46.245075+0000","last_peered":"2024-09-20T19:23:46.245075+0000","last_clean":"2024-09-20T19:23:46.245075+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:23:46.245075+0000","last_undegraded":"2024-09-20T19:23:46.245075+0000","last_fullsized":"2024-09-20T19:23:46.245075+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'118","last_scrub_stamp":"2024-09-20T19:23:04.906510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.906510+0000","objects_scrubbed":10,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:15.867067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.184738706,"stat_sum":{"num_bytes":9008552,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":84394,"num_write":71,"num_write_kb":16068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.3","version":"77'19","reported_seq":395,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172557+0000","last_change":"2024-09-20T19:21:07.855976+0000","last_active":"2024-09-20T19:23:46.172557+0000","last_peered":"2024-09-20T19:23:46.172557+0000","last_clean":"2024-09-20T19:23:46.172557+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:23:46.172557+0000","last_undegraded":"2024-09-20T19:23:46.172557+0000","last_fullsized":"2024-09-20T19:23:46.172557+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:21:07.855828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.855828+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.733445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":307,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171708+0000","last_change":"2024-09-20T19:23:16.461133+0000","last_active":"2024-09-20T19:23:46.171708+0000","last_peered":"2024-09-20T19:23:46.171708+0000","last_clean":"2024-09-20T19:23:46.171708+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:23:46.171708+0000","last_undegraded":"2024-09-20T19:23:46.171708+0000","last_fullsized":"2024-09-20T19:23:46.171708+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:16.461005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:16.461005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.435064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":360,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.190218+0000","last_change":"2024-09-20T19:22:21.758505+0000","last_active":"2024-09-20T19:23:46.190218+0000","last_peered":"2024-09-20T19:23:46.190218+0000","last_clean":"2024-09-20T19:23:46.190218+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:23:46.190218+0000","last_undegraded":"2024-09-20T19:23:46.190218+0000","last_fullsized":"2024-09-20T19:23:46.190218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.758384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.758384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":343,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239017+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:23:46.239017+0000","last_peered":"2024-09-20T19:23:46.239017+0000","last_clean":"2024-09-20T19:23:46.239017+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:23:46.239017+0000","last_undegraded":"2024-09-20T19:23:46.239017+0000","last_fullsized":"2024-09-20T19:23:46.239017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.495223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"237'179","reported_seq":798,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.201491+0000","last_change":"2024-09-20T19:23:45.069481+0000","last_active":"2024-09-20T19:23:46.201491+0000","last_peered":"2024-09-20T19:23:46.201491+0000","last_clean":"2024-09-20T19:23:46.201491+0000","last_became_active":"2024-09-20T19:21:30.244233+0000","last_became_peered":"2024-09-20T19:21:30.244233+0000","last_unstale":"2024-09-20T19:23:46.201491+0000","last_undegraded":"2024-09-20T19:23:46.201491+0000","last_fullsized":"2024-09-20T19:23:46.201491+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'93","last_scrub_stamp":"2024-09-20T19:23:14.039336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.039336+0000","objects_scrubbed":6,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.161243+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011536529999999999,"stat_sum":{"num_bytes":9210236,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":29241,"num_write":61,"num_write_kb":19932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.2","version":"77'10","reported_seq":382,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239399+0000","last_change":"2024-09-20T19:22:17.881823+0000","last_active":"2024-09-20T19:23:46.239399+0000","last_peered":"2024-09-20T19:23:46.239399+0000","last_clean":"2024-09-20T19:23:46.239399+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:23:46.239399+0000","last_undegraded":"2024-09-20T19:23:46.239399+0000","last_fullsized":"2024-09-20T19:23:46.239399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:22:17.881680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:17.881680+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":371,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239833+0000","last_change":"2024-09-20T19:22:21.888862+0000","last_active":"2024-09-20T19:23:46.239833+0000","last_peered":"2024-09-20T19:23:46.239833+0000","last_clean":"2024-09-20T19:23:46.239833+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:23:46.239833+0000","last_undegraded":"2024-09-20T19:23:46.239833+0000","last_fullsized":"2024-09-20T19:23:46.239833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:36.511666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":433,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172265+0000","last_change":"2024-09-20T19:21:10.853855+0000","last_active":"2024-09-20T19:23:46.172265+0000","last_peered":"2024-09-20T19:23:46.172265+0000","last_clean":"2024-09-20T19:23:46.172265+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:23:46.172265+0000","last_undegraded":"2024-09-20T19:23:46.172265+0000","last_fullsized":"2024-09-20T19:23:46.172265+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:10.853722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.853722+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.732391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.174402+0000","last_change":"2024-09-20T19:22:56.797105+0000","last_active":"2024-09-20T19:23:46.174402+0000","last_peered":"2024-09-20T19:23:46.174402+0000","last_clean":"2024-09-20T19:23:46.174402+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:23:46.174402+0000","last_undegraded":"2024-09-20T19:23:46.174402+0000","last_fullsized":"2024-09-20T19:23:46.174402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.054135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"236'119","reported_seq":758,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.637050+0000","last_change":"2024-09-20T19:23:46.637050+0000","last_active":"2024-09-20T19:23:46.637050+0000","last_peered":"2024-09-20T19:23:46.637050+0000","last_clean":"2024-09-20T19:23:46.637050+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:23:46.637050+0000","last_undegraded":"2024-09-20T19:23:46.637050+0000","last_fullsized":"2024-09-20T19:23:46.637050+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'119","last_scrub_stamp":"2024-09-20T19:23:46.636892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.636892+0000","objects_scrubbed":4,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:02.466518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001125912,"stat_sum":{"num_bytes":11798514,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":92,"num_read_kb":61782,"num_write":56,"num_write_kb":22321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.4","version":"77'30","reported_seq":485,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239728+0000","last_change":"2024-09-20T19:22:14.846750+0000","last_active":"2024-09-20T19:23:46.239728+0000","last_peered":"2024-09-20T19:23:46.239728+0000","last_clean":"2024-09-20T19:23:46.239728+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:23:46.239728+0000","last_undegraded":"2024-09-20T19:23:46.239728+0000","last_fullsized":"2024-09-20T19:23:46.239728+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:22:14.846671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.846671+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.459830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.249753+0000","last_change":"2024-09-20T19:23:01.995350+0000","last_active":"2024-09-20T19:23:46.249753+0000","last_peered":"2024-09-20T19:23:46.249753+0000","last_clean":"2024-09-20T19:23:46.249753+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:23:46.249753+0000","last_undegraded":"2024-09-20T19:23:46.249753+0000","last_fullsized":"2024-09-20T19:23:46.249753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:01.995251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.995251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.855295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.173846+0000","last_change":"2024-09-20T19:22:22.664865+0000","last_active":"2024-09-20T19:23:46.173846+0000","last_peered":"2024-09-20T19:23:46.173846+0000","last_clean":"2024-09-20T19:23:46.173846+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:23:46.173846+0000","last_undegraded":"2024-09-20T19:23:46.173846+0000","last_fullsized":"2024-09-20T19:23:46.173846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:22.664804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.664804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":358,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184914+0000","last_change":"2024-09-20T19:23:08.714718+0000","last_active":"2024-09-20T19:23:46.184914+0000","last_peered":"2024-09-20T19:23:46.184914+0000","last_clean":"2024-09-20T19:23:46.184914+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:23:46.184914+0000","last_undegraded":"2024-09-20T19:23:46.184914+0000","last_fullsized":"2024-09-20T19:23:46.184914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.714593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.714593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.378300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"236'230","reported_seq":852,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.174281+0000","last_change":"2024-09-20T19:23:45.062092+0000","last_active":"2024-09-20T19:23:46.174281+0000","last_peered":"2024-09-20T19:23:46.174281+0000","last_clean":"2024-09-20T19:23:46.174281+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:23:46.174281+0000","last_undegraded":"2024-09-20T19:23:46.174281+0000","last_fullsized":"2024-09-20T19:23:46.174281+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'86","last_scrub_stamp":"2024-09-20T19:22:51.812052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.812052+0000","objects_scrubbed":9,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.892253+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0075939129999999999,"stat_sum":{"num_bytes":10421951,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":88,"num_read_kb":49528,"num_write":74,"num_write_kb":31044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.7","version":"77'13","reported_seq":303,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172059+0000","last_change":"2024-09-20T19:21:22.193770+0000","last_active":"2024-09-20T19:23:46.172059+0000","last_peered":"2024-09-20T19:23:46.172059+0000","last_clean":"2024-09-20T19:23:46.172059+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:23:46.172059+0000","last_undegraded":"2024-09-20T19:23:46.172059+0000","last_fullsized":"2024-09-20T19:23:46.172059+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:21:22.193573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:22.193573+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":694,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239289+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:23:46.239289+0000","last_peered":"2024-09-20T19:23:46.239289+0000","last_clean":"2024-09-20T19:23:46.239289+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:23:46.239289+0000","last_undegraded":"2024-09-20T19:23:46.239289+0000","last_fullsized":"2024-09-20T19:23:46.239289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.476879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":323,"num_read_kb":318,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":362,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.170879+0000","last_change":"2024-09-20T19:21:10.216852+0000","last_active":"2024-09-20T19:23:46.170879+0000","last_peered":"2024-09-20T19:23:46.170879+0000","last_clean":"2024-09-20T19:23:46.170879+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:23:46.170879+0000","last_undegraded":"2024-09-20T19:23:46.170879+0000","last_fullsized":"2024-09-20T19:23:46.170879+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:10.216700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.216700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.674411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":354,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171970+0000","last_change":"2024-09-20T19:20:30.146960+0000","last_active":"2024-09-20T19:23:46.171970+0000","last_peered":"2024-09-20T19:23:46.171970+0000","last_clean":"2024-09-20T19:23:46.171970+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:23:46.171970+0000","last_undegraded":"2024-09-20T19:23:46.171970+0000","last_fullsized":"2024-09-20T19:23:46.171970+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:30.146806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:30.146806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":357,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.247021+0000","last_change":"2024-09-20T19:23:11.304539+0000","last_active":"2024-09-20T19:23:46.247021+0000","last_peered":"2024-09-20T19:23:46.247021+0000","last_clean":"2024-09-20T19:23:46.247021+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:23:46.247021+0000","last_undegraded":"2024-09-20T19:23:46.247021+0000","last_fullsized":"2024-09-20T19:23:46.247021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.304478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.304478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.382250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"236'256","reported_seq":868,"reported_epoch":237,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:46.246308+0000","last_change":"2024-09-20T19:23:46.155616+0000","last_active":"2024-09-20T19:23:46.246308+0000","last_peered":"2024-09-20T19:23:46.246308+0000","last_clean":"2024-09-20T19:23:46.246308+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:23:46.246308+0000","last_undegraded":"2024-09-20T19:23:46.246308+0000","last_fullsized":"2024-09-20T19:23:46.246308+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0820198590000001,"stat_sum":{"num_bytes":30922704,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":162,"num_read_kb":100720,"num_write":117,"num_write_kb":49075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"1"},{"start":"33","length":"1"}]},{"pgid":"4.6","version":"77'12","reported_seq":385,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172746+0000","last_change":"2024-09-20T19:22:16.548672+0000","last_active":"2024-09-20T19:23:46.172746+0000","last_peered":"2024-09-20T19:23:46.172746+0000","last_clean":"2024-09-20T19:23:46.172746+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:23:46.172746+0000","last_undegraded":"2024-09-20T19:23:46.172746+0000","last_fullsized":"2024-09-20T19:23:46.172746+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:16.548570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:16.548570+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.801626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.173532+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:23:46.173532+0000","last_peered":"2024-09-20T19:23:46.173532+0000","last_clean":"2024-09-20T19:23:46.173532+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:23:46.173532+0000","last_undegraded":"2024-09-20T19:23:46.173532+0000","last_fullsized":"2024-09-20T19:23:46.173532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.744928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245208+0000","last_change":"2024-09-20T19:22:18.870386+0000","last_active":"2024-09-20T19:23:46.245208+0000","last_peered":"2024-09-20T19:23:46.245208+0000","last_clean":"2024-09-20T19:23:46.245208+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:23:46.245208+0000","last_undegraded":"2024-09-20T19:23:46.245208+0000","last_fullsized":"2024-09-20T19:23:46.245208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.870268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.870268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.666304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.250569+0000","last_change":"2024-09-20T19:22:56.906325+0000","last_active":"2024-09-20T19:23:46.250569+0000","last_peered":"2024-09-20T19:23:46.250569+0000","last_clean":"2024-09-20T19:23:46.250569+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:23:46.250569+0000","last_undegraded":"2024-09-20T19:23:46.250569+0000","last_fullsized":"2024-09-20T19:23:46.250569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.906255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.906255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.054482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239108+0000","last_change":"2024-09-20T19:22:06.926452+0000","last_active":"2024-09-20T19:23:46.239108+0000","last_peered":"2024-09-20T19:23:46.239108+0000","last_clean":"2024-09-20T19:23:46.239108+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:23:46.239108+0000","last_undegraded":"2024-09-20T19:23:46.239108+0000","last_fullsized":"2024-09-20T19:23:46.239108+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:06.926325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:06.926325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.522917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"236'82","reported_seq":710,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172297+0000","last_change":"2024-09-20T19:23:45.066629+0000","last_active":"2024-09-20T19:23:46.172297+0000","last_peered":"2024-09-20T19:23:46.172297+0000","last_clean":"2024-09-20T19:23:46.172297+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:23:46.172297+0000","last_undegraded":"2024-09-20T19:23:46.172297+0000","last_fullsized":"2024-09-20T19:23:46.172297+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'61","last_scrub_stamp":"2024-09-20T19:23:06.675677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:06.675677+0000","objects_scrubbed":4,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.731531+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.011798522000000001,"stat_sum":{"num_bytes":6890804,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":46080,"num_write":32,"num_write_kb":12635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.5","version":"77'16","reported_seq":446,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191192+0000","last_change":"2024-09-20T19:21:12.740077+0000","last_active":"2024-09-20T19:23:46.191192+0000","last_peered":"2024-09-20T19:23:46.191192+0000","last_clean":"2024-09-20T19:23:46.191192+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:23:46.191192+0000","last_undegraded":"2024-09-20T19:23:46.191192+0000","last_fullsized":"2024-09-20T19:23:46.191192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:21:12.739915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:12.739915+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.628021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184902+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:23:46.184902+0000","last_peered":"2024-09-20T19:23:46.184902+0000","last_clean":"2024-09-20T19:23:46.184902+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:23:46.184902+0000","last_undegraded":"2024-09-20T19:23:46.184902+0000","last_fullsized":"2024-09-20T19:23:46.184902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.812423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.248502+0000","last_change":"2024-09-20T19:20:52.723249+0000","last_active":"2024-09-20T19:23:46.248502+0000","last_peered":"2024-09-20T19:23:46.248502+0000","last_clean":"2024-09-20T19:23:46.248502+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:23:46.248502+0000","last_undegraded":"2024-09-20T19:23:46.248502+0000","last_fullsized":"2024-09-20T19:23:46.248502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.723121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.723121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.814578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":401,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245467+0000","last_change":"2024-09-20T19:20:12.275658+0000","last_active":"2024-09-20T19:23:46.245467+0000","last_peered":"2024-09-20T19:23:46.245467+0000","last_clean":"2024-09-20T19:23:46.245467+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:23:46.245467+0000","last_undegraded":"2024-09-20T19:23:46.245467+0000","last_fullsized":"2024-09-20T19:23:46.245467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:20:12.275516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:20:12.275516+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.023904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":353,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246225+0000","last_change":"2024-09-20T19:20:53.868159+0000","last_active":"2024-09-20T19:23:46.246225+0000","last_peered":"2024-09-20T19:23:46.246225+0000","last_clean":"2024-09-20T19:23:46.246225+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:23:46.246225+0000","last_undegraded":"2024-09-20T19:23:46.246225+0000","last_fullsized":"2024-09-20T19:23:46.246225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.868034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.868034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.287685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.250977+0000","last_change":"2024-09-20T19:20:39.803967+0000","last_active":"2024-09-20T19:23:46.250977+0000","last_peered":"2024-09-20T19:23:46.250977+0000","last_clean":"2024-09-20T19:23:46.250977+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:23:46.250977+0000","last_undegraded":"2024-09-20T19:23:46.250977+0000","last_fullsized":"2024-09-20T19:23:46.250977+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:39.803847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:39.803847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.047616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"236'111","reported_seq":741,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.173192+0000","last_change":"2024-09-20T19:23:45.061857+0000","last_active":"2024-09-20T19:23:46.173192+0000","last_peered":"2024-09-20T19:23:46.173192+0000","last_clean":"2024-09-20T19:23:46.173192+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:23:46.173192+0000","last_undegraded":"2024-09-20T19:23:46.173192+0000","last_fullsized":"2024-09-20T19:23:46.173192+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'71","last_scrub_stamp":"2024-09-20T19:23:05.723582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.723582+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:11.077121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.007895895,"stat_sum":{"num_bytes":8084167,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":51498,"num_write":41,"num_write_kb":16957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.a","version":"77'19","reported_seq":387,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191095+0000","last_change":"2024-09-20T19:20:52.718858+0000","last_active":"2024-09-20T19:23:46.191095+0000","last_peered":"2024-09-20T19:23:46.191095+0000","last_clean":"2024-09-20T19:23:46.191095+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:23:46.191095+0000","last_undegraded":"2024-09-20T19:23:46.191095+0000","last_fullsized":"2024-09-20T19:23:46.191095+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:20:52.718711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.718711+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.617546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":370,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171703+0000","last_change":"2024-09-20T19:22:11.289926+0000","last_active":"2024-09-20T19:23:46.171703+0000","last_peered":"2024-09-20T19:23:46.171703+0000","last_clean":"2024-09-20T19:23:46.171703+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:23:46.171703+0000","last_undegraded":"2024-09-20T19:23:46.171703+0000","last_fullsized":"2024-09-20T19:23:46.171703+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.289877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.289877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":354,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171233+0000","last_change":"2024-09-20T19:21:06.345504+0000","last_active":"2024-09-20T19:23:46.171233+0000","last_peered":"2024-09-20T19:23:46.171233+0000","last_clean":"2024-09-20T19:23:46.171233+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:23:46.171233+0000","last_undegraded":"2024-09-20T19:23:46.171233+0000","last_fullsized":"2024-09-20T19:23:46.171233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.345373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.345373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245453+0000","last_change":"2024-09-20T19:23:42.246466+0000","last_active":"2024-09-20T19:23:46.245453+0000","last_peered":"2024-09-20T19:23:46.245453+0000","last_clean":"2024-09-20T19:23:46.245453+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:23:46.245453+0000","last_undegraded":"2024-09-20T19:23:46.245453+0000","last_fullsized":"2024-09-20T19:23:46.245453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.246406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.246406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.009612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"236'282","reported_seq":897,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.170821+0000","last_change":"2024-09-20T19:23:45.255297+0000","last_active":"2024-09-20T19:23:46.170821+0000","last_peered":"2024-09-20T19:23:46.170821+0000","last_clean":"2024-09-20T19:23:46.170821+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:23:46.170821+0000","last_undegraded":"2024-09-20T19:23:46.170821+0000","last_fullsized":"2024-09-20T19:23:46.170821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'112","last_scrub_stamp":"2024-09-20T19:22:52.771031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771031+0000","objects_scrubbed":11,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.254798+0000","scrub_duration":7,"objects_trimmed":2,"snaptrim_duration":0.012176931,"stat_sum":{"num_bytes":11802898,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":74850,"num_write":96,"num_write_kb":43409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.b","version":"77'9","reported_seq":383,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172419+0000","last_change":"2024-09-20T19:22:55.847371+0000","last_active":"2024-09-20T19:23:46.172419+0000","last_peered":"2024-09-20T19:23:46.172419+0000","last_clean":"2024-09-20T19:23:46.172419+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:23:46.172419+0000","last_undegraded":"2024-09-20T19:23:46.172419+0000","last_fullsized":"2024-09-20T19:23:46.172419+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:55.847304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:55.847304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:04.129113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.240170+0000","last_change":"2024-09-20T19:21:05.740388+0000","last_active":"2024-09-20T19:23:46.240170+0000","last_peered":"2024-09-20T19:23:46.240170+0000","last_clean":"2024-09-20T19:23:46.240170+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:23:46.240170+0000","last_undegraded":"2024-09-20T19:23:46.240170+0000","last_fullsized":"2024-09-20T19:23:46.240170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.740278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.740278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.483483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":354,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171081+0000","last_change":"2024-09-20T19:21:17.226835+0000","last_active":"2024-09-20T19:23:46.171081+0000","last_peered":"2024-09-20T19:23:46.171081+0000","last_clean":"2024-09-20T19:23:46.171081+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:23:46.171081+0000","last_undegraded":"2024-09-20T19:23:46.171081+0000","last_fullsized":"2024-09-20T19:23:46.171081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:17.226714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:17.226714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.520900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":368,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.173458+0000","last_change":"2024-09-20T19:23:44.721606+0000","last_active":"2024-09-20T19:23:46.173458+0000","last_peered":"2024-09-20T19:23:46.173458+0000","last_clean":"2024-09-20T19:23:46.173458+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:23:46.173458+0000","last_undegraded":"2024-09-20T19:23:46.173458+0000","last_fullsized":"2024-09-20T19:23:46.173458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:23:44.721438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.721438+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:59.778487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"233'57","reported_seq":682,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239501+0000","last_change":"2024-09-20T19:23:45.049842+0000","last_active":"2024-09-20T19:23:46.239501+0000","last_peered":"2024-09-20T19:23:46.239501+0000","last_clean":"2024-09-20T19:23:46.239501+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:23:46.239501+0000","last_undegraded":"2024-09-20T19:23:46.239501+0000","last_fullsized":"2024-09-20T19:23:46.239501+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'35","last_scrub_stamp":"2024-09-20T19:23:10.333319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.333319+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.644875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00089667200000000005,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":21280,"num_write":19,"num_write_kb":11601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"5"}]},{"pgid":"4.8","version":"77'15","reported_seq":390,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.244481+0000","last_change":"2024-09-20T19:22:22.063998+0000","last_active":"2024-09-20T19:23:46.244481+0000","last_peered":"2024-09-20T19:23:46.244481+0000","last_clean":"2024-09-20T19:23:46.244481+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:23:46.244481+0000","last_undegraded":"2024-09-20T19:23:46.244481+0000","last_fullsized":"2024-09-20T19:23:46.244481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:22.063871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.063871+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.143016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":378,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172123+0000","last_change":"2024-09-20T19:22:02.235649+0000","last_active":"2024-09-20T19:23:46.172123+0000","last_peered":"2024-09-20T19:23:46.172123+0000","last_clean":"2024-09-20T19:23:46.172123+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:23:46.172123+0000","last_undegraded":"2024-09-20T19:23:46.172123+0000","last_fullsized":"2024-09-20T19:23:46.172123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:02.235598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:02.235598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.519095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":439,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245169+0000","last_change":"2024-09-20T19:23:01.188032+0000","last_active":"2024-09-20T19:23:46.245169+0000","last_peered":"2024-09-20T19:23:46.245169+0000","last_clean":"2024-09-20T19:23:46.245169+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:23:46.245169+0000","last_undegraded":"2024-09-20T19:23:46.245169+0000","last_fullsized":"2024-09-20T19:23:46.245169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:01.187968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.187968+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.789452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.249492+0000","last_change":"2024-09-20T19:21:01.739612+0000","last_active":"2024-09-20T19:23:46.249492+0000","last_peered":"2024-09-20T19:23:46.249492+0000","last_clean":"2024-09-20T19:23:46.249492+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:23:46.249492+0000","last_undegraded":"2024-09-20T19:23:46.249492+0000","last_fullsized":"2024-09-20T19:23:46.249492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:01.739492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.739492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.006870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"236'115","reported_seq":722,"reported_epoch":237,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:46.184878+0000","last_change":"2024-09-20T19:23:45.239937+0000","last_active":"2024-09-20T19:23:46.184878+0000","last_peered":"2024-09-20T19:23:46.184878+0000","last_clean":"2024-09-20T19:23:46.184878+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:23:46.184878+0000","last_undegraded":"2024-09-20T19:23:46.184878+0000","last_fullsized":"2024-09-20T19:23:46.184878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.1906979,"stat_sum":{"num_bytes":13149645,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":54,"num_read_kb":29927,"num_write":46,"num_write_kb":18568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2f","length":"2"},{"start":"33","length":"2"},{"start":"36","length":"3"}]},{"pgid":"4.9","version":"77'12","reported_seq":409,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178207+0000","last_change":"2024-09-20T19:22:57.202081+0000","last_active":"2024-09-20T19:23:46.178207+0000","last_peered":"2024-09-20T19:23:46.178207+0000","last_clean":"2024-09-20T19:23:46.178207+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:23:46.178207+0000","last_undegraded":"2024-09-20T19:23:46.178207+0000","last_fullsized":"2024-09-20T19:23:46.178207+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:57.201945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:57.201945+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.217023+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":387,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.180165+0000","last_change":"2024-09-20T19:22:59.196366+0000","last_active":"2024-09-20T19:23:46.180165+0000","last_peered":"2024-09-20T19:23:46.180165+0000","last_clean":"2024-09-20T19:23:46.180165+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:23:46.180165+0000","last_undegraded":"2024-09-20T19:23:46.180165+0000","last_fullsized":"2024-09-20T19:23:46.180165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:22:59.196235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:59.196235+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.646992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":362,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.170799+0000","last_change":"2024-09-20T19:22:18.260176+0000","last_active":"2024-09-20T19:23:46.170799+0000","last_peered":"2024-09-20T19:23:46.170799+0000","last_clean":"2024-09-20T19:23:46.170799+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:23:46.170799+0000","last_undegraded":"2024-09-20T19:23:46.170799+0000","last_fullsized":"2024-09-20T19:23:46.170799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.260046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.260046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.501604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":350,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184326+0000","last_change":"2024-09-20T19:23:00.687733+0000","last_active":"2024-09-20T19:23:46.184326+0000","last_peered":"2024-09-20T19:23:46.184326+0000","last_clean":"2024-09-20T19:23:46.184326+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:23:46.184326+0000","last_undegraded":"2024-09-20T19:23:46.184326+0000","last_fullsized":"2024-09-20T19:23:46.184326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:00.687658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:00.687658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.540716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":383,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172549+0000","last_change":"2024-09-20T19:22:58.800944+0000","last_active":"2024-09-20T19:23:46.172549+0000","last_peered":"2024-09-20T19:23:46.172549+0000","last_clean":"2024-09-20T19:23:46.172549+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:23:46.172549+0000","last_undegraded":"2024-09-20T19:23:46.172549+0000","last_fullsized":"2024-09-20T19:23:46.172549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:58.800874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.800874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:05.321600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":378,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.170955+0000","last_change":"2024-09-20T19:23:07.609407+0000","last_active":"2024-09-20T19:23:46.170955+0000","last_peered":"2024-09-20T19:23:46.170955+0000","last_clean":"2024-09-20T19:23:46.170955+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:23:46.170955+0000","last_undegraded":"2024-09-20T19:23:46.170955+0000","last_fullsized":"2024-09-20T19:23:46.170955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:07.609283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:07.609283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:17.108303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":354,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.185099+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:23:46.185099+0000","last_peered":"2024-09-20T19:23:46.185099+0000","last_clean":"2024-09-20T19:23:46.185099+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:23:46.185099+0000","last_undegraded":"2024-09-20T19:23:46.185099+0000","last_fullsized":"2024-09-20T19:23:46.185099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:35.796722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":358,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171233+0000","last_change":"2024-09-20T19:22:56.840530+0000","last_active":"2024-09-20T19:23:46.171233+0000","last_peered":"2024-09-20T19:23:46.171233+0000","last_clean":"2024-09-20T19:23:46.171233+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:23:46.171233+0000","last_undegraded":"2024-09-20T19:23:46.171233+0000","last_fullsized":"2024-09-20T19:23:46.171233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.840403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.840403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.237174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":368,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191242+0000","last_change":"2024-09-20T19:23:02.640412+0000","last_active":"2024-09-20T19:23:46.191242+0000","last_peered":"2024-09-20T19:23:46.191242+0000","last_clean":"2024-09-20T19:23:46.191242+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:23:46.191242+0000","last_undegraded":"2024-09-20T19:23:46.191242+0000","last_fullsized":"2024-09-20T19:23:46.191242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:02.640270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.640270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.542638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":367,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.185225+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:23:46.185225+0000","last_peered":"2024-09-20T19:23:46.185225+0000","last_clean":"2024-09-20T19:23:46.185225+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:23:46.185225+0000","last_undegraded":"2024-09-20T19:23:46.185225+0000","last_fullsized":"2024-09-20T19:23:46.185225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:27.710208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":353,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.249290+0000","last_change":"2024-09-20T19:21:28.981452+0000","last_active":"2024-09-20T19:23:46.249290+0000","last_peered":"2024-09-20T19:23:46.249290+0000","last_clean":"2024-09-20T19:23:46.249290+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:23:46.249290+0000","last_undegraded":"2024-09-20T19:23:46.249290+0000","last_fullsized":"2024-09-20T19:23:46.249290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.981320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.981320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246806+0000","last_change":"2024-09-20T19:21:29.100795+0000","last_active":"2024-09-20T19:23:46.246806+0000","last_peered":"2024-09-20T19:23:46.246806+0000","last_clean":"2024-09-20T19:23:46.246806+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:23:46.246806+0000","last_undegraded":"2024-09-20T19:23:46.246806+0000","last_fullsized":"2024-09-20T19:23:46.246806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.100718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.100718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:23.284037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":373,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184818+0000","last_change":"2024-09-20T19:20:32.059785+0000","last_active":"2024-09-20T19:23:46.184818+0000","last_peered":"2024-09-20T19:23:46.184818+0000","last_clean":"2024-09-20T19:23:46.184818+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:23:46.184818+0000","last_undegraded":"2024-09-20T19:23:46.184818+0000","last_fullsized":"2024-09-20T19:23:46.184818+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:20:32.059603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:32.059603+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:49.628487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.249469+0000","last_change":"2024-09-20T19:22:21.054003+0000","last_active":"2024-09-20T19:23:46.249469+0000","last_peered":"2024-09-20T19:23:46.249469+0000","last_clean":"2024-09-20T19:23:46.249469+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:23:46.249469+0000","last_undegraded":"2024-09-20T19:23:46.249469+0000","last_fullsized":"2024-09-20T19:23:46.249469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.053899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.053899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":439,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.248490+0000","last_change":"2024-09-20T19:22:13.105842+0000","last_active":"2024-09-20T19:23:46.248490+0000","last_peered":"2024-09-20T19:23:46.248490+0000","last_clean":"2024-09-20T19:23:46.248490+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:23:46.248490+0000","last_undegraded":"2024-09-20T19:23:46.248490+0000","last_fullsized":"2024-09-20T19:23:46.248490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:13.105692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:13.105692+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":367,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172089+0000","last_change":"2024-09-20T19:23:15.737592+0000","last_active":"2024-09-20T19:23:46.172089+0000","last_peered":"2024-09-20T19:23:46.172089+0000","last_clean":"2024-09-20T19:23:46.172089+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:23:46.172089+0000","last_undegraded":"2024-09-20T19:23:46.172089+0000","last_fullsized":"2024-09-20T19:23:46.172089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:15.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:15.737537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.525979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":381,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.248779+0000","last_change":"2024-09-20T19:22:54.923862+0000","last_active":"2024-09-20T19:23:46.248779+0000","last_peered":"2024-09-20T19:23:46.248779+0000","last_clean":"2024-09-20T19:23:46.248779+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:23:46.248779+0000","last_undegraded":"2024-09-20T19:23:46.248779+0000","last_fullsized":"2024-09-20T19:23:46.248779+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:54.923725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.923725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:02.018467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":371,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172768+0000","last_change":"2024-09-20T19:22:20.671257+0000","last_active":"2024-09-20T19:23:46.172768+0000","last_peered":"2024-09-20T19:23:46.172768+0000","last_clean":"2024-09-20T19:23:46.172768+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:23:46.172768+0000","last_undegraded":"2024-09-20T19:23:46.172768+0000","last_fullsized":"2024-09-20T19:23:46.172768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.671196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.671196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":432,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184591+0000","last_change":"2024-09-20T19:22:53.660006+0000","last_active":"2024-09-20T19:23:46.184591+0000","last_peered":"2024-09-20T19:23:46.184591+0000","last_clean":"2024-09-20T19:23:46.184591+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:23:46.184591+0000","last_undegraded":"2024-09-20T19:23:46.184591+0000","last_fullsized":"2024-09-20T19:23:46.184591+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:53.659878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.659878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:06.807822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179779+0000","last_change":"2024-09-20T19:22:11.914350+0000","last_active":"2024-09-20T19:23:46.179779+0000","last_peered":"2024-09-20T19:23:46.179779+0000","last_clean":"2024-09-20T19:23:46.179779+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:23:46.179779+0000","last_undegraded":"2024-09-20T19:23:46.179779+0000","last_fullsized":"2024-09-20T19:23:46.179779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.914224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.914224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.201274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":375,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239162+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:23:46.239162+0000","last_peered":"2024-09-20T19:23:46.239162+0000","last_clean":"2024-09-20T19:23:46.239162+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:23:46.239162+0000","last_undegraded":"2024-09-20T19:23:46.239162+0000","last_fullsized":"2024-09-20T19:23:46.239162+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.610209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":360,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.192199+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:23:46.192199+0000","last_peered":"2024-09-20T19:23:46.192199+0000","last_clean":"2024-09-20T19:23:46.192199+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:23:46.192199+0000","last_undegraded":"2024-09-20T19:23:46.192199+0000","last_fullsized":"2024-09-20T19:23:46.192199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.684147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":354,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184433+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:23:46.184433+0000","last_peered":"2024-09-20T19:23:46.184433+0000","last_clean":"2024-09-20T19:23:46.184433+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:23:46.184433+0000","last_undegraded":"2024-09-20T19:23:46.184433+0000","last_fullsized":"2024-09-20T19:23:46.184433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.835899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.174235+0000","last_change":"2024-09-20T19:23:43.752868+0000","last_active":"2024-09-20T19:23:46.174235+0000","last_peered":"2024-09-20T19:23:46.174235+0000","last_clean":"2024-09-20T19:23:46.174235+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:23:46.174235+0000","last_undegraded":"2024-09-20T19:23:46.174235+0000","last_fullsized":"2024-09-20T19:23:46.174235+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:43.752736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.752736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.607806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":378,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.178136+0000","last_change":"2024-09-20T19:20:52.914253+0000","last_active":"2024-09-20T19:23:46.178136+0000","last_peered":"2024-09-20T19:23:46.178136+0000","last_clean":"2024-09-20T19:23:46.178136+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:23:46.178136+0000","last_undegraded":"2024-09-20T19:23:46.178136+0000","last_fullsized":"2024-09-20T19:23:46.178136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:52.914121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.914121+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.086622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":379,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.240169+0000","last_change":"2024-09-20T19:22:53.163608+0000","last_active":"2024-09-20T19:23:46.240169+0000","last_peered":"2024-09-20T19:23:46.240169+0000","last_clean":"2024-09-20T19:23:46.240169+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:23:46.240169+0000","last_undegraded":"2024-09-20T19:23:46.240169+0000","last_fullsized":"2024-09-20T19:23:46.240169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:53.163555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.163555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:09.815764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239041+0000","last_change":"2024-09-20T19:23:44.572528+0000","last_active":"2024-09-20T19:23:46.239041+0000","last_peered":"2024-09-20T19:23:46.239041+0000","last_clean":"2024-09-20T19:23:46.239041+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:23:46.239041+0000","last_undegraded":"2024-09-20T19:23:46.239041+0000","last_fullsized":"2024-09-20T19:23:46.239041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:44.572407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.572407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.284846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":342,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.183938+0000","last_change":"2024-09-20T19:20:04.097978+0000","last_active":"2024-09-20T19:23:46.183938+0000","last_peered":"2024-09-20T19:23:46.183938+0000","last_clean":"2024-09-20T19:23:46.183938+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:23:46.183938+0000","last_undegraded":"2024-09-20T19:23:46.183938+0000","last_fullsized":"2024-09-20T19:23:46.183938+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:04.097843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:04.097843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":372,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.183984+0000","last_change":"2024-09-20T19:22:50.638425+0000","last_active":"2024-09-20T19:23:46.183984+0000","last_peered":"2024-09-20T19:23:46.183984+0000","last_clean":"2024-09-20T19:23:46.183984+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:23:46.183984+0000","last_undegraded":"2024-09-20T19:23:46.183984+0000","last_fullsized":"2024-09-20T19:23:46.183984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:22:50.638257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:50.638257+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.137027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":369,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.250243+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:23:46.250243+0000","last_peered":"2024-09-20T19:23:46.250243+0000","last_clean":"2024-09-20T19:23:46.250243+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:23:46.250243+0000","last_undegraded":"2024-09-20T19:23:46.250243+0000","last_fullsized":"2024-09-20T19:23:46.250243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.686224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":352,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.190145+0000","last_change":"2024-09-20T19:20:38.745217+0000","last_active":"2024-09-20T19:23:46.190145+0000","last_peered":"2024-09-20T19:23:46.190145+0000","last_clean":"2024-09-20T19:23:46.190145+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:23:46.190145+0000","last_undegraded":"2024-09-20T19:23:46.190145+0000","last_fullsized":"2024-09-20T19:23:46.190145+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:38.745091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:38.745091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:47.791875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":341,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246830+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:23:46.246830+0000","last_peered":"2024-09-20T19:23:46.246830+0000","last_clean":"2024-09-20T19:23:46.246830+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:23:46.246830+0000","last_undegraded":"2024-09-20T19:23:46.246830+0000","last_fullsized":"2024-09-20T19:23:46.246830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.264712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":385,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184095+0000","last_change":"2024-09-20T19:22:08.939934+0000","last_active":"2024-09-20T19:23:46.184095+0000","last_peered":"2024-09-20T19:23:46.184095+0000","last_clean":"2024-09-20T19:23:46.184095+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:23:46.184095+0000","last_undegraded":"2024-09-20T19:23:46.184095+0000","last_fullsized":"2024-09-20T19:23:46.184095+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.939842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.939842+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":368,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.192415+0000","last_change":"2024-09-20T19:22:51.781731+0000","last_active":"2024-09-20T19:23:46.192415+0000","last_peered":"2024-09-20T19:23:46.192415+0000","last_clean":"2024-09-20T19:23:46.192415+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:23:46.192415+0000","last_undegraded":"2024-09-20T19:23:46.192415+0000","last_fullsized":"2024-09-20T19:23:46.192415+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.781680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.781680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:00.226292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":361,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.244649+0000","last_change":"2024-09-20T19:23:14.325927+0000","last_active":"2024-09-20T19:23:46.244649+0000","last_peered":"2024-09-20T19:23:46.244649+0000","last_clean":"2024-09-20T19:23:46.244649+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:23:46.244649+0000","last_undegraded":"2024-09-20T19:23:46.244649+0000","last_fullsized":"2024-09-20T19:23:46.244649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:14.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:20.929643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":358,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184201+0000","last_change":"2024-09-20T19:22:11.898382+0000","last_active":"2024-09-20T19:23:46.184201+0000","last_peered":"2024-09-20T19:23:46.184201+0000","last_clean":"2024-09-20T19:23:46.184201+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:23:46.184201+0000","last_undegraded":"2024-09-20T19:23:46.184201+0000","last_fullsized":"2024-09-20T19:23:46.184201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.898255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.898255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":343,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.246139+0000","last_change":"2024-09-20T19:20:03.328664+0000","last_active":"2024-09-20T19:23:46.246139+0000","last_peered":"2024-09-20T19:23:46.246139+0000","last_clean":"2024-09-20T19:23:46.246139+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:23:46.246139+0000","last_undegraded":"2024-09-20T19:23:46.246139+0000","last_fullsized":"2024-09-20T19:23:46.246139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:03.328531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:03.328531+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:46.997157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":377,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.245684+0000","last_change":"2024-09-20T19:23:11.005669+0000","last_active":"2024-09-20T19:23:46.245684+0000","last_peered":"2024-09-20T19:23:46.245684+0000","last_clean":"2024-09-20T19:23:46.245684+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:23:46.245684+0000","last_undegraded":"2024-09-20T19:23:46.245684+0000","last_fullsized":"2024-09-20T19:23:46.245684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.005587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.005587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.106114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":382,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172494+0000","last_change":"2024-09-20T19:23:10.711312+0000","last_active":"2024-09-20T19:23:46.172494+0000","last_peered":"2024-09-20T19:23:46.172494+0000","last_clean":"2024-09-20T19:23:46.172494+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:23:46.172494+0000","last_undegraded":"2024-09-20T19:23:46.172494+0000","last_fullsized":"2024-09-20T19:23:46.172494+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:10.711138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.711138+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.536408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":441,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191915+0000","last_change":"2024-09-20T19:22:08.792421+0000","last_active":"2024-09-20T19:23:46.191915+0000","last_peered":"2024-09-20T19:23:46.191915+0000","last_clean":"2024-09-20T19:23:46.191915+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:23:46.191915+0000","last_undegraded":"2024-09-20T19:23:46.191915+0000","last_fullsized":"2024-09-20T19:23:46.191915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.792243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.792243+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":359,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239160+0000","last_change":"2024-09-20T19:23:05.361065+0000","last_active":"2024-09-20T19:23:46.239160+0000","last_peered":"2024-09-20T19:23:46.239160+0000","last_clean":"2024-09-20T19:23:46.239160+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:23:46.239160+0000","last_undegraded":"2024-09-20T19:23:46.239160+0000","last_fullsized":"2024-09-20T19:23:46.239160+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:05.360941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.360941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.569089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":371,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184480+0000","last_change":"2024-09-20T19:23:03.692672+0000","last_active":"2024-09-20T19:23:46.184480+0000","last_peered":"2024-09-20T19:23:46.184480+0000","last_clean":"2024-09-20T19:23:46.184480+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:23:46.184480+0000","last_undegraded":"2024-09-20T19:23:46.184480+0000","last_fullsized":"2024-09-20T19:23:46.184480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:03.692537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.692537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:10.678096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":402,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.192180+0000","last_change":"2024-09-20T19:22:22.768858+0000","last_active":"2024-09-20T19:23:46.192180+0000","last_peered":"2024-09-20T19:23:46.192180+0000","last_clean":"2024-09-20T19:23:46.192180+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:23:46.192180+0000","last_undegraded":"2024-09-20T19:23:46.192180+0000","last_fullsized":"2024-09-20T19:23:46.192180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:22.768697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.768697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":355,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.172441+0000","last_change":"2024-09-20T19:20:24.877567+0000","last_active":"2024-09-20T19:23:46.172441+0000","last_peered":"2024-09-20T19:23:46.172441+0000","last_clean":"2024-09-20T19:23:46.172441+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:23:46.172441+0000","last_undegraded":"2024-09-20T19:23:46.172441+0000","last_fullsized":"2024-09-20T19:23:46.172441+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:24.877393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:24.877393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:48.836697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":351,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179382+0000","last_change":"2024-09-20T19:20:59.816653+0000","last_active":"2024-09-20T19:23:46.179382+0000","last_peered":"2024-09-20T19:23:46.179382+0000","last_clean":"2024-09-20T19:23:46.179382+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:23:46.179382+0000","last_undegraded":"2024-09-20T19:23:46.179382+0000","last_fullsized":"2024-09-20T19:23:46.179382+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:59.816510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:59.816510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.141868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":368,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.191684+0000","last_change":"2024-09-20T19:22:18.771300+0000","last_active":"2024-09-20T19:23:46.191684+0000","last_peered":"2024-09-20T19:23:46.191684+0000","last_clean":"2024-09-20T19:23:46.191684+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:23:46.191684+0000","last_undegraded":"2024-09-20T19:23:46.191684+0000","last_fullsized":"2024-09-20T19:23:46.191684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.771181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.771181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":383,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.171531+0000","last_change":"2024-09-20T19:21:21.225432+0000","last_active":"2024-09-20T19:23:46.171531+0000","last_peered":"2024-09-20T19:23:46.171531+0000","last_clean":"2024-09-20T19:23:46.171531+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:23:46.171531+0000","last_undegraded":"2024-09-20T19:23:46.171531+0000","last_fullsized":"2024-09-20T19:23:46.171531+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:21:21.225282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:21.225282+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":363,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.239837+0000","last_change":"2024-09-20T19:22:51.090500+0000","last_active":"2024-09-20T19:23:46.239837+0000","last_peered":"2024-09-20T19:23:46.239837+0000","last_clean":"2024-09-20T19:23:46.239837+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:23:46.239837+0000","last_undegraded":"2024-09-20T19:23:46.239837+0000","last_fullsized":"2024-09-20T19:23:46.239837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.090439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.090439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:23:52.055207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":342,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.185098+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:23:46.185098+0000","last_peered":"2024-09-20T19:23:46.185098+0000","last_clean":"2024-09-20T19:23:46.185098+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:23:46.185098+0000","last_undegraded":"2024-09-20T19:23:46.185098+0000","last_fullsized":"2024-09-20T19:23:46.185098+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.890171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":362,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184565+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:23:46.184565+0000","last_peered":"2024-09-20T19:23:46.184565+0000","last_clean":"2024-09-20T19:23:46.184565+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:23:46.184565+0000","last_undegraded":"2024-09-20T19:23:46.184565+0000","last_fullsized":"2024-09-20T19:23:46.184565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.730587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":381,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.184355+0000","last_change":"2024-09-20T19:21:24.033642+0000","last_active":"2024-09-20T19:23:46.184355+0000","last_peered":"2024-09-20T19:23:46.184355+0000","last_clean":"2024-09-20T19:23:46.184355+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:23:46.184355+0000","last_undegraded":"2024-09-20T19:23:46.184355+0000","last_fullsized":"2024-09-20T19:23:46.184355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:21:24.033570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.033570+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.742580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":355,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-20T19:23:46.179309+0000","last_change":"2024-09-20T19:21:08.918562+0000","last_active":"2024-09-20T19:23:46.179309+0000","last_peered":"2024-09-20T19:23:46.179309+0000","last_clean":"2024-09-20T19:23:46.179309+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:23:46.179309+0000","last_undegraded":"2024-09-20T19:23:46.179309+0000","last_fullsized":"2024-09-20T19:23:46.179309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.918443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.918443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.121645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":153883389,"num_objects":60,"num_object_clones":19,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":2,"num_read":1149,"num_read_kb":809410,"num_write":841,"num_write_kb":317342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":371466240,"data_stored":402274284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2218,"ondisk_log_size":2218,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":323,"num_read_kb":318,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907021,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94516,"kb_used_data":63324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93650636,"statfs":{"total":95995035648,"available":95898251264,"internally_reserved":0,"allocated":64843776,"data_stored":69733597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234072,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69240,"kb_used_data":38048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93675912,"statfs":{"total":95995035648,"available":95924133888,"internally_reserved":0,"allocated":38961152,"data_stored":38180255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561122,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53464,"kb_used_data":22272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691688,"statfs":{"total":95995035648,"available":95940288512,"internally_reserved":0,"allocated":22806528,"data_stored":24802285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855470,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74260,"kb_used_data":38912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93670892,"statfs":{"total":95995035648,"available":95918993408,"internally_reserved":0,"allocated":39845888,"data_stored":43428362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084408,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96624,"kb_used_data":61276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93648528,"statfs":{"total":95995035648,"available":95896092672,"internally_reserved":0,"allocated":62746624,"data_stored":64900942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378758,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94208,"kb_used_data":54696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93650944,"statfs":{"total":95995035648,"available":95898566656,"internally_reserved":0,"allocated":56008704,"data_stored":63640351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66212,"kb_used_data":30864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93678940,"statfs":{"total":95995035648,"available":95927234560,"internally_reserved":0,"allocated":31604736,"data_stored":32046192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110944,"kb_used_data":71432,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93634208,"statfs":{"total":95995035648,"available":95881428992,"internally_reserved":0,"allocated":73146368,"data_stored":76017711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70291456,"data_stored":74172200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29335552,"data_stored":30790407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53751808,"data_stored":62385193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61825024,"data_stored":64981406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":37588992,"data_stored":42174071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20557824,"data_stored":23546981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36122624,"data_stored":36335674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61992960,"data_stored":67888352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:23:49.092 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T19:23:49.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:23:49.094 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-20T19:23:49.094 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:23:49.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:49.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:49.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:49.212 INFO:tasks.rados.rados.0.smithi042.stdout:558: done (3 left) 2024-09-20T19:23:49.212 INFO:tasks.rados.rados.0.smithi042.stdout:559: done (2 left) 2024-09-20T19:23:49.212 INFO:tasks.rados.rados.0.smithi042.stdout:560: done (1 left) 2024-09-20T19:23:49.212 INFO:tasks.rados.rados.0.smithi042.stdout:561: done (0 left) 2024-09-20T19:23:49.213 INFO:tasks.rados.rados.0.smithi042.stdout:562: snap_create 2024-09-20T19:23:49.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:49.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:49.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.390 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:49 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:49.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.279743+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.281179+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.564272+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.565532+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:23:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.644796+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:23:49.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:47.645872+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:23:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.001063+0000 mgr.y (mgr.24491) 222 : cluster [DBG] pgmap v296: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:49.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.184052+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:23:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:49 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.185784+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:23:49.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:49.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:49.684 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:23:49.684 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:23:49.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:49.788 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:49.850 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:23:49.850 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:23:49.868 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:23:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:23:49.884 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:23:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:23:49.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:49.901 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:23:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:23:49.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:49.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:49.922 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:23:49.922 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:23:49.939 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:23:49.939 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:23:49.961 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:23:49.961 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:23:49.980 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:23:49.980 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:23:50.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:50.043 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:50.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:50.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:50.220 INFO:tasks.rados.rados.0.smithi042.stdout:562: done (0 left) 2024-09-20T19:23:50.220 INFO:tasks.rados.rados.0.smithi042.stdout:563: snap_remove snap 62 2024-09-20T19:23:50.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:50.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:50.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T19:23:50.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:50 smithi162 bash[24557]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:50.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[22706]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.551264+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.c scrub starts 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.553401+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.c scrub ok 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.842364+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:48.842635+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: audit 2024-09-20T19:23:49.119783+0000 mgr.y (mgr.24491) 223 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.212101+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-20T19:23:50.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:50 smithi042 bash[32075]: audit 2024-09-20T19:23:49.831788+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.42:0/3263480425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:23:50.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:50.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:50.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:50.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:50.802 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:50.830 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:50.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:50.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:51.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:51.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:51.223 INFO:tasks.rados.rados.0.smithi042.stdout:563: done (0 left) 2024-09-20T19:23:51.223 INFO:tasks.rados.rados.0.smithi042.stdout:564: write (excl) oid 18 current snap is 65 2024-09-20T19:23:51.223 INFO:tasks.rados.rados.0.smithi042.stdout:564: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-20T19:23:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:51.227 INFO:tasks.rados.rados.0.smithi042.stdout:564: writing smithi04268048-18 from 413505 to 1049924 tid 1 2024-09-20T19:23:51.235 INFO:tasks.rados.rados.0.smithi042.stdout:564: writing smithi04268048-18 from 1664805 to 2353869 tid 2 2024-09-20T19:23:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:51.243 INFO:tasks.rados.rados.0.smithi042.stdout:564: writing smithi04268048-18 from 3038061 to 3530132 tid 3 2024-09-20T19:23:51.243 INFO:tasks.rados.rados.0.smithi042.stdout:565: delete oid 3 current snap is 65 2024-09-20T19:23:51.243 INFO:tasks.rados.rados.0.smithi042.stdout:564: finishing write tid 1 to smithi04268048-18 2024-09-20T19:23:51.243 INFO:tasks.rados.rados.0.smithi042.stdout:564: finishing write tid 2 to smithi04268048-18 2024-09-20T19:23:51.246 INFO:tasks.rados.rados.0.smithi042.stdout:565: done (1 left) 2024-09-20T19:23:51.246 INFO:tasks.rados.rados.0.smithi042.stdout:566: read oid 1 snap -1 2024-09-20T19:23:51.246 INFO:tasks.rados.rados.0.smithi042.stdout:566: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:51.246 INFO:tasks.rados.rados.0.smithi042.stdout:567: write (excl) oid 46 current snap is 65 2024-09-20T19:23:51.246 INFO:tasks.rados.rados.0.smithi042.stdout:567: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-20T19:23:51.253 INFO:tasks.rados.rados.0.smithi042.stdout:567: writing smithi04268048-46 from 447119 to 1234036 tid 1 2024-09-20T19:23:51.260 INFO:tasks.rados.rados.0.smithi042.stdout:567: writing smithi04268048-46 from 1665416 to 2358149 tid 2 2024-09-20T19:23:51.266 INFO:tasks.rados.rados.0.smithi042.stdout:567: writing smithi04268048-46 from 2976554 to 3491169 tid 3 2024-09-20T19:23:51.266 INFO:tasks.rados.rados.0.smithi042.stdout:568: read oid 19 snap 58 2024-09-20T19:23:51.266 INFO:tasks.rados.rados.0.smithi042.stdout:568: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:569: snap_create 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: finishing write tid 3 to smithi04268048-18 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: finishing write tid 4 to smithi04268048-18 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: finishing write tid 5 to smithi04268048-18 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: oid 18 updating version 0 to 245 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: oid 18 updating version 245 to 249 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: oid 18 version 249 is already newer than 248 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: oid 18 updating version 249 to 250 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 250 (ObjNum 215 snap 65 seq_num 215) dirty exists 2024-09-20T19:23:51.267 INFO:tasks.rados.rados.0.smithi042.stdout:564: left oid 18 (ObjNum 215 snap 65 seq_num 215) 2024-09-20T19:23:51.268 INFO:tasks.rados.rados.0.smithi042.stdout:566: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:51.271 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 723216 2024-09-20T19:23:51.287 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:51.335 INFO:tasks.rados.rados.0.smithi042.stdout:567: finishing write tid 1 to smithi04268048-46 2024-09-20T19:23:51.335 INFO:tasks.rados.rados.0.smithi042.stdout:567: finishing write tid 2 to smithi04268048-46 2024-09-20T19:23:51.335 INFO:tasks.rados.rados.0.smithi042.stdout:567: finishing write tid 3 to smithi04268048-46 2024-09-20T19:23:51.335 INFO:tasks.rados.rados.0.smithi042.stdout:567: finishing write tid 4 to smithi04268048-46 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: finishing write tid 5 to smithi04268048-46 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: oid 46 updating version 0 to 292 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: oid 46 updating version 292 to 294 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: oid 46 version 294 is already newer than 293 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: oid 46 updating version 294 to 295 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 295 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:567: left oid 46 (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:568: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:23:51.336 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:51.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:51.369 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:51.402 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 186088 2024-09-20T19:23:51.461 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.461 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.464 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.603966+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.b scrub starts 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.604035+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.605215+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.b scrub ok 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.605393+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.641747+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.643737+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.764514+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:49.765541+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.001663+0000 mgr.y (mgr.24491) 224 : cluster [DBG] pgmap v299: 148 pgs: 4 active+clean+snaptrim, 144 active+clean; 147 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:51 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:51.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:51.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.022662+0000 osd.3 (osd.3) 117 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.024855+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:23:51.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.219031+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-20T19:23:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.234945+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:23:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:51 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.236472+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:23:51.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:51.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.881 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:51.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:51.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.013 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:52.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:52.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:52.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.252 INFO:tasks.rados.rados.0.smithi042.stdout:564: done (4 left) 2024-09-20T19:23:52.252 INFO:tasks.rados.rados.0.smithi042.stdout:566: done (3 left) 2024-09-20T19:23:52.252 INFO:tasks.rados.rados.0.smithi042.stdout:567: done (2 left) 2024-09-20T19:23:52.252 INFO:tasks.rados.rados.0.smithi042.stdout:568: done (1 left) 2024-09-20T19:23:52.253 INFO:tasks.rados.rados.0.smithi042.stdout:569: done (0 left) 2024-09-20T19:23:52.253 INFO:tasks.rados.rados.0.smithi042.stdout:570: write oid 14 current snap is 66 2024-09-20T19:23:52.253 INFO:tasks.rados.rados.0.smithi042.stdout:570: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:52 smithi162 bash[24557]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.260 INFO:tasks.rados.rados.0.smithi042.stdout:570: writing smithi04268048-14 from 463926 to 926092 tid 1 2024-09-20T19:23:52.269 INFO:tasks.rados.rados.0.smithi042.stdout:570: writing smithi04268048-14 from 1723898 to 2160289 tid 2 2024-09-20T19:23:52.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:52.277 INFO:tasks.rados.rados.0.smithi042.stdout:570: writing smithi04268048-14 from 2603977 to 3329864 tid 3 2024-09-20T19:23:52.278 INFO:tasks.rados.rados.0.smithi042.stdout:571: read oid 32 snap -1 2024-09-20T19:23:52.278 INFO:tasks.rados.rados.0.smithi042.stdout:571: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:23:52.278 INFO:tasks.rados.rados.0.smithi042.stdout:572: delete oid 28 current snap is 66 2024-09-20T19:23:52.278 INFO:tasks.rados.rados.0.smithi042.stdout:570: finishing write tid 1 to smithi04268048-14 2024-09-20T19:23:52.278 INFO:tasks.rados.rados.0.smithi042.stdout:570: finishing write tid 2 to smithi04268048-14 2024-09-20T19:23:52.282 INFO:tasks.rados.rados.0.smithi042.stdout:572: done (2 left) 2024-09-20T19:23:52.282 INFO:tasks.rados.rados.0.smithi042.stdout:573: write (excl) oid 30 current snap is 66 2024-09-20T19:23:52.282 INFO:tasks.rados.rados.0.smithi042.stdout:573: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-20T19:23:52.290 INFO:tasks.rados.rados.0.smithi042.stdout:573: writing smithi04268048-30 from 497540 to 1110204 tid 1 2024-09-20T19:23:52.299 INFO:tasks.rados.rados.0.smithi042.stdout:573: writing smithi04268048-30 from 1840862 to 2280922 tid 2 2024-09-20T19:23:52.309 INFO:tasks.rados.rados.0.smithi042.stdout:573: writing smithi04268048-30 from 3058823 to 3523607 tid 3 2024-09-20T19:23:52.309 INFO:tasks.rados.rados.0.smithi042.stdout:574: read oid 24 snap 61 2024-09-20T19:23:52.309 INFO:tasks.rados.rados.0.smithi042.stdout:574: expect deleted 2024-09-20T19:23:52.309 INFO:tasks.rados.rados.0.smithi042.stdout:575: read oid 5 snap -1 2024-09-20T19:23:52.309 INFO:tasks.rados.rados.0.smithi042.stdout:575: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:23:52.310 INFO:tasks.rados.rados.0.smithi042.stdout:576: copy_from oid 38 from oid 39 current snap is 66 2024-09-20T19:23:52.310 INFO:tasks.rados.rados.0.smithi042.stdout:570: finishing write tid 3 to smithi04268048-14 2024-09-20T19:23:52.310 INFO:tasks.rados.rados.0.smithi042.stdout:577: write oid 36 current snap is 66 2024-09-20T19:23:52.310 INFO:tasks.rados.rados.0.smithi042.stdout:577: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-20T19:23:52.316 INFO:tasks.rados.rados.0.smithi042.stdout:577: writing smithi04268048-36 from 514347 to 918613 tid 1 2024-09-20T19:23:52.326 INFO:tasks.rados.rados.0.smithi042.stdout:577: writing smithi04268048-36 from 1499344 to 2199415 tid 2 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:577: writing smithi04268048-36 from 2686246 to 3362302 tid 3 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:578: delete oid 23 current snap is 66 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:570: finishing write tid 4 to smithi04268048-14 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:570: finishing write tid 5 to smithi04268048-14 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:570: oid 14 updating version 0 to 152 2024-09-20T19:23:52.334 INFO:tasks.rados.rados.0.smithi042.stdout:570: oid 14 version 152 is already newer than 151 2024-09-20T19:23:52.335 INFO:tasks.rados.rados.0.smithi042.stdout:570: oid 14 version 152 is already newer than 150 2024-09-20T19:23:52.335 INFO:tasks.rados.rados.0.smithi042.stdout:570: oid 14 version 152 is already newer than 149 2024-09-20T19:23:52.335 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 152 (ObjNum 218 snap 66 seq_num 218) dirty exists 2024-09-20T19:23:52.335 INFO:tasks.rados.rados.0.smithi042.stdout:570: left oid 14 (ObjNum 218 snap 66 seq_num 218) 2024-09-20T19:23:52.335 INFO:tasks.rados.rados.0.smithi042.stdout:571: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:23:52.347 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1230072 2024-09-20T19:23:52.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:52.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[32075]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.585641+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.588175+0000 osd.0 (osd.0) 124 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.757080+0000 osd.5 (osd.5) 107 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:50.759064+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.219803+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.330617+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.334290+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.504607+0000 osd.1 (osd.1) 123 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.507206+0000 osd.1 (osd.1) 124 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.657617+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:52 smithi042 bash[22706]: cluster 2024-09-20T19:23:51.659034+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:23:52.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:573: finishing write tid 1 to smithi04268048-30 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:573: finishing write tid 2 to smithi04268048-30 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:573: finishing write tid 3 to smithi04268048-30 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:573: finishing write tid 4 to smithi04268048-30 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:575: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:23:52.521 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:52.530 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:52.565 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:52.610 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:52.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.654 INFO:tasks.rados.rados.0.smithi042.stdout:573: finishing write tid 5 to smithi04268048-30 2024-09-20T19:23:52.654 INFO:tasks.rados.rados.0.smithi042.stdout:573: oid 30 updating version 0 to 252 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:573: oid 30 updating version 252 to 253 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:573: oid 30 updating version 253 to 254 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:573: oid 30 version 254 is already newer than 251 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 254 (ObjNum 220 snap 66 seq_num 220) dirty exists 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:573: left oid 30 (ObjNum 220 snap 66 seq_num 220) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:576: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:577: finishing write tid 1 to smithi04268048-36 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:570: done (7 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:571: done (6 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:573: done (5 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:574: done (4 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:575: done (3 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:578: done (2 left) 2024-09-20T19:23:52.655 INFO:tasks.rados.rados.0.smithi042.stdout:579: read oid 40 snap -1 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:579: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:580: rollback oid 6 current snap is 66 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 59 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:576: finishing copy_from to smithi04268048-38 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 27 (ObjNum 68 snap 10 seq_num 68) dirty exists 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: finishing write tid 2 to smithi04268048-36 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: finishing write tid 3 to smithi04268048-36 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: finishing write tid 4 to smithi04268048-36 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: finishing write tid 5 to smithi04268048-36 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: oid 36 updating version 0 to 287 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: oid 36 updating version 287 to 288 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: oid 36 updating version 288 to 289 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:577: oid 36 updating version 289 to 290 2024-09-20T19:23:52.656 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 290 (ObjNum 221 snap 66 seq_num 221) dirty exists 2024-09-20T19:23:52.657 INFO:tasks.rados.rados.0.smithi042.stdout:577: left oid 36 (ObjNum 221 snap 66 seq_num 221) 2024-09-20T19:23:52.657 INFO:tasks.rados.rados.0.smithi042.stdout:576: done (3 left) 2024-09-20T19:23:52.657 INFO:tasks.rados.rados.0.smithi042.stdout:577: done (2 left) 2024-09-20T19:23:52.657 INFO:tasks.rados.rados.0.smithi042.stdout:581: write oid 7 current snap is 66 2024-09-20T19:23:52.657 INFO:tasks.rados.rados.0.smithi042.stdout:581: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-20T19:23:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:52.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:52.661 INFO:tasks.rados.rados.0.smithi042.stdout:581: writing smithi04268048-7 from 547961 to 1102725 tid 1 2024-09-20T19:23:52.670 INFO:tasks.rados.rados.0.smithi042.stdout:581: writing smithi04268048-7 from 1616308 to 2320048 tid 2 2024-09-20T19:23:52.678 INFO:tasks.rados.rados.0.smithi042.stdout:581: writing smithi04268048-7 from 2741092 to 3439692 tid 3 2024-09-20T19:23:52.678 INFO:tasks.rados.rados.0.smithi042.stdout:582: write (excl) oid 39 current snap is 66 2024-09-20T19:23:52.678 INFO:tasks.rados.rados.0.smithi042.stdout:580: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:23:52.679 INFO:tasks.rados.rados.0.smithi042.stdout:580: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:23:52.679 INFO:tasks.rados.rados.0.smithi042.stdout:582: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-20T19:23:52.686 INFO:tasks.rados.rados.0.smithi042.stdout:582: writing smithi04268048-39 from 564768 to 1194781 tid 1 2024-09-20T19:23:52.692 INFO:tasks.rados.rados.0.smithi042.stdout:582: writing smithi04268048-39 from 1674790 to 2122188 tid 2 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:582: writing smithi04268048-39 from 2768515 to 3394740 tid 3 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:583: delete oid 49 current snap is 66 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:580: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 160 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:579: expect (ObjNum 189 snap 56 seq_num 189) 2024-09-20T19:23:52.701 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:52.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:52.765 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:52.847 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2965612 2024-09-20T19:23:52.856 INFO:tasks.rados.rados.0.smithi042.stdout:581: finishing write tid 1 to smithi04268048-7 2024-09-20T19:23:52.856 INFO:tasks.rados.rados.0.smithi042.stdout:581: finishing write tid 2 to smithi04268048-7 2024-09-20T19:23:52.856 INFO:tasks.rados.rados.0.smithi042.stdout:581: finishing write tid 3 to smithi04268048-7 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: finishing write tid 4 to smithi04268048-7 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: finishing write tid 5 to smithi04268048-7 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: oid 7 updating version 0 to 107 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: oid 7 updating version 107 to 108 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: oid 7 version 108 is already newer than 106 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: oid 7 version 108 is already newer than 105 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 108 (ObjNum 223 snap 66 seq_num 223) dirty exists 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:581: left oid 7 (ObjNum 223 snap 66 seq_num 223) 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:582: finishing write tid 1 to smithi04268048-39 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:582: finishing write tid 2 to smithi04268048-39 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:582: finishing write tid 3 to smithi04268048-39 2024-09-20T19:23:52.857 INFO:tasks.rados.rados.0.smithi042.stdout:582: finishing write tid 4 to smithi04268048-39 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: finishing write tid 5 to smithi04268048-39 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: oid 39 updating version 0 to 151 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: oid 39 version 151 is already newer than 148 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: oid 39 version 151 is already newer than 149 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: oid 39 version 151 is already newer than 150 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 151 (ObjNum 224 snap 66 seq_num 224) dirty exists 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:582: left oid 39 (ObjNum 224 snap 66 seq_num 224) 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:579: done (4 left) 2024-09-20T19:23:52.858 INFO:tasks.rados.rados.0.smithi042.stdout:580: done (3 left) 2024-09-20T19:23:52.859 INFO:tasks.rados.rados.0.smithi042.stdout:581: done (2 left) 2024-09-20T19:23:52.859 INFO:tasks.rados.rados.0.smithi042.stdout:582: done (1 left) 2024-09-20T19:23:52.859 INFO:tasks.rados.rados.0.smithi042.stdout:583: done (0 left) 2024-09-20T19:23:52.859 INFO:tasks.rados.rados.0.smithi042.stdout:584: write (excl) oid 42 current snap is 66 2024-09-20T19:23:52.859 INFO:tasks.rados.rados.0.smithi042.stdout:584: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-20T19:23:52.863 INFO:tasks.rados.rados.0.smithi042.stdout:584: writing smithi04268048-42 from 598382 to 1378893 tid 1 2024-09-20T19:23:52.870 INFO:tasks.rados.rados.0.smithi042.stdout:584: writing smithi04268048-42 from 2075401 to 2526468 tid 2 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:584: writing smithi04268048-42 from 3107008 to 3755777 tid 3 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:585: read oid 4 snap 58 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:585: expect deleted 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:586: delete oid 6 current snap is 66 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:584: finishing write tid 1 to smithi04268048-42 2024-09-20T19:23:52.878 INFO:tasks.rados.rados.0.smithi042.stdout:584: finishing write tid 2 to smithi04268048-42 2024-09-20T19:23:52.880 INFO:tasks.rados.rados.0.smithi042.stdout:586: done (2 left) 2024-09-20T19:23:52.880 INFO:tasks.rados.rados.0.smithi042.stdout:587: snap_create 2024-09-20T19:23:52.882 INFO:tasks.rados.rados.0.smithi042.stdout:584: finishing write tid 3 to smithi04268048-42 2024-09-20T19:23:52.883 INFO:tasks.rados.rados.0.smithi042.stdout:584: finishing write tid 4 to smithi04268048-42 2024-09-20T19:23:52.883 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: finishing write tid 5 to smithi04268048-42 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: oid 42 updating version 0 to 268 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: oid 42 updating version 268 to 270 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: oid 42 version 270 is already newer than 267 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: oid 42 version 270 is already newer than 269 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 270 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:23:52.884 INFO:tasks.rados.rados.0.smithi042.stdout:584: left oid 42 (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:23:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:52.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:53.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:53.032 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:53.049 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:23:53.050 INFO:tasks.rados.rados.0.smithi042.stdout:584: done (2 left) 2024-09-20T19:23:53.050 INFO:tasks.rados.rados.0.smithi042.stdout:585: done (1 left) 2024-09-20T19:23:53.050 INFO:tasks.rados.rados.0.smithi042.stdout:587: done (0 left) 2024-09-20T19:23:53.050 INFO:tasks.rados.rados.0.smithi042.stdout:588: write (excl) oid 30 current snap is 67 2024-09-20T19:23:53.050 INFO:tasks.rados.rados.0.smithi042.stdout:588: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-20T19:23:53.057 INFO:tasks.rados.rados.0.smithi042.stdout:588: writing smithi04268048-30 from 631996 to 1163005 tid 1 2024-09-20T19:23:53.064 INFO:tasks.rados.rados.0.smithi042.stdout:588: writing smithi04268048-30 from 1792365 to 2247101 tid 2 2024-09-20T19:23:53.072 INFO:tasks.rados.rados.0.smithi042.stdout:588: writing smithi04268048-30 from 2761854 to 3433167 tid 3 2024-09-20T19:23:53.072 INFO:tasks.rados.rados.0.smithi042.stdout:589: write (excl) oid 3 current snap is 67 2024-09-20T19:23:53.072 INFO:tasks.rados.rados.0.smithi042.stdout:588: finishing write tid 1 to smithi04268048-30 2024-09-20T19:23:53.072 INFO:tasks.rados.rados.0.smithi042.stdout:589: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-20T19:23:53.079 INFO:tasks.rados.rados.0.smithi042.stdout:589: writing smithi04268048-3 from 648803 to 1371414 tid 1 2024-09-20T19:23:53.087 INFO:tasks.rados.rados.0.smithi042.stdout:589: writing smithi04268048-3 from 1967200 to 2681947 tid 2 2024-09-20T19:23:53.094 INFO:tasks.rados.rados.0.smithi042.stdout:589: writing smithi04268048-3 from 3421983 to 3848803 tid 3 2024-09-20T19:23:53.097 INFO:tasks.rados.rados.0.smithi042.stdout:588: finishing write tid 2 to smithi04268048-30 2024-09-20T19:23:53.098 INFO:tasks.rados.rados.0.smithi042.stdout:588: finishing write tid 3 to smithi04268048-30 2024-09-20T19:23:53.098 INFO:tasks.rados.rados.0.smithi042.stdout:588: finishing write tid 4 to smithi04268048-30 2024-09-20T19:23:53.098 INFO:tasks.rados.rados.0.smithi042.stdout:588: finishing write tid 5 to smithi04268048-30 2024-09-20T19:23:53.098 INFO:tasks.rados.rados.0.smithi042.stdout:588: oid 30 updating version 0 to 257 2024-09-20T19:23:53.098 INFO:tasks.rados.rados.0.smithi042.stdout:588: oid 30 version 257 is already newer than 256 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:588: oid 30 updating version 257 to 258 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:588: oid 30 updating version 258 to 259 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 259 (ObjNum 228 snap 67 seq_num 228) dirty exists 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:588: left oid 30 (ObjNum 228 snap 67 seq_num 228) 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:589: finishing write tid 1 to smithi04268048-3 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:589: finishing write tid 2 to smithi04268048-3 2024-09-20T19:23:53.099 INFO:tasks.rados.rados.0.smithi042.stdout:588: done (1 left) 2024-09-20T19:23:53.100 INFO:tasks.rados.rados.0.smithi042.stdout:590: delete oid 22 current snap is 67 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:590: done (1 left) 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:591: read oid 17 snap -1 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:591: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:592: write oid 48 current snap is 67 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:589: finishing write tid 3 to smithi04268048-3 2024-09-20T19:23:53.101 INFO:tasks.rados.rados.0.smithi042.stdout:592: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-20T19:23:53.105 INFO:tasks.rados.rados.0.smithi042.stdout:592: writing smithi04268048-48 from 682417 to 1155526 tid 1 2024-09-20T19:23:53.111 INFO:tasks.rados.rados.0.smithi042.stdout:592: writing smithi04268048-48 from 1567811 to 2169874 tid 2 2024-09-20T19:23:53.119 INFO:tasks.rados.rados.0.smithi042.stdout:592: writing smithi04268048-48 from 2844123 to 3465605 tid 3 2024-09-20T19:23:53.119 INFO:tasks.rados.rados.0.smithi042.stdout:593: snap_remove snap 61 2024-09-20T19:23:53.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:53.154 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:53.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:53.278 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:53.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[32075]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:53 smithi042 bash[22706]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:53.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.003236+0000 mgr.y (mgr.24491) 225 : cluster [DBG] pgmap v302: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.250679+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.479711+0000 mon.a (mon.0) 1200 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.647105+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.648571+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.843235+0000 mon.c (mon.2) 48 : audit [INF] from='client.? v1:172.21.15.42:0/1517045296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:52.843571+0000 mon.a (mon.0) 1201 : audit [INF] from='client.24947 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.043184+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.043206+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.044536+0000 mon.a (mon.0) 1204 : audit [INF] from='client.24947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.048354+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.217374+0000 mon.a (mon.0) 1206 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.218101+0000 mon.a (mon.0) 1207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:53 smithi162 bash[24557]: audit 2024-09-20T19:23:53.224222+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:53.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:53.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:53.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:53.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:53.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:53.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:53.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:54.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.045 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:54.059 INFO:tasks.rados.rados.0.smithi042.stdout:593: done (3 left) 2024-09-20T19:23:54.059 INFO:tasks.rados.rados.0.smithi042.stdout:594: snap_remove snap 58 2024-09-20T19:23:54.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:54.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:54.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.286 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:54 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:54.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:23:54.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.725095+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:52.726530+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.369 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:54 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.058145+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-20T19:23:54.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:54.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:54.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:54.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.690 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:54.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:54.922 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:54.951 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:54.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:55.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:55.071 INFO:tasks.rados.rados.0.smithi042.stdout:594: done (3 left) 2024-09-20T19:23:55.071 INFO:tasks.rados.rados.0.smithi042.stdout:595: read oid 42 snap -1 2024-09-20T19:23:55.071 INFO:tasks.rados.rados.0.smithi042.stdout:595: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:596: copy_from oid 25 from oid 26 current snap is 67 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:597: copy_from oid 7 from oid 50 current snap is 67 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:598: copy_from oid 19 from oid 37 current snap is 67 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:599: delete oid 21 current snap is 67 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:589: finishing write tid 4 to smithi04268048-3 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:589: finishing write tid 5 to smithi04268048-3 2024-09-20T19:23:55.072 INFO:tasks.rados.rados.0.smithi042.stdout:589: oid 3 updating version 0 to 168 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:589: oid 3 version 168 is already newer than 165 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:589: oid 3 version 168 is already newer than 166 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:589: oid 3 version 168 is already newer than 167 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 168 (ObjNum 229 snap 67 seq_num 229) dirty exists 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:589: left oid 3 (ObjNum 229 snap 67 seq_num 229) 2024-09-20T19:23:55.073 INFO:tasks.rados.rados.0.smithi042.stdout:591: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:23:55.086 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:55.126 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: finishing write tid 1 to smithi04268048-48 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: finishing write tid 2 to smithi04268048-48 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: finishing write tid 3 to smithi04268048-48 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: finishing write tid 4 to smithi04268048-48 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: finishing write tid 5 to smithi04268048-48 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: oid 48 updating version 0 to 261 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: oid 48 updating version 261 to 262 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: oid 48 updating version 262 to 264 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:592: oid 48 version 264 is already newer than 263 2024-09-20T19:23:55.155 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 264 (ObjNum 231 snap 67 seq_num 231) dirty exists 2024-09-20T19:23:55.156 INFO:tasks.rados.rados.0.smithi042.stdout:592: left oid 48 (ObjNum 231 snap 67 seq_num 231) 2024-09-20T19:23:55.156 INFO:tasks.rados.rados.0.smithi042.stdout:595: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:23:55.166 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2679462 2024-09-20T19:23:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:23:55.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:55.214 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:55.262 INFO:tasks.rados.rados.0.smithi042.stdout:598: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:23:55.262 INFO:tasks.rados.rados.0.smithi042.stdout:597: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:23:55.262 INFO:tasks.rados.rados.0.smithi042.stdout:598: finishing copy_from to smithi04268048-19 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 134 (ObjNum 157 snap 45 seq_num 157) dirty exists 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:597: finishing copy_from to smithi04268048-7 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 250 (ObjNum 137 snap 36 seq_num 137) dirty exists 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:596: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:589: done (7 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:591: done (6 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:592: done (5 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:595: done (4 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:597: done (3 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:598: done (2 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:599: done (1 left) 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:600: write (excl) oid 43 current snap is 67 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:596: finishing copy_from to smithi04268048-25 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 104 (ObjNum 108 snap 25 seq_num 108) dirty exists 2024-09-20T19:23:55.263 INFO:tasks.rados.rados.0.smithi042.stdout:600: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-20T19:23:55.271 INFO:tasks.rados.rados.0.smithi042.stdout:600: writing smithi04268048-43 from 716031 to 1339638 tid 1 2024-09-20T19:23:55.280 INFO:tasks.rados.rados.0.smithi042.stdout:600: writing smithi04268048-43 from 2084775 to 2690507 tid 2 2024-09-20T19:23:55.289 INFO:tasks.rados.rados.0.smithi042.stdout:600: writing smithi04268048-43 from 3298969 to 3916031 tid 3 2024-09-20T19:23:55.289 INFO:tasks.rados.rados.0.smithi042.stdout:596: done (1 left) 2024-09-20T19:23:55.289 INFO:tasks.rados.rados.0.smithi042.stdout:601: snap_create 2024-09-20T19:23:55.289 INFO:tasks.rados.rados.0.smithi042.stdout:600: finishing write tid 1 to smithi04268048-43 2024-09-20T19:23:55.289 INFO:tasks.rados.rados.0.smithi042.stdout:600: finishing write tid 2 to smithi04268048-43 2024-09-20T19:23:55.292 INFO:tasks.rados.rados.0.smithi042.stdout:600: finishing write tid 3 to smithi04268048-43 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: finishing write tid 4 to smithi04268048-43 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: finishing write tid 5 to smithi04268048-43 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: oid 43 updating version 0 to 145 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: oid 43 updating version 145 to 146 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: oid 43 updating version 146 to 148 2024-09-20T19:23:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:600: oid 43 version 148 is already newer than 147 2024-09-20T19:23:55.294 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 148 (ObjNum 233 snap 67 seq_num 233) dirty exists 2024-09-20T19:23:55.294 INFO:tasks.rados.rados.0.smithi042.stdout:600: left oid 43 (ObjNum 233 snap 67 seq_num 233) 2024-09-20T19:23:55.317 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:55.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:55.455 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.455 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.455 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.455 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.456 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.457 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:55 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.469 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.822300+0000 osd.5 (osd.5) 109 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:53.824640+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.003934+0000 mgr.y (mgr.24491) 226 : cluster [DBG] pgmap v305: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 26 op/s 2024-09-20T19:23:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.532488+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:23:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.536965+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:23:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:55 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.067644+0000 mon.a (mon.0) 1210 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-20T19:23:55.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:55.599 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:55.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:55.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:55.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:55.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:55.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:56.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:56.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:56.108 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:23:56.137 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:56.273 INFO:tasks.rados.rados.0.smithi042.stdout:600: done (1 left) 2024-09-20T19:23:56.273 INFO:tasks.rados.rados.0.smithi042.stdout:601: done (0 left) 2024-09-20T19:23:56.273 INFO:tasks.rados.rados.0.smithi042.stdout:602: snap_create 2024-09-20T19:23:56.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:56.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:56.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:23:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[32075]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:56 smithi042 bash[22706]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:56.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.534961+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:54.537102+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.220923+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:56 smithi162 bash[24557]: cluster 2024-09-20T19:23:55.223323+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-09-20T19:23:56.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:56.671 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:56.692 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:56.747 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:23:56.747 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:23:56.747 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:23:56.747 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:23:56.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:56.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:56.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:56.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:56.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:57.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:57.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:23:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:23:57.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:57.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:57.281 INFO:tasks.rados.rados.0.smithi042.stdout:602: done (0 left) 2024-09-20T19:23:57.281 INFO:tasks.rados.rados.0.smithi042.stdout:603: read oid 38 snap 59 2024-09-20T19:23:57.281 INFO:tasks.rados.rados.0.smithi042.stdout:603: expect deleted 2024-09-20T19:23:57.282 INFO:tasks.rados.rados.0.smithi042.stdout:604: rollback oid 48 current snap is 69 2024-09-20T19:23:57.282 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 59 2024-09-20T19:23:57.282 INFO:tasks.rados.rados.0.smithi042.stdout:605: snap_remove snap 65 2024-09-20T19:23:57.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:57 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:57.363 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:23:57.364 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:23:57.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:57.457 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:57.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T19:23:57.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:57.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.004648+0000 mgr.y (mgr.24491) 227 : cluster [DBG] pgmap v308: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 161 MiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.273728+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: audit 2024-09-20T19:23:56.959571+0000 mon.a (mon.0) 1212 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: audit 2024-09-20T19:23:56.960742+0000 mon.a (mon.0) 1213 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.045587+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:57 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.047146+0000 osd.3 (osd.3) 120 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:23:57.590 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:57.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:57.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:57.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:57.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:57.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:23:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:57.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:23:58.055 INFO:tasks.rados.rados.0.smithi042.stdout:605: done (2 left) 2024-09-20T19:23:58.055 INFO:tasks.rados.rados.0.smithi042.stdout:606: rollback oid 12 current snap is 69 2024-09-20T19:23:58.055 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 64 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:604: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:604: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:604: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 284 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:603: done (2 left) 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:604: done (1 left) 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:607: rollback oid 18 current snap is 69 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 64 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:608: rollback oid 43 current snap is 69 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 68 2024-09-20T19:23:58.056 INFO:tasks.rados.rados.0.smithi042.stdout:609: delete oid 13 current snap is 69 2024-09-20T19:23:58.068 INFO:tasks.rados.rados.0.smithi042.stdout:609: done (3 left) 2024-09-20T19:23:58.068 INFO:tasks.rados.rados.0.smithi042.stdout:610: rollback oid 36 current snap is 69 2024-09-20T19:23:58.068 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 36 to 63 2024-09-20T19:23:58.068 INFO:tasks.rados.rados.0.smithi042.stdout:611: delete oid 24 current snap is 69 2024-09-20T19:23:58.072 INFO:tasks.rados.rados.0.smithi042.stdout:606: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:23:58.072 INFO:tasks.rados.rados.0.smithi042.stdout:606: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:23:58.073 INFO:tasks.rados.rados.0.smithi042.stdout:606: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:23:58.073 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 152 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:23:58.077 INFO:tasks.rados.rados.0.smithi042.stdout:608: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:23:58.077 INFO:tasks.rados.rados.0.smithi042.stdout:606: done (4 left) 2024-09-20T19:23:58.077 INFO:tasks.rados.rados.0.smithi042.stdout:611: done (3 left) 2024-09-20T19:23:58.077 INFO:tasks.rados.rados.0.smithi042.stdout:612: write oid 25 current snap is 69 2024-09-20T19:23:58.078 INFO:tasks.rados.rados.0.smithi042.stdout:612: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-20T19:23:58.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:58.086 INFO:tasks.rados.rados.0.smithi042.stdout:612: writing smithi04268048-25 from 766452 to 1332159 tid 1 2024-09-20T19:23:58.093 INFO:tasks.rados.rados.0.smithi042.stdout:612: writing smithi04268048-25 from 1860221 to 2329633 tid 2 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:612: writing smithi04268048-25 from 2981238 to 3575433 tid 3 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:608: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:613: read oid 24 snap -1 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:613: expect deleted 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:614: read oid 19 snap -1 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:614: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:615: read oid 1 snap 60 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:615: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:58.101 INFO:tasks.rados.rados.0.smithi042.stdout:616: snap_remove snap 67 2024-09-20T19:23:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:58.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:58.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:58.255 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-20T19:23:57.271767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191327988,"num_objects":275,"num_object_clones":30,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":6,"num_read":2466,"num_read_kb":904552,"num_write":1500,"num_write_kb":369935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":591028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3032,"ondisk_log_size":3032,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":753172,"kb_used_data":470356,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":749208044,"statfs":{"total":767960285184,"available":767189037056,"internally_reserved":0,"allocated":481644544,"data_stored":531067743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22476388,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":72,"num_read_kb":52150,"num_write":61,"num_write_kb":28752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.226728"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670231+0000","last_change":"2024-09-20T19:21:59.826969+0000","last_active":"2024-09-20T19:23:56.670231+0000","last_peered":"2024-09-20T19:23:56.670231+0000","last_clean":"2024-09-20T19:23:56.670231+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:23:56.670231+0000","last_undegraded":"2024-09-20T19:23:56.670231+0000","last_fullsized":"2024-09-20T19:23:56.670231+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:59.826916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:59.826916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":399,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.275737+0000","last_change":"2024-09-20T19:23:50.588385+0000","last_active":"2024-09-20T19:23:56.275737+0000","last_peered":"2024-09-20T19:23:56.275737+0000","last_clean":"2024-09-20T19:23:56.275737+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:23:56.275737+0000","last_undegraded":"2024-09-20T19:23:56.275737+0000","last_fullsized":"2024-09-20T19:23:56.275737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.588194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.588194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.888096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":411,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670792+0000","last_change":"2024-09-20T19:22:14.898580+0000","last_active":"2024-09-20T19:23:56.670792+0000","last_peered":"2024-09-20T19:23:56.670792+0000","last_clean":"2024-09-20T19:23:56.670792+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:23:56.670792+0000","last_undegraded":"2024-09-20T19:23:56.670792+0000","last_fullsized":"2024-09-20T19:23:56.670792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:14.898494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.898494+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.281511+0000","last_change":"2024-09-20T19:23:50.236531+0000","last_active":"2024-09-20T19:23:56.281511+0000","last_peered":"2024-09-20T19:23:56.281511+0000","last_clean":"2024-09-20T19:23:56.281511+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:23:56.281511+0000","last_undegraded":"2024-09-20T19:23:56.281511+0000","last_fullsized":"2024-09-20T19:23:56.281511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.236477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.236477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:07.670048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":390,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671680+0000","last_change":"2024-09-20T19:22:58.622973+0000","last_active":"2024-09-20T19:23:56.671680+0000","last_peered":"2024-09-20T19:23:56.671680+0000","last_clean":"2024-09-20T19:23:56.671680+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:23:56.671680+0000","last_undegraded":"2024-09-20T19:23:56.671680+0000","last_fullsized":"2024-09-20T19:23:56.671680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:58.622796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.622796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.819628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":403,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.281796+0000","last_change":"2024-09-20T19:23:47.200445+0000","last_active":"2024-09-20T19:23:56.281796+0000","last_peered":"2024-09-20T19:23:56.281796+0000","last_clean":"2024-09-20T19:23:56.281796+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:23:56.281796+0000","last_undegraded":"2024-09-20T19:23:56.281796+0000","last_fullsized":"2024-09-20T19:23:56.281796+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:23:47.200185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.200185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.587959+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671117+0000","last_change":"2024-09-20T19:22:03.981228+0000","last_active":"2024-09-20T19:23:56.671117+0000","last_peered":"2024-09-20T19:23:56.671117+0000","last_clean":"2024-09-20T19:23:56.671117+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:23:56.671117+0000","last_undegraded":"2024-09-20T19:23:56.671117+0000","last_fullsized":"2024-09-20T19:23:56.671117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.981101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.981101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.455385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282418+0000","last_change":"2024-09-20T19:22:14.871281+0000","last_active":"2024-09-20T19:23:56.282418+0000","last_peered":"2024-09-20T19:23:56.282418+0000","last_clean":"2024-09-20T19:23:56.282418+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:23:56.282418+0000","last_undegraded":"2024-09-20T19:23:56.282418+0000","last_fullsized":"2024-09-20T19:23:56.282418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:14.871139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.871139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.971798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.678053+0000","last_change":"2024-09-20T19:23:08.305837+0000","last_active":"2024-09-20T19:23:56.678053+0000","last_peered":"2024-09-20T19:23:56.678053+0000","last_clean":"2024-09-20T19:23:56.678053+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:23:56.678053+0000","last_undegraded":"2024-09-20T19:23:56.678053+0000","last_fullsized":"2024-09-20T19:23:56.678053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.305714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.305714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:26.456491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":397,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282724+0000","last_change":"2024-09-20T19:23:02.238503+0000","last_active":"2024-09-20T19:23:56.282724+0000","last_peered":"2024-09-20T19:23:56.282724+0000","last_clean":"2024-09-20T19:23:56.282724+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:23:56.282724+0000","last_undegraded":"2024-09-20T19:23:56.282724+0000","last_fullsized":"2024-09-20T19:23:56.282724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:23:02.238438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.238438+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:03.780590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677075+0000","last_change":"2024-09-20T19:22:08.909960+0000","last_active":"2024-09-20T19:23:56.677075+0000","last_peered":"2024-09-20T19:23:56.677075+0000","last_clean":"2024-09-20T19:23:56.677075+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:23:56.677075+0000","last_undegraded":"2024-09-20T19:23:56.677075+0000","last_fullsized":"2024-09-20T19:23:56.677075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:08.909835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.909835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":377,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676314+0000","last_change":"2024-09-20T19:22:20.013446+0000","last_active":"2024-09-20T19:23:56.676314+0000","last_peered":"2024-09-20T19:23:56.676314+0000","last_clean":"2024-09-20T19:23:56.676314+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:23:56.676314+0000","last_undegraded":"2024-09-20T19:23:56.676314+0000","last_fullsized":"2024-09-20T19:23:56.676314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.013380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.013380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677364+0000","last_change":"2024-09-20T19:22:20.892258+0000","last_active":"2024-09-20T19:23:56.677364+0000","last_peered":"2024-09-20T19:23:56.677364+0000","last_clean":"2024-09-20T19:23:56.677364+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:23:56.677364+0000","last_undegraded":"2024-09-20T19:23:56.677364+0000","last_fullsized":"2024-09-20T19:23:56.677364+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.892102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.892102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":403,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282848+0000","last_change":"2024-09-20T19:22:07.939814+0000","last_active":"2024-09-20T19:23:56.282848+0000","last_peered":"2024-09-20T19:23:56.282848+0000","last_clean":"2024-09-20T19:23:56.282848+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:23:56.282848+0000","last_undegraded":"2024-09-20T19:23:56.282848+0000","last_fullsized":"2024-09-20T19:23:56.282848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:07.939740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:07.939740+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.138214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":373,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676561+0000","last_change":"2024-09-20T19:21:07.802088+0000","last_active":"2024-09-20T19:23:56.676561+0000","last_peered":"2024-09-20T19:23:56.676561+0000","last_clean":"2024-09-20T19:23:56.676561+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:23:56.676561+0000","last_undegraded":"2024-09-20T19:23:56.676561+0000","last_fullsized":"2024-09-20T19:23:56.676561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.801948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.801948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.780869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":371,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272666+0000","last_change":"2024-09-20T19:21:08.846790+0000","last_active":"2024-09-20T19:23:56.272666+0000","last_peered":"2024-09-20T19:23:56.272666+0000","last_clean":"2024-09-20T19:23:56.272666+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:23:56.272666+0000","last_undegraded":"2024-09-20T19:23:56.272666+0000","last_fullsized":"2024-09-20T19:23:56.272666+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.846665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.846665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.808843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"246'81","reported_seq":752,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272366+0000","last_change":"2024-09-20T19:23:55.073833+0000","last_active":"2024-09-20T19:23:56.272366+0000","last_peered":"2024-09-20T19:23:56.272366+0000","last_clean":"2024-09-20T19:23:56.272366+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:23:56.272366+0000","last_undegraded":"2024-09-20T19:23:56.272366+0000","last_fullsized":"2024-09-20T19:23:56.272366+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'45","last_scrub_stamp":"2024-09-20T19:22:54.853379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.853379+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:04.296892+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.011672405,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":11858,"num_write":31,"num_write_kb":8005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.d","version":"77'17","reported_seq":415,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.283811+0000","last_change":"2024-09-20T19:23:55.223430+0000","last_active":"2024-09-20T19:23:56.283811+0000","last_peered":"2024-09-20T19:23:56.283811+0000","last_clean":"2024-09-20T19:23:56.283811+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:23:56.283811+0000","last_undegraded":"2024-09-20T19:23:56.283811+0000","last_fullsized":"2024-09-20T19:23:56.283811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:23:55.223328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:55.223328+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:10.076063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677463+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:23:56.677463+0000","last_peered":"2024-09-20T19:23:56.677463+0000","last_clean":"2024-09-20T19:23:56.677463+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:23:56.677463+0000","last_undegraded":"2024-09-20T19:23:56.677463+0000","last_fullsized":"2024-09-20T19:23:56.677463+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671890+0000","last_change":"2024-09-20T19:23:48.553467+0000","last_active":"2024-09-20T19:23:56.671890+0000","last_peered":"2024-09-20T19:23:56.671890+0000","last_clean":"2024-09-20T19:23:56.671890+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:23:56.671890+0000","last_undegraded":"2024-09-20T19:23:56.671890+0000","last_fullsized":"2024-09-20T19:23:56.671890+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.553406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.553406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.629688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.273183+0000","last_change":"2024-09-20T19:21:25.309366+0000","last_active":"2024-09-20T19:23:56.273183+0000","last_peered":"2024-09-20T19:23:56.273183+0000","last_clean":"2024-09-20T19:23:56.273183+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:23:56.273183+0000","last_undegraded":"2024-09-20T19:23:56.273183+0000","last_fullsized":"2024-09-20T19:23:56.273183+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:25.309306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:25.309306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"246'154","reported_seq":832,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.275173+0000","last_change":"2024-09-20T19:23:55.077082+0000","last_active":"2024-09-20T19:23:56.275173+0000","last_peered":"2024-09-20T19:23:56.275173+0000","last_clean":"2024-09-20T19:23:56.275173+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:23:56.275173+0000","last_undegraded":"2024-09-20T19:23:56.275173+0000","last_fullsized":"2024-09-20T19:23:56.275173+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'100","last_scrub_stamp":"2024-09-20T19:23:11.701907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.701907+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.749827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011651158,"stat_sum":{"num_bytes":11899323,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":88,"num_read_kb":74570,"num_write":58,"num_write_kb":19071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.c","version":"77'10","reported_seq":410,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.283216+0000","last_change":"2024-09-20T19:23:42.253431+0000","last_active":"2024-09-20T19:23:56.283216+0000","last_peered":"2024-09-20T19:23:56.283216+0000","last_clean":"2024-09-20T19:23:56.283216+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:23:56.283216+0000","last_undegraded":"2024-09-20T19:23:56.283216+0000","last_fullsized":"2024-09-20T19:23:56.283216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:42.253363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.253363+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.814248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671935+0000","last_change":"2024-09-20T19:21:16.692876+0000","last_active":"2024-09-20T19:23:56.671935+0000","last_peered":"2024-09-20T19:23:56.671935+0000","last_clean":"2024-09-20T19:23:56.671935+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:23:56.671935+0000","last_undegraded":"2024-09-20T19:23:56.671935+0000","last_fullsized":"2024-09-20T19:23:56.671935+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:16.692759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.692759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.508852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":457,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.273412+0000","last_change":"2024-09-20T19:21:19.204377+0000","last_active":"2024-09-20T19:23:56.273412+0000","last_peered":"2024-09-20T19:23:56.273412+0000","last_clean":"2024-09-20T19:23:56.273412+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:23:56.273412+0000","last_undegraded":"2024-09-20T19:23:56.273412+0000","last_fullsized":"2024-09-20T19:23:56.273412+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:19.204313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:19.204313+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":371,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.283799+0000","last_change":"2024-09-20T19:21:00.850362+0000","last_active":"2024-09-20T19:23:56.283799+0000","last_peered":"2024-09-20T19:23:56.283799+0000","last_clean":"2024-09-20T19:23:56.283799+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:23:56.283799+0000","last_undegraded":"2024-09-20T19:23:56.283799+0000","last_fullsized":"2024-09-20T19:23:56.283799+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.850236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.850236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.099845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"246'167","reported_seq":838,"reported_epoch":247,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:56.675552+0000","last_change":"2024-09-20T19:23:56.082614+0000","last_active":"2024-09-20T19:23:56.675552+0000","last_peered":"2024-09-20T19:23:56.675552+0000","last_clean":"2024-09-20T19:23:56.675552+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:23:56.675552+0000","last_undegraded":"2024-09-20T19:23:56.675552+0000","last_fullsized":"2024-09-20T19:23:56.675552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'57","last_scrub_stamp":"2024-09-20T19:22:52.771766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771766+0000","objects_scrubbed":6,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.964300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99499901800000001,"stat_sum":{"num_bytes":15138036,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":95,"num_read_kb":67282,"num_write":65,"num_write_kb":19898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"5"},{"start":"3f","length":"2"}]},{"pgid":"4.f","version":"77'15","reported_seq":412,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672757+0000","last_change":"2024-09-20T19:23:04.313290+0000","last_active":"2024-09-20T19:23:56.672757+0000","last_peered":"2024-09-20T19:23:56.672757+0000","last_clean":"2024-09-20T19:23:56.672757+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:23:56.672757+0000","last_undegraded":"2024-09-20T19:23:56.672757+0000","last_fullsized":"2024-09-20T19:23:56.672757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.291138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":391,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671394+0000","last_change":"2024-09-20T19:23:45.542703+0000","last_active":"2024-09-20T19:23:56.671394+0000","last_peered":"2024-09-20T19:23:56.671394+0000","last_clean":"2024-09-20T19:23:56.671394+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:23:56.671394+0000","last_undegraded":"2024-09-20T19:23:56.671394+0000","last_fullsized":"2024-09-20T19:23:56.671394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:45.542589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:45.542589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.881406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":463,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.284063+0000","last_change":"2024-09-20T19:23:03.226408+0000","last_active":"2024-09-20T19:23:56.284063+0000","last_peered":"2024-09-20T19:23:56.284063+0000","last_clean":"2024-09-20T19:23:56.284063+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:23:56.284063+0000","last_undegraded":"2024-09-20T19:23:56.284063+0000","last_fullsized":"2024-09-20T19:23:56.284063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.212037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":377,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677516+0000","last_change":"2024-09-20T19:23:03.945590+0000","last_active":"2024-09-20T19:23:56.677516+0000","last_peered":"2024-09-20T19:23:56.677516+0000","last_clean":"2024-09-20T19:23:56.677516+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:23:56.677516+0000","last_undegraded":"2024-09-20T19:23:56.677516+0000","last_fullsized":"2024-09-20T19:23:56.677516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:03.945432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.945432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.503088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"245'91","reported_seq":769,"reported_epoch":247,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:23:56.676416+0000","last_change":"2024-09-20T19:23:56.082423+0000","last_active":"2024-09-20T19:23:56.676416+0000","last_peered":"2024-09-20T19:23:56.676416+0000","last_clean":"2024-09-20T19:23:56.676416+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:23:56.676416+0000","last_undegraded":"2024-09-20T19:23:56.676416+0000","last_fullsized":"2024-09-20T19:23:56.676416+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'64","last_scrub_stamp":"2024-09-20T19:23:13.624271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:13.624271+0000","objects_scrubbed":1,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.070621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99716778299999997,"stat_sum":{"num_bytes":11294271,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":90,"num_read_kb":67051,"num_write":41,"num_write_kb":16360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"5"},{"start":"3f","length":"2"}]},{"pgid":"4.e","version":"77'11","reported_seq":406,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.284219+0000","last_change":"2024-09-20T19:22:58.180428+0000","last_active":"2024-09-20T19:23:56.284219+0000","last_peered":"2024-09-20T19:23:56.284219+0000","last_clean":"2024-09-20T19:23:56.284219+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:23:56.284219+0000","last_undegraded":"2024-09-20T19:23:56.284219+0000","last_fullsized":"2024-09-20T19:23:56.284219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:58.180287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.180287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.237767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":266,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.270708+0000","last_change":"2024-09-20T19:23:52.726605+0000","last_active":"2024-09-20T19:23:56.270708+0000","last_peered":"2024-09-20T19:23:56.270708+0000","last_clean":"2024-09-20T19:23:56.270708+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:23:56.270708+0000","last_undegraded":"2024-09-20T19:23:56.270708+0000","last_fullsized":"2024-09-20T19:23:56.270708+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.726537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.726537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:07.275685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":373,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677936+0000","last_change":"2024-09-20T19:20:58.711594+0000","last_active":"2024-09-20T19:23:56.677936+0000","last_peered":"2024-09-20T19:23:56.677936+0000","last_clean":"2024-09-20T19:23:56.677936+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:23:56.677936+0000","last_undegraded":"2024-09-20T19:23:56.677936+0000","last_fullsized":"2024-09-20T19:23:56.677936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.711472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.711472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.810411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":362,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670356+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:23:56.670356+0000","last_peered":"2024-09-20T19:23:56.670356+0000","last_clean":"2024-09-20T19:23:56.670356+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:23:56.670356+0000","last_undegraded":"2024-09-20T19:23:56.670356+0000","last_fullsized":"2024-09-20T19:23:56.670356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.744413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"244'31","reported_seq":689,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670682+0000","last_change":"2024-09-20T19:23:55.074879+0000","last_active":"2024-09-20T19:23:56.670682+0000","last_peered":"2024-09-20T19:23:56.670682+0000","last_clean":"2024-09-20T19:23:56.670682+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:23:56.670682+0000","last_undegraded":"2024-09-20T19:23:56.670682+0000","last_fullsized":"2024-09-20T19:23:56.670682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001231474,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":14840,"num_write":10,"num_write_kb":1940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.1","version":"77'14","reported_seq":400,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282034+0000","last_change":"2024-09-20T19:23:43.288937+0000","last_active":"2024-09-20T19:23:56.282034+0000","last_peered":"2024-09-20T19:23:56.282034+0000","last_clean":"2024-09-20T19:23:56.282034+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:23:56.282034+0000","last_undegraded":"2024-09-20T19:23:56.282034+0000","last_fullsized":"2024-09-20T19:23:56.282034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:23:43.288805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.288805+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:54.121381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":396,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677252+0000","last_change":"2024-09-20T19:23:47.645931+0000","last_active":"2024-09-20T19:23:56.677252+0000","last_peered":"2024-09-20T19:23:56.677252+0000","last_clean":"2024-09-20T19:23:56.677252+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:23:56.677252+0000","last_undegraded":"2024-09-20T19:23:56.677252+0000","last_fullsized":"2024-09-20T19:23:56.677252+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.645876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.645876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:53.852581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":454,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670125+0000","last_change":"2024-09-20T19:21:07.002029+0000","last_active":"2024-09-20T19:23:56.670125+0000","last_peered":"2024-09-20T19:23:56.670125+0000","last_clean":"2024-09-20T19:23:56.670125+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:23:56.670125+0000","last_undegraded":"2024-09-20T19:23:56.670125+0000","last_fullsized":"2024-09-20T19:23:56.670125+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:07.001951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.001951+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.724883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":369,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676990+0000","last_change":"2024-09-20T19:23:46.300434+0000","last_active":"2024-09-20T19:23:56.676990+0000","last_peered":"2024-09-20T19:23:56.676990+0000","last_clean":"2024-09-20T19:23:56.676990+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:23:56.676990+0000","last_undegraded":"2024-09-20T19:23:56.676990+0000","last_fullsized":"2024-09-20T19:23:56.676990+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:46.300364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.300364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.891484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"246'148","reported_seq":805,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671487+0000","last_change":"2024-09-20T19:23:55.082155+0000","last_active":"2024-09-20T19:23:56.671487+0000","last_peered":"2024-09-20T19:23:56.671487+0000","last_clean":"2024-09-20T19:23:56.671487+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:23:56.671487+0000","last_undegraded":"2024-09-20T19:23:56.671487+0000","last_fullsized":"2024-09-20T19:23:56.671487+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047264780000000001,"stat_sum":{"num_bytes":8085208,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":66536,"num_write":50,"num_write_kb":17546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.0","version":"77'18","reported_seq":418,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670309+0000","last_change":"2024-09-20T19:22:00.803489+0000","last_active":"2024-09-20T19:23:56.670309+0000","last_peered":"2024-09-20T19:23:56.670309+0000","last_clean":"2024-09-20T19:23:56.670309+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:23:56.670309+0000","last_undegraded":"2024-09-20T19:23:56.670309+0000","last_fullsized":"2024-09-20T19:23:56.670309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:22:00.803432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:00.803432+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":392,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672605+0000","last_change":"2024-09-20T19:23:51.507352+0000","last_active":"2024-09-20T19:23:56.672605+0000","last_peered":"2024-09-20T19:23:56.672605+0000","last_clean":"2024-09-20T19:23:56.672605+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:23:56.672605+0000","last_undegraded":"2024-09-20T19:23:56.672605+0000","last_fullsized":"2024-09-20T19:23:56.672605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:51.507216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.507216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:03.128560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.281773+0000","last_change":"2024-09-20T19:22:03.015704+0000","last_active":"2024-09-20T19:23:56.281773+0000","last_peered":"2024-09-20T19:23:56.281773+0000","last_clean":"2024-09-20T19:23:56.281773+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:23:56.281773+0000","last_undegraded":"2024-09-20T19:23:56.281773+0000","last_fullsized":"2024-09-20T19:23:56.281773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.015585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.015585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.008629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282714+0000","last_change":"2024-09-20T19:23:48.185887+0000","last_active":"2024-09-20T19:23:56.282714+0000","last_peered":"2024-09-20T19:23:56.282714+0000","last_clean":"2024-09-20T19:23:56.282714+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:23:56.282714+0000","last_undegraded":"2024-09-20T19:23:56.282714+0000","last_fullsized":"2024-09-20T19:23:56.282714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:01.271009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"245'185","reported_seq":870,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675324+0000","last_change":"2024-09-20T19:23:56.068791+0000","last_active":"2024-09-20T19:23:56.675324+0000","last_peered":"2024-09-20T19:23:56.675324+0000","last_clean":"2024-09-20T19:23:56.675324+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:23:56.675324+0000","last_undegraded":"2024-09-20T19:23:56.675324+0000","last_fullsized":"2024-09-20T19:23:56.675324+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'118","last_scrub_stamp":"2024-09-20T19:23:04.906510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.906510+0000","objects_scrubbed":10,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.956759+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.0042428299999999,"stat_sum":{"num_bytes":15748126,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":120,"num_read_kb":97224,"num_write":82,"num_write_kb":22181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.3","version":"77'19","reported_seq":415,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272419+0000","last_change":"2024-09-20T19:21:07.855976+0000","last_active":"2024-09-20T19:23:56.272419+0000","last_peered":"2024-09-20T19:23:56.272419+0000","last_clean":"2024-09-20T19:23:56.272419+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:23:56.272419+0000","last_undegraded":"2024-09-20T19:23:56.272419+0000","last_fullsized":"2024-09-20T19:23:56.272419+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:21:07.855828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.855828+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.733445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271636+0000","last_change":"2024-09-20T19:23:16.461133+0000","last_active":"2024-09-20T19:23:56.271636+0000","last_peered":"2024-09-20T19:23:56.271636+0000","last_clean":"2024-09-20T19:23:56.271636+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:23:56.271636+0000","last_undegraded":"2024-09-20T19:23:56.271636+0000","last_fullsized":"2024-09-20T19:23:56.271636+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:16.461005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:16.461005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.435064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":380,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675662+0000","last_change":"2024-09-20T19:22:21.758505+0000","last_active":"2024-09-20T19:23:56.675662+0000","last_peered":"2024-09-20T19:23:56.675662+0000","last_clean":"2024-09-20T19:23:56.675662+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:23:56.675662+0000","last_undegraded":"2024-09-20T19:23:56.675662+0000","last_fullsized":"2024-09-20T19:23:56.675662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.758384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.758384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":363,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670961+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:23:56.670961+0000","last_peered":"2024-09-20T19:23:56.670961+0000","last_clean":"2024-09-20T19:23:56.670961+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:23:56.670961+0000","last_undegraded":"2024-09-20T19:23:56.670961+0000","last_fullsized":"2024-09-20T19:23:56.670961+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.495223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"246'196","reported_seq":858,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282392+0000","last_change":"2024-09-20T19:23:55.091139+0000","last_active":"2024-09-20T19:23:56.282392+0000","last_peered":"2024-09-20T19:23:56.282392+0000","last_clean":"2024-09-20T19:23:56.282392+0000","last_became_active":"2024-09-20T19:21:30.244233+0000","last_became_peered":"2024-09-20T19:21:30.244233+0000","last_unstale":"2024-09-20T19:23:56.282392+0000","last_undegraded":"2024-09-20T19:23:56.282392+0000","last_fullsized":"2024-09-20T19:23:56.282392+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'93","last_scrub_stamp":"2024-09-20T19:23:14.039336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.039336+0000","objects_scrubbed":6,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.119362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.013842482999999999,"stat_sum":{"num_bytes":10920649,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":65,"num_read_kb":29242,"num_write":67,"num_write_kb":21384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.2","version":"77'10","reported_seq":410,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671804+0000","last_change":"2024-09-20T19:23:54.537172+0000","last_active":"2024-09-20T19:23:56.671804+0000","last_peered":"2024-09-20T19:23:56.671804+0000","last_clean":"2024-09-20T19:23:56.671804+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:23:56.671804+0000","last_undegraded":"2024-09-20T19:23:56.671804+0000","last_fullsized":"2024-09-20T19:23:56.671804+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:54.536975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.536975+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:18.498094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":391,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672238+0000","last_change":"2024-09-20T19:22:21.888862+0000","last_active":"2024-09-20T19:23:56.672238+0000","last_peered":"2024-09-20T19:23:56.672238+0000","last_clean":"2024-09-20T19:23:56.672238+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:23:56.672238+0000","last_undegraded":"2024-09-20T19:23:56.672238+0000","last_fullsized":"2024-09-20T19:23:56.672238+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:36.511666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":455,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272221+0000","last_change":"2024-09-20T19:21:10.853855+0000","last_active":"2024-09-20T19:23:56.272221+0000","last_peered":"2024-09-20T19:23:56.272221+0000","last_clean":"2024-09-20T19:23:56.272221+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:23:56.272221+0000","last_undegraded":"2024-09-20T19:23:56.272221+0000","last_fullsized":"2024-09-20T19:23:56.272221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:10.853722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.853722+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.732391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272282+0000","last_change":"2024-09-20T19:22:56.797105+0000","last_active":"2024-09-20T19:23:56.272282+0000","last_peered":"2024-09-20T19:23:56.272282+0000","last_clean":"2024-09-20T19:23:56.272282+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:23:56.272282+0000","last_undegraded":"2024-09-20T19:23:56.272282+0000","last_fullsized":"2024-09-20T19:23:56.272282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.054135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"246'129","reported_seq":810,"reported_epoch":247,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:56.676528+0000","last_change":"2024-09-20T19:23:56.080524+0000","last_active":"2024-09-20T19:23:56.676528+0000","last_peered":"2024-09-20T19:23:56.676528+0000","last_clean":"2024-09-20T19:23:56.676528+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:23:56.676528+0000","last_undegraded":"2024-09-20T19:23:56.676528+0000","last_fullsized":"2024-09-20T19:23:56.676528+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'119","last_scrub_stamp":"2024-09-20T19:23:46.636892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.636892+0000","objects_scrubbed":4,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:09.053548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011961160000000001,"stat_sum":{"num_bytes":17015054,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":95,"num_read_kb":61783,"num_write":62,"num_write_kb":27056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"5"},{"start":"40","length":"1"}]},{"pgid":"4.4","version":"77'30","reported_seq":507,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672162+0000","last_change":"2024-09-20T19:22:14.846750+0000","last_active":"2024-09-20T19:23:56.672162+0000","last_peered":"2024-09-20T19:23:56.672162+0000","last_clean":"2024-09-20T19:23:56.672162+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:23:56.672162+0000","last_undegraded":"2024-09-20T19:23:56.672162+0000","last_fullsized":"2024-09-20T19:23:56.672162+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:22:14.846671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.846671+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.459830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676765+0000","last_change":"2024-09-20T19:23:01.995350+0000","last_active":"2024-09-20T19:23:56.676765+0000","last_peered":"2024-09-20T19:23:56.676765+0000","last_clean":"2024-09-20T19:23:56.676765+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:23:56.676765+0000","last_undegraded":"2024-09-20T19:23:56.676765+0000","last_fullsized":"2024-09-20T19:23:56.676765+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:01.995251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.995251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.855295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":391,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.273146+0000","last_change":"2024-09-20T19:23:54.537232+0000","last_active":"2024-09-20T19:23:56.273146+0000","last_peered":"2024-09-20T19:23:56.273146+0000","last_clean":"2024-09-20T19:23:56.273146+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:23:56.273146+0000","last_undegraded":"2024-09-20T19:23:56.273146+0000","last_fullsized":"2024-09-20T19:23:56.273146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:54.537110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.537110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:23.441813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671216+0000","last_change":"2024-09-20T19:23:08.714718+0000","last_active":"2024-09-20T19:23:56.671216+0000","last_peered":"2024-09-20T19:23:56.671216+0000","last_clean":"2024-09-20T19:23:56.671216+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:23:56.671216+0000","last_undegraded":"2024-09-20T19:23:56.671216+0000","last_fullsized":"2024-09-20T19:23:56.671216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.714593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.714593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.378300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"246'280","reported_seq":947,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276667+0000","last_change":"2024-09-20T19:23:55.105059+0000","last_active":"2024-09-20T19:23:56.276667+0000","last_peered":"2024-09-20T19:23:56.276667+0000","last_clean":"2024-09-20T19:23:56.276667+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:23:56.276667+0000","last_undegraded":"2024-09-20T19:23:56.276667+0000","last_fullsized":"2024-09-20T19:23:56.276667+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'86","last_scrub_stamp":"2024-09-20T19:22:51.812052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.812052+0000","objects_scrubbed":9,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:13.286312+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.034244404999999999,"stat_sum":{"num_bytes":28227346,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":101,"num_read_kb":49534,"num_write":107,"num_write_kb":47093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.7","version":"77'13","reported_seq":323,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.273027+0000","last_change":"2024-09-20T19:21:22.193770+0000","last_active":"2024-09-20T19:23:56.273027+0000","last_peered":"2024-09-20T19:23:56.273027+0000","last_clean":"2024-09-20T19:23:56.273027+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:23:56.273027+0000","last_undegraded":"2024-09-20T19:23:56.273027+0000","last_fullsized":"2024-09-20T19:23:56.273027+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:21:22.193573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:22.193573+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":724,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670477+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:23:56.670477+0000","last_peered":"2024-09-20T19:23:56.670477+0000","last_clean":"2024-09-20T19:23:56.670477+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:23:56.670477+0000","last_undegraded":"2024-09-20T19:23:56.670477+0000","last_fullsized":"2024-09-20T19:23:56.670477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.476879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":333,"num_read_kb":328,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271002+0000","last_change":"2024-09-20T19:21:10.216852+0000","last_active":"2024-09-20T19:23:56.271002+0000","last_peered":"2024-09-20T19:23:56.271002+0000","last_clean":"2024-09-20T19:23:56.271002+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:23:56.271002+0000","last_undegraded":"2024-09-20T19:23:56.271002+0000","last_fullsized":"2024-09-20T19:23:56.271002+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:10.216700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.216700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.674411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272641+0000","last_change":"2024-09-20T19:23:47.565593+0000","last_active":"2024-09-20T19:23:56.272641+0000","last_peered":"2024-09-20T19:23:56.272641+0000","last_clean":"2024-09-20T19:23:56.272641+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:23:56.272641+0000","last_undegraded":"2024-09-20T19:23:56.272641+0000","last_fullsized":"2024-09-20T19:23:56.272641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.565537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.565537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:04.991103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":377,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677792+0000","last_change":"2024-09-20T19:23:11.304539+0000","last_active":"2024-09-20T19:23:56.677792+0000","last_peered":"2024-09-20T19:23:56.677792+0000","last_clean":"2024-09-20T19:23:56.677792+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:23:56.677792+0000","last_undegraded":"2024-09-20T19:23:56.677792+0000","last_fullsized":"2024-09-20T19:23:56.677792+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.304478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.304478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.382250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"246'274","reported_seq":950,"reported_epoch":247,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:56.677535+0000","last_change":"2024-09-20T19:23:56.176937+0000","last_active":"2024-09-20T19:23:56.677535+0000","last_peered":"2024-09-20T19:23:56.677535+0000","last_clean":"2024-09-20T19:23:56.677535+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:23:56.677535+0000","last_undegraded":"2024-09-20T19:23:56.677535+0000","last_fullsized":"2024-09-20T19:23:56.677535+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.098133926,"stat_sum":{"num_bytes":22469935,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":181,"num_read_kb":112187,"num_write":122,"num_write_kb":50913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"4"}]},{"pgid":"4.6","version":"77'12","reported_seq":405,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276490+0000","last_change":"2024-09-20T19:22:16.548672+0000","last_active":"2024-09-20T19:23:56.276490+0000","last_peered":"2024-09-20T19:23:56.276490+0000","last_clean":"2024-09-20T19:23:56.276490+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:23:56.276490+0000","last_undegraded":"2024-09-20T19:23:56.276490+0000","last_fullsized":"2024-09-20T19:23:56.276490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:16.548570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:16.548570+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.801626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.274658+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:23:56.274658+0000","last_peered":"2024-09-20T19:23:56.274658+0000","last_clean":"2024-09-20T19:23:56.274658+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:23:56.274658+0000","last_undegraded":"2024-09-20T19:23:56.274658+0000","last_fullsized":"2024-09-20T19:23:56.274658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.744928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675725+0000","last_change":"2024-09-20T19:22:18.870386+0000","last_active":"2024-09-20T19:23:56.675725+0000","last_peered":"2024-09-20T19:23:56.675725+0000","last_clean":"2024-09-20T19:23:56.675725+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:23:56.675725+0000","last_undegraded":"2024-09-20T19:23:56.675725+0000","last_fullsized":"2024-09-20T19:23:56.675725+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.870268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.870268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677950+0000","last_change":"2024-09-20T19:22:56.906325+0000","last_active":"2024-09-20T19:23:56.677950+0000","last_peered":"2024-09-20T19:23:56.677950+0000","last_clean":"2024-09-20T19:23:56.677950+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:23:56.677950+0000","last_undegraded":"2024-09-20T19:23:56.677950+0000","last_fullsized":"2024-09-20T19:23:56.677950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.906255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.906255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.054482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671895+0000","last_change":"2024-09-20T19:22:06.926452+0000","last_active":"2024-09-20T19:23:56.671895+0000","last_peered":"2024-09-20T19:23:56.671895+0000","last_clean":"2024-09-20T19:23:56.671895+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:23:56.671895+0000","last_undegraded":"2024-09-20T19:23:56.671895+0000","last_fullsized":"2024-09-20T19:23:56.671895+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:06.926325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:06.926325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.522917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"246'93","reported_seq":768,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276241+0000","last_change":"2024-09-20T19:23:55.110288+0000","last_active":"2024-09-20T19:23:56.276241+0000","last_peered":"2024-09-20T19:23:56.276241+0000","last_clean":"2024-09-20T19:23:56.276241+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:23:56.276241+0000","last_undegraded":"2024-09-20T19:23:56.276241+0000","last_fullsized":"2024-09-20T19:23:56.276241+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'61","last_scrub_stamp":"2024-09-20T19:23:06.675677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:06.675677+0000","objects_scrubbed":4,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.004556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.044291548,"stat_sum":{"num_bytes":11458736,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":46081,"num_write":38,"num_write_kb":16797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.5","version":"77'16","reported_seq":468,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676575+0000","last_change":"2024-09-20T19:21:12.740077+0000","last_active":"2024-09-20T19:23:56.676575+0000","last_peered":"2024-09-20T19:23:56.676575+0000","last_clean":"2024-09-20T19:23:56.676575+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:23:56.676575+0000","last_undegraded":"2024-09-20T19:23:56.676575+0000","last_fullsized":"2024-09-20T19:23:56.676575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:21:12.739915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:12.739915+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.628021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672025+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:23:56.672025+0000","last_peered":"2024-09-20T19:23:56.672025+0000","last_clean":"2024-09-20T19:23:56.672025+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:23:56.672025+0000","last_undegraded":"2024-09-20T19:23:56.672025+0000","last_fullsized":"2024-09-20T19:23:56.672025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.812423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676792+0000","last_change":"2024-09-20T19:20:52.723249+0000","last_active":"2024-09-20T19:23:56.676792+0000","last_peered":"2024-09-20T19:23:56.676792+0000","last_clean":"2024-09-20T19:23:56.676792+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:23:56.676792+0000","last_undegraded":"2024-09-20T19:23:56.676792+0000","last_fullsized":"2024-09-20T19:23:56.676792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.723121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.723121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.814578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":429,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675904+0000","last_change":"2024-09-20T19:23:51.334458+0000","last_active":"2024-09-20T19:23:56.675904+0000","last_peered":"2024-09-20T19:23:56.675904+0000","last_clean":"2024-09-20T19:23:56.675904+0000","last_became_active":"2024-09-20T19:17:37.192557+0000","last_became_peered":"2024-09-20T19:17:37.192557+0000","last_unstale":"2024-09-20T19:23:56.675904+0000","last_undegraded":"2024-09-20T19:23:56.675904+0000","last_fullsized":"2024-09-20T19:23:56.675904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:56.170021+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":373,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677205+0000","last_change":"2024-09-20T19:20:53.868159+0000","last_active":"2024-09-20T19:23:56.677205+0000","last_peered":"2024-09-20T19:23:56.677205+0000","last_clean":"2024-09-20T19:23:56.677205+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:23:56.677205+0000","last_undegraded":"2024-09-20T19:23:56.677205+0000","last_fullsized":"2024-09-20T19:23:56.677205+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.868034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.868034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.287685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":377,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.678061+0000","last_change":"2024-09-20T19:23:49.765599+0000","last_active":"2024-09-20T19:23:56.678061+0000","last_peered":"2024-09-20T19:23:56.678061+0000","last_clean":"2024-09-20T19:23:56.678061+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:23:56.678061+0000","last_undegraded":"2024-09-20T19:23:56.678061+0000","last_fullsized":"2024-09-20T19:23:56.678061+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.765546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.765546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.098365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"246'130","reported_seq":804,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.273601+0000","last_change":"2024-09-20T19:23:55.088281+0000","last_active":"2024-09-20T19:23:56.273601+0000","last_peered":"2024-09-20T19:23:56.273601+0000","last_clean":"2024-09-20T19:23:56.273601+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:23:56.273601+0000","last_undegraded":"2024-09-20T19:23:56.273601+0000","last_fullsized":"2024-09-20T19:23:56.273601+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'71","last_scrub_stamp":"2024-09-20T19:23:05.723582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.723582+0000","objects_scrubbed":6,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:10.763038+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.028451517999999999,"stat_sum":{"num_bytes":11238958,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":51499,"num_write":47,"num_write_kb":21914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.a","version":"77'19","reported_seq":407,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675915+0000","last_change":"2024-09-20T19:20:52.718858+0000","last_active":"2024-09-20T19:23:56.675915+0000","last_peered":"2024-09-20T19:23:56.675915+0000","last_clean":"2024-09-20T19:23:56.675915+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:23:56.675915+0000","last_undegraded":"2024-09-20T19:23:56.675915+0000","last_fullsized":"2024-09-20T19:23:56.675915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:20:52.718711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.718711+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.617546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":390,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272074+0000","last_change":"2024-09-20T19:22:11.289926+0000","last_active":"2024-09-20T19:23:56.272074+0000","last_peered":"2024-09-20T19:23:56.272074+0000","last_clean":"2024-09-20T19:23:56.272074+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:23:56.272074+0000","last_undegraded":"2024-09-20T19:23:56.272074+0000","last_fullsized":"2024-09-20T19:23:56.272074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.289877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.289877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.274181+0000","last_change":"2024-09-20T19:23:49.605277+0000","last_active":"2024-09-20T19:23:56.274181+0000","last_peered":"2024-09-20T19:23:56.274181+0000","last_clean":"2024-09-20T19:23:56.274181+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:23:56.274181+0000","last_undegraded":"2024-09-20T19:23:56.274181+0000","last_fullsized":"2024-09-20T19:23:56.274181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.605219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.605219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.883340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":369,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675810+0000","last_change":"2024-09-20T19:23:42.246466+0000","last_active":"2024-09-20T19:23:56.675810+0000","last_peered":"2024-09-20T19:23:56.675810+0000","last_clean":"2024-09-20T19:23:56.675810+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:23:56.675810+0000","last_undegraded":"2024-09-20T19:23:56.675810+0000","last_fullsized":"2024-09-20T19:23:56.675810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.246406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.246406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.009612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"246'307","reported_seq":968,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.270961+0000","last_change":"2024-09-20T19:23:55.083334+0000","last_active":"2024-09-20T19:23:56.270961+0000","last_peered":"2024-09-20T19:23:56.270961+0000","last_clean":"2024-09-20T19:23:56.270961+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:23:56.270961+0000","last_undegraded":"2024-09-20T19:23:56.270961+0000","last_fullsized":"2024-09-20T19:23:56.270961+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'112","last_scrub_stamp":"2024-09-20T19:22:52.771031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771031+0000","objects_scrubbed":11,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:09.592155+0000","scrub_duration":7,"objects_trimmed":2,"snaptrim_duration":0.021640610000000001,"stat_sum":{"num_bytes":13126267,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":115,"num_read_kb":84645,"num_write":104,"num_write_kb":45358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.b","version":"77'9","reported_seq":403,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272324+0000","last_change":"2024-09-20T19:22:55.847371+0000","last_active":"2024-09-20T19:23:56.272324+0000","last_peered":"2024-09-20T19:23:56.272324+0000","last_clean":"2024-09-20T19:23:56.272324+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:23:56.272324+0000","last_undegraded":"2024-09-20T19:23:56.272324+0000","last_fullsized":"2024-09-20T19:23:56.272324+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:55.847304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:55.847304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:04.129113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672636+0000","last_change":"2024-09-20T19:21:05.740388+0000","last_active":"2024-09-20T19:23:56.672636+0000","last_peered":"2024-09-20T19:23:56.672636+0000","last_clean":"2024-09-20T19:23:56.672636+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:23:56.672636+0000","last_undegraded":"2024-09-20T19:23:56.672636+0000","last_fullsized":"2024-09-20T19:23:56.672636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.740278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.740278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.483483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":374,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271387+0000","last_change":"2024-09-20T19:21:17.226835+0000","last_active":"2024-09-20T19:23:56.271387+0000","last_peered":"2024-09-20T19:23:56.271387+0000","last_clean":"2024-09-20T19:23:56.271387+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:23:56.271387+0000","last_undegraded":"2024-09-20T19:23:56.271387+0000","last_fullsized":"2024-09-20T19:23:56.271387+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:17.226714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:17.226714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.520900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":388,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.274511+0000","last_change":"2024-09-20T19:23:44.721606+0000","last_active":"2024-09-20T19:23:56.274511+0000","last_peered":"2024-09-20T19:23:56.274511+0000","last_clean":"2024-09-20T19:23:56.274511+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:23:56.274511+0000","last_undegraded":"2024-09-20T19:23:56.274511+0000","last_fullsized":"2024-09-20T19:23:56.274511+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:23:44.721438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.721438+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:59.778487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"233'57","reported_seq":728,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671605+0000","last_change":"2024-09-20T19:23:55.085038+0000","last_active":"2024-09-20T19:23:56.671605+0000","last_peered":"2024-09-20T19:23:56.671605+0000","last_clean":"2024-09-20T19:23:56.671605+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:23:56.671605+0000","last_undegraded":"2024-09-20T19:23:56.671605+0000","last_fullsized":"2024-09-20T19:23:56.671605+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'35","last_scrub_stamp":"2024-09-20T19:23:10.333319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.333319+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.249690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015538469999999999,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":21280,"num_write":19,"num_write_kb":11601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"6"},{"start":"3f","length":"2"}]},{"pgid":"4.8","version":"77'15","reported_seq":410,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676087+0000","last_change":"2024-09-20T19:22:22.063998+0000","last_active":"2024-09-20T19:23:56.676087+0000","last_peered":"2024-09-20T19:23:56.676087+0000","last_clean":"2024-09-20T19:23:56.676087+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:23:56.676087+0000","last_undegraded":"2024-09-20T19:23:56.676087+0000","last_fullsized":"2024-09-20T19:23:56.676087+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:22.063871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.063871+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":398,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272842+0000","last_change":"2024-09-20T19:22:02.235649+0000","last_active":"2024-09-20T19:23:56.272842+0000","last_peered":"2024-09-20T19:23:56.272842+0000","last_clean":"2024-09-20T19:23:56.272842+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:23:56.272842+0000","last_undegraded":"2024-09-20T19:23:56.272842+0000","last_fullsized":"2024-09-20T19:23:56.272842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:02.235598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:02.235598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.519095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":461,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676770+0000","last_change":"2024-09-20T19:23:01.188032+0000","last_active":"2024-09-20T19:23:56.676770+0000","last_peered":"2024-09-20T19:23:56.676770+0000","last_clean":"2024-09-20T19:23:56.676770+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:23:56.676770+0000","last_undegraded":"2024-09-20T19:23:56.676770+0000","last_fullsized":"2024-09-20T19:23:56.676770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:01.187968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.187968+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:01.789452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":369,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676383+0000","last_change":"2024-09-20T19:21:01.739612+0000","last_active":"2024-09-20T19:23:56.676383+0000","last_peered":"2024-09-20T19:23:56.676383+0000","last_clean":"2024-09-20T19:23:56.676383+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:23:56.676383+0000","last_undegraded":"2024-09-20T19:23:56.676383+0000","last_fullsized":"2024-09-20T19:23:56.676383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:01.739492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.739492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.006870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"246'131","reported_seq":788,"reported_epoch":247,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:23:56.670549+0000","last_change":"2024-09-20T19:23:56.067725+0000","last_active":"2024-09-20T19:23:56.670549+0000","last_peered":"2024-09-20T19:23:56.670549+0000","last_clean":"2024-09-20T19:23:56.670549+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:23:56.670549+0000","last_undegraded":"2024-09-20T19:23:56.670549+0000","last_fullsized":"2024-09-20T19:23:56.670549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.215168+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99433126000000005,"stat_sum":{"num_bytes":11252415,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":66,"num_read_kb":47887,"num_write":48,"num_write_kb":21145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"5"},{"start":"3f","length":"2"}]},{"pgid":"4.9","version":"77'12","reported_seq":429,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282000+0000","last_change":"2024-09-20T19:22:57.202081+0000","last_active":"2024-09-20T19:23:56.282000+0000","last_peered":"2024-09-20T19:23:56.282000+0000","last_clean":"2024-09-20T19:23:56.282000+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:23:56.282000+0000","last_undegraded":"2024-09-20T19:23:56.282000+0000","last_fullsized":"2024-09-20T19:23:56.282000+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:57.201945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:57.201945+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.217023+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":407,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.284589+0000","last_change":"2024-09-20T19:22:59.196366+0000","last_active":"2024-09-20T19:23:56.284589+0000","last_peered":"2024-09-20T19:23:56.284589+0000","last_clean":"2024-09-20T19:23:56.284589+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:23:56.284589+0000","last_undegraded":"2024-09-20T19:23:56.284589+0000","last_fullsized":"2024-09-20T19:23:56.284589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:22:59.196235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:59.196235+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:12.646992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271351+0000","last_change":"2024-09-20T19:22:18.260176+0000","last_active":"2024-09-20T19:23:56.271351+0000","last_peered":"2024-09-20T19:23:56.271351+0000","last_clean":"2024-09-20T19:23:56.271351+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:23:56.271351+0000","last_undegraded":"2024-09-20T19:23:56.271351+0000","last_fullsized":"2024-09-20T19:23:56.271351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.260046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.260046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.501604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670428+0000","last_change":"2024-09-20T19:23:00.687733+0000","last_active":"2024-09-20T19:23:56.670428+0000","last_peered":"2024-09-20T19:23:56.670428+0000","last_clean":"2024-09-20T19:23:56.670428+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:23:56.670428+0000","last_undegraded":"2024-09-20T19:23:56.670428+0000","last_fullsized":"2024-09-20T19:23:56.670428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:00.687658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:00.687658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.540716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":403,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276411+0000","last_change":"2024-09-20T19:22:58.800944+0000","last_active":"2024-09-20T19:23:56.276411+0000","last_peered":"2024-09-20T19:23:56.276411+0000","last_clean":"2024-09-20T19:23:56.276411+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:23:56.276411+0000","last_undegraded":"2024-09-20T19:23:56.276411+0000","last_fullsized":"2024-09-20T19:23:56.276411+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:58.800874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.800874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:05.321600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":398,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271838+0000","last_change":"2024-09-20T19:23:07.609407+0000","last_active":"2024-09-20T19:23:56.271838+0000","last_peered":"2024-09-20T19:23:56.271838+0000","last_clean":"2024-09-20T19:23:56.271838+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:23:56.271838+0000","last_undegraded":"2024-09-20T19:23:56.271838+0000","last_fullsized":"2024-09-20T19:23:56.271838+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:07.609283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:07.609283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:17.108303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":374,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671401+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:23:56.671401+0000","last_peered":"2024-09-20T19:23:56.671401+0000","last_clean":"2024-09-20T19:23:56.671401+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:23:56.671401+0000","last_undegraded":"2024-09-20T19:23:56.671401+0000","last_fullsized":"2024-09-20T19:23:56.671401+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:35.796722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.271516+0000","last_change":"2024-09-20T19:22:56.840530+0000","last_active":"2024-09-20T19:23:56.271516+0000","last_peered":"2024-09-20T19:23:56.271516+0000","last_clean":"2024-09-20T19:23:56.271516+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:23:56.271516+0000","last_undegraded":"2024-09-20T19:23:56.271516+0000","last_fullsized":"2024-09-20T19:23:56.271516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.840403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.840403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.237174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":388,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676360+0000","last_change":"2024-09-20T19:23:02.640412+0000","last_active":"2024-09-20T19:23:56.676360+0000","last_peered":"2024-09-20T19:23:56.676360+0000","last_clean":"2024-09-20T19:23:56.676360+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:23:56.676360+0000","last_undegraded":"2024-09-20T19:23:56.676360+0000","last_fullsized":"2024-09-20T19:23:56.676360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:02.640270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.640270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.542638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":387,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671534+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:23:56.671534+0000","last_peered":"2024-09-20T19:23:56.671534+0000","last_clean":"2024-09-20T19:23:56.671534+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:23:56.671534+0000","last_undegraded":"2024-09-20T19:23:56.671534+0000","last_fullsized":"2024-09-20T19:23:56.671534+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:27.710208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677869+0000","last_change":"2024-09-20T19:23:50.759135+0000","last_active":"2024-09-20T19:23:56.677869+0000","last_peered":"2024-09-20T19:23:56.677869+0000","last_clean":"2024-09-20T19:23:56.677869+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:23:56.677869+0000","last_undegraded":"2024-09-20T19:23:56.677869+0000","last_fullsized":"2024-09-20T19:23:56.677869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.808215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":369,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677702+0000","last_change":"2024-09-20T19:21:29.100795+0000","last_active":"2024-09-20T19:23:56.677702+0000","last_peered":"2024-09-20T19:23:56.677702+0000","last_clean":"2024-09-20T19:23:56.677702+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:23:56.677702+0000","last_undegraded":"2024-09-20T19:23:56.677702+0000","last_fullsized":"2024-09-20T19:23:56.677702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.100718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.100718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:23.284037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":401,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671050+0000","last_change":"2024-09-20T19:23:50.024939+0000","last_active":"2024-09-20T19:23:56.671050+0000","last_peered":"2024-09-20T19:23:56.671050+0000","last_clean":"2024-09-20T19:23:56.671050+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:23:56.671050+0000","last_undegraded":"2024-09-20T19:23:56.671050+0000","last_fullsized":"2024-09-20T19:23:56.671050+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:50.024860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.024860+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:14.939068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":397,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.858288+0000","last_change":"2024-09-20T19:23:56.858288+0000","last_active":"2024-09-20T19:23:56.858288+0000","last_peered":"2024-09-20T19:23:56.858288+0000","last_clean":"2024-09-20T19:23:56.858288+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:23:56.858288+0000","last_undegraded":"2024-09-20T19:23:56.858288+0000","last_fullsized":"2024-09-20T19:23:56.858288+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:56.858227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.858227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:16.206316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":469,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677724+0000","last_change":"2024-09-20T19:23:53.824722+0000","last_active":"2024-09-20T19:23:56.677724+0000","last_peered":"2024-09-20T19:23:56.677724+0000","last_clean":"2024-09-20T19:23:56.677724+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:23:56.677724+0000","last_undegraded":"2024-09-20T19:23:56.677724+0000","last_fullsized":"2024-09-20T19:23:56.677724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:53.824647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:53.824647+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.400274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":387,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276000+0000","last_change":"2024-09-20T19:23:15.737592+0000","last_active":"2024-09-20T19:23:56.276000+0000","last_peered":"2024-09-20T19:23:56.276000+0000","last_clean":"2024-09-20T19:23:56.276000+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:23:56.276000+0000","last_undegraded":"2024-09-20T19:23:56.276000+0000","last_fullsized":"2024-09-20T19:23:56.276000+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:15.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:15.737537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.525979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":401,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677834+0000","last_change":"2024-09-20T19:22:54.923862+0000","last_active":"2024-09-20T19:23:56.677834+0000","last_peered":"2024-09-20T19:23:56.677834+0000","last_clean":"2024-09-20T19:23:56.677834+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:23:56.677834+0000","last_undegraded":"2024-09-20T19:23:56.677834+0000","last_fullsized":"2024-09-20T19:23:56.677834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:54.923725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.923725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:02.018467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":391,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272570+0000","last_change":"2024-09-20T19:22:20.671257+0000","last_active":"2024-09-20T19:23:56.272570+0000","last_peered":"2024-09-20T19:23:56.272570+0000","last_clean":"2024-09-20T19:23:56.272570+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:23:56.272570+0000","last_undegraded":"2024-09-20T19:23:56.272570+0000","last_fullsized":"2024-09-20T19:23:56.272570+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.671196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.671196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":454,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670633+0000","last_change":"2024-09-20T19:22:53.660006+0000","last_active":"2024-09-20T19:23:56.670633+0000","last_peered":"2024-09-20T19:23:56.670633+0000","last_clean":"2024-09-20T19:23:56.670633+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:23:56.670633+0000","last_undegraded":"2024-09-20T19:23:56.670633+0000","last_fullsized":"2024-09-20T19:23:56.670633+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:53.659878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.659878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:06.807822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.284149+0000","last_change":"2024-09-20T19:22:11.914350+0000","last_active":"2024-09-20T19:23:56.284149+0000","last_peered":"2024-09-20T19:23:56.284149+0000","last_clean":"2024-09-20T19:23:56.284149+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:23:56.284149+0000","last_undegraded":"2024-09-20T19:23:56.284149+0000","last_fullsized":"2024-09-20T19:23:56.284149+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.914224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.914224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.201274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":395,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671787+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:23:56.671787+0000","last_peered":"2024-09-20T19:23:56.671787+0000","last_clean":"2024-09-20T19:23:56.671787+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:23:56.671787+0000","last_undegraded":"2024-09-20T19:23:56.671787+0000","last_fullsized":"2024-09-20T19:23:56.671787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.610209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":380,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675884+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:23:56.675884+0000","last_peered":"2024-09-20T19:23:56.675884+0000","last_clean":"2024-09-20T19:23:56.675884+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:23:56.675884+0000","last_undegraded":"2024-09-20T19:23:56.675884+0000","last_fullsized":"2024-09-20T19:23:56.675884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.684147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":374,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671505+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:23:56.671505+0000","last_peered":"2024-09-20T19:23:56.671505+0000","last_clean":"2024-09-20T19:23:56.671505+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:23:56.671505+0000","last_undegraded":"2024-09-20T19:23:56.671505+0000","last_fullsized":"2024-09-20T19:23:56.671505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.835899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.272018+0000","last_change":"2024-09-20T19:23:43.752868+0000","last_active":"2024-09-20T19:23:56.272018+0000","last_peered":"2024-09-20T19:23:56.272018+0000","last_clean":"2024-09-20T19:23:56.272018+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:23:56.272018+0000","last_undegraded":"2024-09-20T19:23:56.272018+0000","last_fullsized":"2024-09-20T19:23:56.272018+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:43.752736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.752736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.607806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":398,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.282780+0000","last_change":"2024-09-20T19:20:52.914253+0000","last_active":"2024-09-20T19:23:56.282780+0000","last_peered":"2024-09-20T19:23:56.282780+0000","last_clean":"2024-09-20T19:23:56.282780+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:23:56.282780+0000","last_undegraded":"2024-09-20T19:23:56.282780+0000","last_fullsized":"2024-09-20T19:23:56.282780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:52.914121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.914121+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.086622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":399,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672576+0000","last_change":"2024-09-20T19:22:53.163608+0000","last_active":"2024-09-20T19:23:56.672576+0000","last_peered":"2024-09-20T19:23:56.672576+0000","last_clean":"2024-09-20T19:23:56.672576+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:23:56.672576+0000","last_undegraded":"2024-09-20T19:23:56.672576+0000","last_fullsized":"2024-09-20T19:23:56.672576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:53.163555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.163555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:09.815764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671672+0000","last_change":"2024-09-20T19:23:44.572528+0000","last_active":"2024-09-20T19:23:56.671672+0000","last_peered":"2024-09-20T19:23:56.671672+0000","last_clean":"2024-09-20T19:23:56.671672+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:23:56.671672+0000","last_undegraded":"2024-09-20T19:23:56.671672+0000","last_fullsized":"2024-09-20T19:23:56.671672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:44.572407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.572407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.284846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670330+0000","last_change":"2024-09-20T19:23:47.003723+0000","last_active":"2024-09-20T19:23:56.670330+0000","last_peered":"2024-09-20T19:23:56.670330+0000","last_clean":"2024-09-20T19:23:56.670330+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:23:56.670330+0000","last_undegraded":"2024-09-20T19:23:56.670330+0000","last_fullsized":"2024-09-20T19:23:56.670330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.003602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.003602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.662168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":392,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670929+0000","last_change":"2024-09-20T19:22:50.638425+0000","last_active":"2024-09-20T19:23:56.670929+0000","last_peered":"2024-09-20T19:23:56.670929+0000","last_clean":"2024-09-20T19:23:56.670929+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:23:56.670929+0000","last_undegraded":"2024-09-20T19:23:56.670929+0000","last_fullsized":"2024-09-20T19:23:56.670929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:22:50.638257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:50.638257+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.137027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":389,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677979+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:23:56.677979+0000","last_peered":"2024-09-20T19:23:56.677979+0000","last_clean":"2024-09-20T19:23:56.677979+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:23:56.677979+0000","last_undegraded":"2024-09-20T19:23:56.677979+0000","last_fullsized":"2024-09-20T19:23:56.677979+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.686224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":380,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676181+0000","last_change":"2024-09-20T19:23:49.643862+0000","last_active":"2024-09-20T19:23:56.676181+0000","last_peered":"2024-09-20T19:23:56.676181+0000","last_clean":"2024-09-20T19:23:56.676181+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:23:56.676181+0000","last_undegraded":"2024-09-20T19:23:56.676181+0000","last_fullsized":"2024-09-20T19:23:56.676181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.643746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.643746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.482157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":361,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677005+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:23:56.677005+0000","last_peered":"2024-09-20T19:23:56.677005+0000","last_clean":"2024-09-20T19:23:56.677005+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:23:56.677005+0000","last_undegraded":"2024-09-20T19:23:56.677005+0000","last_fullsized":"2024-09-20T19:23:56.677005+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.264712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":405,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670793+0000","last_change":"2024-09-20T19:22:08.939934+0000","last_active":"2024-09-20T19:23:56.670793+0000","last_peered":"2024-09-20T19:23:56.670793+0000","last_clean":"2024-09-20T19:23:56.670793+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:23:56.670793+0000","last_undegraded":"2024-09-20T19:23:56.670793+0000","last_fullsized":"2024-09-20T19:23:56.670793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.939842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.939842+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":388,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677166+0000","last_change":"2024-09-20T19:22:51.781731+0000","last_active":"2024-09-20T19:23:56.677166+0000","last_peered":"2024-09-20T19:23:56.677166+0000","last_clean":"2024-09-20T19:23:56.677166+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:23:56.677166+0000","last_undegraded":"2024-09-20T19:23:56.677166+0000","last_fullsized":"2024-09-20T19:23:56.677166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.781680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.781680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:00.226292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.675308+0000","last_change":"2024-09-20T19:23:14.325927+0000","last_active":"2024-09-20T19:23:56.675308+0000","last_peered":"2024-09-20T19:23:56.675308+0000","last_clean":"2024-09-20T19:23:56.675308+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:23:56.675308+0000","last_undegraded":"2024-09-20T19:23:56.675308+0000","last_fullsized":"2024-09-20T19:23:56.675308+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:14.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:20.929643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670299+0000","last_change":"2024-09-20T19:22:11.898382+0000","last_active":"2024-09-20T19:23:56.670299+0000","last_peered":"2024-09-20T19:23:56.670299+0000","last_clean":"2024-09-20T19:23:56.670299+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:23:56.670299+0000","last_undegraded":"2024-09-20T19:23:56.670299+0000","last_fullsized":"2024-09-20T19:23:56.670299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.898255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.898255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":371,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.677093+0000","last_change":"2024-09-20T19:23:47.281254+0000","last_active":"2024-09-20T19:23:56.677093+0000","last_peered":"2024-09-20T19:23:56.677093+0000","last_clean":"2024-09-20T19:23:56.677093+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:23:56.677093+0000","last_undegraded":"2024-09-20T19:23:56.677093+0000","last_fullsized":"2024-09-20T19:23:56.677093+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:23:47.281195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.281195+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:08.244407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":397,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676115+0000","last_change":"2024-09-20T19:23:11.005669+0000","last_active":"2024-09-20T19:23:56.676115+0000","last_peered":"2024-09-20T19:23:56.676115+0000","last_clean":"2024-09-20T19:23:56.676115+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:23:56.676115+0000","last_undegraded":"2024-09-20T19:23:56.676115+0000","last_fullsized":"2024-09-20T19:23:56.676115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.005587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.005587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.106114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":402,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276350+0000","last_change":"2024-09-20T19:23:10.711312+0000","last_active":"2024-09-20T19:23:56.276350+0000","last_peered":"2024-09-20T19:23:56.276350+0000","last_clean":"2024-09-20T19:23:56.276350+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:23:56.276350+0000","last_undegraded":"2024-09-20T19:23:56.276350+0000","last_fullsized":"2024-09-20T19:23:56.276350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:10.711138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.711138+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:14.536408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":471,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676708+0000","last_change":"2024-09-20T19:23:51.659148+0000","last_active":"2024-09-20T19:23:56.676708+0000","last_peered":"2024-09-20T19:23:56.676708+0000","last_clean":"2024-09-20T19:23:56.676708+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:23:56.676708+0000","last_undegraded":"2024-09-20T19:23:56.676708+0000","last_fullsized":"2024-09-20T19:23:56.676708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:51.659044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.659044+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:12.110176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":379,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672144+0000","last_change":"2024-09-20T19:23:05.361065+0000","last_active":"2024-09-20T19:23:56.672144+0000","last_peered":"2024-09-20T19:23:56.672144+0000","last_clean":"2024-09-20T19:23:56.672144+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:23:56.672144+0000","last_undegraded":"2024-09-20T19:23:56.672144+0000","last_fullsized":"2024-09-20T19:23:56.672144+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:05.360941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.360941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.569089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":391,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671133+0000","last_change":"2024-09-20T19:23:03.692672+0000","last_active":"2024-09-20T19:23:56.671133+0000","last_peered":"2024-09-20T19:23:56.671133+0000","last_clean":"2024-09-20T19:23:56.671133+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:23:56.671133+0000","last_undegraded":"2024-09-20T19:23:56.671133+0000","last_fullsized":"2024-09-20T19:23:56.671133+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:03.692537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.692537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:10.678096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":422,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676808+0000","last_change":"2024-09-20T19:22:22.768858+0000","last_active":"2024-09-20T19:23:56.676808+0000","last_peered":"2024-09-20T19:23:56.676808+0000","last_clean":"2024-09-20T19:23:56.676808+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:23:56.676808+0000","last_undegraded":"2024-09-20T19:23:56.676808+0000","last_fullsized":"2024-09-20T19:23:56.676808+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:22.768697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.768697+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.276220+0000","last_change":"2024-09-20T19:23:49.605522+0000","last_active":"2024-09-20T19:23:56.276220+0000","last_peered":"2024-09-20T19:23:56.276220+0000","last_clean":"2024-09-20T19:23:56.276220+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:23:56.276220+0000","last_undegraded":"2024-09-20T19:23:56.276220+0000","last_fullsized":"2024-09-20T19:23:56.276220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.605402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.605402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:03.424112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":371,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.283959+0000","last_change":"2024-09-20T19:20:59.816653+0000","last_active":"2024-09-20T19:23:56.283959+0000","last_peered":"2024-09-20T19:23:56.283959+0000","last_clean":"2024-09-20T19:23:56.283959+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:23:56.283959+0000","last_undegraded":"2024-09-20T19:23:56.283959+0000","last_fullsized":"2024-09-20T19:23:56.283959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:59.816510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:59.816510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.141868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":396,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.676073+0000","last_change":"2024-09-20T19:23:52.648685+0000","last_active":"2024-09-20T19:23:56.676073+0000","last_peered":"2024-09-20T19:23:56.676073+0000","last_clean":"2024-09-20T19:23:56.676073+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:23:56.676073+0000","last_undegraded":"2024-09-20T19:23:56.676073+0000","last_fullsized":"2024-09-20T19:23:56.676073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.648578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.648578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:20.151508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":411,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.744113+0000","last_change":"2024-09-20T19:23:56.744113+0000","last_active":"2024-09-20T19:23:56.744113+0000","last_peered":"2024-09-20T19:23:56.744113+0000","last_clean":"2024-09-20T19:23:56.744113+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:23:56.744113+0000","last_undegraded":"2024-09-20T19:23:56.744113+0000","last_fullsized":"2024-09-20T19:23:56.744113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:56.744020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.744020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.330507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":383,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.672385+0000","last_change":"2024-09-20T19:22:51.090500+0000","last_active":"2024-09-20T19:23:56.672385+0000","last_peered":"2024-09-20T19:23:56.672385+0000","last_clean":"2024-09-20T19:23:56.672385+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:23:56.672385+0000","last_undegraded":"2024-09-20T19:23:56.672385+0000","last_fullsized":"2024-09-20T19:23:56.672385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.090439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.090439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":362,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671357+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:23:56.671357+0000","last_peered":"2024-09-20T19:23:56.671357+0000","last_clean":"2024-09-20T19:23:56.671357+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:23:56.671357+0000","last_undegraded":"2024-09-20T19:23:56.671357+0000","last_fullsized":"2024-09-20T19:23:56.671357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.890171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.671250+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:23:56.671250+0000","last_peered":"2024-09-20T19:23:56.671250+0000","last_clean":"2024-09-20T19:23:56.671250+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:23:56.671250+0000","last_undegraded":"2024-09-20T19:23:56.671250+0000","last_fullsized":"2024-09-20T19:23:56.671250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.730587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":401,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.670750+0000","last_change":"2024-09-20T19:21:24.033642+0000","last_active":"2024-09-20T19:23:56.670750+0000","last_peered":"2024-09-20T19:23:56.670750+0000","last_clean":"2024-09-20T19:23:56.670750+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:23:56.670750+0000","last_undegraded":"2024-09-20T19:23:56.670750+0000","last_fullsized":"2024-09-20T19:23:56.670750+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:21:24.033570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.033570+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.742580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":375,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-20T19:23:56.283220+0000","last_change":"2024-09-20T19:21:08.918562+0000","last_active":"2024-09-20T19:23:56.283220+0000","last_peered":"2024-09-20T19:23:56.283220+0000","last_clean":"2024-09-20T19:23:56.283220+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:23:56.283220+0000","last_undegraded":"2024-09-20T19:23:56.283220+0000","last_fullsized":"2024-09-20T19:23:56.283220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.918443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.918443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.121645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":190731481,"num_objects":76,"num_object_clones":30,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":6,"num_read":1270,"num_read_kb":903499,"num_write":951,"num_write_kb":368262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":457482240,"data_stored":513694638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2454,"ondisk_log_size":2454,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":333,"num_read_kb":328,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907023,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92316,"kb_used_data":61124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93652836,"statfs":{"total":95995035648,"available":95900504064,"internally_reserved":0,"allocated":62590976,"data_stored":68532295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234074,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77424,"kb_used_data":46232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93667728,"statfs":{"total":95995035648,"available":95915753472,"internally_reserved":0,"allocated":47341568,"data_stored":51211531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561124,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56004,"kb_used_data":24808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689148,"statfs":{"total":95995035648,"available":95937687552,"internally_reserved":0,"allocated":25403392,"data_stored":25674678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855472,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99548,"kb_used_data":60040,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93645604,"statfs":{"total":95995035648,"available":95893098496,"internally_reserved":0,"allocated":61480960,"data_stored":72130550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084410,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118720,"kb_used_data":83368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93626432,"statfs":{"total":95995035648,"available":95873466368,"internally_reserved":0,"allocated":85368832,"data_stored":94089885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378760,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85432,"kb_used_data":45920,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93659720,"statfs":{"total":95995035648,"available":95907553280,"internally_reserved":0,"allocated":47022080,"data_stored":51899300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91204,"kb_used_data":55856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93653948,"statfs":{"total":95995035648,"available":95901642752,"internally_reserved":0,"allocated":57196544,"data_stored":62606348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738528,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132524,"kb_used_data":93008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93612628,"statfs":{"total":95995035648,"available":95859331072,"internally_reserved":0,"allocated":95240192,"data_stored":104923156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":92262400,"data_stored":103004295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54816768,"data_stored":61284483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44642304,"data_stored":50570792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":79605760,"data_stored":87764295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59113472,"data_stored":70810179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":23031808,"data_stored":24346024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":44380160,"data_stored":49293600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":59629568,"data_stored":66620970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:23:58.261 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T19:23:58.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-20T19:23:58.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:58 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:58.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.741856+0000 osd.2 (osd.2) 97 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.744016+0000 osd.2 (osd.2) 98 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.856765+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:56.858224+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.280426+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-20T19:23:58.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: audit 2024-09-20T19:23:57.340915+0000 mgr.y (mgr.24491) 228 : audit [DBG] from='client.24953 v1:172.21.15.42:0/3208977517' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.509383+0000 osd.1 (osd.1) 127 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:57.510538+0000 osd.1 (osd.1) 128 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.040973+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.044061+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:58 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.054093+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-20T19:23:58.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:23:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:23:58.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:58.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:58.669 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:23:58.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:58.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:23:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:58.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:23:58.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:59.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:616: done (7 left) 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:617: read oid 27 snap -1 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:617: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:618: snap_create 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:610: finishing rollback tid 0 to smithi04268048-36 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:607: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:23:59.078 INFO:tasks.rados.rados.0.smithi042.stdout:610: finishing rollback tid 1 to smithi04268048-36 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:610: finishing rollback tid 2 to smithi04268048-36 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 294 (ObjNum 151 snap 42 seq_num 151) dirty exists 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:607: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:607: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 288 (ObjNum 212 snap 63 seq_num 212) dirty exists 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:608: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 156 (ObjNum 233 snap 67 seq_num 233) dirty exists 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: finishing write tid 1 to smithi04268048-25 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: finishing write tid 2 to smithi04268048-25 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: finishing write tid 3 to smithi04268048-25 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: finishing write tid 4 to smithi04268048-25 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: finishing write tid 5 to smithi04268048-25 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: oid 25 updating version 0 to 299 2024-09-20T19:23:59.079 INFO:tasks.rados.rados.0.smithi042.stdout:612: oid 25 version 299 is already newer than 298 2024-09-20T19:23:59.080 INFO:tasks.rados.rados.0.smithi042.stdout:612: oid 25 version 299 is already newer than 297 2024-09-20T19:23:59.080 INFO:tasks.rados.rados.0.smithi042.stdout:612: oid 25 version 299 is already newer than 294 2024-09-20T19:23:59.080 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 299 (ObjNum 236 snap 69 seq_num 236) dirty exists 2024-09-20T19:23:59.080 INFO:tasks.rados.rados.0.smithi042.stdout:612: left oid 25 (ObjNum 236 snap 69 seq_num 236) 2024-09-20T19:23:59.080 INFO:tasks.rados.rados.0.smithi042.stdout:614: expect (ObjNum 157 snap 45 seq_num 157) 2024-09-20T19:23:59.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:59.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:23:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:23:59.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:59.245 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:59.246 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:23:59.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:59.305 INFO:tasks.rados.rados.0.smithi042.stdout:615: expect (ObjNum 134 snap 35 seq_num 134) 2024-09-20T19:23:59.305 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:59.346 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1221219 2024-09-20T19:23:59.359 INFO:tasks.rados.rados.0.smithi042.stdout:617: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:23:59.359 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:59.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:59.378 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:23:59 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:23:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:23:59.409 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:59.458 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:23:59.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:59.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:59.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.005399+0000 mgr.y (mgr.24491) 229 : cluster [DBG] pgmap v311: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.794294+0000 osd.3 (osd.3) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.794703+0000 osd.3 (osd.3) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:23:59 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.075353+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-20T19:23:59.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:23:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:23:59.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:23:59.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:23:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:23:59.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:00.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:00.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:00.078 INFO:tasks.rados.rados.0.smithi042.stdout:607: done (8 left) 2024-09-20T19:24:00.078 INFO:tasks.rados.rados.0.smithi042.stdout:608: done (7 left) 2024-09-20T19:24:00.078 INFO:tasks.rados.rados.0.smithi042.stdout:610: done (6 left) 2024-09-20T19:24:00.078 INFO:tasks.rados.rados.0.smithi042.stdout:612: done (5 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:613: done (4 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:614: done (3 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:615: done (2 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:617: done (1 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:618: done (0 left) 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:619: write (excl) oid 13 current snap is 70 2024-09-20T19:24:00.079 INFO:tasks.rados.rados.0.smithi042.stdout:619: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-20T19:24:00.096 INFO:tasks.rados.rados.0.smithi042.stdout:619: writing smithi04268048-13 from 783259 to 1424215 tid 1 2024-09-20T19:24:00.110 INFO:tasks.rados.rados.0.smithi042.stdout:619: writing smithi04268048-13 from 1918703 to 2531773 tid 2 2024-09-20T19:24:00.119 INFO:tasks.rados.rados.0.smithi042.stdout:619: writing smithi04268048-13 from 3008661 to 3414128 tid 3 2024-09-20T19:24:00.126 INFO:tasks.rados.rados.0.smithi042.stdout:619: writing smithi04268048-13 from 3821939 to 3983259 tid 4 2024-09-20T19:24:00.127 INFO:tasks.rados.rados.0.smithi042.stdout:620: write oid 16 current snap is 70 2024-09-20T19:24:00.127 INFO:tasks.rados.rados.0.smithi042.stdout:620: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-20T19:24:00.137 INFO:tasks.rados.rados.0.smithi042.stdout:620: writing smithi04268048-16 from 400066 to 1116271 tid 1 2024-09-20T19:24:00.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:620: writing smithi04268048-16 from 1860832 to 2000066 tid 2 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:621: write (excl) oid 8 current snap is 70 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 1 to smithi04268048-13 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 2 to smithi04268048-13 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 3 to smithi04268048-13 2024-09-20T19:24:00.146 INFO:tasks.rados.rados.0.smithi042.stdout:621: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-20T19:24:00.154 INFO:tasks.rados.rados.0.smithi042.stdout:621: writing smithi04268048-8 from 416873 to 1208327 tid 1 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:621: writing smithi04268048-8 from 1919314 to 2016873 tid 2 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:622: write (excl) oid 1 current snap is 70 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 4 to smithi04268048-13 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 5 to smithi04268048-13 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: finishing write tid 6 to smithi04268048-13 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: oid 13 updating version 0 to 299 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: oid 13 version 299 is already newer than 297 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: oid 13 version 299 is already newer than 296 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: oid 13 version 299 is already newer than 295 2024-09-20T19:24:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:619: oid 13 version 299 is already newer than 298 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 299 (ObjNum 237 snap 70 seq_num 237) dirty exists 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:619: left oid 13 (ObjNum 237 snap 70 seq_num 237) 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:620: finishing write tid 1 to smithi04268048-16 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:620: finishing write tid 2 to smithi04268048-16 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:620: finishing write tid 3 to smithi04268048-16 2024-09-20T19:24:00.161 INFO:tasks.rados.rados.0.smithi042.stdout:622: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-20T19:24:00.165 INFO:tasks.rados.rados.0.smithi042.stdout:622: writing smithi04268048-1 from 433680 to 900383 tid 1 2024-09-20T19:24:00.168 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:00.172 INFO:tasks.rados.rados.0.smithi042.stdout:622: writing smithi04268048-1 from 1577796 to 2033680 tid 2 2024-09-20T19:24:00.172 INFO:tasks.rados.rados.0.smithi042.stdout:619: done (3 left) 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:623: copy_from oid 19 from oid 15 current snap is 70 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:620: finishing write tid 4 to smithi04268048-16 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:620: oid 16 updating version 0 to 201 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:620: oid 16 updating version 201 to 203 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:620: oid 16 version 203 is already newer than 202 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 203 (ObjNum 238 snap 70 seq_num 238) dirty exists 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:620: left oid 16 (ObjNum 238 snap 70 seq_num 238) 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: finishing write tid 1 to smithi04268048-8 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: finishing write tid 2 to smithi04268048-8 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: finishing write tid 3 to smithi04268048-8 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: finishing write tid 4 to smithi04268048-8 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: oid 8 updating version 0 to 175 2024-09-20T19:24:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:621: oid 8 version 175 is already newer than 173 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:621: oid 8 version 175 is already newer than 174 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 175 (ObjNum 239 snap 70 seq_num 239) dirty exists 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:621: left oid 8 (ObjNum 239 snap 70 seq_num 239) 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:622: finishing write tid 1 to smithi04268048-1 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:620: done (3 left) 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:621: done (2 left) 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:624: rollback oid 30 current snap is 70 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 64 2024-09-20T19:24:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:625: delete oid 45 current snap is 70 2024-09-20T19:24:00.176 INFO:tasks.rados.rados.0.smithi042.stdout:624: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:24:00.176 INFO:tasks.rados.rados.0.smithi042.stdout:625: done (3 left) 2024-09-20T19:24:00.177 INFO:tasks.rados.rados.0.smithi042.stdout:626: write oid 40 current snap is 70 2024-09-20T19:24:00.177 INFO:tasks.rados.rados.0.smithi042.stdout:626: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-20T19:24:00.183 INFO:tasks.rados.rados.0.smithi042.stdout:626: writing smithi04268048-40 from 467294 to 1084495 tid 1 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:626: writing smithi04268048-40 from 1578407 to 2067294 tid 2 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:627: read oid 36 snap -1 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:627: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:628: rollback oid 8 current snap is 70 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 69 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:624: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 308 (ObjNum 97 snap 0 seq_num 3066624432) dirty dne 2024-09-20T19:24:00.190 INFO:tasks.rados.rados.0.smithi042.stdout:624: done (5 left) 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:629: write oid 37 current snap is 70 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: finishing write tid 2 to smithi04268048-1 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: finishing write tid 3 to smithi04268048-1 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: finishing write tid 4 to smithi04268048-1 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: oid 1 updating version 0 to 179 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: oid 1 version 179 is already newer than 177 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: oid 1 version 179 is already newer than 178 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 179 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:622: left oid 1 (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:00.191 INFO:tasks.rados.rados.0.smithi042.stdout:629: seq_num 243 ranges {484101=692450,1636889=447212} 2024-09-20T19:24:00.196 INFO:tasks.rados.rados.0.smithi042.stdout:629: writing smithi04268048-37 from 484101 to 1176551 tid 1 2024-09-20T19:24:00.202 INFO:tasks.rados.rados.0.smithi042.stdout:629: writing smithi04268048-37 from 1636889 to 2084101 tid 2 2024-09-20T19:24:00.202 INFO:tasks.rados.rados.0.smithi042.stdout:622: done (5 left) 2024-09-20T19:24:00.202 INFO:tasks.rados.rados.0.smithi042.stdout:630: write oid 9 current snap is 70 2024-09-20T19:24:00.202 INFO:tasks.rados.rados.0.smithi042.stdout:623: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:24:00.202 INFO:tasks.rados.rados.0.smithi042.stdout:630: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-20T19:24:00.208 INFO:tasks.rados.rados.0.smithi042.stdout:630: writing smithi04268048-9 from 500908 to 984960 tid 1 2024-09-20T19:24:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:630: writing smithi04268048-9 from 1411724 to 2100908 tid 2 2024-09-20T19:24:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:631: delete oid 14 current snap is 70 2024-09-20T19:24:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:628: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:24:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:628: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:24:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:627: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:24:00.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:00.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:00.338 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:00.356 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:00.356 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:00.372 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:00.373 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:00.391 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:00.391 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:00.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:00.411 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:00.411 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:626: finishing write tid 1 to smithi04268048-40 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:628: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 182 (ObjNum 68 snap 10 seq_num 68) dirty exists 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:626: finishing write tid 2 to smithi04268048-40 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:626: finishing write tid 3 to smithi04268048-40 2024-09-20T19:24:00.417 INFO:tasks.rados.rados.0.smithi042.stdout:626: finishing write tid 4 to smithi04268048-40 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:626: oid 40 updating version 0 to 189 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:626: oid 40 updating version 189 to 191 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:626: oid 40 version 191 is already newer than 190 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 191 (ObjNum 242 snap 70 seq_num 242) dirty exists 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:626: left oid 40 (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:623: finishing copy_from to smithi04268048-19 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 186 (ObjNum 210 snap 63 seq_num 210) dirty exists 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:623: done (6 left) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:626: done (5 left) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:627: done (4 left) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:628: done (3 left) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:631: done (2 left) 2024-09-20T19:24:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:632: read oid 43 snap -1 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:632: expect (ObjNum 233 snap 67 seq_num 233) 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: finishing write tid 1 to smithi04268048-37 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: finishing write tid 2 to smithi04268048-37 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: finishing write tid 3 to smithi04268048-37 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: finishing write tid 4 to smithi04268048-37 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: oid 37 updating version 0 to 158 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: oid 37 version 158 is already newer than 156 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: oid 37 version 158 is already newer than 157 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 158 (ObjNum 243 snap 70 seq_num 243) dirty exists 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:629: left oid 37 (ObjNum 243 snap 70 seq_num 243) 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:630: finishing write tid 1 to smithi04268048-9 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:630: finishing write tid 2 to smithi04268048-9 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:630: finishing write tid 3 to smithi04268048-9 2024-09-20T19:24:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:630: finishing write tid 4 to smithi04268048-9 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:630: oid 9 updating version 0 to 282 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:630: oid 9 version 282 is already newer than 280 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:630: oid 9 version 282 is already newer than 281 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 282 (ObjNum 244 snap 70 seq_num 244) dirty exists 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:630: left oid 9 (ObjNum 244 snap 70 seq_num 244) 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:633: read oid 1 snap -1 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:633: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:629: done (3 left) 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:630: done (2 left) 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:634: write (excl) oid 11 current snap is 70 2024-09-20T19:24:00.420 INFO:tasks.rados.rados.0.smithi042.stdout:634: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-20T19:24:00.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:00.427 INFO:tasks.rados.rados.0.smithi042.stdout:634: writing smithi04268048-11 from 534522 to 1169072 tid 1 2024-09-20T19:24:00.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.434 INFO:tasks.rados.rados.0.smithi042.stdout:634: writing smithi04268048-11 from 1812335 to 2134522 tid 2 2024-09-20T19:24:00.434 INFO:tasks.rados.rados.0.smithi042.stdout:635: copy_from oid 31 from oid 36 current snap is 70 2024-09-20T19:24:00.434 INFO:tasks.rados.rados.0.smithi042.stdout:632: expect (ObjNum 233 snap 67 seq_num 233) 2024-09-20T19:24:00.435 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:00.440 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:00.440 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:00.460 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:00.479 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:00.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:00.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:00.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.844643+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:24:00.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:58.846855+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: audit 2024-09-20T19:23:59.128414+0000 mgr.y (mgr.24491) 230 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.647003+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:23:59.649119+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:00.077446+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-20T19:24:00.612 INFO:tasks.rados.rados.0.smithi042.stdout:633: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:00.642 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:00.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:00.672 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:00.703 INFO:tasks.rados.rados.0.smithi042.stdout:634: finishing write tid 1 to smithi04268048-11 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: finishing write tid 2 to smithi04268048-11 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: finishing write tid 3 to smithi04268048-11 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: finishing write tid 4 to smithi04268048-11 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: oid 11 updating version 0 to 285 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: oid 11 updating version 285 to 286 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: oid 11 version 286 is already newer than 284 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 286 (ObjNum 246 snap 70 seq_num 246) dirty exists 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:634: left oid 11 (ObjNum 246 snap 70 seq_num 246) 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:635: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:635: finishing copy_from to smithi04268048-31 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 294 (ObjNum 151 snap 42 seq_num 151) dirty exists 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:632: done (3 left) 2024-09-20T19:24:00.704 INFO:tasks.rados.rados.0.smithi042.stdout:633: done (2 left) 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:634: done (1 left) 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:635: done (0 left) 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:636: read oid 5 snap -1 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:636: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:637: rollback oid 34 current snap is 70 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 68 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:638: snap_create 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:637: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:24:00.705 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 0 (ObjNum 81 snap 0 seq_num 1415352016) dirty dne 2024-09-20T19:24:00.707 INFO:tasks.rados.rados.0.smithi042.stdout:636: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:00.750 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:00.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:00.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:00.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:00.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:01.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:01.155 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:01.180 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:01.283 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:01.312 INFO:tasks.rados.rados.0.smithi042.stdout:636: done (2 left) 2024-09-20T19:24:01.312 INFO:tasks.rados.rados.0.smithi042.stdout:637: done (1 left) 2024-09-20T19:24:01.312 INFO:tasks.rados.rados.0.smithi042.stdout:638: done (0 left) 2024-09-20T19:24:01.312 INFO:tasks.rados.rados.0.smithi042.stdout:639: snap_remove snap 66 2024-09-20T19:24:01.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:01.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:01.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:01 smithi162 bash[24557]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:01 smithi162 bash[24557]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:01 smithi162 bash[24557]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:01 smithi162 bash[24557]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:01.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[32075]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[32075]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[32075]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[32075]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[22706]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[22706]: cluster 2024-09-20T19:24:00.006231+0000 mgr.y (mgr.24491) 231 : cluster [DBG] pgmap v314: 148 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 143 active+clean; 182 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 33 op/s 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[22706]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:01 smithi042 bash[22706]: audit 2024-09-20T19:24:00.320568+0000 mon.a (mon.0) 1218 : audit [DBG] from='client.? v1:172.21.15.42:0/1804584993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:01.713 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:01.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:01.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:01.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:01.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:01.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:02.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:02.103 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:02.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:02.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:639: done (0 left) 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:640: rollback oid 28 current snap is 71 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 68 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:641: read oid 15 snap -1 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:641: expect (ObjNum 210 snap 63 seq_num 210) 2024-09-20T19:24:02.319 INFO:tasks.rados.rados.0.smithi042.stdout:642: snap_remove snap 70 2024-09-20T19:24:02.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:02.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:02 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:02.468 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.312560+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.335214+0000 osd.7 (osd.7) 93 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.346339+0000 osd.7 (osd.7) 94 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.119988+0000 osd.3 (osd.3) 125 : cluster [DBG] 7.a scrub starts 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:02 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.123470+0000 osd.3 (osd.3) 126 : cluster [DBG] 7.a scrub ok 2024-09-20T19:24:02.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:02.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:02.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:02.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:02.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:02.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:02.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:02.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:03.058 INFO:tasks.rados.rados.0.smithi042.stdout:642: done (2 left) 2024-09-20T19:24:03.059 INFO:tasks.rados.rados.0.smithi042.stdout:643: write oid 27 current snap is 71 2024-09-20T19:24:03.059 INFO:tasks.rados.rados.0.smithi042.stdout:643: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-20T19:24:03.079 INFO:tasks.rados.rados.0.smithi042.stdout:643: writing smithi04268048-27 from 551329 to 1261128 tid 1 2024-09-20T19:24:03.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:03.095 INFO:tasks.rados.rados.0.smithi042.stdout:643: writing smithi04268048-27 from 1870817 to 2151329 tid 2 2024-09-20T19:24:03.095 INFO:tasks.rados.rados.0.smithi042.stdout:644: read oid 31 snap 63 2024-09-20T19:24:03.096 INFO:tasks.rados.rados.0.smithi042.stdout:644: expect (ObjNum 203 snap 61 seq_num 203) 2024-09-20T19:24:03.096 INFO:tasks.rados.rados.0.smithi042.stdout:645: snap_remove snap 64 2024-09-20T19:24:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:03.222 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:03.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:03.480 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[32075]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:03 smithi042 bash[22706]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.507 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:03.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:03.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.757732+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.d scrub starts 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:01.760080+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.d scrub ok 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.007795+0000 mgr.y (mgr.24491) 232 : cluster [DBG] pgmap v317: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.317553+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.058010+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.080838+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:24:03.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.700 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.085881+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:24:03.700 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.700 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:03 smithi162 bash[24557]: audit 2024-09-20T19:24:03.188947+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:03.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:24:03.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:03.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:03.864 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.874 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:03.874 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:03.889 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:03.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:03.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:04.050 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:645: done (4 left) 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:646: copy_from oid 10 from oid 4 current snap is 71 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:647: read oid 29 snap 63 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:647: expect (ObjNum 214 snap 63 seq_num 214) 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:648: copy_from oid 43 from oid 1 current snap is 71 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:649: copy_from oid 16 from oid 25 current snap is 71 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:650: write oid 23 current snap is 71 2024-09-20T19:24:04.061 INFO:tasks.rados.rados.0.smithi042.stdout:650: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-20T19:24:04.064 INFO:tasks.rados.rados.0.smithi042.stdout:650: writing smithi04268048-23 from 568136 to 1353184 tid 1 2024-09-20T19:24:04.069 INFO:tasks.rados.rados.0.smithi042.stdout:650: writing smithi04268048-23 from 1929299 to 2168136 tid 2 2024-09-20T19:24:04.069 INFO:tasks.rados.rados.0.smithi042.stdout:651: snap_remove snap 69 2024-09-20T19:24:04.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:04.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:04.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:04.266 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:04.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:04 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:04.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.785739+0000 osd.2 (osd.2) 101 : cluster [DBG] 6.f scrub starts 2024-09-20T19:24:04.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:02.787963+0000 osd.2 (osd.2) 102 : cluster [DBG] 6.f scrub ok 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.686407+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.690134+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.049258+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: audit 2024-09-20T19:24:04.050848+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? v1:172.21.15.42:0/1366050519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:04 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.054770+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-20T19:24:04.523 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:04.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:04.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:04.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T19:24:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:04.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:04.783 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:04.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:04.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:05.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:05.064 INFO:tasks.rados.rados.0.smithi042.stdout:651: done (9 left) 2024-09-20T19:24:05.064 INFO:tasks.rados.rados.0.smithi042.stdout:652: read oid 36 snap -1 2024-09-20T19:24:05.064 INFO:tasks.rados.rados.0.smithi042.stdout:652: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:24:05.065 INFO:tasks.rados.rados.0.smithi042.stdout:653: write oid 48 current snap is 71 2024-09-20T19:24:05.065 INFO:tasks.rados.rados.0.smithi042.stdout:641: expect (ObjNum 210 snap 63 seq_num 210) 2024-09-20T19:24:05.065 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:05.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:05.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:640: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 123 (ObjNum 129 snap 0 seq_num 1415352752) dirty dne 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: finishing write tid 1 to smithi04268048-27 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: finishing write tid 2 to smithi04268048-27 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: finishing write tid 3 to smithi04268048-27 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: finishing write tid 4 to smithi04268048-27 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: oid 27 updating version 0 to 292 2024-09-20T19:24:05.353 INFO:tasks.rados.rados.0.smithi042.stdout:643: oid 27 updating version 292 to 294 2024-09-20T19:24:05.354 INFO:tasks.rados.rados.0.smithi042.stdout:643: oid 27 version 294 is already newer than 293 2024-09-20T19:24:05.354 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 294 (ObjNum 247 snap 71 seq_num 247) dirty exists 2024-09-20T19:24:05.354 INFO:tasks.rados.rados.0.smithi042.stdout:643: left oid 27 (ObjNum 247 snap 71 seq_num 247) 2024-09-20T19:24:05.354 INFO:tasks.rados.rados.0.smithi042.stdout:644: expect (ObjNum 203 snap 61 seq_num 203) 2024-09-20T19:24:05.354 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.404 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:05.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:05 smithi162 bash[24557]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:648: finishing copy_from racing read to smithi04268048-43 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: finishing write tid 1 to smithi04268048-23 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: finishing write tid 2 to smithi04268048-23 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: finishing write tid 3 to smithi04268048-23 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: finishing write tid 4 to smithi04268048-23 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: oid 23 updating version 0 to 171 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: oid 23 version 171 is already newer than 169 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: oid 23 version 171 is already newer than 170 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 171 (ObjNum 248 snap 71 seq_num 248) dirty exists 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:650: left oid 23 (ObjNum 248 snap 71 seq_num 248) 2024-09-20T19:24:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:648: finishing copy_from to smithi04268048-43 2024-09-20T19:24:05.562 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 179 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:24:05.562 INFO:tasks.rados.rados.0.smithi042.stdout:649: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:24:05.562 INFO:tasks.rados.rados.0.smithi042.stdout:647: expect (ObjNum 214 snap 63 seq_num 214) 2024-09-20T19:24:05.562 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:05.613 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:05.666 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[32075]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.783166+0000 osd.2 (osd.2) 103 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:03.786557+0000 osd.2 (osd.2) 104 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:04.008234+0000 mgr.y (mgr.24491) 233 : cluster [DBG] pgmap v320: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.9 MiB/s wr, 34 op/s 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:05 smithi042 bash[22706]: cluster 2024-09-20T19:24:05.063864+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-20T19:24:05.686 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:05.720 INFO:tasks.rados.rados.0.smithi042.stdout:646: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:24:05.720 INFO:tasks.rados.rados.0.smithi042.stdout:646: finishing copy_from to smithi04268048-10 2024-09-20T19:24:05.720 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 162 (ObjNum 155 snap 44 seq_num 155) dirty exists 2024-09-20T19:24:05.720 INFO:tasks.rados.rados.0.smithi042.stdout:649: finishing copy_from to smithi04268048-16 2024-09-20T19:24:05.720 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 299 (ObjNum 236 snap 69 seq_num 236) dirty exists 2024-09-20T19:24:05.721 INFO:tasks.rados.rados.0.smithi042.stdout:652: expect (ObjNum 151 snap 42 seq_num 151) 2024-09-20T19:24:05.787 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 689592 2024-09-20T19:24:05.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:05.843 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:05.909 INFO:tasks.rados.rados.0.smithi042.stdout:653: seq_num 249 ranges {584943=460297,1587781=597162} 2024-09-20T19:24:05.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.926 INFO:tasks.rados.rados.0.smithi042.stdout:653: writing smithi04268048-48 from 584943 to 1045240 tid 1 2024-09-20T19:24:05.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:05.945 INFO:tasks.rados.rados.0.smithi042.stdout:653: writing smithi04268048-48 from 1587781 to 2184943 tid 2 2024-09-20T19:24:05.945 INFO:tasks.rados.rados.0.smithi042.stdout:640: done (10 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:641: done (9 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:643: done (8 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:644: done (7 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:646: done (6 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:647: done (5 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:648: done (4 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:649: done (3 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:650: done (2 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:652: done (1 left) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:654: read oid 41 snap -1 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:654: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:655: read oid 4 snap -1 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:655: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:05.946 INFO:tasks.rados.rados.0.smithi042.stdout:656: snap_remove snap 59 2024-09-20T19:24:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:05.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:06.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:06.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:06.179 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:06.205 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:06.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:06.336 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:06.357 INFO:teuthology.orchestra.run.smithi042.stderr:set pool 1 pg_num to 11 2024-09-20T19:24:06.358 INFO:tasks.rados.rados.0.smithi042.stdout:656: done (3 left) 2024-09-20T19:24:06.359 INFO:tasks.rados.rados.0.smithi042.stdout:657: write (excl) oid 25 current snap is 71 2024-09-20T19:24:06.359 INFO:tasks.rados.rados.0.smithi042.stdout:653: finishing write tid 1 to smithi04268048-48 2024-09-20T19:24:06.359 INFO:tasks.rados.rados.0.smithi042.stdout:657: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-20T19:24:06.365 INFO:tasks.rados.rados.0.smithi042.stdout:657: writing smithi04268048-25 from 601750 to 1137296 tid 1 2024-09-20T19:24:06.372 INFO:tasks.rados.rados.0.smithi042.stdout:657: writing smithi04268048-25 from 1929910 to 2201750 tid 2 2024-09-20T19:24:06.372 INFO:tasks.rados.rados.0.smithi042.stdout:658: write (excl) oid 16 current snap is 71 2024-09-20T19:24:06.372 INFO:tasks.rados.rados.0.smithi042.stdout:658: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-20T19:24:06.380 INFO:tasks.rados.rados.0.smithi042.stdout:658: writing smithi04268048-16 from 618557 to 1345705 tid 1 2024-09-20T19:24:06.386 INFO:tasks.rados.rados.0.smithi042.stdout:658: writing smithi04268048-16 from 2104745 to 2218557 tid 2 2024-09-20T19:24:06.386 INFO:tasks.rados.rados.0.smithi042.stdout:659: rollback oid 31 current snap is 71 2024-09-20T19:24:06.386 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 60 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: finishing write tid 2 to smithi04268048-48 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: finishing write tid 3 to smithi04268048-48 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: finishing write tid 4 to smithi04268048-48 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: oid 48 updating version 0 to 329 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: oid 48 updating version 329 to 330 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: oid 48 updating version 330 to 331 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 331 (ObjNum 249 snap 71 seq_num 249) dirty exists 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:653: left oid 48 (ObjNum 249 snap 71 seq_num 249) 2024-09-20T19:24:06.387 INFO:tasks.rados.rados.0.smithi042.stdout:655: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:06.395 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1089028 2024-09-20T19:24:06.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:06.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[22706]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[22706]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[22706]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[22706]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[32075]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[32075]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[32075]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:06 smithi042 bash[32075]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:06.472 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:06.521 INFO:tasks.rados.rados.0.smithi042.stdout:654: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:06.546 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:06.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:06.586 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: finishing write tid 1 to smithi04268048-25 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: finishing write tid 2 to smithi04268048-25 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: finishing write tid 3 to smithi04268048-25 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: finishing write tid 4 to smithi04268048-25 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: oid 25 updating version 0 to 338 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: oid 25 version 338 is already newer than 337 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: oid 25 version 338 is already newer than 332 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 338 (ObjNum 250 snap 71 seq_num 250) dirty exists 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:657: left oid 25 (ObjNum 250 snap 71 seq_num 250) 2024-09-20T19:24:06.612 INFO:tasks.rados.rados.0.smithi042.stdout:658: finishing write tid 1 to smithi04268048-16 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:658: finishing write tid 2 to smithi04268048-16 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:659: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:658: finishing write tid 3 to smithi04268048-16 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:653: done (5 left) 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:654: done (4 left) 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:655: done (3 left) 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:657: done (2 left) 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:660: read oid 26 snap 63 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:660: expect (ObjNum 108 snap 25 seq_num 108) 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:661: rollback oid 15 current snap is 71 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 63 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:659: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:658: finishing write tid 4 to smithi04268048-16 2024-09-20T19:24:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:658: oid 16 updating version 0 to 300 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:658: oid 16 updating version 300 to 302 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:658: oid 16 version 302 is already newer than 301 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 302 (ObjNum 251 snap 71 seq_num 251) dirty exists 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:658: left oid 16 (ObjNum 251 snap 71 seq_num 251) 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:659: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 297 (ObjNum 188 snap 56 seq_num 188) dirty exists 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:658: done (3 left) 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:659: done (2 left) 2024-09-20T19:24:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:662: snap_create 2024-09-20T19:24:06.615 INFO:tasks.rados.rados.0.smithi042.stdout:660: expect (ObjNum 108 snap 25 seq_num 108) 2024-09-20T19:24:06.615 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:06.689 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:06.744 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:06.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:06 smithi162 bash[24557]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:06 smithi162 bash[24557]: audit 2024-09-20T19:24:05.828703+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.42:0/2195820630' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:06 smithi162 bash[24557]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:06 smithi162 bash[24557]: audit 2024-09-20T19:24:05.829278+0000 mon.a (mon.0) 1227 : audit [INF] from='client.24971 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-20T19:24:06.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:06.814 INFO:tasks.rados.rados.0.smithi042.stdout:661: finishing rollback tid 0 to smithi04268048-15 2024-09-20T19:24:06.814 INFO:tasks.rados.rados.0.smithi042.stdout:661: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:24:06.814 INFO:tasks.rados.rados.0.smithi042.stdout:661: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:24:06.814 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 305 (ObjNum 210 snap 63 seq_num 210) dirty exists 2024-09-20T19:24:06.837 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:06.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:06.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:07.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:07.197 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:07.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.351 INFO:tasks.rados.rados.0.smithi042.stdout:660: done (2 left) 2024-09-20T19:24:07.351 INFO:tasks.rados.rados.0.smithi042.stdout:661: done (1 left) 2024-09-20T19:24:07.351 INFO:tasks.rados.rados.0.smithi042.stdout:662: done (0 left) 2024-09-20T19:24:07.351 INFO:tasks.rados.rados.0.smithi042.stdout:663: write (excl) oid 2 current snap is 72 2024-09-20T19:24:07.351 INFO:tasks.rados.rados.0.smithi042.stdout:663: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-20T19:24:07.359 INFO:tasks.rados.rados.0.smithi042.stdout:663: writing smithi04268048-2 from 635364 to 1037761 tid 1 2024-09-20T19:24:07.360 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:07.365 INFO:tasks.rados.rados.0.smithi042.stdout:663: writing smithi04268048-2 from 1763227 to 2235364 tid 2 2024-09-20T19:24:07.366 INFO:tasks.rados.rados.0.smithi042.stdout:664: snap_create 2024-09-20T19:24:07.372 INFO:tasks.rados.rados.0.smithi042.stdout:663: finishing write tid 1 to smithi04268048-2 2024-09-20T19:24:07.372 INFO:tasks.rados.rados.0.smithi042.stdout:663: finishing write tid 2 to smithi04268048-2 2024-09-20T19:24:07.372 INFO:tasks.rados.rados.0.smithi042.stdout:663: finishing write tid 3 to smithi04268048-2 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:663: finishing write tid 4 to smithi04268048-2 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:663: oid 2 updating version 0 to 153 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:663: oid 2 version 153 is already newer than 152 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:663: oid 2 updating version 153 to 154 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 154 (ObjNum 252 snap 72 seq_num 252) dirty exists 2024-09-20T19:24:07.373 INFO:tasks.rados.rados.0.smithi042.stdout:663: left oid 2 (ObjNum 252 snap 72 seq_num 252) 2024-09-20T19:24:07.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[22706]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:07 smithi042 bash[32075]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:24:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:07.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:07.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: cluster 2024-09-20T19:24:06.008891+0000 mgr.y (mgr.24491) 234 : cluster [DBG] pgmap v323: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 205 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:07.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: audit 2024-09-20T19:24:06.343422+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24971 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T19:24:07.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:07 smithi162 bash[24557]: cluster 2024-09-20T19:24:06.354224+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-20T19:24:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:07.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:07.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:07.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:07.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:07.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:07.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:08.057 INFO:tasks.rados.rados.0.smithi042.stdout:663: done (1 left) 2024-09-20T19:24:08.057 INFO:tasks.rados.rados.0.smithi042.stdout:664: done (0 left) 2024-09-20T19:24:08.057 INFO:tasks.rados.rados.0.smithi042.stdout:665: write (excl) oid 47 current snap is 73 2024-09-20T19:24:08.057 INFO:tasks.rados.rados.0.smithi042.stdout:665: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-20T19:24:08.064 INFO:tasks.rados.rados.0.smithi042.stdout:665: writing smithi04268048-47 from 652171 to 1129817 tid 1 2024-09-20T19:24:08.071 INFO:tasks.rados.rados.0.smithi042.stdout:665: writing smithi04268048-47 from 1821709 to 2252171 tid 2 2024-09-20T19:24:08.071 INFO:tasks.rados.rados.0.smithi042.stdout:666: rollback oid 39 current snap is 73 2024-09-20T19:24:08.071 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 71 2024-09-20T19:24:08.072 INFO:tasks.rados.rados.0.smithi042.stdout:667: read oid 15 snap -1 2024-09-20T19:24:08.072 INFO:tasks.rados.rados.0.smithi042.stdout:667: expect (ObjNum 210 snap 63 seq_num 210) 2024-09-20T19:24:08.072 INFO:tasks.rados.rados.0.smithi042.stdout:668: copy_from oid 37 from oid 5 current snap is 73 2024-09-20T19:24:08.072 INFO:tasks.rados.rados.0.smithi042.stdout:669: delete oid 27 current snap is 73 2024-09-20T19:24:08.076 INFO:tasks.rados.rados.0.smithi042.stdout:665: finishing write tid 1 to smithi04268048-47 2024-09-20T19:24:08.081 INFO:tasks.rados.rados.0.smithi042.stdout:666: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:24:08.082 INFO:tasks.rados.rados.0.smithi042.stdout:666: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:24:08.083 INFO:tasks.rados.rados.0.smithi042.stdout:668: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:24:08.083 INFO:tasks.rados.rados.0.smithi042.stdout:666: finishing rollback tid 2 to smithi04268048-39 2024-09-20T19:24:08.083 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 157 (ObjNum 224 snap 66 seq_num 224) dirty exists 2024-09-20T19:24:08.088 INFO:tasks.rados.rados.0.smithi042.stdout:665: finishing write tid 2 to smithi04268048-47 2024-09-20T19:24:08.088 INFO:tasks.rados.rados.0.smithi042.stdout:665: finishing write tid 3 to smithi04268048-47 2024-09-20T19:24:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:665: finishing write tid 4 to smithi04268048-47 2024-09-20T19:24:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:665: oid 47 updating version 0 to 303 2024-09-20T19:24:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:665: oid 47 version 303 is already newer than 302 2024-09-20T19:24:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:665: oid 47 updating version 303 to 304 2024-09-20T19:24:08.089 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 304 (ObjNum 253 snap 73 seq_num 253) dirty exists 2024-09-20T19:24:08.090 INFO:tasks.rados.rados.0.smithi042.stdout:665: left oid 47 (ObjNum 253 snap 73 seq_num 253) 2024-09-20T19:24:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:668: finishing copy_from to smithi04268048-37 2024-09-20T19:24:08.092 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:24:08.096 INFO:tasks.rados.rados.0.smithi042.stdout:667: expect (ObjNum 210 snap 63 seq_num 210) 2024-09-20T19:24:08.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:08.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:08.169 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:08.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:08.253 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:665: done (4 left) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:666: done (3 left) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:667: done (2 left) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:668: done (1 left) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:669: done (0 left) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:670: read oid 41 snap -1 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:670: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:671: rollback oid 8 current snap is 73 2024-09-20T19:24:08.338 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 63 2024-09-20T19:24:08.339 INFO:tasks.rados.rados.0.smithi042.stdout:672: snap_create 2024-09-20T19:24:08.344 INFO:tasks.rados.rados.0.smithi042.stdout:670: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:08.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:08.446 INFO:tasks.rados.rados.0.smithi042.stdout:671: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:24:08.446 INFO:tasks.rados.rados.0.smithi042.stdout:671: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:24:08.446 INFO:tasks.rados.rados.0.smithi042.stdout:671: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:24:08.446 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 300 (ObjNum 68 snap 10 seq_num 68) dirty exists 2024-09-20T19:24:08.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:08.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:08 smithi162 bash[24557]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[32075]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: cluster 2024-09-20T19:24:07.350845+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: audit 2024-09-20T19:24:08.010086+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: audit 2024-09-20T19:24:08.050200+0000 mon.a (mon.0) 1232 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:08 smithi042 bash[22706]: cluster 2024-09-20T19:24:08.054011+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-20T19:24:08.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:08.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:08.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:08.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:09.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:09.069 INFO:tasks.rados.rados.0.smithi042.stdout:670: done (2 left) 2024-09-20T19:24:09.069 INFO:tasks.rados.rados.0.smithi042.stdout:671: done (1 left) 2024-09-20T19:24:09.069 INFO:tasks.rados.rados.0.smithi042.stdout:672: done (0 left) 2024-09-20T19:24:09.069 INFO:tasks.rados.rados.0.smithi042.stdout:673: snap_remove snap 72 2024-09-20T19:24:09.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:09.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:09.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:09.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:09.439 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:09.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:09.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:09 smithi162 bash[24557]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:09 smithi162 bash[24557]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:09 smithi162 bash[24557]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:09 smithi162 bash[24557]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[32075]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[32075]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[32075]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[32075]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[22706]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[22706]: cluster 2024-09-20T19:24:08.009415+0000 mgr.y (mgr.24491) 235 : cluster [DBG] pgmap v326: 148 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[22706]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:09 smithi042 bash[22706]: cluster 2024-09-20T19:24:09.067675+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-20T19:24:09.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:09.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:09.702 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:09.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:09.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:09.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.956 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:09.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:10.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:10.076 INFO:tasks.rados.rados.0.smithi042.stdout:673: done (0 left) 2024-09-20T19:24:10.077 INFO:tasks.rados.rados.0.smithi042.stdout:674: rollback oid 45 current snap is 74 2024-09-20T19:24:10.077 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 63 2024-09-20T19:24:10.077 INFO:tasks.rados.rados.0.smithi042.stdout:675: write (excl) oid 19 current snap is 74 2024-09-20T19:24:10.077 INFO:tasks.rados.rados.0.smithi042.stdout:675: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-20T19:24:10.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:10.103 INFO:tasks.rados.rados.0.smithi042.stdout:675: writing smithi04268048-19 from 685785 to 1313929 tid 1 2024-09-20T19:24:10.121 INFO:tasks.rados.rados.0.smithi042.stdout:675: writing smithi04268048-19 from 1822320 to 2285785 tid 2 2024-09-20T19:24:10.121 INFO:tasks.rados.rados.0.smithi042.stdout:676: copy_from oid 11 from oid 32 current snap is 74 2024-09-20T19:24:10.121 INFO:tasks.rados.rados.0.smithi042.stdout:674: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:24:10.121 INFO:tasks.rados.rados.0.smithi042.stdout:674: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 327 (ObjNum 197 snap 59 seq_num 197) dirty exists 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:675: finishing write tid 1 to smithi04268048-19 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:674: done (2 left) 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:677: read oid 27 snap 73 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:677: expect deleted 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:678: read oid 43 snap -1 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:678: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:679: read oid 1 snap -1 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:679: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:680: copy_from oid 8 from oid 6 current snap is 74 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:681: read oid 4 snap -1 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:681: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:10.122 INFO:tasks.rados.rados.0.smithi042.stdout:682: rollback oid 9 current snap is 74 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 73 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:683: read oid 25 snap -1 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:683: expect (ObjNum 250 snap 71 seq_num 250) 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:684: rollback oid 35 current snap is 74 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 68 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:685: read oid 30 snap -1 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:685: expect deleted 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:686: read oid 10 snap -1 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:686: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:687: copy_from oid 36 from oid 40 current snap is 74 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:688: read oid 2 snap -1 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:688: expect (ObjNum 252 snap 72 seq_num 252) 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:689: read oid 50 snap 71 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:689: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:24:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:690: copy_from oid 38 from oid 16 current snap is 74 2024-09-20T19:24:10.124 INFO:tasks.rados.rados.0.smithi042.stdout:677: done (15 left) 2024-09-20T19:24:10.124 INFO:tasks.rados.rados.0.smithi042.stdout:691: write (excl) oid 48 current snap is 74 2024-09-20T19:24:10.124 INFO:tasks.rados.rados.0.smithi042.stdout:691: seq_num 256 ranges {702592=703393,1880802=421790} 2024-09-20T19:24:10.131 INFO:tasks.rados.rados.0.smithi042.stdout:691: writing smithi04268048-48 from 702592 to 1405985 tid 1 2024-09-20T19:24:10.138 INFO:tasks.rados.rados.0.smithi042.stdout:691: writing smithi04268048-48 from 1880802 to 2302592 tid 2 2024-09-20T19:24:10.139 INFO:tasks.rados.rados.0.smithi042.stdout: waiting on 16 2024-09-20T19:24:10.139 INFO:tasks.rados.rados.0.smithi042.stdout:678: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:10.147 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1433909 2024-09-20T19:24:10.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:10.219 INFO:tasks.rados.rados.0.smithi042.stdout:683: expect (ObjNum 250 snap 71 seq_num 250) 2024-09-20T19:24:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:10.260 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2044350 2024-09-20T19:24:10.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:10.347 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:10.369 INFO:tasks.rados.rados.0.smithi042.stdout:675: finishing write tid 2 to smithi04268048-19 2024-09-20T19:24:10.369 INFO:tasks.rados.rados.0.smithi042.stdout:679: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:10.373 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 702206 2024-09-20T19:24:10.395 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:10.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:680: finishing copy_from to smithi04268048-8 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:680: got expected ENOENT (src dne) 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:680: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:684: finishing rollback tid 0 to smithi04268048-35 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:684: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:24:10.455 INFO:tasks.rados.rados.0.smithi042.stdout:684: finishing rollback tid 2 to smithi04268048-35 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 62 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: finishing write tid 3 to smithi04268048-19 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: finishing write tid 4 to smithi04268048-19 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: oid 19 updating version 0 to 189 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: oid 19 version 189 is already newer than 187 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: oid 19 version 189 is already newer than 188 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 189 (ObjNum 255 snap 74 seq_num 255) dirty exists 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:675: left oid 19 (ObjNum 255 snap 74 seq_num 255) 2024-09-20T19:24:10.456 INFO:tasks.rados.rados.0.smithi042.stdout:686: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:10.475 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:10.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:690: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: finishing write tid 1 to smithi04268048-48 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:687: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: finishing write tid 2 to smithi04268048-48 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: finishing write tid 3 to smithi04268048-48 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: finishing write tid 4 to smithi04268048-48 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: oid 48 updating version 0 to 344 2024-09-20T19:24:10.590 INFO:tasks.rados.rados.0.smithi042.stdout:691: oid 48 version 344 is already newer than 343 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:691: oid 48 version 344 is already newer than 342 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 344 (ObjNum 256 snap 74 seq_num 256) dirty exists 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:691: left oid 48 (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:687: finishing copy_from to smithi04268048-36 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 191 (ObjNum 242 snap 70 seq_num 242) dirty exists 2024-09-20T19:24:10.591 INFO:tasks.rados.rados.0.smithi042.stdout:688: expect (ObjNum 252 snap 72 seq_num 252) 2024-09-20T19:24:10.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:10.611 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:10.648 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1477023 2024-09-20T19:24:10.667 INFO:tasks.rados.rados.0.smithi042.stdout:681: expect (ObjNum 155 snap 44 seq_num 155) 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[22706]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:10 smithi042 bash[32075]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.707 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:10.715 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:10.763 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:10.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:09.139637+0000 mgr.y (mgr.24491) 236 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:10.010815+0000 mon.a (mon.0) 1235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: audit 2024-09-20T19:24:10.065726+0000 mon.a (mon.0) 1236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:10 smithi162 bash[24557]: cluster 2024-09-20T19:24:10.073703+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-20T19:24:10.811 INFO:tasks.rados.rados.0.smithi042.stdout:690: finishing copy_from to smithi04268048-38 2024-09-20T19:24:10.811 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 302 (ObjNum 251 snap 71 seq_num 251) dirty exists 2024-09-20T19:24:10.811 INFO:tasks.rados.rados.0.smithi042.stdout:689: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:24:10.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:10.835 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 356430 2024-09-20T19:24:10.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:682: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:682: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:682: finishing rollback tid 2 to smithi04268048-9 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 314 (ObjNum 244 snap 70 seq_num 244) dirty exists 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:676: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:24:10.882 INFO:tasks.rados.rados.0.smithi042.stdout:676: finishing copy_from to smithi04268048-11 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 286 (ObjNum 196 snap 59 seq_num 196) dirty exists 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:675: done (15 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:676: done (14 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:678: done (13 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:679: done (12 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:680: done (11 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:681: done (10 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:682: done (9 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:683: done (8 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:684: done (7 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:685: done (6 left) 2024-09-20T19:24:10.883 INFO:tasks.rados.rados.0.smithi042.stdout:686: done (5 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:687: done (4 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:688: done (3 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:689: done (2 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:690: done (1 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:691: done (0 left) 2024-09-20T19:24:10.884 INFO:tasks.rados.rados.0.smithi042.stdout:692: snap_create 2024-09-20T19:24:10.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:10.971 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:10.971 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:10.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:10.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:11.070 INFO:tasks.rados.rados.0.smithi042.stdout:692: done (0 left) 2024-09-20T19:24:11.071 INFO:tasks.rados.rados.0.smithi042.stdout:693: write oid 9 current snap is 75 2024-09-20T19:24:11.071 INFO:tasks.rados.rados.0.smithi042.stdout:693: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-20T19:24:11.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.099 INFO:tasks.rados.rados.0.smithi042.stdout:693: writing smithi04268048-9 from 719399 to 1498041 tid 1 2024-09-20T19:24:11.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:11.110 INFO:tasks.rados.rados.0.smithi042.stdout:693: writing smithi04268048-9 from 1939284 to 2319399 tid 2 2024-09-20T19:24:11.110 INFO:tasks.rados.rados.0.smithi042.stdout:694: read oid 11 snap 68 2024-09-20T19:24:11.111 INFO:tasks.rados.rados.0.smithi042.stdout:694: expect (ObjNum 198 snap 59 seq_num 198) 2024-09-20T19:24:11.111 INFO:tasks.rados.rados.0.smithi042.stdout:695: snap_create 2024-09-20T19:24:11.111 INFO:tasks.rados.rados.0.smithi042.stdout:693: finishing write tid 1 to smithi04268048-9 2024-09-20T19:24:11.118 INFO:tasks.rados.rados.0.smithi042.stdout:693: finishing write tid 2 to smithi04268048-9 2024-09-20T19:24:11.118 INFO:tasks.rados.rados.0.smithi042.stdout:693: finishing write tid 3 to smithi04268048-9 2024-09-20T19:24:11.118 INFO:tasks.rados.rados.0.smithi042.stdout:693: finishing write tid 4 to smithi04268048-9 2024-09-20T19:24:11.118 INFO:tasks.rados.rados.0.smithi042.stdout:693: oid 9 updating version 0 to 323 2024-09-20T19:24:11.118 INFO:tasks.rados.rados.0.smithi042.stdout:693: oid 9 version 323 is already newer than 322 2024-09-20T19:24:11.119 INFO:tasks.rados.rados.0.smithi042.stdout:693: oid 9 updating version 323 to 324 2024-09-20T19:24:11.119 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 324 (ObjNum 257 snap 75 seq_num 257) dirty exists 2024-09-20T19:24:11.119 INFO:tasks.rados.rados.0.smithi042.stdout:693: left oid 9 (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:11.119 INFO:tasks.rados.rados.0.smithi042.stdout:694: expect (ObjNum 198 snap 59 seq_num 198) 2024-09-20T19:24:11.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:11.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:11.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:11.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:11 smithi162 bash[24557]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:11 smithi162 bash[24557]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:11 smithi162 bash[24557]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:11 smithi162 bash[24557]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:11.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:11.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:11.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[22706]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[22706]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[22706]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[22706]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[32075]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[32075]: cluster 2024-09-20T19:24:10.010179+0000 mgr.y (mgr.24491) 237 : cluster [DBG] pgmap v329: 158 pgs: 10 unknown, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 140 active+clean; 206 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-20T19:24:11.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[32075]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:11 smithi042 bash[32075]: cluster 2024-09-20T19:24:11.070325+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-20T19:24:11.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T19:24:11.721 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:11.744 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:11.783 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:11.783 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:11.801 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:11.801 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:11.819 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:11.819 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:11.836 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:11.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:11.857 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:11.857 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:11.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:11.879 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:11.896 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:11.896 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:11.914 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:11.915 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:11.934 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:11.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:11.993 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:11.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:12.074 INFO:tasks.rados.rados.0.smithi042.stdout:693: done (2 left) 2024-09-20T19:24:12.074 INFO:tasks.rados.rados.0.smithi042.stdout:694: done (1 left) 2024-09-20T19:24:12.074 INFO:tasks.rados.rados.0.smithi042.stdout:695: done (0 left) 2024-09-20T19:24:12.075 INFO:tasks.rados.rados.0.smithi042.stdout:696: read oid 35 snap 73 2024-09-20T19:24:12.075 INFO:tasks.rados.rados.0.smithi042.stdout:696: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:12.075 INFO:tasks.rados.rados.0.smithi042.stdout:697: write oid 23 current snap is 76 2024-09-20T19:24:12.075 INFO:tasks.rados.rados.0.smithi042.stdout:697: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-20T19:24:12.094 INFO:tasks.rados.rados.0.smithi042.stdout:697: writing smithi04268048-23 from 736206 to 1190097 tid 1 2024-09-20T19:24:12.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.112 INFO:tasks.rados.rados.0.smithi042.stdout:697: writing smithi04268048-23 from 1881413 to 2336206 tid 2 2024-09-20T19:24:12.113 INFO:tasks.rados.rados.0.smithi042.stdout:698: copy_from oid 49 from oid 17 current snap is 76 2024-09-20T19:24:12.113 INFO:tasks.rados.rados.0.smithi042.stdout:696: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:12.123 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:12.189 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2794857 2024-09-20T19:24:12.226 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: finishing write tid 1 to smithi04268048-23 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: finishing write tid 2 to smithi04268048-23 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: finishing write tid 3 to smithi04268048-23 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: finishing write tid 4 to smithi04268048-23 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: oid 23 updating version 0 to 173 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: oid 23 version 173 is already newer than 172 2024-09-20T19:24:12.238 INFO:tasks.rados.rados.0.smithi042.stdout:697: oid 23 updating version 173 to 174 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 174 (ObjNum 258 snap 76 seq_num 258) dirty exists 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:697: left oid 23 (ObjNum 258 snap 76 seq_num 258) 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:696: done (2 left) 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:697: done (1 left) 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:699: read oid 20 snap -1 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:699: expect deleted 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:700: write oid 7 current snap is 76 2024-09-20T19:24:12.239 INFO:tasks.rados.rados.0.smithi042.stdout:700: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-20T19:24:12.249 INFO:tasks.rados.rados.0.smithi042.stdout:700: writing smithi04268048-7 from 753013 to 1398506 tid 1 2024-09-20T19:24:12.253 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:700: writing smithi04268048-7 from 2056248 to 2353013 tid 2 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:701: rollback oid 28 current snap is 76 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 75 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:698: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:699: done (3 left) 2024-09-20T19:24:12.256 INFO:tasks.rados.rados.0.smithi042.stdout:702: snap_create 2024-09-20T19:24:12.261 INFO:tasks.rados.rados.0.smithi042.stdout:701: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:24:12.261 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 123 (ObjNum 97 snap 0 seq_num 1415177680) dirty dne 2024-09-20T19:24:12.261 INFO:tasks.rados.rados.0.smithi042.stdout:698: finishing copy_from to smithi04268048-49 2024-09-20T19:24:12.261 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 67 (ObjNum 159 snap 47 seq_num 159) dirty exists 2024-09-20T19:24:12.262 INFO:tasks.rados.rados.0.smithi042.stdout:700: finishing write tid 1 to smithi04268048-7 2024-09-20T19:24:12.262 INFO:tasks.rados.rados.0.smithi042.stdout:700: finishing write tid 2 to smithi04268048-7 2024-09-20T19:24:12.262 INFO:tasks.rados.rados.0.smithi042.stdout:700: finishing write tid 3 to smithi04268048-7 2024-09-20T19:24:12.262 INFO:tasks.rados.rados.0.smithi042.stdout:700: finishing write tid 4 to smithi04268048-7 2024-09-20T19:24:12.263 INFO:tasks.rados.rados.0.smithi042.stdout:700: oid 7 updating version 0 to 253 2024-09-20T19:24:12.263 INFO:tasks.rados.rados.0.smithi042.stdout:700: oid 7 version 253 is already newer than 252 2024-09-20T19:24:12.263 INFO:tasks.rados.rados.0.smithi042.stdout:700: oid 7 version 253 is already newer than 251 2024-09-20T19:24:12.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 253 (ObjNum 259 snap 76 seq_num 259) dirty exists 2024-09-20T19:24:12.263 INFO:tasks.rados.rados.0.smithi042.stdout:700: left oid 7 (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:24:12.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:12.393 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:24:12.393 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:24:12.393 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-20T19:24:12.393 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-20T19:24:12.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:24:12.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[22706]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:12 smithi042 bash[32075]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:12.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.766243+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.42:0/1742819322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.978192+0000 mon.a (mon.0) 1240 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: audit 2024-09-20T19:24:11.979173+0000 mon.a (mon.0) 1241 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:12 smithi162 bash[24557]: cluster 2024-09-20T19:24:12.072313+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-20T19:24:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:12.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:12.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:12.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:12.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:13.003 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:13.056 INFO:tasks.rados.rados.0.smithi042.stdout:698: done (3 left) 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:700: done (2 left) 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:701: done (1 left) 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:702: done (0 left) 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:703: read oid 9 snap -1 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:703: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:13.057 INFO:tasks.rados.rados.0.smithi042.stdout:704: snap_create 2024-09-20T19:24:13.074 INFO:tasks.rados.rados.0.smithi042.stdout:703: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:13.088 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1288017 2024-09-20T19:24:13.105 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:13.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:13.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1632711 2024-09-20T19:24:13.160 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:13.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:13.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:13.395 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:13.424 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:13.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:13.552 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:13 smithi162 bash[24557]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:13 smithi162 bash[24557]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:13 smithi162 bash[24557]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:13 smithi162 bash[24557]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:13.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[32075]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[32075]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[32075]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[32075]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[22706]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[22706]: cluster 2024-09-20T19:24:12.010964+0000 mgr.y (mgr.24491) 238 : cluster [DBG] pgmap v332: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:13.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[22706]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:13 smithi042 bash[22706]: cluster 2024-09-20T19:24:13.056349+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-20T19:24:13.683 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:13.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:13.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:13.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:13.945 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:13.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:14.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:14.064 INFO:tasks.rados.rados.0.smithi042.stdout:703: done (1 left) 2024-09-20T19:24:14.064 INFO:tasks.rados.rados.0.smithi042.stdout:704: done (0 left) 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:705: read oid 20 snap -1 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:705: expect deleted 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:706: read oid 24 snap -1 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:706: expect deleted 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:707: write (excl) oid 18 current snap is 78 2024-09-20T19:24:14.065 INFO:tasks.rados.rados.0.smithi042.stdout:707: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-20T19:24:14.077 INFO:tasks.rados.rados.0.smithi042.stdout:707: writing smithi04268048-18 from 769820 to 1490562 tid 1 2024-09-20T19:24:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:14.086 INFO:tasks.rados.rados.0.smithi042.stdout:707: writing smithi04268048-18 from 2114730 to 2369820 tid 2 2024-09-20T19:24:14.086 INFO:tasks.rados.rados.0.smithi042.stdout:708: delete oid 16 current snap is 78 2024-09-20T19:24:14.086 INFO:tasks.rados.rados.0.smithi042.stdout:707: finishing write tid 1 to smithi04268048-18 2024-09-20T19:24:14.089 INFO:tasks.rados.rados.0.smithi042.stdout:707: finishing write tid 2 to smithi04268048-18 2024-09-20T19:24:14.090 INFO:tasks.rados.rados.0.smithi042.stdout:705: done (3 left) 2024-09-20T19:24:14.090 INFO:tasks.rados.rados.0.smithi042.stdout:706: done (2 left) 2024-09-20T19:24:14.090 INFO:tasks.rados.rados.0.smithi042.stdout:708: done (1 left) 2024-09-20T19:24:14.090 INFO:tasks.rados.rados.0.smithi042.stdout:709: delete oid 13 current snap is 78 2024-09-20T19:24:14.091 INFO:tasks.rados.rados.0.smithi042.stdout:707: finishing write tid 3 to smithi04268048-18 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:707: finishing write tid 4 to smithi04268048-18 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:707: oid 18 updating version 0 to 347 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:707: oid 18 updating version 347 to 348 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:707: oid 18 version 348 is already newer than 346 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 348 (ObjNum 260 snap 78 seq_num 260) dirty exists 2024-09-20T19:24:14.092 INFO:tasks.rados.rados.0.smithi042.stdout:707: left oid 18 (ObjNum 260 snap 78 seq_num 260) 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:707: done (1 left) 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:709: done (0 left) 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:710: rollback oid 6 current snap is 78 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 60 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:711: copy_from oid 47 from oid 14 current snap is 78 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:712: write (excl) oid 34 current snap is 78 2024-09-20T19:24:14.093 INFO:tasks.rados.rados.0.smithi042.stdout:712: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-20T19:24:14.104 INFO:tasks.rados.rados.0.smithi042.stdout:712: writing smithi04268048-34 from 420241 to 966730 tid 1 2024-09-20T19:24:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:712: writing smithi04268048-34 from 1373823 to 1918237 tid 2 2024-09-20T19:24:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:713: snap_create 2024-09-20T19:24:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:710: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:24:14.115 INFO:tasks.rados.rados.0.smithi042.stdout:710: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:24:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 302 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:24:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:711: finishing copy_from to smithi04268048-47 2024-09-20T19:24:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:711: got expected ENOENT (src dne) 2024-09-20T19:24:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:711: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:24:14.116 INFO:tasks.rados.rados.0.smithi042.stdout:712: finishing write tid 1 to smithi04268048-34 2024-09-20T19:24:14.119 INFO:tasks.rados.rados.0.smithi042.stdout:712: finishing write tid 2 to smithi04268048-34 2024-09-20T19:24:14.119 INFO:tasks.rados.rados.0.smithi042.stdout:712: finishing write tid 3 to smithi04268048-34 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:712: finishing write tid 4 to smithi04268048-34 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:712: oid 34 updating version 0 to 64 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:712: oid 34 updating version 64 to 65 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:712: oid 34 version 65 is already newer than 63 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 65 (ObjNum 263 snap 78 seq_num 263) dirty exists 2024-09-20T19:24:14.120 INFO:tasks.rados.rados.0.smithi042.stdout:712: left oid 34 (ObjNum 263 snap 78 seq_num 263) 2024-09-20T19:24:14.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:14.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:14.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:14.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:14.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:14.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:14.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:14.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:14.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:14.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:710: done (3 left) 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:711: done (2 left) 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:712: done (1 left) 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:713: done (0 left) 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:714: rollback oid 15 current snap is 79 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 71 2024-09-20T19:24:15.067 INFO:tasks.rados.rados.0.smithi042.stdout:715: delete oid 24 current snap is 79 2024-09-20T19:24:15.068 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:24:15.080 INFO:tasks.rados.rados.0.smithi042.stdout:714: finishing rollback tid 0 to smithi04268048-15 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:714: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:714: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 309 (ObjNum 210 snap 63 seq_num 210) dirty exists 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:714: done (1 left) 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:715: done (0 left) 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:716: write oid 11 current snap is 79 2024-09-20T19:24:15.083 INFO:tasks.rados.rados.0.smithi042.stdout:716: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-20T19:24:15.090 INFO:tasks.rados.rados.0.smithi042.stdout:716: writing smithi04268048-11 from 453855 to 1150842 tid 1 2024-09-20T19:24:15.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:716: writing smithi04268048-11 from 1890787 to 2438870 tid 2 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:717: read oid 44 snap -1 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:717: expect deleted 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:718: read oid 48 snap -1 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:718: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:719: read oid 17 snap -1 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:719: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:720: copy_from oid 10 from oid 22 current snap is 79 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:716: finishing write tid 1 to smithi04268048-11 2024-09-20T19:24:15.098 INFO:tasks.rados.rados.0.smithi042.stdout:721: snap_create 2024-09-20T19:24:15.100 INFO:tasks.rados.rados.0.smithi042.stdout:718: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:15 smithi162 bash[24557]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:15.177 INFO:tasks.rados.rados.0.smithi042.stdout:720: finishing copy_from to smithi04268048-10 2024-09-20T19:24:15.177 INFO:tasks.rados.rados.0.smithi042.stdout:720: got expected ENOENT (src dne) 2024-09-20T19:24:15.178 INFO:tasks.rados.rados.0.smithi042.stdout:720: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:24:15.178 INFO:tasks.rados.rados.0.smithi042.stdout:719: expect (ObjNum 159 snap 47 seq_num 159) 2024-09-20T19:24:15.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:15.243 INFO:tasks.rados.rados.0.smithi042.stdout:716: finishing write tid 2 to smithi04268048-11 2024-09-20T19:24:15.243 INFO:tasks.rados.rados.0.smithi042.stdout:716: finishing write tid 3 to smithi04268048-11 2024-09-20T19:24:15.243 INFO:tasks.rados.rados.0.smithi042.stdout:716: finishing write tid 4 to smithi04268048-11 2024-09-20T19:24:15.244 INFO:tasks.rados.rados.0.smithi042.stdout:716: oid 11 updating version 0 to 331 2024-09-20T19:24:15.244 INFO:tasks.rados.rados.0.smithi042.stdout:716: oid 11 version 331 is already newer than 330 2024-09-20T19:24:15.244 INFO:tasks.rados.rados.0.smithi042.stdout:716: oid 11 updating version 331 to 332 2024-09-20T19:24:15.244 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 332 (ObjNum 265 snap 79 seq_num 265) dirty exists 2024-09-20T19:24:15.244 INFO:tasks.rados.rados.0.smithi042.stdout:716: left oid 11 (ObjNum 265 snap 79 seq_num 265) 2024-09-20T19:24:15.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:15.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[22706]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: cluster 2024-09-20T19:24:14.011625+0000 mgr.y (mgr.24491) 239 : cluster [DBG] pgmap v335: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.7 MiB/s wr, 30 op/s; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: cluster 2024-09-20T19:24:14.062636+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: audit 2024-09-20T19:24:14.729710+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.42:0/817798758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:15 smithi042 bash[32075]: audit 2024-09-20T19:24:14.730155+0000 mon.a (mon.0) 1245 : audit [INF] from='client.24983 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:15.374 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:15.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:15.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:15.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:24:15.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:15.637 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:15.740 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:15.765 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:15.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:15.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:15.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:15.998 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:16.026 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:716: done (5 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:717: done (4 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:718: done (3 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:719: done (2 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:720: done (1 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:721: done (0 left) 2024-09-20T19:24:16.088 INFO:tasks.rados.rados.0.smithi042.stdout:722: snap_remove snap 75 2024-09-20T19:24:16.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:16.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22706]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[32075]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: cluster 2024-09-20T19:24:15.055175+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: audit 2024-09-20T19:24:15.057337+0000 mon.a (mon.0) 1247 : audit [INF] from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:16 smithi162 bash[24557]: cluster 2024-09-20T19:24:15.065773+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-20T19:24:16.347 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:16.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:16.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.485 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:16.491 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:16.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.620 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:16.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:16.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:16.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:16.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:16.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:16.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:17.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:17.093 INFO:tasks.rados.rados.0.smithi042.stdout:722: done (0 left) 2024-09-20T19:24:17.093 INFO:tasks.rados.rados.0.smithi042.stdout:723: copy_from oid 11 from oid 12 current snap is 80 2024-09-20T19:24:17.095 INFO:tasks.rados.rados.0.smithi042.stdout:724: read oid 48 snap -1 2024-09-20T19:24:17.095 INFO:tasks.rados.rados.0.smithi042.stdout:724: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:24:17.095 INFO:tasks.rados.rados.0.smithi042.stdout:725: write oid 4 current snap is 80 2024-09-20T19:24:17.095 INFO:tasks.rados.rados.0.smithi042.stdout:725: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-20T19:24:17.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:17.117 INFO:tasks.rados.rados.0.smithi042.stdout:725: writing smithi04268048-4 from 470662 to 1242898 tid 1 2024-09-20T19:24:17.134 INFO:tasks.rados.rados.0.smithi042.stdout:725: writing smithi04268048-4 from 1832916 to 2241010 tid 2 2024-09-20T19:24:17.134 INFO:tasks.rados.rados.0.smithi042.stdout:726: snap_create 2024-09-20T19:24:17.134 INFO:tasks.rados.rados.0.smithi042.stdout:724: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:24:17.137 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:723: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:723: finishing copy_from to smithi04268048-11 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 152 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:725: finishing write tid 1 to smithi04268048-4 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:725: finishing write tid 2 to smithi04268048-4 2024-09-20T19:24:17.217 INFO:tasks.rados.rados.0.smithi042.stdout:725: finishing write tid 3 to smithi04268048-4 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:725: finishing write tid 4 to smithi04268048-4 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:725: oid 4 updating version 0 to 339 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:725: oid 4 updating version 339 to 340 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:725: oid 4 version 340 is already newer than 338 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 340 (ObjNum 266 snap 80 seq_num 266) dirty exists 2024-09-20T19:24:17.218 INFO:tasks.rados.rados.0.smithi042.stdout:725: left oid 4 (ObjNum 266 snap 80 seq_num 266) 2024-09-20T19:24:17.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.266 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:17.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:17 smithi162 bash[24557]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:17 smithi162 bash[24557]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:17 smithi162 bash[24557]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:17 smithi162 bash[24557]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[32075]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[32075]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[32075]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[32075]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[22706]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[22706]: cluster 2024-09-20T19:24:16.012224+0000 mgr.y (mgr.24491) 240 : cluster [DBG] pgmap v338: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 226 MiB data, 819 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[22706]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:17 smithi042 bash[22706]: cluster 2024-09-20T19:24:16.088360+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-20T19:24:17.498 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.527 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:17.630 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:17.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:17.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:17.891 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:17.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:17.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:24:18.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.027 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:24:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:24:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:24:18.027 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:24:18.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:18.063 INFO:tasks.rados.rados.0.smithi042.stdout:723: done (3 left) 2024-09-20T19:24:18.063 INFO:tasks.rados.rados.0.smithi042.stdout:724: done (2 left) 2024-09-20T19:24:18.063 INFO:tasks.rados.rados.0.smithi042.stdout:725: done (1 left) 2024-09-20T19:24:18.063 INFO:tasks.rados.rados.0.smithi042.stdout:726: done (0 left) 2024-09-20T19:24:18.063 INFO:tasks.rados.rados.0.smithi042.stdout:727: snap_remove snap 71 2024-09-20T19:24:18.149 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[22706]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[22706]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[22706]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[22706]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.159 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[32075]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.160 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[32075]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.160 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[32075]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.160 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:18 smithi042 bash[32075]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:18.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:18.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:18 smithi162 bash[24557]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:18 smithi162 bash[24557]: cluster 2024-09-20T19:24:17.089224+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-20T19:24:18.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:18 smithi162 bash[24557]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:18 smithi162 bash[24557]: cluster 2024-09-20T19:24:18.063458+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-20T19:24:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:18.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.430 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:18.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:18.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:18.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:18.688 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:18.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T19:24:18.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:18.922 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.951 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:18.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:19.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.074 INFO:tasks.rados.rados.0.smithi042.stdout:727: done (0 left) 2024-09-20T19:24:19.074 INFO:tasks.rados.rados.0.smithi042.stdout:728: snap_create 2024-09-20T19:24:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:19.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:19.209 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:19.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.324 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:19 smithi162 bash[24557]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.324 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:19 smithi162 bash[24557]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.324 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:19 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.324 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:19 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:19.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:19.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[22706]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[22706]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[32075]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[32075]: cluster 2024-09-20T19:24:18.012877+0000 mgr.y (mgr.24491) 241 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:19.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:19 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.072804+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-20T19:24:19.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:19.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:19.598 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:19.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:19.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:19.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:19.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:19.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:19.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:20.069 INFO:tasks.rados.rados.0.smithi042.stdout:728: done (0 left) 2024-09-20T19:24:20.069 INFO:tasks.rados.rados.0.smithi042.stdout:729: read oid 33 snap -1 2024-09-20T19:24:20.070 INFO:tasks.rados.rados.0.smithi042.stdout:729: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:20.070 INFO:tasks.rados.rados.0.smithi042.stdout:730: write (excl) oid 29 current snap is 82 2024-09-20T19:24:20.070 INFO:tasks.rados.rados.0.smithi042.stdout:730: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-20T19:24:20.088 INFO:tasks.rados.rados.0.smithi042.stdout:730: writing smithi04268048-29 from 487469 to 1051307 tid 1 2024-09-20T19:24:20.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:20.105 INFO:tasks.rados.rados.0.smithi042.stdout:730: writing smithi04268048-29 from 1607751 to 2159503 tid 2 2024-09-20T19:24:20.105 INFO:tasks.rados.rados.0.smithi042.stdout:731: rollback oid 5 current snap is 82 2024-09-20T19:24:20.105 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 74 2024-09-20T19:24:20.105 INFO:tasks.rados.rados.0.smithi042.stdout:729: expect (ObjNum 186 snap 56 seq_num 186) 2024-09-20T19:24:20.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:20.131 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 589965 2024-09-20T19:24:20.192 INFO:tasks.rados.rados.0.smithi042.stdout:730: finishing write tid 1 to smithi04268048-29 2024-09-20T19:24:20.192 INFO:tasks.rados.rados.0.smithi042.stdout:730: finishing write tid 2 to smithi04268048-29 2024-09-20T19:24:20.192 INFO:tasks.rados.rados.0.smithi042.stdout:730: finishing write tid 3 to smithi04268048-29 2024-09-20T19:24:20.192 INFO:tasks.rados.rados.0.smithi042.stdout:730: finishing write tid 4 to smithi04268048-29 2024-09-20T19:24:20.192 INFO:tasks.rados.rados.0.smithi042.stdout:730: oid 29 updating version 0 to 303 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:730: oid 29 updating version 303 to 305 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:730: oid 29 version 305 is already newer than 304 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 305 (ObjNum 267 snap 82 seq_num 267) dirty exists 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:730: left oid 29 (ObjNum 267 snap 82 seq_num 267) 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:731: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:729: done (2 left) 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:730: done (1 left) 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:732: read oid 43 snap 73 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:732: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:733: write oid 11 current snap is 82 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:731: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:731: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:24:20.193 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 184 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:24:20.194 INFO:tasks.rados.rados.0.smithi042.stdout:733: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-20T19:24:20.198 INFO:tasks.rados.rados.0.smithi042.stdout:733: writing smithi04268048-11 from 504276 to 1143363 tid 1 2024-09-20T19:24:20.206 INFO:tasks.rados.rados.0.smithi042.stdout:733: writing smithi04268048-11 from 1666233 to 2361643 tid 2 2024-09-20T19:24:20.206 INFO:tasks.rados.rados.0.smithi042.stdout:731: done (2 left) 2024-09-20T19:24:20.206 INFO:tasks.rados.rados.0.smithi042.stdout:734: snap_remove snap 68 2024-09-20T19:24:20.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:20.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.231 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.231 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.231 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:20 smithi162 bash[24557]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:20.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.359 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[32075]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: audit 2024-09-20T19:24:19.148390+0000 mgr.y (mgr.24491) 242 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.661849+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:19.662205+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:20 smithi042 bash[22706]: cluster 2024-09-20T19:24:20.067736+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-20T19:24:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:20.446 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:24:20.449 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:24:20.469 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:20.475 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:20.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:20.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:20.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:20.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:20.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:20.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:20.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:20.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:20.987 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:21.119 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:21.146 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:21.160 INFO:tasks.rados.rados.0.smithi042.stdout:734: done (2 left) 2024-09-20T19:24:21.160 INFO:tasks.rados.rados.0.smithi042.stdout:735: write (excl) oid 13 current snap is 82 2024-09-20T19:24:21.160 INFO:tasks.rados.rados.0.smithi042.stdout:735: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-20T19:24:21.167 INFO:tasks.rados.rados.0.smithi042.stdout:735: writing smithi04268048-13 from 521083 to 1235419 tid 1 2024-09-20T19:24:21.173 INFO:tasks.rados.rados.0.smithi042.stdout:735: writing smithi04268048-13 from 1724715 to 2280136 tid 2 2024-09-20T19:24:21.173 INFO:tasks.rados.rados.0.smithi042.stdout:736: write (excl) oid 1 current snap is 82 2024-09-20T19:24:21.173 INFO:tasks.rados.rados.0.smithi042.stdout:736: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-20T19:24:21.181 INFO:tasks.rados.rados.0.smithi042.stdout:736: writing smithi04268048-1 from 537890 to 1327475 tid 1 2024-09-20T19:24:21.188 INFO:tasks.rados.rados.0.smithi042.stdout:736: writing smithi04268048-1 from 1783197 to 2482276 tid 2 2024-09-20T19:24:21.188 INFO:tasks.rados.rados.0.smithi042.stdout:737: snap_remove snap 81 2024-09-20T19:24:21.196 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[22706]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[22706]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[22706]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[22706]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[32075]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[32075]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[32075]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:21 smithi042 bash[32075]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:21.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:21 smithi162 bash[24557]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:21 smithi162 bash[24557]: cluster 2024-09-20T19:24:20.013440+0000 mgr.y (mgr.24491) 243 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 237 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-20T19:24:21.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:21 smithi162 bash[24557]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.260 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:21 smithi162 bash[24557]: audit 2024-09-20T19:24:20.435983+0000 mgr.y (mgr.24491) 244 : audit [DBG] from='client.24989 v1:172.21.15.42:0/1936835370' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:24:21.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:21.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:21.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:21.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:21.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:21.772 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:21.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:21.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:21.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:21.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:22.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.065 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:22.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-20T19:24:20.061938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":249028310,"num_objects":314,"num_object_clones":68,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":8,"num_read":2759,"num_read_kb":1145352,"num_write":1646,"num_write_kb":422031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4237,"ondisk_log_size":4237,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":862812,"kb_used_data":584180,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":749098404,"statfs":{"total":767960285184,"available":767076765696,"internally_reserved":0,"allocated":598200320,"data_stored":794570257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11786290,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":17076,"num_write":26,"num_write_kb":7657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.982229"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":426,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231284+0000","last_change":"2024-09-20T19:24:03.086154+0000","last_active":"2024-09-20T19:24:16.231284+0000","last_peered":"2024-09-20T19:24:16.231284+0000","last_clean":"2024-09-20T19:24:16.231284+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:24:16.231284+0000","last_undegraded":"2024-09-20T19:24:16.231284+0000","last_fullsized":"2024-09-20T19:24:16.231284+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:03.085896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.085896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:14.760862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":439,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231496+0000","last_change":"2024-09-20T19:23:50.588385+0000","last_active":"2024-09-20T19:24:16.231496+0000","last_peered":"2024-09-20T19:24:16.231496+0000","last_clean":"2024-09-20T19:24:16.231496+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:24:16.231496+0000","last_undegraded":"2024-09-20T19:24:16.231496+0000","last_fullsized":"2024-09-20T19:24:16.231496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.588194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.588194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.888096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":451,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231522+0000","last_change":"2024-09-20T19:22:14.898580+0000","last_active":"2024-09-20T19:24:16.231522+0000","last_peered":"2024-09-20T19:24:16.231522+0000","last_clean":"2024-09-20T19:24:16.231522+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:24:16.231522+0000","last_undegraded":"2024-09-20T19:24:16.231522+0000","last_fullsized":"2024-09-20T19:24:16.231522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:14.898494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.898494+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:15.113413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087962+0000","last_change":"2024-09-20T19:23:50.236531+0000","last_active":"2024-09-20T19:24:16.087962+0000","last_peered":"2024-09-20T19:24:16.087962+0000","last_clean":"2024-09-20T19:24:16.087962+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:24:16.087962+0000","last_undegraded":"2024-09-20T19:24:16.087962+0000","last_fullsized":"2024-09-20T19:24:16.087962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.236477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.236477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:07.670048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235266+0000","last_change":"2024-09-20T19:22:58.622973+0000","last_active":"2024-09-20T19:24:16.235266+0000","last_peered":"2024-09-20T19:24:16.235266+0000","last_clean":"2024-09-20T19:24:16.235266+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:24:16.235266+0000","last_undegraded":"2024-09-20T19:24:16.235266+0000","last_fullsized":"2024-09-20T19:24:16.235266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:58.622796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.622796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":443,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088264+0000","last_change":"2024-09-20T19:23:47.200445+0000","last_active":"2024-09-20T19:24:16.088264+0000","last_peered":"2024-09-20T19:24:16.088264+0000","last_clean":"2024-09-20T19:24:16.088264+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:24:16.088264+0000","last_undegraded":"2024-09-20T19:24:16.088264+0000","last_fullsized":"2024-09-20T19:24:16.088264+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:23:47.200185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.200185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.587959+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.082922+0000","last_change":"2024-09-20T19:22:03.981228+0000","last_active":"2024-09-20T19:24:16.082922+0000","last_peered":"2024-09-20T19:24:16.082922+0000","last_clean":"2024-09-20T19:24:16.082922+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:24:16.082922+0000","last_undegraded":"2024-09-20T19:24:16.082922+0000","last_fullsized":"2024-09-20T19:24:16.082922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.981101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.981101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.455385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088355+0000","last_change":"2024-09-20T19:22:14.871281+0000","last_active":"2024-09-20T19:24:16.088355+0000","last_peered":"2024-09-20T19:24:16.088355+0000","last_clean":"2024-09-20T19:24:16.088355+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:24:16.088355+0000","last_undegraded":"2024-09-20T19:24:16.088355+0000","last_fullsized":"2024-09-20T19:24:16.088355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:14.871139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.871139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.971798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.242119+0000","last_change":"2024-09-20T19:23:08.305837+0000","last_active":"2024-09-20T19:24:16.242119+0000","last_peered":"2024-09-20T19:24:16.242119+0000","last_clean":"2024-09-20T19:24:16.242119+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:24:16.242119+0000","last_undegraded":"2024-09-20T19:24:16.242119+0000","last_fullsized":"2024-09-20T19:24:16.242119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.305714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.305714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:26.456491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":437,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088614+0000","last_change":"2024-09-20T19:23:02.238503+0000","last_active":"2024-09-20T19:24:16.088614+0000","last_peered":"2024-09-20T19:24:16.088614+0000","last_clean":"2024-09-20T19:24:16.088614+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:24:16.088614+0000","last_undegraded":"2024-09-20T19:24:16.088614+0000","last_fullsized":"2024-09-20T19:24:16.088614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:23:02.238438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.238438+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:07.321039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237356+0000","last_change":"2024-09-20T19:22:08.909960+0000","last_active":"2024-09-20T19:24:16.237356+0000","last_peered":"2024-09-20T19:24:16.237356+0000","last_clean":"2024-09-20T19:24:16.237356+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:24:16.237356+0000","last_undegraded":"2024-09-20T19:24:16.237356+0000","last_fullsized":"2024-09-20T19:24:16.237356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:08.909835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.909835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:08.254350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":425,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235200+0000","last_change":"2024-09-20T19:23:58.846995+0000","last_active":"2024-09-20T19:24:16.235200+0000","last_peered":"2024-09-20T19:24:16.235200+0000","last_clean":"2024-09-20T19:24:16.235200+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:24:16.235200+0000","last_undegraded":"2024-09-20T19:24:16.235200+0000","last_fullsized":"2024-09-20T19:24:16.235200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:58.846866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:58.846866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:59.685012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240248+0000","last_change":"2024-09-20T19:22:20.892258+0000","last_active":"2024-09-20T19:24:16.240248+0000","last_peered":"2024-09-20T19:24:16.240248+0000","last_clean":"2024-09-20T19:24:16.240248+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:24:16.240248+0000","last_undegraded":"2024-09-20T19:24:16.240248+0000","last_fullsized":"2024-09-20T19:24:16.240248+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.892102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.892102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":443,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089091+0000","last_change":"2024-09-20T19:22:07.939814+0000","last_active":"2024-09-20T19:24:16.089091+0000","last_peered":"2024-09-20T19:24:16.089091+0000","last_clean":"2024-09-20T19:24:16.089091+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:24:16.089091+0000","last_undegraded":"2024-09-20T19:24:16.089091+0000","last_fullsized":"2024-09-20T19:24:16.089091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:07.939740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:07.939740+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.138214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":413,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236141+0000","last_change":"2024-09-20T19:21:07.802088+0000","last_active":"2024-09-20T19:24:16.236141+0000","last_peered":"2024-09-20T19:24:16.236141+0000","last_clean":"2024-09-20T19:24:16.236141+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:24:16.236141+0000","last_undegraded":"2024-09-20T19:24:16.236141+0000","last_fullsized":"2024-09-20T19:24:16.236141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:07.801948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.801948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.780869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229154+0000","last_change":"2024-09-20T19:21:08.846790+0000","last_active":"2024-09-20T19:24:16.229154+0000","last_peered":"2024-09-20T19:24:16.229154+0000","last_clean":"2024-09-20T19:24:16.229154+0000","last_became_active":"2024-09-20T19:18:13.351712+0000","last_became_peered":"2024-09-20T19:18:13.351712+0000","last_unstale":"2024-09-20T19:24:16.229154+0000","last_undegraded":"2024-09-20T19:24:16.229154+0000","last_fullsized":"2024-09-20T19:24:16.229154+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.846665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.846665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.808843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":29,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235591+0000","last_change":"2024-09-20T19:24:11.110948+0000","last_active":"2024-09-20T19:24:16.235591+0000","last_peered":"2024-09-20T19:24:16.235591+0000","last_clean":"2024-09-20T19:24:16.235591+0000","last_became_active":"2024-09-20T19:24:11.093457+0000","last_became_peered":"2024-09-20T19:24:11.093457+0000","last_unstale":"2024-09-20T19:24:16.235591+0000","last_undegraded":"2024-09-20T19:24:16.235591+0000","last_fullsized":"2024-09-20T19:24:16.235591+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"246'81","reported_seq":844,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087186+0000","last_change":"2024-09-20T19:24:10.077322+0000","last_active":"2024-09-20T19:24:16.087186+0000","last_peered":"2024-09-20T19:24:16.087186+0000","last_clean":"2024-09-20T19:24:16.087186+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:24:16.087186+0000","last_undegraded":"2024-09-20T19:24:16.087186+0000","last_fullsized":"2024-09-20T19:24:16.087186+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'45","last_scrub_stamp":"2024-09-20T19:22:54.853379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.853379+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013750839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":11858,"num_write":31,"num_write_kb":8005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.d","version":"77'17","reported_seq":455,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089761+0000","last_change":"2024-09-20T19:23:55.223430+0000","last_active":"2024-09-20T19:24:16.089761+0000","last_peered":"2024-09-20T19:24:16.089761+0000","last_clean":"2024-09-20T19:24:16.089761+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:24:16.089761+0000","last_undegraded":"2024-09-20T19:24:16.089761+0000","last_fullsized":"2024-09-20T19:24:16.089761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:23:55.223328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:55.223328+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:10.076063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238597+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:24:16.238597+0000","last_peered":"2024-09-20T19:24:16.238597+0000","last_clean":"2024-09-20T19:24:16.238597+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:24:16.238597+0000","last_undegraded":"2024-09-20T19:24:16.238597+0000","last_fullsized":"2024-09-20T19:24:16.238597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.295233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083540+0000","last_change":"2024-09-20T19:23:48.553467+0000","last_active":"2024-09-20T19:24:16.083540+0000","last_peered":"2024-09-20T19:24:16.083540+0000","last_clean":"2024-09-20T19:24:16.083540+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:24:16.083540+0000","last_undegraded":"2024-09-20T19:24:16.083540+0000","last_fullsized":"2024-09-20T19:24:16.083540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.553406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.553406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.629688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":418,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087518+0000","last_change":"2024-09-20T19:24:02.788104+0000","last_active":"2024-09-20T19:24:16.087518+0000","last_peered":"2024-09-20T19:24:16.087518+0000","last_clean":"2024-09-20T19:24:16.087518+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:24:16.087518+0000","last_undegraded":"2024-09-20T19:24:16.087518+0000","last_fullsized":"2024-09-20T19:24:16.087518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:02.787974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:02.787974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:15.665700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":484,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236973+0000","last_change":"2024-09-20T19:24:11.093721+0000","last_active":"2024-09-20T19:24:16.236973+0000","last_peered":"2024-09-20T19:24:16.236973+0000","last_clean":"2024-09-20T19:24:16.236973+0000","last_became_active":"2024-09-20T19:24:11.093089+0000","last_became_peered":"2024-09-20T19:24:11.093089+0000","last_unstale":"2024-09-20T19:24:16.236973+0000","last_undegraded":"2024-09-20T19:24:16.236973+0000","last_fullsized":"2024-09-20T19:24:16.236973+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"261'178","reported_seq":948,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231043+0000","last_change":"2024-09-20T19:24:10.102641+0000","last_active":"2024-09-20T19:24:16.231043+0000","last_peered":"2024-09-20T19:24:16.231043+0000","last_clean":"2024-09-20T19:24:16.231043+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:24:16.231043+0000","last_undegraded":"2024-09-20T19:24:16.231043+0000","last_fullsized":"2024-09-20T19:24:16.231043+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'100","last_scrub_stamp":"2024-09-20T19:23:11.701907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.701907+0000","objects_scrubbed":3,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.184364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0050955649999999998,"stat_sum":{"num_bytes":13156601,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":99,"num_read_kb":85739,"num_write":64,"num_write_kb":22976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.c","version":"77'10","reported_seq":450,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089684+0000","last_change":"2024-09-20T19:23:42.253431+0000","last_active":"2024-09-20T19:24:16.089684+0000","last_peered":"2024-09-20T19:24:16.089684+0000","last_clean":"2024-09-20T19:24:16.089684+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:24:16.089684+0000","last_undegraded":"2024-09-20T19:24:16.089684+0000","last_fullsized":"2024-09-20T19:24:16.089684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:42.253363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.253363+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.814248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083705+0000","last_change":"2024-09-20T19:21:16.692876+0000","last_active":"2024-09-20T19:24:16.083705+0000","last_peered":"2024-09-20T19:24:16.083705+0000","last_clean":"2024-09-20T19:24:16.083705+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:24:16.083705+0000","last_undegraded":"2024-09-20T19:24:16.083705+0000","last_fullsized":"2024-09-20T19:24:16.083705+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:16.692759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:16.692759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.508852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":509,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087113+0000","last_change":"2024-09-20T19:24:01.760145+0000","last_active":"2024-09-20T19:24:16.087113+0000","last_peered":"2024-09-20T19:24:16.087113+0000","last_clean":"2024-09-20T19:24:16.087113+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:24:16.087113+0000","last_undegraded":"2024-09-20T19:24:16.087113+0000","last_fullsized":"2024-09-20T19:24:16.087113+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:01.760085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:01.760085+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:01.886823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089901+0000","last_change":"2024-09-20T19:21:00.850362+0000","last_active":"2024-09-20T19:24:16.089901+0000","last_peered":"2024-09-20T19:24:16.089901+0000","last_clean":"2024-09-20T19:24:16.089901+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:24:16.089901+0000","last_undegraded":"2024-09-20T19:24:16.089901+0000","last_fullsized":"2024-09-20T19:24:16.089901+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:00.850236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:00.850236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.099845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":22,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234796+0000","last_change":"2024-09-20T19:24:11.089419+0000","last_active":"2024-09-20T19:24:16.234796+0000","last_peered":"2024-09-20T19:24:16.234796+0000","last_clean":"2024-09-20T19:24:16.234796+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:24:16.234796+0000","last_undegraded":"2024-09-20T19:24:16.234796+0000","last_fullsized":"2024-09-20T19:24:16.234796+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:14.791530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"265'215","reported_seq":989,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.367911+0000","last_change":"2024-09-20T19:24:16.368338+0000","last_active":"2024-09-20T19:24:16.367911+0000","last_peered":"2024-09-20T19:24:16.367911+0000","last_clean":"2024-09-20T19:24:16.367911+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:24:16.367911+0000","last_undegraded":"2024-09-20T19:24:16.367911+0000","last_fullsized":"2024-09-20T19:24:16.367911+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'57","last_scrub_stamp":"2024-09-20T19:22:52.771766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771766+0000","objects_scrubbed":6,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.13005967299999999,"stat_sum":{"num_bytes":25872263,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":136,"num_read_kb":103016,"num_write":81,"num_write_kb":24148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.f","version":"77'15","reported_seq":452,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.084697+0000","last_change":"2024-09-20T19:23:04.313290+0000","last_active":"2024-09-20T19:24:16.084697+0000","last_peered":"2024-09-20T19:24:16.084697+0000","last_clean":"2024-09-20T19:24:16.084697+0000","last_became_active":"2024-09-20T19:18:09.301879+0000","last_became_peered":"2024-09-20T19:18:09.301879+0000","last_unstale":"2024-09-20T19:24:16.084697+0000","last_undegraded":"2024-09-20T19:24:16.084697+0000","last_fullsized":"2024-09-20T19:24:16.084697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.291138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083140+0000","last_change":"2024-09-20T19:23:45.542703+0000","last_active":"2024-09-20T19:24:16.083140+0000","last_peered":"2024-09-20T19:24:16.083140+0000","last_clean":"2024-09-20T19:24:16.083140+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:24:16.083140+0000","last_undegraded":"2024-09-20T19:24:16.083140+0000","last_fullsized":"2024-09-20T19:24:16.083140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:45.542589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:45.542589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.881406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":507,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.090169+0000","last_change":"2024-09-20T19:23:03.226408+0000","last_active":"2024-09-20T19:24:16.090169+0000","last_peered":"2024-09-20T19:24:16.090169+0000","last_clean":"2024-09-20T19:24:16.090169+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:24:16.090169+0000","last_undegraded":"2024-09-20T19:24:16.090169+0000","last_fullsized":"2024-09-20T19:24:16.090169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.212037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":417,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239779+0000","last_change":"2024-09-20T19:23:03.945590+0000","last_active":"2024-09-20T19:24:16.239779+0000","last_peered":"2024-09-20T19:24:16.239779+0000","last_clean":"2024-09-20T19:24:16.239779+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:24:16.239779+0000","last_undegraded":"2024-09-20T19:24:16.239779+0000","last_fullsized":"2024-09-20T19:24:16.239779+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:03.945432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.945432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"263'110","reported_seq":886,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238305+0000","last_change":"2024-09-20T19:24:10.356154+0000","last_active":"2024-09-20T19:24:16.238305+0000","last_peered":"2024-09-20T19:24:16.238305+0000","last_clean":"2024-09-20T19:24:16.238305+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:24:16.238305+0000","last_undegraded":"2024-09-20T19:24:16.238305+0000","last_fullsized":"2024-09-20T19:24:16.238305+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'64","last_scrub_stamp":"2024-09-20T19:23:13.624271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:13.624271+0000","objects_scrubbed":1,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.629135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.26356655800000001,"stat_sum":{"num_bytes":14371091,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":68552,"num_write":49,"num_write_kb":18250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.e","version":"77'11","reported_seq":446,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.090200+0000","last_change":"2024-09-20T19:22:58.180428+0000","last_active":"2024-09-20T19:24:16.090200+0000","last_peered":"2024-09-20T19:24:16.090200+0000","last_clean":"2024-09-20T19:24:16.090200+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:24:16.090200+0000","last_undegraded":"2024-09-20T19:24:16.090200+0000","last_fullsized":"2024-09-20T19:24:16.090200+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:58.180287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.180287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.237767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":307,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.081521+0000","last_change":"2024-09-20T19:23:52.726605+0000","last_active":"2024-09-20T19:24:16.081521+0000","last_peered":"2024-09-20T19:24:16.081521+0000","last_clean":"2024-09-20T19:24:16.081521+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:24:16.081521+0000","last_undegraded":"2024-09-20T19:24:16.081521+0000","last_fullsized":"2024-09-20T19:24:16.081521+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.726537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.726537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:07.275685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":413,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240611+0000","last_change":"2024-09-20T19:20:58.711594+0000","last_active":"2024-09-20T19:24:16.240611+0000","last_peered":"2024-09-20T19:24:16.240611+0000","last_clean":"2024-09-20T19:24:16.240611+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:24:16.240611+0000","last_undegraded":"2024-09-20T19:24:16.240611+0000","last_fullsized":"2024-09-20T19:24:16.240611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:58.711472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:58.711472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.810411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":402,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234687+0000","last_change":"2024-09-20T19:20:16.026450+0000","last_active":"2024-09-20T19:24:16.234687+0000","last_peered":"2024-09-20T19:24:16.234687+0000","last_clean":"2024-09-20T19:24:16.234687+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:24:16.234687+0000","last_undegraded":"2024-09-20T19:24:16.234687+0000","last_fullsized":"2024-09-20T19:24:16.234687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:16.026389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:16.026389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.744413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":27,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234455+0000","last_change":"2024-09-20T19:24:11.099405+0000","last_active":"2024-09-20T19:24:16.234455+0000","last_peered":"2024-09-20T19:24:16.234455+0000","last_clean":"2024-09-20T19:24:16.234455+0000","last_became_active":"2024-09-20T19:24:11.088414+0000","last_became_peered":"2024-09-20T19:24:11.088414+0000","last_unstale":"2024-09-20T19:24:16.234455+0000","last_undegraded":"2024-09-20T19:24:16.234455+0000","last_fullsized":"2024-09-20T19:24:16.234455+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:13.045554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"244'31","reported_seq":789,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.233135+0000","last_change":"2024-09-20T19:24:10.358665+0000","last_active":"2024-09-20T19:24:16.233135+0000","last_peered":"2024-09-20T19:24:16.233135+0000","last_clean":"2024-09-20T19:24:16.233135+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:24:16.233135+0000","last_undegraded":"2024-09-20T19:24:16.233135+0000","last_fullsized":"2024-09-20T19:24:16.233135+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'31","last_scrub_stamp":"2024-09-20T19:23:57.047151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:57.047151+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:08.903970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.27124635600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":14840,"num_write":10,"num_write_kb":1940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.1","version":"77'14","reported_seq":440,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088827+0000","last_change":"2024-09-20T19:23:43.288937+0000","last_active":"2024-09-20T19:24:16.088827+0000","last_peered":"2024-09-20T19:24:16.088827+0000","last_clean":"2024-09-20T19:24:16.088827+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:24:16.088827+0000","last_undegraded":"2024-09-20T19:24:16.088827+0000","last_fullsized":"2024-09-20T19:24:16.088827+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:23:43.288805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.288805+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:54.121381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":436,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239350+0000","last_change":"2024-09-20T19:23:47.645931+0000","last_active":"2024-09-20T19:24:16.239350+0000","last_peered":"2024-09-20T19:24:16.239350+0000","last_clean":"2024-09-20T19:24:16.239350+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:24:16.239350+0000","last_undegraded":"2024-09-20T19:24:16.239350+0000","last_fullsized":"2024-09-20T19:24:16.239350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.645876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.645876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:53.852581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":499,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229877+0000","last_change":"2024-09-20T19:21:07.002029+0000","last_active":"2024-09-20T19:24:16.229877+0000","last_peered":"2024-09-20T19:24:16.229877+0000","last_clean":"2024-09-20T19:24:16.229877+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:24:16.229877+0000","last_undegraded":"2024-09-20T19:24:16.229877+0000","last_fullsized":"2024-09-20T19:24:16.229877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:07.001951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.001951+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.724883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":408,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236827+0000","last_change":"2024-09-20T19:23:46.300434+0000","last_active":"2024-09-20T19:24:16.236827+0000","last_peered":"2024-09-20T19:24:16.236827+0000","last_clean":"2024-09-20T19:24:16.236827+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:24:16.236827+0000","last_undegraded":"2024-09-20T19:24:16.236827+0000","last_fullsized":"2024-09-20T19:24:16.236827+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:46.300364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.300364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.891484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":22,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088984+0000","last_change":"2024-09-20T19:24:11.087136+0000","last_active":"2024-09-20T19:24:16.088984+0000","last_peered":"2024-09-20T19:24:16.088984+0000","last_clean":"2024-09-20T19:24:16.088984+0000","last_became_active":"2024-09-20T19:24:11.086665+0000","last_became_peered":"2024-09-20T19:24:11.086665+0000","last_unstale":"2024-09-20T19:24:16.088984+0000","last_undegraded":"2024-09-20T19:24:16.088984+0000","last_fullsized":"2024-09-20T19:24:16.088984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:11.366393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"259'178","reported_seq":935,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235080+0000","last_change":"2024-09-20T19:24:10.359266+0000","last_active":"2024-09-20T19:24:16.235080+0000","last_peered":"2024-09-20T19:24:16.235080+0000","last_clean":"2024-09-20T19:24:16.235080+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:24:16.235080+0000","last_undegraded":"2024-09-20T19:24:16.235080+0000","last_fullsized":"2024-09-20T19:24:16.235080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'148","last_scrub_stamp":"2024-09-20T19:23:58.044067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:58.044067+0000","objects_scrubbed":3,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:19.490998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.26867115800000002,"stat_sum":{"num_bytes":10118888,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":80726,"num_write":55,"num_write_kb":19533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.0","version":"77'18","reported_seq":459,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229984+0000","last_change":"2024-09-20T19:22:00.803489+0000","last_active":"2024-09-20T19:24:16.229984+0000","last_peered":"2024-09-20T19:24:16.229984+0000","last_clean":"2024-09-20T19:24:16.229984+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:24:16.229984+0000","last_undegraded":"2024-09-20T19:24:16.229984+0000","last_fullsized":"2024-09-20T19:24:16.229984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:22:00.803432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:00.803432+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.099791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":432,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.084534+0000","last_change":"2024-09-20T19:23:51.507352+0000","last_active":"2024-09-20T19:24:16.084534+0000","last_peered":"2024-09-20T19:24:16.084534+0000","last_clean":"2024-09-20T19:24:16.084534+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:24:16.084534+0000","last_undegraded":"2024-09-20T19:24:16.084534+0000","last_fullsized":"2024-09-20T19:24:16.084534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:51.507216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.507216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:03.128560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088212+0000","last_change":"2024-09-20T19:22:03.015704+0000","last_active":"2024-09-20T19:24:16.088212+0000","last_peered":"2024-09-20T19:24:16.088212+0000","last_clean":"2024-09-20T19:24:16.088212+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:24:16.088212+0000","last_undegraded":"2024-09-20T19:24:16.088212+0000","last_fullsized":"2024-09-20T19:24:16.088212+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:03.015585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:03.015585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.008629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088671+0000","last_change":"2024-09-20T19:23:48.185887+0000","last_active":"2024-09-20T19:24:16.088671+0000","last_peered":"2024-09-20T19:24:16.088671+0000","last_clean":"2024-09-20T19:24:16.088671+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:24:16.088671+0000","last_undegraded":"2024-09-20T19:24:16.088671+0000","last_fullsized":"2024-09-20T19:24:16.088671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:01.271009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":124,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237443+0000","last_change":"2024-09-20T19:24:11.099489+0000","last_active":"2024-09-20T19:24:16.237443+0000","last_peered":"2024-09-20T19:24:16.237443+0000","last_clean":"2024-09-20T19:24:16.237443+0000","last_became_active":"2024-09-20T19:24:11.099054+0000","last_became_peered":"2024-09-20T19:24:11.099054+0000","last_unstale":"2024-09-20T19:24:16.237443+0000","last_undegraded":"2024-09-20T19:24:16.237443+0000","last_fullsized":"2024-09-20T19:24:16.237443+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:13.610537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"261'209","reported_seq":991,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235127+0000","last_change":"2024-09-20T19:24:10.359644+0000","last_active":"2024-09-20T19:24:16.235127+0000","last_peered":"2024-09-20T19:24:16.235127+0000","last_clean":"2024-09-20T19:24:16.235127+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:24:16.235127+0000","last_undegraded":"2024-09-20T19:24:16.235127+0000","last_fullsized":"2024-09-20T19:24:16.235127+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'118","last_scrub_stamp":"2024-09-20T19:23:04.906510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.906510+0000","objects_scrubbed":10,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.873214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.27065835599999999,"stat_sum":{"num_bytes":21677856,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":141,"num_read_kb":124016,"num_write":88,"num_write_kb":27974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.3","version":"77'19","reported_seq":455,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230298+0000","last_change":"2024-09-20T19:21:07.855976+0000","last_active":"2024-09-20T19:24:16.230298+0000","last_peered":"2024-09-20T19:24:16.230298+0000","last_clean":"2024-09-20T19:24:16.230298+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:24:16.230298+0000","last_undegraded":"2024-09-20T19:24:16.230298+0000","last_fullsized":"2024-09-20T19:24:16.230298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:21:07.855828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:07.855828+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.733445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":368,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087055+0000","last_change":"2024-09-20T19:23:16.461133+0000","last_active":"2024-09-20T19:24:16.087055+0000","last_peered":"2024-09-20T19:24:16.087055+0000","last_clean":"2024-09-20T19:24:16.087055+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:24:16.087055+0000","last_undegraded":"2024-09-20T19:24:16.087055+0000","last_fullsized":"2024-09-20T19:24:16.087055+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:16.461005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:16.461005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.435064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236675+0000","last_change":"2024-09-20T19:23:59.649288+0000","last_active":"2024-09-20T19:24:16.236675+0000","last_peered":"2024-09-20T19:24:16.236675+0000","last_clean":"2024-09-20T19:24:16.236675+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:24:16.236675+0000","last_undegraded":"2024-09-20T19:24:16.236675+0000","last_fullsized":"2024-09-20T19:24:16.236675+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:59.649128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:59.649128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:17.346503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":403,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.082922+0000","last_change":"2024-09-20T19:19:59.552180+0000","last_active":"2024-09-20T19:24:16.082922+0000","last_peered":"2024-09-20T19:24:16.082922+0000","last_clean":"2024-09-20T19:24:16.082922+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:24:16.082922+0000","last_undegraded":"2024-09-20T19:24:16.082922+0000","last_fullsized":"2024-09-20T19:24:16.082922+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:59.552040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:59.552040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.495223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":124,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237762+0000","last_change":"2024-09-20T19:24:11.094519+0000","last_active":"2024-09-20T19:24:16.237762+0000","last_peered":"2024-09-20T19:24:16.237762+0000","last_clean":"2024-09-20T19:24:16.237762+0000","last_became_active":"2024-09-20T19:24:11.094059+0000","last_became_peered":"2024-09-20T19:24:11.094059+0000","last_unstale":"2024-09-20T19:24:16.237762+0000","last_undegraded":"2024-09-20T19:24:16.237762+0000","last_fullsized":"2024-09-20T19:24:16.237762+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:11.678831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"266'226","reported_seq":986,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089238+0000","last_change":"2024-09-20T19:24:10.080268+0000","last_active":"2024-09-20T19:24:16.089238+0000","last_peered":"2024-09-20T19:24:16.089238+0000","last_clean":"2024-09-20T19:24:16.089238+0000","last_became_active":"2024-09-20T19:21:30.244233+0000","last_became_peered":"2024-09-20T19:21:30.244233+0000","last_unstale":"2024-09-20T19:24:16.089238+0000","last_undegraded":"2024-09-20T19:24:16.089238+0000","last_fullsized":"2024-09-20T19:24:16.089238+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'93","last_scrub_stamp":"2024-09-20T19:23:14.039336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.039336+0000","objects_scrubbed":6,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.061036+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0043580229999999999,"stat_sum":{"num_bytes":13139206,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":86,"num_read_kb":53849,"num_write":82,"num_write_kb":26917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.2","version":"77'10","reported_seq":450,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083615+0000","last_change":"2024-09-20T19:23:54.537172+0000","last_active":"2024-09-20T19:24:16.083615+0000","last_peered":"2024-09-20T19:24:16.083615+0000","last_clean":"2024-09-20T19:24:16.083615+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:24:16.083615+0000","last_undegraded":"2024-09-20T19:24:16.083615+0000","last_fullsized":"2024-09-20T19:24:16.083615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:54.536975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.536975+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:18.498094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083952+0000","last_change":"2024-09-20T19:22:21.888862+0000","last_active":"2024-09-20T19:24:16.083952+0000","last_peered":"2024-09-20T19:24:16.083952+0000","last_clean":"2024-09-20T19:24:16.083952+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:24:16.083952+0000","last_undegraded":"2024-09-20T19:24:16.083952+0000","last_fullsized":"2024-09-20T19:24:16.083952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:36.511666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":499,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229914+0000","last_change":"2024-09-20T19:21:10.853855+0000","last_active":"2024-09-20T19:24:16.229914+0000","last_peered":"2024-09-20T19:24:16.229914+0000","last_clean":"2024-09-20T19:24:16.229914+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:24:16.229914+0000","last_undegraded":"2024-09-20T19:24:16.229914+0000","last_fullsized":"2024-09-20T19:24:16.229914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:21:10.853722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.853722+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.732391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229085+0000","last_change":"2024-09-20T19:22:56.797105+0000","last_active":"2024-09-20T19:24:16.229085+0000","last_peered":"2024-09-20T19:24:16.229085+0000","last_clean":"2024-09-20T19:24:16.229085+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:24:16.229085+0000","last_undegraded":"2024-09-20T19:24:16.229085+0000","last_fullsized":"2024-09-20T19:24:16.229085+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.054135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":22,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234466+0000","last_change":"2024-09-20T19:24:11.090619+0000","last_active":"2024-09-20T19:24:16.234466+0000","last_peered":"2024-09-20T19:24:16.234466+0000","last_clean":"2024-09-20T19:24:16.234466+0000","last_became_active":"2024-09-20T19:24:11.090208+0000","last_became_peered":"2024-09-20T19:24:11.090208+0000","last_unstale":"2024-09-20T19:24:16.234466+0000","last_undegraded":"2024-09-20T19:24:16.234466+0000","last_fullsized":"2024-09-20T19:24:16.234466+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:15.836410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"263'145","reported_seq":932,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238536+0000","last_change":"2024-09-20T19:24:14.111607+0000","last_active":"2024-09-20T19:24:16.238536+0000","last_peered":"2024-09-20T19:24:16.238536+0000","last_clean":"2024-09-20T19:24:16.238536+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:24:16.238536+0000","last_undegraded":"2024-09-20T19:24:16.238536+0000","last_fullsized":"2024-09-20T19:24:16.238536+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'119","last_scrub_stamp":"2024-09-20T19:23:46.636892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.636892+0000","objects_scrubbed":4,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:14.071391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014351123,"stat_sum":{"num_bytes":17889588,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":121,"num_read_kb":79717,"num_write":68,"num_write_kb":27911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.4","version":"77'30","reported_seq":551,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083904+0000","last_change":"2024-09-20T19:22:14.846750+0000","last_active":"2024-09-20T19:24:16.083904+0000","last_peered":"2024-09-20T19:24:16.083904+0000","last_clean":"2024-09-20T19:24:16.083904+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:24:16.083904+0000","last_undegraded":"2024-09-20T19:24:16.083904+0000","last_fullsized":"2024-09-20T19:24:16.083904+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:22:14.846671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:14.846671+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.459830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":429,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236422+0000","last_change":"2024-09-20T19:23:01.995350+0000","last_active":"2024-09-20T19:24:16.236422+0000","last_peered":"2024-09-20T19:24:16.236422+0000","last_clean":"2024-09-20T19:24:16.236422+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:24:16.236422+0000","last_undegraded":"2024-09-20T19:24:16.236422+0000","last_fullsized":"2024-09-20T19:24:16.236422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:01.995251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.995251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.855295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230670+0000","last_change":"2024-09-20T19:23:54.537232+0000","last_active":"2024-09-20T19:24:16.230670+0000","last_peered":"2024-09-20T19:24:16.230670+0000","last_clean":"2024-09-20T19:24:16.230670+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:24:16.230670+0000","last_undegraded":"2024-09-20T19:24:16.230670+0000","last_fullsized":"2024-09-20T19:24:16.230670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:54.537110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.537110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:23.441813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":418,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235025+0000","last_change":"2024-09-20T19:23:08.714718+0000","last_active":"2024-09-20T19:24:16.235025+0000","last_peered":"2024-09-20T19:24:16.235025+0000","last_clean":"2024-09-20T19:24:16.235025+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:24:16.235025+0000","last_undegraded":"2024-09-20T19:24:16.235025+0000","last_fullsized":"2024-09-20T19:24:16.235025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:08.714593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:08.714593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:34.378300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":484,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237997+0000","last_change":"2024-09-20T19:24:11.088472+0000","last_active":"2024-09-20T19:24:16.237997+0000","last_peered":"2024-09-20T19:24:16.237997+0000","last_clean":"2024-09-20T19:24:16.237997+0000","last_became_active":"2024-09-20T19:24:11.088125+0000","last_became_peered":"2024-09-20T19:24:11.088125+0000","last_unstale":"2024-09-20T19:24:16.237997+0000","last_undegraded":"2024-09-20T19:24:16.237997+0000","last_fullsized":"2024-09-20T19:24:16.237997+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:12.184865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":21,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"265'348","reported_seq":1106,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230165+0000","last_change":"2024-09-20T19:24:10.097756+0000","last_active":"2024-09-20T19:24:16.230165+0000","last_peered":"2024-09-20T19:24:16.230165+0000","last_clean":"2024-09-20T19:24:16.230165+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:24:16.230165+0000","last_undegraded":"2024-09-20T19:24:16.230165+0000","last_fullsized":"2024-09-20T19:24:16.230165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'86","last_scrub_stamp":"2024-09-20T19:22:51.812052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.812052+0000","objects_scrubbed":9,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005171681,"stat_sum":{"num_bytes":32748510,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":124,"num_read_kb":54547,"num_write":134,"num_write_kb":52562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.7","version":"77'13","reported_seq":372,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087432+0000","last_change":"2024-09-20T19:24:03.786688+0000","last_active":"2024-09-20T19:24:16.087432+0000","last_peered":"2024-09-20T19:24:16.087432+0000","last_clean":"2024-09-20T19:24:16.087432+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:24:16.087432+0000","last_undegraded":"2024-09-20T19:24:16.087432+0000","last_fullsized":"2024-09-20T19:24:16.087432+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:24:03.786567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.786567+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:24.540040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":785,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083146+0000","last_change":"2024-09-20T19:20:48.666744+0000","last_active":"2024-09-20T19:24:16.083146+0000","last_peered":"2024-09-20T19:24:16.083146+0000","last_clean":"2024-09-20T19:24:16.083146+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:24:16.083146+0000","last_undegraded":"2024-09-20T19:24:16.083146+0000","last_fullsized":"2024-09-20T19:24:16.083146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:20:48.666584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:48.666584+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.476879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":422,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.086721+0000","last_change":"2024-09-20T19:21:10.216852+0000","last_active":"2024-09-20T19:24:16.086721+0000","last_peered":"2024-09-20T19:24:16.086721+0000","last_clean":"2024-09-20T19:24:16.086721+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:24:16.086721+0000","last_undegraded":"2024-09-20T19:24:16.086721+0000","last_fullsized":"2024-09-20T19:24:16.086721+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:10.216700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:10.216700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.674411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":422,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087388+0000","last_change":"2024-09-20T19:23:47.565593+0000","last_active":"2024-09-20T19:24:16.087388+0000","last_peered":"2024-09-20T19:24:16.087388+0000","last_clean":"2024-09-20T19:24:16.087388+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:24:16.087388+0000","last_undegraded":"2024-09-20T19:24:16.087388+0000","last_fullsized":"2024-09-20T19:24:16.087388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.565537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.565537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:04.991103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":416,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.241290+0000","last_change":"2024-09-20T19:23:11.304539+0000","last_active":"2024-09-20T19:24:16.241290+0000","last_peered":"2024-09-20T19:24:16.241290+0000","last_clean":"2024-09-20T19:24:16.241290+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:24:16.241290+0000","last_undegraded":"2024-09-20T19:24:16.241290+0000","last_fullsized":"2024-09-20T19:24:16.241290+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.304478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.304478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":22,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083948+0000","last_change":"2024-09-20T19:24:11.091335+0000","last_active":"2024-09-20T19:24:16.083948+0000","last_peered":"2024-09-20T19:24:16.083948+0000","last_clean":"2024-09-20T19:24:16.083948+0000","last_became_active":"2024-09-20T19:24:11.090709+0000","last_became_peered":"2024-09-20T19:24:11.090709+0000","last_unstale":"2024-09-20T19:24:16.083948+0000","last_undegraded":"2024-09-20T19:24:16.083948+0000","last_fullsized":"2024-09-20T19:24:16.083948+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:13.298038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"266'332","reported_seq":1131,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.241855+0000","last_change":"2024-09-20T19:24:14.367835+0000","last_active":"2024-09-20T19:24:16.241855+0000","last_peered":"2024-09-20T19:24:16.241855+0000","last_clean":"2024-09-20T19:24:16.241855+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:24:16.241855+0000","last_undegraded":"2024-09-20T19:24:16.241855+0000","last_fullsized":"2024-09-20T19:24:16.241855+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'286","last_scrub_stamp":"2024-09-20T19:24:01.346351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:01.346351+0000","objects_scrubbed":10,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:03.017169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.26400115800000001,"stat_sum":{"num_bytes":30667880,"num_objects":16,"num_object_clones":9,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":244,"num_read_kb":167611,"num_write":150,"num_write_kb":60418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.6","version":"77'12","reported_seq":445,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231631+0000","last_change":"2024-09-20T19:22:16.548672+0000","last_active":"2024-09-20T19:24:16.231631+0000","last_peered":"2024-09-20T19:24:16.231631+0000","last_clean":"2024-09-20T19:24:16.231631+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:24:16.231631+0000","last_undegraded":"2024-09-20T19:24:16.231631+0000","last_fullsized":"2024-09-20T19:24:16.231631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:16.548570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:16.548570+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.801626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":429,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230277+0000","last_change":"2024-09-20T19:20:25.853565+0000","last_active":"2024-09-20T19:24:16.230277+0000","last_peered":"2024-09-20T19:24:16.230277+0000","last_clean":"2024-09-20T19:24:16.230277+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:24:16.230277+0000","last_undegraded":"2024-09-20T19:24:16.230277+0000","last_fullsized":"2024-09-20T19:24:16.230277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:20:25.853427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:25.853427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.744928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":429,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235815+0000","last_change":"2024-09-20T19:22:18.870386+0000","last_active":"2024-09-20T19:24:16.235815+0000","last_peered":"2024-09-20T19:24:16.235815+0000","last_clean":"2024-09-20T19:24:16.235815+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:24:16.235815+0000","last_undegraded":"2024-09-20T19:24:16.235815+0000","last_fullsized":"2024-09-20T19:24:16.235815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.870268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.870268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":421,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.241074+0000","last_change":"2024-09-20T19:22:56.906325+0000","last_active":"2024-09-20T19:24:16.241074+0000","last_peered":"2024-09-20T19:24:16.241074+0000","last_clean":"2024-09-20T19:24:16.241074+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:24:16.241074+0000","last_undegraded":"2024-09-20T19:24:16.241074+0000","last_fullsized":"2024-09-20T19:24:16.241074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.906255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.906255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.054482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083533+0000","last_change":"2024-09-20T19:22:06.926452+0000","last_active":"2024-09-20T19:24:16.083533+0000","last_peered":"2024-09-20T19:24:16.083533+0000","last_clean":"2024-09-20T19:24:16.083533+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:24:16.083533+0000","last_undegraded":"2024-09-20T19:24:16.083533+0000","last_fullsized":"2024-09-20T19:24:16.083533+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:06.926325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:06.926325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:33.522917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"263'105","reported_seq":875,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231194+0000","last_change":"2024-09-20T19:24:10.098936+0000","last_active":"2024-09-20T19:24:16.231194+0000","last_peered":"2024-09-20T19:24:16.231194+0000","last_clean":"2024-09-20T19:24:16.231194+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:24:16.231194+0000","last_undegraded":"2024-09-20T19:24:16.231194+0000","last_fullsized":"2024-09-20T19:24:16.231194+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'61","last_scrub_stamp":"2024-09-20T19:23:06.675677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:06.675677+0000","objects_scrubbed":4,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.089131+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015579540000000001,"stat_sum":{"num_bytes":10135621,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":62105,"num_write":42,"num_write_kb":17718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.5","version":"77'16","reported_seq":512,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239013+0000","last_change":"2024-09-20T19:21:12.740077+0000","last_active":"2024-09-20T19:24:16.239013+0000","last_peered":"2024-09-20T19:24:16.239013+0000","last_clean":"2024-09-20T19:24:16.239013+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:24:16.239013+0000","last_undegraded":"2024-09-20T19:24:16.239013+0000","last_fullsized":"2024-09-20T19:24:16.239013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:21:12.739915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:12.739915+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.628021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":421,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234239+0000","last_change":"2024-09-20T19:20:15.003594+0000","last_active":"2024-09-20T19:24:16.234239+0000","last_peered":"2024-09-20T19:24:16.234239+0000","last_clean":"2024-09-20T19:24:16.234239+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:24:16.234239+0000","last_undegraded":"2024-09-20T19:24:16.234239+0000","last_fullsized":"2024-09-20T19:24:16.234239+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:20:15.003448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:20:15.003448+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.812423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":421,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237113+0000","last_change":"2024-09-20T19:20:52.723249+0000","last_active":"2024-09-20T19:24:16.237113+0000","last_peered":"2024-09-20T19:24:16.237113+0000","last_clean":"2024-09-20T19:24:16.237113+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:24:16.237113+0000","last_undegraded":"2024-09-20T19:24:16.237113+0000","last_fullsized":"2024-09-20T19:24:16.237113+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:52.723121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.723121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:37.814578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":478,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236160+0000","last_change":"2024-09-20T19:24:09.094725+0000","last_active":"2024-09-20T19:24:16.236160+0000","last_peered":"2024-09-20T19:24:16.236160+0000","last_clean":"2024-09-20T19:24:16.236160+0000","last_became_active":"2024-09-20T19:24:09.093719+0000","last_became_peered":"2024-09-20T19:24:09.093719+0000","last_unstale":"2024-09-20T19:24:16.236160+0000","last_undegraded":"2024-09-20T19:24:16.236160+0000","last_fullsized":"2024-09-20T19:24:16.236160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:25:11.218092+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":412,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239540+0000","last_change":"2024-09-20T19:20:53.868159+0000","last_active":"2024-09-20T19:24:16.239540+0000","last_peered":"2024-09-20T19:24:16.239540+0000","last_clean":"2024-09-20T19:24:16.239540+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:24:16.239540+0000","last_undegraded":"2024-09-20T19:24:16.239540+0000","last_fullsized":"2024-09-20T19:24:16.239540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:53.868034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:20:53.868034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.287685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":417,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.241376+0000","last_change":"2024-09-20T19:23:49.765599+0000","last_active":"2024-09-20T19:24:16.241376+0000","last_peered":"2024-09-20T19:24:16.241376+0000","last_clean":"2024-09-20T19:24:16.241376+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:24:16.241376+0000","last_undegraded":"2024-09-20T19:24:16.241376+0000","last_fullsized":"2024-09-20T19:24:16.241376+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.765546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.765546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.098365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"265'163","reported_seq":930,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229607+0000","last_change":"2024-09-20T19:24:10.099519+0000","last_active":"2024-09-20T19:24:16.229607+0000","last_peered":"2024-09-20T19:24:16.229607+0000","last_clean":"2024-09-20T19:24:16.229607+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:24:16.229607+0000","last_undegraded":"2024-09-20T19:24:16.229607+0000","last_fullsized":"2024-09-20T19:24:16.229607+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'71","last_scrub_stamp":"2024-09-20T19:23:05.723582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.723582+0000","objects_scrubbed":6,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0067479300000000001,"stat_sum":{"num_bytes":18393848,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":79,"num_read_kb":68516,"num_write":58,"num_write_kb":25712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.a","version":"77'19","reported_seq":447,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237473+0000","last_change":"2024-09-20T19:20:52.718858+0000","last_active":"2024-09-20T19:24:16.237473+0000","last_peered":"2024-09-20T19:24:16.237473+0000","last_clean":"2024-09-20T19:24:16.237473+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:24:16.237473+0000","last_undegraded":"2024-09-20T19:24:16.237473+0000","last_fullsized":"2024-09-20T19:24:16.237473+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:20:52.718711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.718711+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.617546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":430,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087115+0000","last_change":"2024-09-20T19:22:11.289926+0000","last_active":"2024-09-20T19:24:16.087115+0000","last_peered":"2024-09-20T19:24:16.087115+0000","last_clean":"2024-09-20T19:24:16.087115+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:24:16.087115+0000","last_undegraded":"2024-09-20T19:24:16.087115+0000","last_fullsized":"2024-09-20T19:24:16.087115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.289877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.289877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:04.801284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":422,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.086566+0000","last_change":"2024-09-20T19:23:49.605277+0000","last_active":"2024-09-20T19:24:16.086566+0000","last_peered":"2024-09-20T19:24:16.086566+0000","last_clean":"2024-09-20T19:24:16.086566+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:24:16.086566+0000","last_undegraded":"2024-09-20T19:24:16.086566+0000","last_fullsized":"2024-09-20T19:24:16.086566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.605219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.605219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.883340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":408,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237499+0000","last_change":"2024-09-20T19:23:42.246466+0000","last_active":"2024-09-20T19:24:16.237499+0000","last_peered":"2024-09-20T19:24:16.237499+0000","last_clean":"2024-09-20T19:24:16.237499+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:24:16.237499+0000","last_undegraded":"2024-09-20T19:24:16.237499+0000","last_fullsized":"2024-09-20T19:24:16.237499+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.246406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.246406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:48.009612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"263'328","reported_seq":1080,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.081647+0000","last_change":"2024-09-20T19:24:10.071931+0000","last_active":"2024-09-20T19:24:16.081647+0000","last_peered":"2024-09-20T19:24:16.081647+0000","last_clean":"2024-09-20T19:24:16.081647+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:24:16.081647+0000","last_undegraded":"2024-09-20T19:24:16.081647+0000","last_fullsized":"2024-09-20T19:24:16.081647+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'112","last_scrub_stamp":"2024-09-20T19:22:52.771031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:22:52.771031+0000","objects_scrubbed":11,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002236151,"stat_sum":{"num_bytes":19109559,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":84645,"num_write":107,"num_write_kb":47968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.b","version":"77'9","reported_seq":443,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230116+0000","last_change":"2024-09-20T19:22:55.847371+0000","last_active":"2024-09-20T19:24:16.230116+0000","last_peered":"2024-09-20T19:24:16.230116+0000","last_clean":"2024-09-20T19:24:16.230116+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:24:16.230116+0000","last_undegraded":"2024-09-20T19:24:16.230116+0000","last_fullsized":"2024-09-20T19:24:16.230116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:55.847304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:55.847304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:15.696283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.084504+0000","last_change":"2024-09-20T19:21:05.740388+0000","last_active":"2024-09-20T19:24:16.084504+0000","last_peered":"2024-09-20T19:24:16.084504+0000","last_clean":"2024-09-20T19:24:16.084504+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:24:16.084504+0000","last_undegraded":"2024-09-20T19:24:16.084504+0000","last_fullsized":"2024-09-20T19:24:16.084504+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.740278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.740278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:29.483483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":414,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.086609+0000","last_change":"2024-09-20T19:21:17.226835+0000","last_active":"2024-09-20T19:24:16.086609+0000","last_peered":"2024-09-20T19:24:16.086609+0000","last_clean":"2024-09-20T19:24:16.086609+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:24:16.086609+0000","last_undegraded":"2024-09-20T19:24:16.086609+0000","last_fullsized":"2024-09-20T19:24:16.086609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:17.226714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:17.226714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.520900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229913+0000","last_change":"2024-09-20T19:23:44.721606+0000","last_active":"2024-09-20T19:24:16.229913+0000","last_peered":"2024-09-20T19:24:16.229913+0000","last_clean":"2024-09-20T19:24:16.229913+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:24:16.229913+0000","last_undegraded":"2024-09-20T19:24:16.229913+0000","last_fullsized":"2024-09-20T19:24:16.229913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:23:44.721438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.721438+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:59.778487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"265'65","reported_seq":831,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083377+0000","last_change":"2024-09-20T19:24:10.072989+0000","last_active":"2024-09-20T19:24:16.083377+0000","last_peered":"2024-09-20T19:24:16.083377+0000","last_clean":"2024-09-20T19:24:16.083377+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:24:16.083377+0000","last_undegraded":"2024-09-20T19:24:16.083377+0000","last_fullsized":"2024-09-20T19:24:16.083377+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'35","last_scrub_stamp":"2024-09-20T19:23:10.333319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.333319+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001901624,"stat_sum":{"num_bytes":5277398,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":26989,"num_write":25,"num_write_kb":12667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.8","version":"77'15","reported_seq":450,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236632+0000","last_change":"2024-09-20T19:22:22.063998+0000","last_active":"2024-09-20T19:24:16.236632+0000","last_peered":"2024-09-20T19:24:16.236632+0000","last_clean":"2024-09-20T19:24:16.236632+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:24:16.236632+0000","last_undegraded":"2024-09-20T19:24:16.236632+0000","last_fullsized":"2024-09-20T19:24:16.236632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:22.063871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.063871+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:04.860933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":438,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.087555+0000","last_change":"2024-09-20T19:22:02.235649+0000","last_active":"2024-09-20T19:24:16.087555+0000","last_peered":"2024-09-20T19:24:16.087555+0000","last_clean":"2024-09-20T19:24:16.087555+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:24:16.087555+0000","last_undegraded":"2024-09-20T19:24:16.087555+0000","last_fullsized":"2024-09-20T19:24:16.087555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:02.235598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:02.235598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:24.519095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":504,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236343+0000","last_change":"2024-09-20T19:23:01.188032+0000","last_active":"2024-09-20T19:24:16.236343+0000","last_peered":"2024-09-20T19:24:16.236343+0000","last_clean":"2024-09-20T19:24:16.236343+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:24:16.236343+0000","last_undegraded":"2024-09-20T19:24:16.236343+0000","last_fullsized":"2024-09-20T19:24:16.236343+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:01.187968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:01.187968+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":409,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235264+0000","last_change":"2024-09-20T19:21:01.739612+0000","last_active":"2024-09-20T19:24:16.235264+0000","last_peered":"2024-09-20T19:24:16.235264+0000","last_clean":"2024-09-20T19:24:16.235264+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:24:16.235264+0000","last_undegraded":"2024-09-20T19:24:16.235264+0000","last_fullsized":"2024-09-20T19:24:16.235264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:01.739492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:01.739492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.006870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"263'155","reported_seq":914,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.232408+0000","last_change":"2024-09-20T19:24:10.365716+0000","last_active":"2024-09-20T19:24:16.232408+0000","last_peered":"2024-09-20T19:24:16.232408+0000","last_clean":"2024-09-20T19:24:16.232408+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:24:16.232408+0000","last_undegraded":"2024-09-20T19:24:16.232408+0000","last_fullsized":"2024-09-20T19:24:16.232408+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'137","last_scrub_stamp":"2024-09-20T19:24:02.123477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:02.123477+0000","objects_scrubbed":6,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:30.219217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.27869142200000002,"stat_sum":{"num_bytes":15873494,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":77,"num_read_kb":57561,"num_write":53,"num_write_kb":25659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"2"},{"start":"4a","length":"1"}]},{"pgid":"4.9","version":"77'12","reported_seq":469,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088712+0000","last_change":"2024-09-20T19:22:57.202081+0000","last_active":"2024-09-20T19:24:16.088712+0000","last_peered":"2024-09-20T19:24:16.088712+0000","last_clean":"2024-09-20T19:24:16.088712+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:24:16.088712+0000","last_undegraded":"2024-09-20T19:24:16.088712+0000","last_fullsized":"2024-09-20T19:24:16.088712+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:57.201945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:57.201945+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:25.217023+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":447,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.090668+0000","last_change":"2024-09-20T19:22:59.196366+0000","last_active":"2024-09-20T19:24:16.090668+0000","last_peered":"2024-09-20T19:24:16.090668+0000","last_clean":"2024-09-20T19:24:16.090668+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:24:16.090668+0000","last_undegraded":"2024-09-20T19:24:16.090668+0000","last_fullsized":"2024-09-20T19:24:16.090668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:22:59.196235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:59.196235+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:15.330093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":422,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.081783+0000","last_change":"2024-09-20T19:22:18.260176+0000","last_active":"2024-09-20T19:24:16.081783+0000","last_peered":"2024-09-20T19:24:16.081783+0000","last_clean":"2024-09-20T19:24:16.081783+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:24:16.081783+0000","last_undegraded":"2024-09-20T19:24:16.081783+0000","last_fullsized":"2024-09-20T19:24:16.081783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:18.260046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:18.260046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.501604+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":410,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234295+0000","last_change":"2024-09-20T19:23:00.687733+0000","last_active":"2024-09-20T19:24:16.234295+0000","last_peered":"2024-09-20T19:24:16.234295+0000","last_clean":"2024-09-20T19:24:16.234295+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:24:16.234295+0000","last_undegraded":"2024-09-20T19:24:16.234295+0000","last_fullsized":"2024-09-20T19:24:16.234295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:00.687658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:00.687658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:30.540716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":443,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231578+0000","last_change":"2024-09-20T19:22:58.800944+0000","last_active":"2024-09-20T19:24:16.231578+0000","last_peered":"2024-09-20T19:24:16.231578+0000","last_clean":"2024-09-20T19:24:16.231578+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:24:16.231578+0000","last_undegraded":"2024-09-20T19:24:16.231578+0000","last_fullsized":"2024-09-20T19:24:16.231578+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:58.800874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.800874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":438,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.081916+0000","last_change":"2024-09-20T19:23:07.609407+0000","last_active":"2024-09-20T19:24:16.081916+0000","last_peered":"2024-09-20T19:24:16.081916+0000","last_clean":"2024-09-20T19:24:16.081916+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:24:16.081916+0000","last_undegraded":"2024-09-20T19:24:16.081916+0000","last_fullsized":"2024-09-20T19:24:16.081916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:07.609283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:07.609283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:17.108303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":414,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235646+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:24:16.235646+0000","last_peered":"2024-09-20T19:24:16.235646+0000","last_clean":"2024-09-20T19:24:16.235646+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:24:16.235646+0000","last_undegraded":"2024-09-20T19:24:16.235646+0000","last_fullsized":"2024-09-20T19:24:16.235646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:35.796722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":418,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.086615+0000","last_change":"2024-09-20T19:22:56.840530+0000","last_active":"2024-09-20T19:24:16.086615+0000","last_peered":"2024-09-20T19:24:16.086615+0000","last_clean":"2024-09-20T19:24:16.086615+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:24:16.086615+0000","last_undegraded":"2024-09-20T19:24:16.086615+0000","last_fullsized":"2024-09-20T19:24:16.086615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.840403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.840403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237344+0000","last_change":"2024-09-20T19:23:02.640412+0000","last_active":"2024-09-20T19:24:16.237344+0000","last_peered":"2024-09-20T19:24:16.237344+0000","last_clean":"2024-09-20T19:24:16.237344+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:24:16.237344+0000","last_undegraded":"2024-09-20T19:24:16.237344+0000","last_fullsized":"2024-09-20T19:24:16.237344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:02.640270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.640270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:21.542638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":427,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235792+0000","last_change":"2024-09-20T19:20:47.001496+0000","last_active":"2024-09-20T19:24:16.235792+0000","last_peered":"2024-09-20T19:24:16.235792+0000","last_clean":"2024-09-20T19:24:16.235792+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:24:16.235792+0000","last_undegraded":"2024-09-20T19:24:16.235792+0000","last_fullsized":"2024-09-20T19:24:16.235792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:20:47.001319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:47.001319+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:27.710208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":421,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240376+0000","last_change":"2024-09-20T19:23:50.759135+0000","last_active":"2024-09-20T19:24:16.240376+0000","last_peered":"2024-09-20T19:24:16.240376+0000","last_clean":"2024-09-20T19:24:16.240376+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:24:16.240376+0000","last_undegraded":"2024-09-20T19:24:16.240376+0000","last_fullsized":"2024-09-20T19:24:16.240376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.808215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":408,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240957+0000","last_change":"2024-09-20T19:21:29.100795+0000","last_active":"2024-09-20T19:24:16.240957+0000","last_peered":"2024-09-20T19:24:16.240957+0000","last_clean":"2024-09-20T19:24:16.240957+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:24:16.240957+0000","last_undegraded":"2024-09-20T19:24:16.240957+0000","last_fullsized":"2024-09-20T19:24:16.240957+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:29.100718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:21:29.100718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:23.284037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":441,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235659+0000","last_change":"2024-09-20T19:23:50.024939+0000","last_active":"2024-09-20T19:24:16.235659+0000","last_peered":"2024-09-20T19:24:16.235659+0000","last_clean":"2024-09-20T19:24:16.235659+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:24:16.235659+0000","last_undegraded":"2024-09-20T19:24:16.235659+0000","last_fullsized":"2024-09-20T19:24:16.235659+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:50.024860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.024860+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:14.939068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":437,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235974+0000","last_change":"2024-09-20T19:23:56.858288+0000","last_active":"2024-09-20T19:24:16.235974+0000","last_peered":"2024-09-20T19:24:16.235974+0000","last_clean":"2024-09-20T19:24:16.235974+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:24:16.235974+0000","last_undegraded":"2024-09-20T19:24:16.235974+0000","last_fullsized":"2024-09-20T19:24:16.235974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:56.858227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.858227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:16.206316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":513,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240064+0000","last_change":"2024-09-20T19:23:53.824722+0000","last_active":"2024-09-20T19:24:16.240064+0000","last_peered":"2024-09-20T19:24:16.240064+0000","last_clean":"2024-09-20T19:24:16.240064+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:24:16.240064+0000","last_undegraded":"2024-09-20T19:24:16.240064+0000","last_fullsized":"2024-09-20T19:24:16.240064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:53.824647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:53.824647+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.400274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":427,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230923+0000","last_change":"2024-09-20T19:23:15.737592+0000","last_active":"2024-09-20T19:24:16.230923+0000","last_peered":"2024-09-20T19:24:16.230923+0000","last_clean":"2024-09-20T19:24:16.230923+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:24:16.230923+0000","last_undegraded":"2024-09-20T19:24:16.230923+0000","last_fullsized":"2024-09-20T19:24:16.230923+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:15.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:15.737537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.525979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":441,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240329+0000","last_change":"2024-09-20T19:22:54.923862+0000","last_active":"2024-09-20T19:24:16.240329+0000","last_peered":"2024-09-20T19:24:16.240329+0000","last_clean":"2024-09-20T19:24:16.240329+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:24:16.240329+0000","last_undegraded":"2024-09-20T19:24:16.240329+0000","last_fullsized":"2024-09-20T19:24:16.240329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:54.923725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.923725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.841757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230371+0000","last_change":"2024-09-20T19:22:20.671257+0000","last_active":"2024-09-20T19:24:16.230371+0000","last_peered":"2024-09-20T19:24:16.230371+0000","last_clean":"2024-09-20T19:24:16.230371+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:24:16.230371+0000","last_undegraded":"2024-09-20T19:24:16.230371+0000","last_fullsized":"2024-09-20T19:24:16.230371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.671196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.671196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:04.731374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":498,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234908+0000","last_change":"2024-09-20T19:22:53.660006+0000","last_active":"2024-09-20T19:24:16.234908+0000","last_peered":"2024-09-20T19:24:16.234908+0000","last_clean":"2024-09-20T19:24:16.234908+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:24:16.234908+0000","last_undegraded":"2024-09-20T19:24:16.234908+0000","last_fullsized":"2024-09-20T19:24:16.234908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:53.659878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.659878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.090234+0000","last_change":"2024-09-20T19:22:11.914350+0000","last_active":"2024-09-20T19:24:16.090234+0000","last_peered":"2024-09-20T19:24:16.090234+0000","last_clean":"2024-09-20T19:24:16.090234+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:24:16.090234+0000","last_undegraded":"2024-09-20T19:24:16.090234+0000","last_fullsized":"2024-09-20T19:24:16.090234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.914224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.914224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:39.201274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":435,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083516+0000","last_change":"2024-09-20T19:20:19.567151+0000","last_active":"2024-09-20T19:24:16.083516+0000","last_peered":"2024-09-20T19:24:16.083516+0000","last_clean":"2024-09-20T19:24:16.083516+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:24:16.083516+0000","last_undegraded":"2024-09-20T19:24:16.083516+0000","last_fullsized":"2024-09-20T19:24:16.083516+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:20:19.567010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:19.567010+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.610209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238782+0000","last_change":"2024-09-20T19:20:28.700102+0000","last_active":"2024-09-20T19:24:16.238782+0000","last_peered":"2024-09-20T19:24:16.238782+0000","last_clean":"2024-09-20T19:24:16.238782+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:24:16.238782+0000","last_undegraded":"2024-09-20T19:24:16.238782+0000","last_fullsized":"2024-09-20T19:24:16.238782+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:28.699924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:20:28.699924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.684147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":414,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234311+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:24:16.234311+0000","last_peered":"2024-09-20T19:24:16.234311+0000","last_clean":"2024-09-20T19:24:16.234311+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:24:16.234311+0000","last_undegraded":"2024-09-20T19:24:16.234311+0000","last_fullsized":"2024-09-20T19:24:16.234311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:38.835899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.228905+0000","last_change":"2024-09-20T19:23:43.752868+0000","last_active":"2024-09-20T19:24:16.228905+0000","last_peered":"2024-09-20T19:24:16.228905+0000","last_clean":"2024-09-20T19:24:16.228905+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:24:16.228905+0000","last_undegraded":"2024-09-20T19:24:16.228905+0000","last_fullsized":"2024-09-20T19:24:16.228905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:43.752736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:43.752736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:57.607806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":438,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.088802+0000","last_change":"2024-09-20T19:20:52.914253+0000","last_active":"2024-09-20T19:24:16.088802+0000","last_peered":"2024-09-20T19:24:16.088802+0000","last_clean":"2024-09-20T19:24:16.088802+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:24:16.088802+0000","last_undegraded":"2024-09-20T19:24:16.088802+0000","last_fullsized":"2024-09-20T19:24:16.088802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:20:52.914121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:20:52.914121+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.086622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":439,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.084424+0000","last_change":"2024-09-20T19:22:53.163608+0000","last_active":"2024-09-20T19:24:16.084424+0000","last_peered":"2024-09-20T19:24:16.084424+0000","last_clean":"2024-09-20T19:24:16.084424+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:24:16.084424+0000","last_undegraded":"2024-09-20T19:24:16.084424+0000","last_fullsized":"2024-09-20T19:24:16.084424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:22:53.163555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:53.163555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:15.366054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083355+0000","last_change":"2024-09-20T19:23:44.572528+0000","last_active":"2024-09-20T19:24:16.083355+0000","last_peered":"2024-09-20T19:24:16.083355+0000","last_clean":"2024-09-20T19:24:16.083355+0000","last_became_active":"2024-09-20T19:18:11.306393+0000","last_became_peered":"2024-09-20T19:18:11.306393+0000","last_unstale":"2024-09-20T19:24:16.083355+0000","last_undegraded":"2024-09-20T19:24:16.083355+0000","last_fullsized":"2024-09-20T19:24:16.083355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:44.572407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.572407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.284846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":410,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230194+0000","last_change":"2024-09-20T19:23:47.003723+0000","last_active":"2024-09-20T19:24:16.230194+0000","last_peered":"2024-09-20T19:24:16.230194+0000","last_clean":"2024-09-20T19:24:16.230194+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:24:16.230194+0000","last_undegraded":"2024-09-20T19:24:16.230194+0000","last_fullsized":"2024-09-20T19:24:16.230194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.003602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.003602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.662168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":432,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.229939+0000","last_change":"2024-09-20T19:22:50.638425+0000","last_active":"2024-09-20T19:24:16.229939+0000","last_peered":"2024-09-20T19:24:16.229939+0000","last_clean":"2024-09-20T19:24:16.229939+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:24:16.229939+0000","last_undegraded":"2024-09-20T19:24:16.229939+0000","last_fullsized":"2024-09-20T19:24:16.229939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:22:50.638257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:50.638257+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:19.137027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":429,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.240611+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:24:16.240611+0000","last_peered":"2024-09-20T19:24:16.240611+0000","last_clean":"2024-09-20T19:24:16.240611+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:24:16.240611+0000","last_undegraded":"2024-09-20T19:24:16.240611+0000","last_fullsized":"2024-09-20T19:24:16.240611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:06.686224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.236352+0000","last_change":"2024-09-20T19:23:49.643862+0000","last_active":"2024-09-20T19:24:16.236352+0000","last_peered":"2024-09-20T19:24:16.236352+0000","last_clean":"2024-09-20T19:24:16.236352+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:24:16.236352+0000","last_undegraded":"2024-09-20T19:24:16.236352+0000","last_fullsized":"2024-09-20T19:24:16.236352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.643746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.643746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:05.482157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":400,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.237091+0000","last_change":"2024-09-20T19:19:54.255225+0000","last_active":"2024-09-20T19:24:16.237091+0000","last_peered":"2024-09-20T19:24:16.237091+0000","last_clean":"2024-09-20T19:24:16.237091+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:24:16.237091+0000","last_undegraded":"2024-09-20T19:24:16.237091+0000","last_fullsized":"2024-09-20T19:24:16.237091+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:19:54.255108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:19:54.255108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:40.264712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":445,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.230724+0000","last_change":"2024-09-20T19:22:08.939934+0000","last_active":"2024-09-20T19:24:16.230724+0000","last_peered":"2024-09-20T19:24:16.230724+0000","last_clean":"2024-09-20T19:24:16.230724+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:24:16.230724+0000","last_undegraded":"2024-09-20T19:24:16.230724+0000","last_fullsized":"2024-09-20T19:24:16.230724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.939842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.939842+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:10.009099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":428,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239197+0000","last_change":"2024-09-20T19:22:51.781731+0000","last_active":"2024-09-20T19:24:16.239197+0000","last_peered":"2024-09-20T19:24:16.239197+0000","last_clean":"2024-09-20T19:24:16.239197+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:24:16.239197+0000","last_undegraded":"2024-09-20T19:24:16.239197+0000","last_fullsized":"2024-09-20T19:24:16.239197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:51.781680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:51.781680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234718+0000","last_change":"2024-09-20T19:23:14.325927+0000","last_active":"2024-09-20T19:24:16.234718+0000","last_peered":"2024-09-20T19:24:16.234718+0000","last_clean":"2024-09-20T19:24:16.234718+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:24:16.234718+0000","last_undegraded":"2024-09-20T19:24:16.234718+0000","last_fullsized":"2024-09-20T19:24:16.234718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:14.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:14.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:20.929643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":418,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231240+0000","last_change":"2024-09-20T19:22:11.898382+0000","last_active":"2024-09-20T19:24:16.231240+0000","last_peered":"2024-09-20T19:24:16.231240+0000","last_clean":"2024-09-20T19:24:16.231240+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:24:16.231240+0000","last_undegraded":"2024-09-20T19:24:16.231240+0000","last_fullsized":"2024-09-20T19:24:16.231240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.898255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.898255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:16.064123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":410,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239224+0000","last_change":"2024-09-20T19:23:47.281254+0000","last_active":"2024-09-20T19:24:16.239224+0000","last_peered":"2024-09-20T19:24:16.239224+0000","last_clean":"2024-09-20T19:24:16.239224+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:24:16.239224+0000","last_undegraded":"2024-09-20T19:24:16.239224+0000","last_fullsized":"2024-09-20T19:24:16.239224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:23:47.281195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.281195+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:08.244407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":437,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235845+0000","last_change":"2024-09-20T19:23:11.005669+0000","last_active":"2024-09-20T19:24:16.235845+0000","last_peered":"2024-09-20T19:24:16.235845+0000","last_clean":"2024-09-20T19:24:16.235845+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:24:16.235845+0000","last_undegraded":"2024-09-20T19:24:16.235845+0000","last_fullsized":"2024-09-20T19:24:16.235845+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.005587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.005587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:28.106114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":443,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231509+0000","last_change":"2024-09-20T19:23:10.711312+0000","last_active":"2024-09-20T19:24:16.231509+0000","last_peered":"2024-09-20T19:24:16.231509+0000","last_clean":"2024-09-20T19:24:16.231509+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:24:16.231509+0000","last_undegraded":"2024-09-20T19:24:16.231509+0000","last_fullsized":"2024-09-20T19:24:16.231509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:10.711138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:10.711138+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":515,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238834+0000","last_change":"2024-09-20T19:23:51.659148+0000","last_active":"2024-09-20T19:24:16.238834+0000","last_peered":"2024-09-20T19:24:16.238834+0000","last_clean":"2024-09-20T19:24:16.238834+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:24:16.238834+0000","last_undegraded":"2024-09-20T19:24:16.238834+0000","last_fullsized":"2024-09-20T19:24:16.238834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:51.659044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.659044+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:12.110176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":419,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.083930+0000","last_change":"2024-09-20T19:23:05.361065+0000","last_active":"2024-09-20T19:24:16.083930+0000","last_peered":"2024-09-20T19:24:16.083930+0000","last_clean":"2024-09-20T19:24:16.083930+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:24:16.083930+0000","last_undegraded":"2024-09-20T19:24:16.083930+0000","last_fullsized":"2024-09-20T19:24:16.083930+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:05.360941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:05.360941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:18.569089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231914+0000","last_change":"2024-09-20T19:23:03.692672+0000","last_active":"2024-09-20T19:24:16.231914+0000","last_peered":"2024-09-20T19:24:16.231914+0000","last_clean":"2024-09-20T19:24:16.231914+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:24:16.231914+0000","last_undegraded":"2024-09-20T19:24:16.231914+0000","last_fullsized":"2024-09-20T19:24:16.231914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:03.692537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.692537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":470,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.239083+0000","last_change":"2024-09-20T19:24:03.690277+0000","last_active":"2024-09-20T19:24:16.239083+0000","last_peered":"2024-09-20T19:24:16.239083+0000","last_clean":"2024-09-20T19:24:16.239083+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:24:16.239083+0000","last_undegraded":"2024-09-20T19:24:16.239083+0000","last_fullsized":"2024-09-20T19:24:16.239083+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:03.690144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.690144+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:19.793418+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":424,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.231401+0000","last_change":"2024-09-20T19:23:49.605522+0000","last_active":"2024-09-20T19:24:16.231401+0000","last_peered":"2024-09-20T19:24:16.231401+0000","last_clean":"2024-09-20T19:24:16.231401+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:24:16.231401+0000","last_undegraded":"2024-09-20T19:24:16.231401+0000","last_fullsized":"2024-09-20T19:24:16.231401+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.605402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.605402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:03.424112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":412,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.090026+0000","last_change":"2024-09-20T19:20:59.816653+0000","last_active":"2024-09-20T19:24:16.090026+0000","last_peered":"2024-09-20T19:24:16.090026+0000","last_clean":"2024-09-20T19:24:16.090026+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:24:16.090026+0000","last_undegraded":"2024-09-20T19:24:16.090026+0000","last_fullsized":"2024-09-20T19:24:16.090026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:59.816510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:59.816510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.141868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":436,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.238045+0000","last_change":"2024-09-20T19:23:52.648685+0000","last_active":"2024-09-20T19:24:16.238045+0000","last_peered":"2024-09-20T19:24:16.238045+0000","last_clean":"2024-09-20T19:24:16.238045+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:24:16.238045+0000","last_undegraded":"2024-09-20T19:24:16.238045+0000","last_fullsized":"2024-09-20T19:24:16.238045+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.648578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.648578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:20.151508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":451,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.086945+0000","last_change":"2024-09-20T19:23:56.744113+0000","last_active":"2024-09-20T19:24:16.086945+0000","last_peered":"2024-09-20T19:24:16.086945+0000","last_clean":"2024-09-20T19:24:16.086945+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:24:16.086945+0000","last_undegraded":"2024-09-20T19:24:16.086945+0000","last_fullsized":"2024-09-20T19:24:16.086945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:56.744020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.744020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:00.330507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":431,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.084237+0000","last_change":"2024-09-20T19:23:57.510594+0000","last_active":"2024-09-20T19:24:16.084237+0000","last_peered":"2024-09-20T19:24:16.084237+0000","last_clean":"2024-09-20T19:24:16.084237+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:24:16.084237+0000","last_undegraded":"2024-09-20T19:24:16.084237+0000","last_fullsized":"2024-09-20T19:24:16.084237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:57.510542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:57.510542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:25:22.607363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":402,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.235564+0000","last_change":"2024-09-20T19:20:07.131659+0000","last_active":"2024-09-20T19:24:16.235564+0000","last_peered":"2024-09-20T19:24:16.235564+0000","last_clean":"2024-09-20T19:24:16.235564+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:24:16.235564+0000","last_undegraded":"2024-09-20T19:24:16.235564+0000","last_fullsized":"2024-09-20T19:24:16.235564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:20:07.131537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:20:07.131537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:41.890171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":422,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234644+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:24:16.234644+0000","last_peered":"2024-09-20T19:24:16.234644+0000","last_clean":"2024-09-20T19:24:16.234644+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:24:16.234644+0000","last_undegraded":"2024-09-20T19:24:16.234644+0000","last_fullsized":"2024-09-20T19:24:16.234644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:32.730587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":441,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.234620+0000","last_change":"2024-09-20T19:21:24.033642+0000","last_active":"2024-09-20T19:24:16.234620+0000","last_peered":"2024-09-20T19:24:16.234620+0000","last_clean":"2024-09-20T19:24:16.234620+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:24:16.234620+0000","last_undegraded":"2024-09-20T19:24:16.234620+0000","last_fullsized":"2024-09-20T19:24:16.234620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:21:24.033570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:21:24.033570+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:22.742580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":415,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-20T19:24:16.089631+0000","last_change":"2024-09-20T19:21:08.918562+0000","last_active":"2024-09-20T19:24:16.089631+0000","last_peered":"2024-09-20T19:24:16.089631+0000","last_clean":"2024-09-20T19:24:16.089631+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:24:16.089631+0000","last_undegraded":"2024-09-20T19:24:16.089631+0000","last_fullsized":"2024-09-20T19:24:16.089631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:08.918443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:08.918443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:24:31.121645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":248431803,"num_objects":115,"num_object_clones":68,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":8,"num_read":1554,"num_read_kb":1144287,"num_write":1097,"num_write_kb":420358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575410176,"data_stored":780999246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2869,"ondisk_log_size":2869,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":223,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907027,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109788,"kb_used_data":74436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93635364,"statfs":{"total":95995035648,"available":95882612736,"internally_reserved":0,"allocated":76222464,"data_stored":102602366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234078,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83996,"kb_used_data":52808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93661156,"statfs":{"total":95995035648,"available":95909023744,"internally_reserved":0,"allocated":54075392,"data_stored":82539833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561128,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68216,"kb_used_data":37028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93676936,"statfs":{"total":95995035648,"available":95925182464,"internally_reserved":0,"allocated":37916672,"data_stored":42422324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855476,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113316,"kb_used_data":73808,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93631836,"statfs":{"total":95995035648,"available":95879000064,"internally_reserved":0,"allocated":75579392,"data_stored":109288755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":133576,"kb_used_data":98228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93611576,"statfs":{"total":95995035648,"available":95858253824,"internally_reserved":0,"allocated":100585472,"data_stored":133932571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378764,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107924,"kb_used_data":64256,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93637228,"statfs":{"total":95995035648,"available":95884521472,"internally_reserved":0,"allocated":65798144,"data_stored":85031311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106984,"kb_used_data":71636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93638168,"statfs":{"total":95995035648,"available":95885484032,"internally_reserved":0,"allocated":73355264,"data_stored":92253201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139012,"kb_used_data":111980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93606140,"statfs":{"total":95995035648,"available":95852687360,"internally_reserved":0,"allocated":114667520,"data_stored":146499896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112037888,"data_stored":145023835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70717440,"data_stored":90776498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62906368,"data_stored":83292547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97357824,"data_stored":131873493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":72953856,"data_stored":107813546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34705408,"data_stored":40355734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":51462144,"data_stored":81064702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73269248,"data_stored":100798891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:24:22.107 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-20T19:24:22.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:737: done (4 left) 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:738: read oid 45 snap -1 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:738: expect (ObjNum 197 snap 59 seq_num 197) 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:739: rollback oid 17 current snap is 82 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 73 2024-09-20T19:24:22.171 INFO:tasks.rados.rados.0.smithi042.stdout:732: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:24:22.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:22 smithi042 bash[32075]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:22 smithi042 bash[32075]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:22 smithi042 bash[22706]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:22 smithi042 bash[22706]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:22.209 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:22.244 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 100652 2024-09-20T19:24:22.273 INFO:tasks.rados.rados.0.smithi042.stdout:733: finishing write tid 1 to smithi04268048-11 2024-09-20T19:24:22.273 INFO:tasks.rados.rados.0.smithi042.stdout:733: finishing write tid 2 to smithi04268048-11 2024-09-20T19:24:22.273 INFO:tasks.rados.rados.0.smithi042.stdout:733: finishing write tid 3 to smithi04268048-11 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:733: finishing write tid 4 to smithi04268048-11 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:733: oid 11 updating version 0 to 346 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:733: oid 11 updating version 346 to 347 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:733: oid 11 updating version 347 to 348 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 348 (ObjNum 268 snap 82 seq_num 268) dirty exists 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:733: left oid 11 (ObjNum 268 snap 82 seq_num 268) 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: finishing write tid 1 to smithi04268048-13 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: finishing write tid 2 to smithi04268048-13 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: finishing write tid 3 to smithi04268048-13 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: finishing write tid 4 to smithi04268048-13 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: oid 13 updating version 0 to 303 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: oid 13 version 303 is already newer than 302 2024-09-20T19:24:22.274 INFO:tasks.rados.rados.0.smithi042.stdout:735: oid 13 version 303 is already newer than 301 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 303 (ObjNum 269 snap 82 seq_num 269) dirty exists 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:735: left oid 13 (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: finishing write tid 1 to smithi04268048-1 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: finishing write tid 2 to smithi04268048-1 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: finishing write tid 3 to smithi04268048-1 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: finishing write tid 4 to smithi04268048-1 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: oid 1 updating version 0 to 303 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: oid 1 updating version 303 to 304 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: oid 1 updating version 304 to 305 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 305 (ObjNum 270 snap 82 seq_num 270) dirty exists 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:736: left oid 1 (ObjNum 270 snap 82 seq_num 270) 2024-09-20T19:24:22.275 INFO:tasks.rados.rados.0.smithi042.stdout:738: expect (ObjNum 197 snap 59 seq_num 197) 2024-09-20T19:24:22.286 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1261341 2024-09-20T19:24:22.300 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:22.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:22 smithi162 bash[24557]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:22 smithi162 bash[24557]: cluster 2024-09-20T19:24:21.154670+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-20T19:24:22.325 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2454193 2024-09-20T19:24:22.421 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 213852 2024-09-20T19:24:22.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:739: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:739: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:739: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 177 (ObjNum 159 snap 47 seq_num 159) dirty exists 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:732: done (5 left) 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:733: done (4 left) 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:735: done (3 left) 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:736: done (2 left) 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:738: done (1 left) 2024-09-20T19:24:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:739: done (0 left) 2024-09-20T19:24:22.500 INFO:tasks.rados.rados.0.smithi042.stdout:740: read oid 30 snap -1 2024-09-20T19:24:22.500 INFO:tasks.rados.rados.0.smithi042.stdout:740: expect deleted 2024-09-20T19:24:22.500 INFO:tasks.rados.rados.0.smithi042.stdout:741: snap_remove snap 78 2024-09-20T19:24:22.550 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:22.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:22.585 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:22.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:22.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:22.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:22.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:22.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:22.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:22.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:23.072 INFO:tasks.rados.rados.0.smithi042.stdout:741: done (1 left) 2024-09-20T19:24:23.073 INFO:tasks.rados.rados.0.smithi042.stdout:742: write (excl) oid 27 current snap is 82 2024-09-20T19:24:23.073 INFO:tasks.rados.rados.0.smithi042.stdout:742: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-20T19:24:23.074 INFO:tasks.rados.rados.0.smithi042.stdout:742: writing smithi04268048-27 from 554697 to 1019531 tid 1 2024-09-20T19:24:23.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.082 INFO:tasks.rados.rados.0.smithi042.stdout:742: writing smithi04268048-27 from 1725326 to 2284416 tid 2 2024-09-20T19:24:23.082 INFO:tasks.rados.rados.0.smithi042.stdout:743: write oid 17 current snap is 82 2024-09-20T19:24:23.082 INFO:tasks.rados.rados.0.smithi042.stdout:743: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-20T19:24:23.089 INFO:tasks.rados.rados.0.smithi042.stdout:743: writing smithi04268048-17 from 571504 to 1111587 tid 1 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:743: writing smithi04268048-17 from 1783808 to 2486556 tid 2 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:744: delete oid 15 current snap is 82 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:742: finishing write tid 1 to smithi04268048-27 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:742: finishing write tid 2 to smithi04268048-27 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:742: finishing write tid 3 to smithi04268048-27 2024-09-20T19:24:23.098 INFO:tasks.rados.rados.0.smithi042.stdout:742: finishing write tid 4 to smithi04268048-27 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:742: oid 27 updating version 0 to 354 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:742: oid 27 updating version 354 to 355 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:742: oid 27 version 355 is already newer than 353 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 355 (ObjNum 271 snap 82 seq_num 271) dirty exists 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:742: left oid 27 (ObjNum 271 snap 82 seq_num 271) 2024-09-20T19:24:23.099 INFO:tasks.rados.rados.0.smithi042.stdout:743: finishing write tid 1 to smithi04268048-17 2024-09-20T19:24:23.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:23.104 INFO:tasks.rados.rados.0.smithi042.stdout:740: done (3 left) 2024-09-20T19:24:23.104 INFO:tasks.rados.rados.0.smithi042.stdout:742: done (2 left) 2024-09-20T19:24:23.104 INFO:tasks.rados.rados.0.smithi042.stdout:744: done (1 left) 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:745: read oid 48 snap 63 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:745: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:746: copy_from oid 32 from oid 28 current snap is 82 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:747: read oid 42 snap -1 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:747: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:748: write (excl) oid 21 current snap is 82 2024-09-20T19:24:23.105 INFO:tasks.rados.rados.0.smithi042.stdout:748: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-20T19:24:23.110 INFO:tasks.rados.rados.0.smithi042.stdout:748: writing smithi04268048-21 from 605118 to 1012052 tid 1 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:748: writing smithi04268048-21 from 1617125 to 2323542 tid 2 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:749: delete oid 25 current snap is 82 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:746: finishing copy_from to smithi04268048-32 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:746: got expected ENOENT (src dne) 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:746: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:743: finishing write tid 2 to smithi04268048-17 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:743: finishing write tid 3 to smithi04268048-17 2024-09-20T19:24:23.119 INFO:tasks.rados.rados.0.smithi042.stdout:743: finishing write tid 4 to smithi04268048-17 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:743: oid 17 updating version 0 to 178 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:743: oid 17 updating version 178 to 179 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:743: oid 17 updating version 179 to 180 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 180 (ObjNum 272 snap 82 seq_num 272) dirty exists 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:743: left oid 17 (ObjNum 272 snap 82 seq_num 272) 2024-09-20T19:24:23.120 INFO:tasks.rados.rados.0.smithi042.stdout:745: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-20T19:24:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:23.120 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:23.152 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:23.152 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:23.167 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:23.182 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:23.183 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:23.200 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:23.200 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:23.201 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:23.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:23.220 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[32075]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:23 smithi042 bash[22706]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:23.233 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:23.233 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:23.245 INFO:tasks.rados.rados.0.smithi042.stdout:748: finishing write tid 1 to smithi04268048-21 2024-09-20T19:24:23.245 INFO:tasks.rados.rados.0.smithi042.stdout:747: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:24:23.245 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:23.266 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:23.266 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:23.289 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:23.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:22.014625+0000 mgr.y (mgr.24491) 245 : cluster [DBG] pgmap v347: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:22.170784+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.061156+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:23 smithi162 bash[24557]: audit 2024-09-20T19:24:23.095848+0000 mon.a (mon.0) 1257 : audit [DBG] from='client.? v1:172.21.15.42:0/3937318391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:23.321 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:23.321 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:23.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.342 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:23.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:748: finishing write tid 2 to smithi04268048-21 2024-09-20T19:24:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:748: finishing write tid 3 to smithi04268048-21 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:748: finishing write tid 4 to smithi04268048-21 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:748: oid 21 updating version 0 to 364 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:748: oid 21 version 364 is already newer than 363 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:748: oid 21 updating version 364 to 365 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 365 (ObjNum 274 snap 82 seq_num 274) dirty exists 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:748: left oid 21 (ObjNum 274 snap 82 seq_num 274) 2024-09-20T19:24:23.384 INFO:tasks.rados.rados.0.smithi042.stdout:743: done (5 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:745: done (4 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:746: done (3 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:747: done (2 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:748: done (1 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:749: done (0 left) 2024-09-20T19:24:23.385 INFO:tasks.rados.rados.0.smithi042.stdout:750: snap_create 2024-09-20T19:24:23.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:23.475 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:23.554 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:23.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.598 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:23.701 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:23.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:23.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:23.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:23.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:23.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:24.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:24.123 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:24.173 INFO:tasks.rados.rados.0.smithi042.stdout:750: done (0 left) 2024-09-20T19:24:24.174 INFO:tasks.rados.rados.0.smithi042.stdout:751: read oid 2 snap -1 2024-09-20T19:24:24.174 INFO:tasks.rados.rados.0.smithi042.stdout:751: expect (ObjNum 252 snap 72 seq_num 252) 2024-09-20T19:24:24.174 INFO:tasks.rados.rados.0.smithi042.stdout:752: snap_remove snap 76 2024-09-20T19:24:24.226 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:24.357 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:24.382 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:24.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:24.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:24.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:24.643 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:24.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:24.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:24.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:24.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:24.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:24.991 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:24.991 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:25.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:25.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:25.136 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:25.185 INFO:tasks.rados.rados.0.smithi042.stdout:752: done (1 left) 2024-09-20T19:24:25.185 INFO:tasks.rados.rados.0.smithi042.stdout:753: read oid 36 snap -1 2024-09-20T19:24:25.185 INFO:tasks.rados.rados.0.smithi042.stdout:753: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:24:25.185 INFO:tasks.rados.rados.0.smithi042.stdout:754: snap_remove snap 73 2024-09-20T19:24:25.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:25.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:25.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:25.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:25 smithi162 bash[24557]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[22706]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.877433+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:23.878914+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:24.015296+0000 mgr.y (mgr.24491) 246 : cluster [DBG] pgmap v350: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-20T19:24:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:25 smithi042 bash[32075]: cluster 2024-09-20T19:24:24.170555+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-20T19:24:25.523 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:25.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.676 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:25.780 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T19:24:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:25.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:25.934 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:25.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:26.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:26.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:26.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:26.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:26.185 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:24:26.197 INFO:tasks.rados.rados.0.smithi042.stdout:754: done (2 left) 2024-09-20T19:24:26.197 INFO:tasks.rados.rados.0.smithi042.stdout:755: delete oid 38 current snap is 83 2024-09-20T19:24:26.197 INFO:tasks.rados.rados.0.smithi042.stdout:751: expect (ObjNum 252 snap 72 seq_num 252) 2024-09-20T19:24:26.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:26.231 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:26.270 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1516832 2024-09-20T19:24:26.289 INFO:tasks.rados.rados.0.smithi042.stdout:753: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:24:26.289 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:26.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:26.322 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[32075]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[32075]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[32075]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[32075]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[22706]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[22706]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[22706]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:26 smithi042 bash[22706]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:751: done (2 left) 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:753: done (1 left) 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:755: done (0 left) 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:756: read oid 19 snap -1 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:756: expect (ObjNum 255 snap 74 seq_num 255) 2024-09-20T19:24:26.367 INFO:tasks.rados.rados.0.smithi042.stdout:757: snap_create 2024-09-20T19:24:26.370 INFO:tasks.rados.rados.0.smithi042.stdout:756: expect (ObjNum 255 snap 74 seq_num 255) 2024-09-20T19:24:26.370 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:26.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:26.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:26 smithi162 bash[24557]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:26 smithi162 bash[24557]: cluster 2024-09-20T19:24:25.179436+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-20T19:24:26.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:26 smithi162 bash[24557]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:26 smithi162 bash[24557]: audit 2024-09-20T19:24:25.581643+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:24:26.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:26.471 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 240083 2024-09-20T19:24:26.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:26.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:26.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:26.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:26.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:26.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:26.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:26.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:26.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:27.087 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:27.108 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:24:27.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:24:27.108 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-20T19:24:27.108 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T19:24:27.108 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T19:24:27.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-20T19:24:27.115 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:27.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:24:27.199 INFO:tasks.rados.rados.0.smithi042.stdout:756: done (1 left) 2024-09-20T19:24:27.199 INFO:tasks.rados.rados.0.smithi042.stdout:757: done (0 left) 2024-09-20T19:24:27.200 INFO:tasks.rados.rados.0.smithi042.stdout:758: write (excl) oid 44 current snap is 84 2024-09-20T19:24:27.200 INFO:tasks.rados.rados.0.smithi042.stdout:758: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-20T19:24:27.207 INFO:tasks.rados.rados.0.smithi042.stdout:758: writing smithi04268048-44 from 655539 to 1288220 tid 1 2024-09-20T19:24:27.215 INFO:tasks.rados.rados.0.smithi042.stdout:758: writing smithi04268048-44 from 2076218 to 2646315 tid 2 2024-09-20T19:24:27.215 INFO:tasks.rados.rados.0.smithi042.stdout:759: write (excl) oid 11 current snap is 84 2024-09-20T19:24:27.215 INFO:tasks.rados.rados.0.smithi042.stdout:758: finishing write tid 1 to smithi04268048-44 2024-09-20T19:24:27.216 INFO:tasks.rados.rados.0.smithi042.stdout:759: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-20T19:24:27.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:27.223 INFO:tasks.rados.rados.0.smithi042.stdout:759: writing smithi04268048-11 from 672346 to 1380276 tid 1 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.229 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:27 smithi162 bash[24557]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:759: writing smithi04268048-11 from 2134700 to 2672346 tid 2 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:760: rollback oid 46 current snap is 84 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 83 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:758: finishing write tid 2 to smithi04268048-44 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:758: finishing write tid 3 to smithi04268048-44 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:761: read oid 13 snap -1 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:761: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:762: rollback oid 33 current snap is 84 2024-09-20T19:24:27.232 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 83 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: finishing write tid 4 to smithi04268048-44 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: oid 44 updating version 0 to 331 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: oid 44 updating version 331 to 332 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: oid 44 updating version 332 to 333 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 333 (ObjNum 277 snap 84 seq_num 277) dirty exists 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: left oid 44 (ObjNum 277 snap 84 seq_num 277) 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:759: finishing write tid 1 to smithi04268048-11 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:758: done (4 left) 2024-09-20T19:24:27.233 INFO:tasks.rados.rados.0.smithi042.stdout:763: snap_remove snap 74 2024-09-20T19:24:27.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:27.346 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.356 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[32075]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.015924+0000 mgr.y (mgr.24491) 247 : cluster [DBG] pgmap v353: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 224 MiB data, 826 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.176804+0000 mon.a (mon.0) 1261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: audit 2024-09-20T19:24:26.184772+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.42:0/1924297930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.190755+0000 mon.a (mon.0) 1263 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.471369+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.472970+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:27 smithi042 bash[22706]: audit 2024-09-20T19:24:26.996185+0000 mon.a (mon.0) 1264 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:27.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:27.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:27.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:27.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:27.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:27.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:27.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:27.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:763: done (4 left) 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:764: rollback oid 47 current snap is 84 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 47 to 60 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:760: finishing rollback tid 0 to smithi04268048-46 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:760: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:24:28.067 INFO:tasks.rados.rados.0.smithi042.stdout:761: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:28.084 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2032470 2024-09-20T19:24:28.119 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:28.122 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:28.148 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:28.153 INFO:tasks.rados.rados.0.smithi042.stdout:759: finishing write tid 2 to smithi04268048-11 2024-09-20T19:24:28.153 INFO:tasks.rados.rados.0.smithi042.stdout:760: finishing rollback tid 2 to smithi04268048-46 2024-09-20T19:24:28.153 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 337 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:24:28.153 INFO:tasks.rados.rados.0.smithi042.stdout:759: finishing write tid 3 to smithi04268048-11 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:760: done (4 left) 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:761: done (3 left) 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:765: write oid 17 current snap is 84 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:759: finishing write tid 4 to smithi04268048-11 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:759: oid 11 updating version 0 to 367 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:759: oid 11 updating version 367 to 369 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:759: oid 11 version 369 is already newer than 368 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 369 (ObjNum 278 snap 84 seq_num 278) dirty exists 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:759: left oid 11 (ObjNum 278 snap 84 seq_num 278) 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:762: finishing rollback tid 0 to smithi04268048-33 2024-09-20T19:24:28.154 INFO:tasks.rados.rados.0.smithi042.stdout:765: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-20T19:24:28.162 INFO:tasks.rados.rados.0.smithi042.stdout:765: writing smithi04268048-17 from 689153 to 1472332 tid 1 2024-09-20T19:24:28.167 INFO:tasks.rados.rados.0.smithi042.stdout:765: writing smithi04268048-17 from 2076829 to 2534242 tid 2 2024-09-20T19:24:28.167 INFO:tasks.rados.rados.0.smithi042.stdout:759: done (3 left) 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:766: snap_create 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:762: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:762: finishing rollback tid 2 to smithi04268048-33 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 160 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:764: finishing rollback tid 0 to smithi04268048-47 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:764: finishing rollback tid 1 to smithi04268048-47 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:764: finishing rollback tid 2 to smithi04268048-47 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 373 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:24:28.168 INFO:tasks.rados.rados.0.smithi042.stdout:765: finishing write tid 1 to smithi04268048-17 2024-09-20T19:24:28.172 INFO:tasks.rados.rados.0.smithi042.stdout:765: finishing write tid 2 to smithi04268048-17 2024-09-20T19:24:28.172 INFO:tasks.rados.rados.0.smithi042.stdout:765: finishing write tid 3 to smithi04268048-17 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:765: finishing write tid 4 to smithi04268048-17 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:765: oid 17 updating version 0 to 183 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:765: oid 17 version 183 is already newer than 181 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:765: oid 17 version 183 is already newer than 182 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 183 (ObjNum 279 snap 84 seq_num 279) dirty exists 2024-09-20T19:24:28.173 INFO:tasks.rados.rados.0.smithi042.stdout:765: left oid 17 (ObjNum 279 snap 84 seq_num 279) 2024-09-20T19:24:28.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.263 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.264 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:28 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.278 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:28.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.410 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:28.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:28.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.714357+0000 osd.2 (osd.2) 105 : cluster [DBG] 5.a scrub starts 2024-09-20T19:24:28.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.716046+0000 osd.2 (osd.2) 106 : cluster [DBG] 5.a scrub ok 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.866348+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:26.871626+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.198936+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:28 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.067353+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-20T19:24:28.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:28.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:28.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:28.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:28.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:28.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:28.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:29.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:29.071 INFO:tasks.rados.rados.0.smithi042.stdout:762: done (3 left) 2024-09-20T19:24:29.072 INFO:tasks.rados.rados.0.smithi042.stdout:764: done (2 left) 2024-09-20T19:24:29.072 INFO:tasks.rados.rados.0.smithi042.stdout:765: done (1 left) 2024-09-20T19:24:29.072 INFO:tasks.rados.rados.0.smithi042.stdout:766: done (0 left) 2024-09-20T19:24:29.072 INFO:tasks.rados.rados.0.smithi042.stdout:767: delete oid 8 current snap is 85 2024-09-20T19:24:29.077 INFO:tasks.rados.rados.0.smithi042.stdout:767: done (0 left) 2024-09-20T19:24:29.077 INFO:tasks.rados.rados.0.smithi042.stdout:768: delete oid 12 current snap is 85 2024-09-20T19:24:29.083 INFO:tasks.rados.rados.0.smithi042.stdout:768: done (0 left) 2024-09-20T19:24:29.083 INFO:tasks.rados.rados.0.smithi042.stdout:769: delete oid 7 current snap is 85 2024-09-20T19:24:29.092 INFO:tasks.rados.rados.0.smithi042.stdout:769: done (0 left) 2024-09-20T19:24:29.092 INFO:tasks.rados.rados.0.smithi042.stdout:770: snap_create 2024-09-20T19:24:29.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:24:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:29.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:29 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:29.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:29.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:29.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.395270+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:27.396271+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.016699+0000 mgr.y (mgr.24491) 248 : cluster [DBG] pgmap v356: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:29 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.069819+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-20T19:24:29.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.706 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:29.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:29.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:29.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:30.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:30.073 INFO:tasks.rados.rados.0.smithi042.stdout:770: done (0 left) 2024-09-20T19:24:30.073 INFO:tasks.rados.rados.0.smithi042.stdout:771: copy_from oid 22 from oid 47 current snap is 86 2024-09-20T19:24:30.073 INFO:tasks.rados.rados.0.smithi042.stdout:772: read oid 27 snap 85 2024-09-20T19:24:30.073 INFO:tasks.rados.rados.0.smithi042.stdout:772: expect (ObjNum 271 snap 82 seq_num 271) 2024-09-20T19:24:30.074 INFO:tasks.rados.rados.0.smithi042.stdout:773: snap_remove snap 82 2024-09-20T19:24:30.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:30.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:30.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:30.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:30 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:30.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.557293+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.560006+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.909384+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:28.915857+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.127265+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.130896+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: audit 2024-09-20T19:24:29.159734+0000 mgr.y (mgr.24491) 249 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:30 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.071730+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-20T19:24:30.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:30.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:30.597 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:30.715 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:30.742 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:30.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:30.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:30.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:31.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:31.078 INFO:tasks.rados.rados.0.smithi042.stdout:773: done (2 left) 2024-09-20T19:24:31.078 INFO:tasks.rados.rados.0.smithi042.stdout:774: snap_create 2024-09-20T19:24:31.078 INFO:tasks.rados.rados.0.smithi042.stdout:772: expect (ObjNum 271 snap 82 seq_num 271) 2024-09-20T19:24:31.080 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 623214 2024-09-20T19:24:31.106 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:31.146 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2379170 2024-09-20T19:24:31.152 INFO:tasks.rados.rados.0.smithi042.stdout:771: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:24:31.152 INFO:tasks.rados.rados.0.smithi042.stdout:771: finishing copy_from to smithi04268048-22 2024-09-20T19:24:31.152 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 373 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:24:31.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:31.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:31.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.375 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:31 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.830623+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.e scrub starts 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:29.833726+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.e scrub ok 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.017360+0000 mgr.y (mgr.24491) 250 : cluster [DBG] pgmap v359: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 225 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 825 KiB/s wr, 21 op/s 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.102306+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.106235+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.431299+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.435217+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:31 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.074319+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-20T19:24:31.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:31.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:31.618 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:31.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:31.744 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:31.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:31.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:31.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:31.977 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:31.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:32.002 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:32.077 INFO:tasks.rados.rados.0.smithi042.stdout:771: done (2 left) 2024-09-20T19:24:32.077 INFO:tasks.rados.rados.0.smithi042.stdout:772: done (1 left) 2024-09-20T19:24:32.077 INFO:tasks.rados.rados.0.smithi042.stdout:774: done (0 left) 2024-09-20T19:24:32.077 INFO:tasks.rados.rados.0.smithi042.stdout:775: write (excl) oid 15 current snap is 87 2024-09-20T19:24:32.077 INFO:tasks.rados.rados.0.smithi042.stdout:775: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-20T19:24:32.080 INFO:tasks.rados.rados.0.smithi042.stdout:775: writing smithi04268048-15 from 756381 to 1156909 tid 1 2024-09-20T19:24:32.089 INFO:tasks.rados.rados.0.smithi042.stdout:775: writing smithi04268048-15 from 1910757 to 2375508 tid 2 2024-09-20T19:24:32.089 INFO:tasks.rados.rados.0.smithi042.stdout:776: read oid 42 snap -1 2024-09-20T19:24:32.089 INFO:tasks.rados.rados.0.smithi042.stdout:776: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:24:32.089 INFO:tasks.rados.rados.0.smithi042.stdout:775: finishing write tid 1 to smithi04268048-15 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:777: rollback oid 37 current snap is 87 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 37 to 85 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:778: rollback oid 30 current snap is 87 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 83 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:779: read oid 7 snap -1 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:779: expect deleted 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:780: copy_from oid 25 from oid 45 current snap is 87 2024-09-20T19:24:32.090 INFO:tasks.rados.rados.0.smithi042.stdout:781: delete oid 19 current snap is 87 2024-09-20T19:24:32.097 INFO:tasks.rados.rados.0.smithi042.stdout:775: finishing write tid 2 to smithi04268048-15 2024-09-20T19:24:32.098 INFO:tasks.rados.rados.0.smithi042.stdout:781: done (6 left) 2024-09-20T19:24:32.098 INFO:tasks.rados.rados.0.smithi042.stdout:782: snap_remove snap 84 2024-09-20T19:24:32.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:32.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:32.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:32.261 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:32.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:32.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.376 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:32 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.391 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:32.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.317716+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:24:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.320183+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.884055+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:30.887734+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:32 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.073152+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-20T19:24:32.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:32.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:32.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:32.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:32.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:32.883 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:32.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:32.998 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:33.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:33.065 INFO:tasks.rados.rados.0.smithi042.stdout:782: done (6 left) 2024-09-20T19:24:33.065 INFO:tasks.rados.rados.0.smithi042.stdout:783: read oid 32 snap -1 2024-09-20T19:24:33.065 INFO:tasks.rados.rados.0.smithi042.stdout:783: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:24:33.066 INFO:tasks.rados.rados.0.smithi042.stdout:784: snap_remove snap 86 2024-09-20T19:24:33.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.159 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:33.159 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:33.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:33.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:33 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:33.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:33.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.847112+0000 osd.2 (osd.2) 109 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:31.849472+0000 osd.2 (osd.2) 110 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:24:33.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.018532+0000 mgr.y (mgr.24491) 251 : cluster [DBG] pgmap v362: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.086763+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.089802+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.367839+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.e scrub starts 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.370257+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.e scrub ok 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:33 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.064528+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-20T19:24:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:33.751 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:24:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T19:24:33.752 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:24:33.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:33.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:33.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:33.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:34.043 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:784: done (7 left) 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:785: snap_create 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:775: finishing write tid 3 to smithi04268048-15 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:775: finishing write tid 4 to smithi04268048-15 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:775: oid 15 updating version 0 to 312 2024-09-20T19:24:34.068 INFO:tasks.rados.rados.0.smithi042.stdout:775: oid 15 updating version 312 to 313 2024-09-20T19:24:34.069 INFO:tasks.rados.rados.0.smithi042.stdout:775: oid 15 version 313 is already newer than 311 2024-09-20T19:24:34.069 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:24:34.069 INFO:tasks.rados.rados.0.smithi042.stdout:775: left oid 15 (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:24:34.069 INFO:tasks.rados.rados.0.smithi042.stdout:776: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:24:34.069 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:34.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:34.182 INFO:tasks.rados.rados.0.smithi042.stdout:778: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:24:34.182 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:24:34.182 INFO:tasks.rados.rados.0.smithi042.stdout:777: finishing rollback tid 0 to smithi04268048-37 2024-09-20T19:24:34.182 INFO:tasks.rados.rados.0.smithi042.stdout:777: finishing rollback tid 1 to smithi04268048-37 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:777: finishing rollback tid 2 to smithi04268048-37 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 204 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:780: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:780: finishing copy_from to smithi04268048-25 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 327 (ObjNum 197 snap 59 seq_num 197) dirty exists 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:783: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:24:34.183 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:34.265 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:34.300 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:34.310 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.311 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.312 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:34 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:24:34.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:34.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.852822+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.860561+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.905630+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.a scrub starts 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:32.907714+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.a scrub ok 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.376581+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:33.379102+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:34 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.067219+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-20T19:24:34.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:34.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:34.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:34.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:34.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:34.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:34.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:34.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:34.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:35.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:775: done (7 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:776: done (6 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:777: done (5 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:778: done (4 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:779: done (3 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:780: done (2 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:783: done (1 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:785: done (0 left) 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:786: copy_from oid 17 from oid 11 current snap is 88 2024-09-20T19:24:35.075 INFO:tasks.rados.rados.0.smithi042.stdout:787: delete oid 3 current snap is 88 2024-09-20T19:24:35.078 INFO:tasks.rados.rados.0.smithi042.stdout:787: done (1 left) 2024-09-20T19:24:35.078 INFO:tasks.rados.rados.0.smithi042.stdout:788: write (excl) oid 35 current snap is 88 2024-09-20T19:24:35.078 INFO:tasks.rados.rados.0.smithi042.stdout:788: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-20T19:24:35.090 INFO:tasks.rados.rados.0.smithi042.stdout:788: writing smithi04268048-35 from 406802 to 1033077 tid 1 2024-09-20T19:24:35.100 INFO:tasks.rados.rados.0.smithi042.stdout:788: writing smithi04268048-35 from 1686203 to 2414634 tid 2 2024-09-20T19:24:35.100 INFO:tasks.rados.rados.0.smithi042.stdout:789: copy_from oid 28 from oid 9 current snap is 88 2024-09-20T19:24:35.100 INFO:tasks.rados.rados.0.smithi042.stdout:786: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:24:35.100 INFO:tasks.rados.rados.0.smithi042.stdout:786: finishing copy_from to smithi04268048-17 2024-09-20T19:24:35.100 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 369 (ObjNum 278 snap 84 seq_num 278) dirty exists 2024-09-20T19:24:35.101 INFO:tasks.rados.rados.0.smithi042.stdout:788: finishing write tid 1 to smithi04268048-35 2024-09-20T19:24:35.101 INFO:tasks.rados.rados.0.smithi042.stdout:786: done (2 left) 2024-09-20T19:24:35.101 INFO:tasks.rados.rados.0.smithi042.stdout:790: snap_create 2024-09-20T19:24:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:35.105 INFO:tasks.rados.rados.0.smithi042.stdout:788: finishing write tid 2 to smithi04268048-35 2024-09-20T19:24:35.105 INFO:tasks.rados.rados.0.smithi042.stdout:788: finishing write tid 3 to smithi04268048-35 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:788: finishing write tid 4 to smithi04268048-35 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:788: oid 35 updating version 0 to 74 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:788: oid 35 updating version 74 to 75 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:788: oid 35 version 75 is already newer than 73 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 75 (ObjNum 286 snap 88 seq_num 286) dirty exists 2024-09-20T19:24:35.106 INFO:tasks.rados.rados.0.smithi042.stdout:788: left oid 35 (ObjNum 286 snap 88 seq_num 286) 2024-09-20T19:24:35.115 INFO:tasks.rados.rados.0.smithi042.stdout:789: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:24:35.119 INFO:tasks.rados.rados.0.smithi042.stdout:789: finishing copy_from to smithi04268048-28 2024-09-20T19:24:35.119 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 324 (ObjNum 257 snap 75 seq_num 257) dirty exists 2024-09-20T19:24:35.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:35.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:35.351 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:35.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:35.369 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:35.386 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:35.386 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:35.402 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:35.420 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:35.420 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:35.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:35 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.440 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:35.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:35.462 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:35.484 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:35.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.019251+0000 mgr.y (mgr.24491) 252 : cluster [DBG] pgmap v365: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.332373+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.334737+0000 osd.7 (osd.7) 102 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.552387+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.554252+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.072558+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.098530+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:35 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.098859+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:35.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:35.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:35.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:35.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:35.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:35.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:35.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:788: done (2 left) 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:789: done (1 left) 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:790: done (0 left) 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:791: copy_from oid 37 from oid 31 current snap is 89 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:792: read oid 19 snap -1 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:792: expect deleted 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:793: read oid 13 snap -1 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:793: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:36.073 INFO:tasks.rados.rados.0.smithi042.stdout:794: snap_create 2024-09-20T19:24:36.077 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:24:36.080 INFO:tasks.rados.rados.0.smithi042.stdout:793: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:36.080 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:36.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:36.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:36.150 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:36.211 INFO:tasks.rados.rados.0.smithi042.stdout:791: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:24:36.211 INFO:tasks.rados.rados.0.smithi042.stdout:791: finishing copy_from to smithi04268048-37 2024-09-20T19:24:36.211 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 297 (ObjNum 188 snap 56 seq_num 188) dirty exists 2024-09-20T19:24:36.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:36.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:36.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.369 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:36 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:36.425 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:36.426 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.426 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.426 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.771240+0000 osd.2 (osd.2) 113 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.773563+0000 osd.2 (osd.2) 114 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.896865+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:34.899345+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.283414+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.285403+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.333060+0000 mon.a (mon.0) 1274 : audit [DBG] from='client.? v1:172.21.15.42:0/2692675892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.372142+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.373216+0000 osd.4 (osd.4) 132 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.534008+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.c scrub starts 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.542754+0000 osd.6 (osd.6) 94 : cluster [DBG] 7.c scrub ok 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.807952+0000 mon.c (mon.2) 51 : audit [INF] from='client.? v1:172.21.15.42:0/1267760416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:35.808564+0000 mon.a (mon.0) 1275 : audit [INF] from='client.25013 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.065573+0000 mon.a (mon.0) 1276 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.065621+0000 mon.a (mon.0) 1277 : cluster [INF] Cluster is now healthy 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: audit 2024-09-20T19:24:36.067602+0000 mon.a (mon.0) 1278 : audit [INF] from='client.25013 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:36 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.071380+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-20T19:24:36.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:36.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:36.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:36.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:36.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:36.900 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:36.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:37.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:37.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:37.079 INFO:tasks.rados.rados.0.smithi042.stdout:791: done (3 left) 2024-09-20T19:24:37.080 INFO:tasks.rados.rados.0.smithi042.stdout:792: done (2 left) 2024-09-20T19:24:37.080 INFO:tasks.rados.rados.0.smithi042.stdout:793: done (1 left) 2024-09-20T19:24:37.080 INFO:tasks.rados.rados.0.smithi042.stdout:794: done (0 left) 2024-09-20T19:24:37.080 INFO:tasks.rados.rados.0.smithi042.stdout:795: write oid 6 current snap is 90 2024-09-20T19:24:37.080 INFO:tasks.rados.rados.0.smithi042.stdout:795: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-20T19:24:37.090 INFO:tasks.rados.rados.0.smithi042.stdout:795: writing smithi04268048-6 from 423609 to 1125133 tid 1 2024-09-20T19:24:37.099 INFO:tasks.rados.rados.0.smithi042.stdout:795: writing smithi04268048-6 from 1628332 to 2100421 tid 2 2024-09-20T19:24:37.106 INFO:tasks.rados.rados.0.smithi042.stdout:795: writing smithi04268048-6 from 2648987 to 2823609 tid 3 2024-09-20T19:24:37.106 INFO:tasks.rados.rados.0.smithi042.stdout:796: snap_remove snap 80 2024-09-20T19:24:37.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:37.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:37.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:37 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:37.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.406 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.788189+0000 osd.2 (osd.2) 115 : cluster [DBG] 7.e scrub starts 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:35.790662+0000 osd.2 (osd.2) 116 : cluster [DBG] 7.e scrub ok 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.019836+0000 mgr.y (mgr.24491) 253 : cluster [DBG] pgmap v368: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 230 MiB data, 811 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.356818+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.359285+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.407 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:37 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.079432+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-20T19:24:37.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:37.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:37.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:37.681 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:37.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:37.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:37.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:37.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:38.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:796: done (1 left) 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:797: delete oid 27 current snap is 90 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: finishing write tid 1 to smithi04268048-6 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: finishing write tid 2 to smithi04268048-6 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: finishing write tid 3 to smithi04268048-6 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: finishing write tid 4 to smithi04268048-6 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: finishing write tid 5 to smithi04268048-6 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: oid 6 updating version 0 to 308 2024-09-20T19:24:38.076 INFO:tasks.rados.rados.0.smithi042.stdout:795: oid 6 updating version 308 to 310 2024-09-20T19:24:38.077 INFO:tasks.rados.rados.0.smithi042.stdout:795: oid 6 version 310 is already newer than 309 2024-09-20T19:24:38.077 INFO:tasks.rados.rados.0.smithi042.stdout:795: oid 6 version 310 is already newer than 307 2024-09-20T19:24:38.077 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 310 (ObjNum 287 snap 90 seq_num 287) dirty exists 2024-09-20T19:24:38.077 INFO:tasks.rados.rados.0.smithi042.stdout:795: left oid 6 (ObjNum 287 snap 90 seq_num 287) 2024-09-20T19:24:38.081 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:38.090 INFO:tasks.rados.rados.0.smithi042.stdout:795: done (1 left) 2024-09-20T19:24:38.090 INFO:tasks.rados.rados.0.smithi042.stdout:797: done (0 left) 2024-09-20T19:24:38.090 INFO:tasks.rados.rados.0.smithi042.stdout:798: snap_remove snap 79 2024-09-20T19:24:38.188 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:38.215 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:38.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:38.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:38.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:24:38.448 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:38.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.484 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.485 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.485 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.485 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.485 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.485 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:38 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.836023+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:36.836653+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.334511+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:24:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.337839+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:24:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:38 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.068826+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-20T19:24:38.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:38.623 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:38.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:38.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:38.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:38.881 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:38.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:38.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.012 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:39.089 INFO:tasks.rados.rados.0.smithi042.stdout:798: done (0 left) 2024-09-20T19:24:39.089 INFO:tasks.rados.rados.0.smithi042.stdout:799: snap_create 2024-09-20T19:24:39.106 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:39.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:39.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:24:39.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:39.245 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:39.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:39.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:39.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.384 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:39 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:39.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:39.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.866462+0000 osd.5 (osd.5) 119 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:37.868225+0000 osd.5 (osd.5) 120 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.020560+0000 mgr.y (mgr.24491) 254 : cluster [DBG] pgmap v371: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.560390+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub starts 2024-09-20T19:24:39.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.562118+0000 osd.1 (osd.1) 136 : cluster [DBG] 2.d scrub ok 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.573583+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.575798+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:39 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.088361+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-20T19:24:39.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:39.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:39.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:39.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:39.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:40.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:40.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:40.049 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:24:40.050 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:24:40.050 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-20T19:24:40.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-20T19:24:40.091 INFO:tasks.rados.rados.0.smithi042.stdout:799: done (0 left) 2024-09-20T19:24:40.091 INFO:tasks.rados.rados.0.smithi042.stdout:800: delete oid 14 current snap is 91 2024-09-20T19:24:40.103 INFO:tasks.rados.rados.0.smithi042.stdout:800: done (0 left) 2024-09-20T19:24:40.103 INFO:tasks.rados.rados.0.smithi042.stdout:801: delete oid 43 current snap is 91 2024-09-20T19:24:40.107 INFO:tasks.rados.rados.0.smithi042.stdout:801: done (0 left) 2024-09-20T19:24:40.107 INFO:tasks.rados.rados.0.smithi042.stdout:802: snap_remove snap 90 2024-09-20T19:24:40.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:40.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:40.273 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:40.274 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.274 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.274 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.274 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.274 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.275 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:40 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.297 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:40.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:40.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.904333+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.f scrub starts 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.906643+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.f scrub ok 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:38.997799+0000 osd.0 (osd.0) 137 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.000750+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: audit 2024-09-20T19:24:39.168347+0000 mgr.y (mgr.24491) 255 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.313016+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.314600+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.337264+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.339327+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.520607+0000 osd.1 (osd.1) 137 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.523018+0000 osd.1 (osd.1) 138 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:40 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.086939+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-20T19:24:40.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:40.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:40.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:40.636 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:24:40.636 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:24:40.636 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:24:40.636 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:24:40.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:40.683 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:40.786 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:40.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:40.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:40.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:40.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:41.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:41.077 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:41.089 INFO:tasks.rados.rados.0.smithi042.stdout:802: done (0 left) 2024-09-20T19:24:41.089 INFO:tasks.rados.rados.0.smithi042.stdout:803: write (excl) oid 35 current snap is 91 2024-09-20T19:24:41.089 INFO:tasks.rados.rados.0.smithi042.stdout:803: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-20T19:24:41.096 INFO:tasks.rados.rados.0.smithi042.stdout:803: writing smithi04268048-35 from 490837 to 1209710 tid 1 2024-09-20T19:24:41.102 INFO:tasks.rados.rados.0.smithi042.stdout:803: writing smithi04268048-35 from 1862260 to 2341687 tid 2 2024-09-20T19:24:41.105 INFO:tasks.rados.rados.0.smithi042.stdout:803: writing smithi04268048-35 from 2875032 to 2890837 tid 3 2024-09-20T19:24:41.106 INFO:tasks.rados.rados.0.smithi042.stdout:804: snap_create 2024-09-20T19:24:41.109 INFO:tasks.rados.rados.0.smithi042.stdout:803: finishing write tid 1 to smithi04268048-35 2024-09-20T19:24:41.109 INFO:tasks.rados.rados.0.smithi042.stdout:803: finishing write tid 2 to smithi04268048-35 2024-09-20T19:24:41.109 INFO:tasks.rados.rados.0.smithi042.stdout:803: finishing write tid 3 to smithi04268048-35 2024-09-20T19:24:41.109 INFO:tasks.rados.rados.0.smithi042.stdout:803: finishing write tid 4 to smithi04268048-35 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: finishing write tid 5 to smithi04268048-35 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: oid 35 updating version 0 to 82 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: oid 35 version 82 is already newer than 81 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: oid 35 updating version 82 to 84 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: oid 35 version 84 is already newer than 83 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 84 (ObjNum 291 snap 91 seq_num 291) dirty exists 2024-09-20T19:24:41.110 INFO:tasks.rados.rados.0.smithi042.stdout:803: left oid 35 (ObjNum 291 snap 91 seq_num 291) 2024-09-20T19:24:41.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:41.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:41.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:41.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:41.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:24:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[22706]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:41 smithi042 bash[32075]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.967708+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:24:41.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:39.971833+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:24:41.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.021406+0000 mgr.y (mgr.24491) 256 : cluster [DBG] pgmap v374: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 244 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.520396+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.a scrub starts 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.522445+0000 osd.1 (osd.1) 140 : cluster [DBG] 2.a scrub ok 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:41 smithi162 bash[24557]: cluster 2024-09-20T19:24:41.089476+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-20T19:24:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:41.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:41.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:41.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:41.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:41.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:41.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:42.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.102 INFO:tasks.rados.rados.0.smithi042.stdout:803: done (1 left) 2024-09-20T19:24:42.102 INFO:tasks.rados.rados.0.smithi042.stdout:804: done (0 left) 2024-09-20T19:24:42.102 INFO:tasks.rados.rados.0.smithi042.stdout:805: snap_create 2024-09-20T19:24:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:42.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:42.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:42.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:42 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.806007+0000 osd.2 (osd.2) 119 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:40.807640+0000 osd.2 (osd.2) 120 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: audit 2024-09-20T19:24:42.030919+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: audit 2024-09-20T19:24:42.033075+0000 mon.a (mon.0) 1286 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:42 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.096956+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-20T19:24:42.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:42.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:42.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:42.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:42.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:42.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:42.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:42.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:43.016 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:43.075 INFO:tasks.rados.rados.0.smithi042.stdout:805: done (0 left) 2024-09-20T19:24:43.075 INFO:tasks.rados.rados.0.smithi042.stdout:806: write oid 41 current snap is 93 2024-09-20T19:24:43.075 INFO:tasks.rados.rados.0.smithi042.stdout:806: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-20T19:24:43.094 INFO:tasks.rados.rados.0.smithi042.stdout:806: writing smithi04268048-41 from 507644 to 1301766 tid 1 2024-09-20T19:24:43.113 INFO:tasks.rados.rados.0.smithi042.stdout:806: writing smithi04268048-41 from 1920742 to 2543827 tid 2 2024-09-20T19:24:43.113 INFO:tasks.rados.rados.0.smithi042.stdout:807: write (excl) oid 32 current snap is 93 2024-09-20T19:24:43.113 INFO:tasks.rados.rados.0.smithi042.stdout:806: finishing write tid 1 to smithi04268048-41 2024-09-20T19:24:43.113 INFO:tasks.rados.rados.0.smithi042.stdout:807: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-20T19:24:43.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:43.127 INFO:tasks.rados.rados.0.smithi042.stdout:807: writing smithi04268048-32 from 524451 to 993822 tid 1 2024-09-20T19:24:43.142 INFO:tasks.rados.rados.0.smithi042.stdout:807: writing smithi04268048-32 from 1579224 to 2062320 tid 2 2024-09-20T19:24:43.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:43.153 INFO:tasks.rados.rados.0.smithi042.stdout:807: writing smithi04268048-32 from 2529878 to 2924451 tid 3 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:808: read oid 7 snap -1 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:808: expect deleted 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:809: write oid 36 current snap is 93 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: finishing write tid 2 to smithi04268048-41 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: finishing write tid 3 to smithi04268048-41 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: finishing write tid 4 to smithi04268048-41 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: oid 41 updating version 0 to 165 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: oid 41 updating version 165 to 167 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: oid 41 version 167 is already newer than 166 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 167 (ObjNum 292 snap 93 seq_num 292) dirty exists 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:806: left oid 41 (ObjNum 292 snap 93 seq_num 292) 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:807: finishing write tid 1 to smithi04268048-32 2024-09-20T19:24:43.154 INFO:tasks.rados.rados.0.smithi042.stdout:807: finishing write tid 2 to smithi04268048-32 2024-09-20T19:24:43.155 INFO:tasks.rados.rados.0.smithi042.stdout:809: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-20T19:24:43.165 INFO:tasks.rados.rados.0.smithi042.stdout:809: writing smithi04268048-36 from 541258 to 1085878 tid 1 2024-09-20T19:24:43.177 INFO:tasks.rados.rados.0.smithi042.stdout:809: writing smithi04268048-36 from 1637706 to 2264460 tid 2 2024-09-20T19:24:43.184 INFO:tasks.rados.rados.0.smithi042.stdout:809: writing smithi04268048-36 from 2840948 to 2941258 tid 3 2024-09-20T19:24:43.184 INFO:tasks.rados.rados.0.smithi042.stdout:806: done (3 left) 2024-09-20T19:24:43.184 INFO:tasks.rados.rados.0.smithi042.stdout:810: rollback oid 47 current snap is 93 2024-09-20T19:24:43.184 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 47 to 91 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: finishing write tid 3 to smithi04268048-32 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: finishing write tid 4 to smithi04268048-32 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: finishing write tid 5 to smithi04268048-32 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: oid 32 updating version 0 to 350 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: oid 32 version 350 is already newer than 347 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: oid 32 version 350 is already newer than 348 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: oid 32 version 350 is already newer than 349 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 350 (ObjNum 293 snap 93 seq_num 293) dirty exists 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: left oid 32 (ObjNum 293 snap 93 seq_num 293) 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:809: finishing write tid 1 to smithi04268048-36 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:809: finishing write tid 2 to smithi04268048-36 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:807: done (3 left) 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:808: done (2 left) 2024-09-20T19:24:43.185 INFO:tasks.rados.rados.0.smithi042.stdout:811: snap_create 2024-09-20T19:24:43.188 INFO:tasks.rados.rados.0.smithi042.stdout:809: finishing write tid 3 to smithi04268048-36 2024-09-20T19:24:43.188 INFO:tasks.rados.rados.0.smithi042.stdout:809: finishing write tid 4 to smithi04268048-36 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: finishing write tid 5 to smithi04268048-36 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: oid 36 updating version 0 to 304 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: oid 36 updating version 304 to 306 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: oid 36 version 306 is already newer than 305 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: oid 36 updating version 306 to 307 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 307 (ObjNum 294 snap 93 seq_num 294) dirty exists 2024-09-20T19:24:43.189 INFO:tasks.rados.rados.0.smithi042.stdout:809: left oid 36 (ObjNum 294 snap 93 seq_num 294) 2024-09-20T19:24:43.190 INFO:tasks.rados.rados.0.smithi042.stdout:810: finishing rollback tid 0 to smithi04268048-47 2024-09-20T19:24:43.190 INFO:tasks.rados.rados.0.smithi042.stdout:810: finishing rollback tid 1 to smithi04268048-47 2024-09-20T19:24:43.190 INFO:tasks.rados.rados.0.smithi042.stdout:810: finishing rollback tid 2 to smithi04268048-47 2024-09-20T19:24:43.190 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 401 (ObjNum 66 snap 10 seq_num 66) dirty exists 2024-09-20T19:24:43.234 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:43.234 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:43.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:43.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:43.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:43 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:43.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.023142+0000 mgr.y (mgr.24491) 257 : cluster [DBG] pgmap v377: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.623637+0000 osd.6 (osd.6) 97 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.625443+0000 osd.6 (osd.6) 98 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.028048+0000 osd.3 (osd.3) 135 : cluster [DBG] 6.c scrub starts 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.030123+0000 osd.3 (osd.3) 136 : cluster [DBG] 6.c scrub ok 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:43 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.073994+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-20T19:24:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:43.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:43.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:43.789 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:43.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:43.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:43.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:44.025 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:44.053 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:44.072 INFO:tasks.rados.rados.0.smithi042.stdout:809: done (2 left) 2024-09-20T19:24:44.073 INFO:tasks.rados.rados.0.smithi042.stdout:810: done (1 left) 2024-09-20T19:24:44.073 INFO:tasks.rados.rados.0.smithi042.stdout:811: done (0 left) 2024-09-20T19:24:44.073 INFO:tasks.rados.rados.0.smithi042.stdout:812: snap_remove snap 87 2024-09-20T19:24:44.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:44.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:44.255 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:44.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:44.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:44.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:44 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.786437+0000 osd.2 (osd.2) 121 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:42.788601+0000 osd.2 (osd.2) 122 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:24:44.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.311669+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.313645+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.582167+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:43.584138+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:44 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.073081+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-20T19:24:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:44.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:44.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:44.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:44.695 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:44.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:44.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:44.928 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:44.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:44.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:45.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.078 INFO:tasks.rados.rados.0.smithi042.stdout:812: done (0 left) 2024-09-20T19:24:45.079 INFO:tasks.rados.rados.0.smithi042.stdout:813: delete oid 23 current snap is 94 2024-09-20T19:24:45.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:45.095 INFO:tasks.rados.rados.0.smithi042.stdout:813: done (0 left) 2024-09-20T19:24:45.095 INFO:tasks.rados.rados.0.smithi042.stdout:814: write oid 31 current snap is 94 2024-09-20T19:24:45.095 INFO:tasks.rados.rados.0.smithi042.stdout:814: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-20T19:24:45.102 INFO:tasks.rados.rados.0.smithi042.stdout:814: writing smithi04268048-31 from 574872 to 1269990 tid 1 2024-09-20T19:24:45.111 INFO:tasks.rados.rados.0.smithi042.stdout:814: writing smithi04268048-31 from 2038317 to 2668740 tid 2 2024-09-20T19:24:45.111 INFO:tasks.rados.rados.0.smithi042.stdout:815: copy_from oid 29 from oid 21 current snap is 94 2024-09-20T19:24:45.111 INFO:tasks.rados.rados.0.smithi042.stdout:814: finishing write tid 1 to smithi04268048-31 2024-09-20T19:24:45.111 INFO:tasks.rados.rados.0.smithi042.stdout:816: snap_remove snap 89 2024-09-20T19:24:45.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:45.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:45.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:45.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:45 smithi162 bash[24557]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.582 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:45.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:45.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[32075]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.023849+0000 mgr.y (mgr.24491) 258 : cluster [DBG] pgmap v380: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 4 op/s 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.485855+0000 osd.1 (osd.1) 141 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.491355+0000 osd.1 (osd.1) 142 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.559301+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.d scrub starts 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.564755+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.d scrub ok 2024-09-20T19:24:45.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:45 smithi042 bash[22706]: cluster 2024-09-20T19:24:45.075179+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-20T19:24:45.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:45.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:45.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:45.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:45.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:46.082 INFO:tasks.rados.rados.0.smithi042.stdout:816: done (2 left) 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:817: read oid 34 snap -1 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:817: expect (ObjNum 263 snap 78 seq_num 263) 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:818: copy_from oid 37 from oid 28 current snap is 94 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:819: read oid 16 snap -1 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:819: expect deleted 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:820: write oid 50 current snap is 94 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:814: finishing write tid 2 to smithi04268048-31 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:814: finishing write tid 3 to smithi04268048-31 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:814: finishing write tid 4 to smithi04268048-31 2024-09-20T19:24:46.083 INFO:tasks.rados.rados.0.smithi042.stdout:814: oid 31 updating version 0 to 313 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:814: oid 31 version 313 is already newer than 311 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:814: oid 31 version 313 is already newer than 312 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 313 (ObjNum 296 snap 94 seq_num 296) dirty exists 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:814: left oid 31 (ObjNum 296 snap 94 seq_num 296) 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:815: finishing copy_from racing read to smithi04268048-29 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:815: finishing copy_from to smithi04268048-29 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 365 (ObjNum 274 snap 82 seq_num 274) dirty exists 2024-09-20T19:24:46.084 INFO:tasks.rados.rados.0.smithi042.stdout:820: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-20T19:24:46.094 INFO:tasks.rados.rados.0.smithi042.stdout:820: writing smithi04268048-50 from 591679 to 1078399 tid 1 2024-09-20T19:24:46.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:46.108 INFO:tasks.rados.rados.0.smithi042.stdout:820: writing smithi04268048-50 from 1813152 to 2303586 tid 2 2024-09-20T19:24:46.108 INFO:tasks.rados.rados.0.smithi042.stdout:814: done (5 left) 2024-09-20T19:24:46.108 INFO:tasks.rados.rados.0.smithi042.stdout:815: done (4 left) 2024-09-20T19:24:46.108 INFO:tasks.rados.rados.0.smithi042.stdout:821: delete oid 25 current snap is 94 2024-09-20T19:24:46.108 INFO:tasks.rados.rados.0.smithi042.stdout:817: expect (ObjNum 263 snap 78 seq_num 263) 2024-09-20T19:24:46.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:46.143 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1195377 2024-09-20T19:24:46.177 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: finishing write tid 1 to smithi04268048-50 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: finishing write tid 2 to smithi04268048-50 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: finishing write tid 3 to smithi04268048-50 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: finishing write tid 4 to smithi04268048-50 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: oid 50 updating version 0 to 407 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: oid 50 version 407 is already newer than 405 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:820: oid 50 version 407 is already newer than 406 2024-09-20T19:24:46.198 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 407 (ObjNum 297 snap 94 seq_num 297) dirty exists 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:820: left oid 50 (ObjNum 297 snap 94 seq_num 297) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:818: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:818: finishing copy_from to smithi04268048-37 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 324 (ObjNum 257 snap 75 seq_num 257) dirty exists 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:817: done (4 left) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:818: done (3 left) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:819: done (2 left) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:820: done (1 left) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:821: done (0 left) 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:822: write oid 30 current snap is 94 2024-09-20T19:24:46.199 INFO:tasks.rados.rados.0.smithi042.stdout:822: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-20T19:24:46.213 INFO:tasks.rados.rados.0.smithi042.stdout:822: writing smithi04268048-30 from 625293 to 1262511 tid 1 2024-09-20T19:24:46.226 INFO:tasks.rados.rados.0.smithi042.stdout:822: writing smithi04268048-30 from 1813763 to 2307866 tid 2 2024-09-20T19:24:46.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:822: writing smithi04268048-30 from 2978063 to 3025293 tid 3 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:823: copy_from oid 11 from oid 4 current snap is 94 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:822: finishing write tid 1 to smithi04268048-30 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:822: finishing write tid 2 to smithi04268048-30 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:824: copy_from oid 17 from oid 50 current snap is 94 2024-09-20T19:24:46.235 INFO:tasks.rados.rados.0.smithi042.stdout:825: rollback oid 29 current snap is 94 2024-09-20T19:24:46.236 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 85 2024-09-20T19:24:46.236 INFO:tasks.rados.rados.0.smithi042.stdout:826: write (excl) oid 10 current snap is 94 2024-09-20T19:24:46.236 INFO:tasks.rados.rados.0.smithi042.stdout:826: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-20T19:24:46.249 INFO:tasks.rados.rados.0.smithi042.stdout:826: writing smithi04268048-10 from 642100 to 1354567 tid 1 2024-09-20T19:24:46.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:46.262 INFO:tasks.rados.rados.0.smithi042.stdout:826: writing smithi04268048-10 from 1872245 to 2510006 tid 2 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:827: rollback oid 45 current snap is 94 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 63 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: finishing write tid 3 to smithi04268048-30 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: finishing write tid 4 to smithi04268048-30 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: finishing write tid 5 to smithi04268048-30 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: oid 30 updating version 0 to 392 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: oid 30 version 392 is already newer than 390 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: oid 30 version 392 is already newer than 391 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: oid 30 version 392 is already newer than 389 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 392 (ObjNum 299 snap 94 seq_num 299) dirty exists 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:822: left oid 30 (ObjNum 299 snap 94 seq_num 299) 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:825: finishing rollback tid 0 to smithi04268048-29 2024-09-20T19:24:46.263 INFO:tasks.rados.rados.0.smithi042.stdout:825: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:825: finishing rollback tid 2 to smithi04268048-29 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 368 (ObjNum 267 snap 82 seq_num 267) dirty exists 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:823: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:822: done (5 left) 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:825: done (4 left) 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:828: read oid 41 snap -1 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:828: expect (ObjNum 292 snap 93 seq_num 292) 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:829: rollback oid 7 current snap is 94 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 83 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:824: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:823: finishing copy_from to smithi04268048-11 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 340 (ObjNum 266 snap 80 seq_num 266) dirty exists 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:823: done (5 left) 2024-09-20T19:24:46.264 INFO:tasks.rados.rados.0.smithi042.stdout:830: write oid 6 current snap is 94 2024-09-20T19:24:46.265 INFO:tasks.rados.rados.0.smithi042.stdout:830: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-20T19:24:46.277 INFO:tasks.rados.rados.0.smithi042.stdout:830: writing smithi04268048-6 from 658907 to 1446623 tid 1 2024-09-20T19:24:46.286 INFO:tasks.rados.rados.0.smithi042.stdout:830: writing smithi04268048-6 from 1930727 to 2428499 tid 2 2024-09-20T19:24:46.292 INFO:tasks.rados.rados.0.smithi042.stdout:830: writing smithi04268048-6 from 3032909 to 3058907 tid 3 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:831: snap_create 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:829: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:826: finishing write tid 1 to smithi04268048-10 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:829: finishing rollback tid 2 to smithi04268048-7 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 256 (ObjNum 259 snap 76 seq_num 259) dirty exists 2024-09-20T19:24:46.293 INFO:tasks.rados.rados.0.smithi042.stdout:828: expect (ObjNum 292 snap 93 seq_num 292) 2024-09-20T19:24:46.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:46.383 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:46.416 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1470237 2024-09-20T19:24:46.440 INFO:tasks.rados.rados.0.smithi042.stdout:824: finishing copy_from to smithi04268048-17 2024-09-20T19:24:46.440 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 407 (ObjNum 297 snap 94 seq_num 297) dirty exists 2024-09-20T19:24:46.440 INFO:tasks.rados.rados.0.smithi042.stdout:827: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:24:46.440 INFO:tasks.rados.rados.0.smithi042.stdout:827: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: finishing write tid 2 to smithi04268048-10 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: finishing write tid 3 to smithi04268048-10 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: finishing write tid 4 to smithi04268048-10 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: oid 10 updating version 0 to 371 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: oid 10 version 371 is already newer than 370 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: oid 10 version 371 is already newer than 369 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 371 (ObjNum 300 snap 94 seq_num 300) dirty exists 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:826: left oid 10 (ObjNum 300 snap 94 seq_num 300) 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:827: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 358 (ObjNum 197 snap 59 seq_num 197) dirty exists 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:830: finishing write tid 1 to smithi04268048-6 2024-09-20T19:24:46.441 INFO:tasks.rados.rados.0.smithi042.stdout:830: finishing write tid 2 to smithi04268048-6 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: finishing write tid 3 to smithi04268048-6 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: finishing write tid 4 to smithi04268048-6 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: finishing write tid 5 to smithi04268048-6 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: oid 6 updating version 0 to 315 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: oid 6 version 315 is already newer than 314 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: oid 6 updating version 315 to 316 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: oid 6 updating version 316 to 317 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 317 (ObjNum 301 snap 94 seq_num 301) dirty exists 2024-09-20T19:24:46.442 INFO:tasks.rados.rados.0.smithi042.stdout:830: left oid 6 (ObjNum 301 snap 94 seq_num 301) 2024-09-20T19:24:46.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:46.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:46 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:46.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:46.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.922330+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:44.924853+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.016046+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.018389+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:24:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:46 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.081072+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-20T19:24:46.744 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:46.769 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:46.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:46.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:46.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:47.003 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:47.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:47.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:47.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:47.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:824: done (6 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:826: done (5 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:827: done (4 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:828: done (3 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:829: done (2 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:830: done (1 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:831: done (0 left) 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:832: write (excl) oid 25 current snap is 95 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:832: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-20T19:24:47.301 INFO:tasks.rados.rados.0.smithi042.stdout:832: writing smithi04268048-25 from 675714 to 1138679 tid 1 2024-09-20T19:24:47.306 INFO:tasks.rados.rados.0.smithi042.stdout:832: writing smithi04268048-25 from 1589209 to 2230639 tid 2 2024-09-20T19:24:47.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:832: writing smithi04268048-25 from 2943979 to 3075714 tid 3 2024-09-20T19:24:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:833: read oid 15 snap 93 2024-09-20T19:24:47.312 INFO:tasks.rados.rados.0.smithi042.stdout:833: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:24:47.312 INFO:tasks.rados.rados.0.smithi042.stdout:834: delete oid 44 current snap is 95 2024-09-20T19:24:47.315 INFO:tasks.rados.rados.0.smithi042.stdout:834: done (2 left) 2024-09-20T19:24:47.316 INFO:tasks.rados.rados.0.smithi042.stdout:835: read oid 26 snap 94 2024-09-20T19:24:47.316 INFO:tasks.rados.rados.0.smithi042.stdout:835: expect (ObjNum 108 snap 25 seq_num 108) 2024-09-20T19:24:47.316 INFO:tasks.rados.rados.0.smithi042.stdout:836: snap_create 2024-09-20T19:24:47.318 INFO:tasks.rados.rados.0.smithi042.stdout:832: finishing write tid 1 to smithi04268048-25 2024-09-20T19:24:47.318 INFO:tasks.rados.rados.0.smithi042.stdout:832: finishing write tid 2 to smithi04268048-25 2024-09-20T19:24:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:835: expect (ObjNum 108 snap 25 seq_num 108) 2024-09-20T19:24:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 20866 2024-09-20T19:24:47.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:47.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.423 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:47 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:47.520 INFO:tasks.rados.rados.0.smithi042.stdout:832: finishing write tid 3 to smithi04268048-25 2024-09-20T19:24:47.520 INFO:tasks.rados.rados.0.smithi042.stdout:832: finishing write tid 4 to smithi04268048-25 2024-09-20T19:24:47.520 INFO:tasks.rados.rados.0.smithi042.stdout:832: finishing write tid 5 to smithi04268048-25 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:832: oid 25 updating version 0 to 396 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:832: oid 25 version 396 is already newer than 395 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:832: oid 25 version 396 is already newer than 393 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:832: oid 25 version 396 is already newer than 394 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 396 (ObjNum 302 snap 95 seq_num 302) dirty exists 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:832: left oid 25 (ObjNum 302 snap 95 seq_num 302) 2024-09-20T19:24:47.521 INFO:tasks.rados.rados.0.smithi042.stdout:833: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:24:47.529 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:47.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:47.547 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:47.547 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:47.551 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:47.560 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.013522+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.015723+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.024579+0000 mgr.y (mgr.24491) 259 : cluster [DBG] pgmap v383: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 245 MiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.295900+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.297116+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.976714+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:24:47.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.565 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:46.978787+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:24:47.565 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.565 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:47 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.292107+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-20T19:24:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:47.566 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:47.566 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:47.583 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:47.593 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1946344 2024-09-20T19:24:47.601 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:47.619 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:47.640 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:47.640 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:47.662 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:47.662 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:47.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:47.675 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:47.675 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:47.686 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:47.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:47.807 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:47.910 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:47.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:47.989 INFO:teuthology.orchestra.run.smithi042.stdout:moved 0 / 474 (0%) 2024-09-20T19:24:47.989 INFO:teuthology.orchestra.run.smithi042.stdout:avg 59.25 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:stddev 5.35607 -> 5.35607 (expected baseline 7.20026) 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:min osd.2 with 52 -> 52 pgs (0.877637 -> 0.877637 * mean) 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:max osd.3 with 70 -> 70 pgs (1.18143 -> 1.18143 * mean) 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:oload 1000 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:max_change 1 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:max_change_osds 4 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:average_utilization 0.0012 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stdout:overload_utilization 0.0121 2024-09-20T19:24:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:no change 2024-09-20T19:24:48.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:48.070 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:24:48.070 INFO:tasks.rados.rados.0.smithi042.stdout:832: done (3 left) 2024-09-20T19:24:48.070 INFO:tasks.rados.rados.0.smithi042.stdout:833: done (2 left) 2024-09-20T19:24:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:835: done (1 left) 2024-09-20T19:24:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:836: done (0 left) 2024-09-20T19:24:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:837: write oid 22 current snap is 96 2024-09-20T19:24:48.071 INFO:tasks.rados.rados.0.smithi042.stdout:837: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-20T19:24:48.079 INFO:tasks.rados.rados.0.smithi042.stdout:837: writing smithi04268048-22 from 709328 to 1322791 tid 1 2024-09-20T19:24:48.087 INFO:tasks.rados.rados.0.smithi042.stdout:837: writing smithi04268048-22 from 1989820 to 2634919 tid 2 2024-09-20T19:24:48.087 INFO:tasks.rados.rados.0.smithi042.stdout:838: read oid 28 snap -1 2024-09-20T19:24:48.087 INFO:tasks.rados.rados.0.smithi042.stdout:838: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:48.087 INFO:tasks.rados.rados.0.smithi042.stdout:839: snap_create 2024-09-20T19:24:48.089 INFO:tasks.rados.rados.0.smithi042.stdout:837: finishing write tid 1 to smithi04268048-22 2024-09-20T19:24:48.093 INFO:tasks.rados.rados.0.smithi042.stdout:838: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:48.103 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 981507 2024-09-20T19:24:48.152 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 109132 2024-09-20T19:24:48.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:48.184 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:48.186 INFO:tasks.rados.rados.0.smithi042.stdout:837: finishing write tid 2 to smithi04268048-22 2024-09-20T19:24:48.186 INFO:tasks.rados.rados.0.smithi042.stdout:837: finishing write tid 3 to smithi04268048-22 2024-09-20T19:24:48.186 INFO:tasks.rados.rados.0.smithi042.stdout:837: finishing write tid 4 to smithi04268048-22 2024-09-20T19:24:48.186 INFO:tasks.rados.rados.0.smithi042.stdout:837: oid 22 updating version 0 to 374 2024-09-20T19:24:48.186 INFO:tasks.rados.rados.0.smithi042.stdout:837: oid 22 updating version 374 to 375 2024-09-20T19:24:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:837: oid 22 updating version 375 to 376 2024-09-20T19:24:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 376 (ObjNum 304 snap 96 seq_num 304) dirty exists 2024-09-20T19:24:48.187 INFO:tasks.rados.rados.0.smithi042.stdout:837: left oid 22 (ObjNum 304 snap 96 seq_num 304) 2024-09-20T19:24:48.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:48.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:48.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:24:48.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:48 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:48.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:48.575 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.508405+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? v1:172.21.15.42:0/1163348064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.511891+0000 osd.1 (osd.1) 143 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:47.513031+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.986885+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.42:0/1957307442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:47.987216+0000 mon.a (mon.0) 1294 : audit [INF] from='client.25025 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.066518+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.576 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: audit 2024-09-20T19:24:48.067545+0000 mon.a (mon.0) 1296 : audit [INF] from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:24:48.577 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.577 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:48 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.070926+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-20T19:24:48.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:48.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:48.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:48.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:48.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:48.975 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:48.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:49.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:837: done (2 left) 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:838: done (1 left) 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:839: done (0 left) 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:840: read oid 36 snap -1 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:840: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:841: rollback oid 12 current snap is 97 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 93 2024-09-20T19:24:49.084 INFO:tasks.rados.rados.0.smithi042.stdout:842: snap_remove snap 95 2024-09-20T19:24:49.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:49.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:49.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:49.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[32075]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:49 smithi042 bash[22706]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:49.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:49.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: audit 2024-09-20T19:24:47.988508+0000 mgr.y (mgr.24491) 260 : audit [DBG] from='client.15174 v1:172.21.15.42:0/1006408902' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: cluster 2024-09-20T19:24:48.025082+0000 mgr.y (mgr.24491) 261 : cluster [DBG] pgmap v386: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:49 smithi162 bash[24557]: cluster 2024-09-20T19:24:49.083596+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-20T19:24:49.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:49.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:49.630 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:49.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.759 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:49.835 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-20T19:24:49.835 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-20T19:24:49.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:49.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:49.991 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.018 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:50.082 INFO:tasks.rados.rados.0.smithi042.stdout:842: done (2 left) 2024-09-20T19:24:50.082 INFO:tasks.rados.rados.0.smithi042.stdout:843: snap_create 2024-09-20T19:24:50.082 INFO:tasks.rados.rados.0.smithi042.stdout:840: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-20T19:24:50.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:50.124 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:50.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:50.188 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 732459 2024-09-20T19:24:50.228 INFO:tasks.rados.rados.0.smithi042.stdout:841: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:24:50.228 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 184 (ObjNum 225 snap 0 seq_num 3047886256) dirty dne 2024-09-20T19:24:50.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:50.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:50.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:50.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:50 smithi162 bash[24557]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:50 smithi162 bash[24557]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:50 smithi162 bash[24557]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:50 smithi162 bash[24557]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:50.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:50.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[22706]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[22706]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[22706]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[22706]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.650 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[32075]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.650 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[32075]: audit 2024-09-20T19:24:49.176319+0000 mgr.y (mgr.24491) 262 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:24:50.650 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[32075]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.650 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:50 smithi042 bash[32075]: cluster 2024-09-20T19:24:50.080874+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-20T19:24:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:50.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:24:50.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:50.798 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:50.901 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:50.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:50.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:51.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:840: done (2 left) 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:841: done (1 left) 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:843: done (0 left) 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:844: read oid 41 snap -1 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:844: expect (ObjNum 292 snap 93 seq_num 292) 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:845: write (excl) oid 3 current snap is 98 2024-09-20T19:24:51.085 INFO:tasks.rados.rados.0.smithi042.stdout:845: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-20T19:24:51.091 INFO:tasks.rados.rados.0.smithi042.stdout:845: writing smithi04268048-3 from 726135 to 1131200 tid 1 2024-09-20T19:24:51.101 INFO:tasks.rados.rados.0.smithi042.stdout:845: writing smithi04268048-3 from 1764655 to 2553412 tid 2 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:845: writing smithi04268048-3 from 3026248 to 3126135 tid 3 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:846: read oid 25 snap -1 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:846: expect (ObjNum 302 snap 95 seq_num 302) 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:847: read oid 10 snap -1 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:847: expect (ObjNum 300 snap 94 seq_num 300) 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:848: snap_create 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:844: expect (ObjNum 292 snap 93 seq_num 292) 2024-09-20T19:24:51.105 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:51.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:51.208 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: finishing write tid 1 to smithi04268048-3 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: finishing write tid 2 to smithi04268048-3 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: finishing write tid 3 to smithi04268048-3 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: finishing write tid 4 to smithi04268048-3 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: finishing write tid 5 to smithi04268048-3 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: oid 3 updating version 0 to 410 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: oid 3 updating version 410 to 411 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: oid 3 version 411 is already newer than 408 2024-09-20T19:24:51.249 INFO:tasks.rados.rados.0.smithi042.stdout:845: oid 3 version 411 is already newer than 409 2024-09-20T19:24:51.250 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 411 (ObjNum 305 snap 98 seq_num 305) dirty exists 2024-09-20T19:24:51.250 INFO:tasks.rados.rados.0.smithi042.stdout:845: left oid 3 (ObjNum 305 snap 98 seq_num 305) 2024-09-20T19:24:51.250 INFO:tasks.rados.rados.0.smithi042.stdout:846: expect (ObjNum 302 snap 95 seq_num 302) 2024-09-20T19:24:51.258 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1699284 2024-09-20T19:24:51.273 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 300999 2024-09-20T19:24:51.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.304 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 243885 2024-09-20T19:24:51.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:51.338 INFO:tasks.rados.rados.0.smithi042.stdout:847: expect (ObjNum 300 snap 94 seq_num 300) 2024-09-20T19:24:51.338 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:51.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[32075]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[32075]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[32075]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[32075]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[22706]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[22706]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[22706]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:51 smithi042 bash[22706]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.439 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:51.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:51.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:51 smithi162 bash[24557]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:51 smithi162 bash[24557]: cluster 2024-09-20T19:24:50.025484+0000 mgr.y (mgr.24491) 263 : cluster [DBG] pgmap v389: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 272 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-20T19:24:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:51 smithi162 bash[24557]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:51 smithi162 bash[24557]: cluster 2024-09-20T19:24:51.082717+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-20T19:24:51.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:51.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:51.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:51.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:51.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:51.864 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:24:51.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:51.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:52.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:52.085 INFO:tasks.rados.rados.0.smithi042.stdout:844: done (4 left) 2024-09-20T19:24:52.085 INFO:tasks.rados.rados.0.smithi042.stdout:845: done (3 left) 2024-09-20T19:24:52.085 INFO:tasks.rados.rados.0.smithi042.stdout:846: done (2 left) 2024-09-20T19:24:52.085 INFO:tasks.rados.rados.0.smithi042.stdout:847: done (1 left) 2024-09-20T19:24:52.085 INFO:tasks.rados.rados.0.smithi042.stdout:848: done (0 left) 2024-09-20T19:24:52.086 INFO:tasks.rados.rados.0.smithi042.stdout:849: write oid 37 current snap is 99 2024-09-20T19:24:52.086 INFO:tasks.rados.rados.0.smithi042.stdout:849: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-20T19:24:52.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:52.091 INFO:tasks.rados.rados.0.smithi042.stdout:849: writing smithi04268048-37 from 742942 to 1223256 tid 1 2024-09-20T19:24:52.098 INFO:tasks.rados.rados.0.smithi042.stdout:849: writing smithi04268048-37 from 1823137 to 2471905 tid 2 2024-09-20T19:24:52.099 INFO:tasks.rados.rados.0.smithi042.stdout:850: write oid 2 current snap is 99 2024-09-20T19:24:52.099 INFO:tasks.rados.rados.0.smithi042.stdout:849: finishing write tid 1 to smithi04268048-37 2024-09-20T19:24:52.099 INFO:tasks.rados.rados.0.smithi042.stdout:850: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-20T19:24:52.106 INFO:tasks.rados.rados.0.smithi042.stdout:850: writing smithi04268048-2 from 759749 to 1315312 tid 1 2024-09-20T19:24:52.113 INFO:tasks.rados.rados.0.smithi042.stdout:850: writing smithi04268048-2 from 1765266 to 2557692 tid 2 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:850: writing smithi04268048-2 from 2964741 to 3159749 tid 3 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:851: read oid 40 snap -1 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:851: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:852: read oid 5 snap -1 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:852: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:853: rollback oid 3 current snap is 99 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 85 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:849: finishing write tid 2 to smithi04268048-37 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:849: finishing write tid 3 to smithi04268048-37 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:849: finishing write tid 4 to smithi04268048-37 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:849: oid 37 updating version 0 to 326 2024-09-20T19:24:52.117 INFO:tasks.rados.rados.0.smithi042.stdout:849: oid 37 updating version 326 to 327 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:849: oid 37 version 327 is already newer than 325 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 327 (ObjNum 306 snap 99 seq_num 306) dirty exists 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:849: left oid 37 (ObjNum 306 snap 99 seq_num 306) 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:849: done (4 left) 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:854: read oid 13 snap -1 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:854: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:52.118 INFO:tasks.rados.rados.0.smithi042.stdout:855: delete oid 42 current snap is 99 2024-09-20T19:24:52.124 INFO:tasks.rados.rados.0.smithi042.stdout:850: finishing write tid 1 to smithi04268048-2 2024-09-20T19:24:52.125 INFO:tasks.rados.rados.0.smithi042.stdout:854: expect (ObjNum 269 snap 82 seq_num 269) 2024-09-20T19:24:52.129 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1462577 2024-09-20T19:24:52.145 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:52.175 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1289479 2024-09-20T19:24:52.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:52.193 INFO:tasks.rados.rados.0.smithi042.stdout:852: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-20T19:24:52.216 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:52.236 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:52.279 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:52.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:52.322 INFO:tasks.rados.rados.0.smithi042.stdout:853: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:24:52.322 INFO:tasks.rados.rados.0.smithi042.stdout:853: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:24:52.322 INFO:tasks.rados.rados.0.smithi042.stdout:850: finishing write tid 2 to smithi04268048-2 2024-09-20T19:24:52.322 INFO:tasks.rados.rados.0.smithi042.stdout:851: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:24:52.327 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 636579 2024-09-20T19:24:52.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:52.360 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1288227 2024-09-20T19:24:52.381 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1080268 2024-09-20T19:24:52.397 INFO:tasks.rados.rados.0.smithi042.stdout:853: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 414 (ObjNum 229 snap 67 seq_num 229) dirty exists 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: finishing write tid 3 to smithi04268048-2 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: finishing write tid 4 to smithi04268048-2 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: finishing write tid 5 to smithi04268048-2 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: oid 2 updating version 0 to 173 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: oid 2 updating version 173 to 174 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: oid 2 version 174 is already newer than 171 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: oid 2 version 174 is already newer than 172 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 174 (ObjNum 307 snap 99 seq_num 307) dirty exists 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: left oid 2 (ObjNum 307 snap 99 seq_num 307) 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:850: done (5 left) 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:851: done (4 left) 2024-09-20T19:24:52.398 INFO:tasks.rados.rados.0.smithi042.stdout:852: done (3 left) 2024-09-20T19:24:52.399 INFO:tasks.rados.rados.0.smithi042.stdout:853: done (2 left) 2024-09-20T19:24:52.399 INFO:tasks.rados.rados.0.smithi042.stdout:854: done (1 left) 2024-09-20T19:24:52.399 INFO:tasks.rados.rados.0.smithi042.stdout:855: done (0 left) 2024-09-20T19:24:52.399 INFO:tasks.rados.rados.0.smithi042.stdout:856: write (excl) oid 44 current snap is 99 2024-09-20T19:24:52.399 INFO:tasks.rados.rados.0.smithi042.stdout:856: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-20T19:24:52.415 INFO:tasks.rados.rados.0.smithi042.stdout:856: writing smithi04268048-44 from 793363 to 1499424 tid 1 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:856: writing smithi04268048-44 from 2282230 to 3078325 tid 2 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:857: rollback oid 11 current snap is 99 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 88 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:856: finishing write tid 1 to smithi04268048-44 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:858: read oid 10 snap -1 2024-09-20T19:24:52.434 INFO:tasks.rados.rados.0.smithi042.stdout:858: expect (ObjNum 300 snap 94 seq_num 300) 2024-09-20T19:24:52.435 INFO:tasks.rados.rados.0.smithi042.stdout:859: write oid 21 current snap is 99 2024-09-20T19:24:52.435 INFO:tasks.rados.rados.0.smithi042.stdout:859: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-20T19:24:52.448 INFO:tasks.rados.rados.0.smithi042.stdout:859: writing smithi04268048-21 from 410170 to 1191480 tid 1 2024-09-20T19:24:52.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:52.463 INFO:tasks.rados.rados.0.smithi042.stdout:859: writing smithi04268048-21 from 1940712 to 2596818 tid 2 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:859: writing smithi04268048-21 from 3163363 to 3210170 tid 3 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:860: copy_from oid 34 from oid 15 current snap is 99 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:856: finishing write tid 2 to smithi04268048-44 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:857: finishing rollback tid 0 to smithi04268048-11 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:856: finishing write tid 3 to smithi04268048-44 2024-09-20T19:24:52.468 INFO:tasks.rados.rados.0.smithi042.stdout:856: finishing write tid 4 to smithi04268048-44 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:856: oid 44 updating version 0 to 361 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:856: oid 44 updating version 361 to 363 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:856: oid 44 version 363 is already newer than 362 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 363 (ObjNum 309 snap 99 seq_num 309) dirty exists 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:856: left oid 44 (ObjNum 309 snap 99 seq_num 309) 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:857: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:857: finishing rollback tid 2 to smithi04268048-11 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 419 (ObjNum 278 snap 84 seq_num 278) dirty exists 2024-09-20T19:24:52.469 INFO:tasks.rados.rados.0.smithi042.stdout:858: expect (ObjNum 300 snap 94 seq_num 300) 2024-09-20T19:24:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:52.543 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1933516 2024-09-20T19:24:52.560 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:52.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:52.595 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:859: finishing write tid 1 to smithi04268048-21 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:859: finishing write tid 2 to smithi04268048-21 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:856: done (4 left) 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:857: done (3 left) 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:858: done (2 left) 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:861: write oid 22 current snap is 99 2024-09-20T19:24:52.617 INFO:tasks.rados.rados.0.smithi042.stdout:859: finishing write tid 3 to smithi04268048-21 2024-09-20T19:24:52.618 INFO:tasks.rados.rados.0.smithi042.stdout:861: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-20T19:24:52.621 INFO:tasks.rados.rados.0.smithi042.stdout:861: writing smithi04268048-22 from 426977 to 883536 tid 1 2024-09-20T19:24:52.629 INFO:tasks.rados.rados.0.smithi042.stdout:861: writing smithi04268048-22 from 1482841 to 2282605 tid 2 2024-09-20T19:24:52.635 INFO:tasks.rados.rados.0.smithi042.stdout:861: writing smithi04268048-22 from 2958080 to 3226977 tid 3 2024-09-20T19:24:52.635 INFO:tasks.rados.rados.0.smithi042.stdout:862: snap_remove snap 83 2024-09-20T19:24:52.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:52.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:52.828 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:52.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:52.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:52.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:52.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:53.078 INFO:tasks.rados.rados.0.smithi042.stdout:862: done (3 left) 2024-09-20T19:24:53.079 INFO:tasks.rados.rados.0.smithi042.stdout:863: snap_remove snap 94 2024-09-20T19:24:53.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:53.198 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:24:53.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:53.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:53.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:53 smithi162 bash[24557]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:53.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:24:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[22706]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:52.027236+0000 mgr.y (mgr.24491) 264 : cluster [DBG] pgmap v392: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 926 KiB/s wr, 21 op/s 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:52.084475+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-20T19:24:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:53 smithi042 bash[32075]: cluster 2024-09-20T19:24:53.077659+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-20T19:24:53.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:53.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:53.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:53.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:53.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:53.991 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:54.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:863: done (3 left) 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:864: copy_from oid 13 from oid 14 current snap is 99 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: finishing write tid 4 to smithi04268048-21 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: finishing write tid 5 to smithi04268048-21 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: oid 21 updating version 0 to 400 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: oid 21 version 400 is already newer than 399 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: oid 21 version 400 is already newer than 398 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: oid 21 updating version 400 to 401 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 401 (ObjNum 310 snap 99 seq_num 310) dirty exists 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: left oid 21 (ObjNum 310 snap 99 seq_num 310) 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:860: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:24:54.092 INFO:tasks.rados.rados.0.smithi042.stdout:859: done (3 left) 2024-09-20T19:24:54.093 INFO:tasks.rados.rados.0.smithi042.stdout:865: rollback oid 18 current snap is 99 2024-09-20T19:24:54.093 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 63 2024-09-20T19:24:54.093 INFO:tasks.rados.rados.0.smithi042.stdout:866: snap_remove snap 98 2024-09-20T19:24:54.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:54.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:54 smithi162 bash[24557]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:54 smithi162 bash[24557]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.139 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:54.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:54.261 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:54.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:54.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:54 smithi042 bash[32075]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:54 smithi042 bash[32075]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.374 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:54 smithi042 bash[22706]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.374 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:54 smithi042 bash[22706]: audit 2024-09-20T19:24:53.657972+0000 mon.a (mon.0) 1303 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:24:54.386 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:54.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:54.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:54.633 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:54.634 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:54.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:54.711 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:24:54.712 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:24:54.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:54.765 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:54.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:54.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:54.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:55.094 INFO:tasks.rados.rados.0.smithi042.stdout:866: done (4 left) 2024-09-20T19:24:55.094 INFO:tasks.rados.rados.0.smithi042.stdout:867: write (excl) oid 10 current snap is 99 2024-09-20T19:24:55.094 INFO:tasks.rados.rados.0.smithi042.stdout:867: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-20T19:24:55.103 INFO:tasks.rados.rados.0.smithi042.stdout:867: writing smithi04268048-10 from 443784 to 1091945 tid 1 2024-09-20T19:24:55.115 INFO:tasks.rados.rados.0.smithi042.stdout:867: writing smithi04268048-10 from 1657676 to 2317451 tid 2 2024-09-20T19:24:55.123 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:55.124 INFO:tasks.rados.rados.0.smithi042.stdout:867: writing smithi04268048-10 from 2818209 to 3243784 tid 3 2024-09-20T19:24:55.124 INFO:tasks.rados.rados.0.smithi042.stdout:868: write (excl) oid 35 current snap is 99 2024-09-20T19:24:55.124 INFO:tasks.rados.rados.0.smithi042.stdout:868: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-20T19:24:55.135 INFO:tasks.rados.rados.0.smithi042.stdout:868: writing smithi04268048-35 from 460591 to 1184001 tid 1 2024-09-20T19:24:55.143 INFO:tasks.rados.rados.0.smithi042.stdout:868: writing smithi04268048-35 from 1716158 to 2119591 tid 2 2024-09-20T19:24:55.150 INFO:tasks.rados.rados.0.smithi042.stdout:868: writing smithi04268048-35 from 2729279 to 3169419 tid 3 2024-09-20T19:24:55.150 INFO:tasks.rados.rados.0.smithi042.stdout:869: write (excl) oid 21 current snap is 99 2024-09-20T19:24:55.150 INFO:tasks.rados.rados.0.smithi042.stdout:860: finishing copy_from to smithi04268048-34 2024-09-20T19:24:55.150 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: finishing write tid 1 to smithi04268048-22 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: finishing write tid 2 to smithi04268048-22 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: finishing write tid 3 to smithi04268048-22 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: finishing write tid 4 to smithi04268048-22 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: finishing write tid 5 to smithi04268048-22 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: oid 22 updating version 0 to 380 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: oid 22 version 380 is already newer than 377 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: oid 22 version 380 is already newer than 378 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: oid 22 version 380 is already newer than 379 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 380 (ObjNum 311 snap 99 seq_num 311) dirty exists 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:861: left oid 22 (ObjNum 311 snap 99 seq_num 311) 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:864: finishing copy_from to smithi04268048-13 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:864: got expected ENOENT (src dne) 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:864: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:24:55.151 INFO:tasks.rados.rados.0.smithi042.stdout:865: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:865: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:865: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 407 (ObjNum 212 snap 63 seq_num 212) dirty exists 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: finishing write tid 1 to smithi04268048-10 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: finishing write tid 2 to smithi04268048-10 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: finishing write tid 3 to smithi04268048-10 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: finishing write tid 4 to smithi04268048-10 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: finishing write tid 5 to smithi04268048-10 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: oid 10 updating version 0 to 373 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: oid 10 updating version 373 to 374 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: oid 10 version 374 is already newer than 372 2024-09-20T19:24:55.152 INFO:tasks.rados.rados.0.smithi042.stdout:867: oid 10 updating version 374 to 375 2024-09-20T19:24:55.153 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 375 (ObjNum 312 snap 99 seq_num 312) dirty exists 2024-09-20T19:24:55.153 INFO:tasks.rados.rados.0.smithi042.stdout:867: left oid 10 (ObjNum 312 snap 99 seq_num 312) 2024-09-20T19:24:55.153 INFO:tasks.rados.rados.0.smithi042.stdout:868: finishing write tid 1 to smithi04268048-35 2024-09-20T19:24:55.153 INFO:tasks.rados.rados.0.smithi042.stdout:868: finishing write tid 2 to smithi04268048-35 2024-09-20T19:24:55.153 INFO:tasks.rados.rados.0.smithi042.stdout:869: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-20T19:24:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.154 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:55.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:55.157 INFO:tasks.rados.rados.0.smithi042.stdout:869: writing smithi04268048-21 from 477398 to 1276057 tid 1 2024-09-20T19:24:55.163 INFO:tasks.rados.rados.0.smithi042.stdout:869: writing smithi04268048-21 from 1774640 to 2438084 tid 2 2024-09-20T19:24:55.168 INFO:tasks.rados.rados.0.smithi042.stdout:869: writing smithi04268048-21 from 2873055 to 3277398 tid 3 2024-09-20T19:24:55.168 INFO:tasks.rados.rados.0.smithi042.stdout:860: done (6 left) 2024-09-20T19:24:55.168 INFO:tasks.rados.rados.0.smithi042.stdout:861: done (5 left) 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:864: done (4 left) 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:865: done (3 left) 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:867: done (2 left) 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:870: snap_create 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:868: finishing write tid 3 to smithi04268048-35 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:868: finishing write tid 4 to smithi04268048-35 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:868: finishing write tid 5 to smithi04268048-35 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:868: oid 35 updating version 0 to 317 2024-09-20T19:24:55.169 INFO:tasks.rados.rados.0.smithi042.stdout:868: oid 35 version 317 is already newer than 315 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:868: oid 35 version 317 is already newer than 314 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:868: oid 35 version 317 is already newer than 316 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 317 (ObjNum 313 snap 99 seq_num 313) dirty exists 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:868: left oid 35 (ObjNum 313 snap 99 seq_num 313) 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:869: finishing write tid 1 to smithi04268048-21 2024-09-20T19:24:55.170 INFO:tasks.rados.rados.0.smithi042.stdout:869: finishing write tid 2 to smithi04268048-21 2024-09-20T19:24:55.171 INFO:tasks.rados.rados.0.smithi042.stdout:869: finishing write tid 3 to smithi04268048-21 2024-09-20T19:24:55.171 INFO:tasks.rados.rados.0.smithi042.stdout:869: finishing write tid 4 to smithi04268048-21 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: finishing write tid 5 to smithi04268048-21 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: oid 21 updating version 0 to 416 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: oid 21 version 416 is already newer than 415 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: oid 21 version 416 is already newer than 414 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: oid 21 updating version 416 to 417 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 417 (ObjNum 314 snap 99 seq_num 314) dirty exists 2024-09-20T19:24:55.172 INFO:tasks.rados.rados.0.smithi042.stdout:869: left oid 21 (ObjNum 314 snap 99 seq_num 314) 2024-09-20T19:24:55.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:55.283 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:55 smithi162 bash[24557]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:55.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[22706]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:54.027809+0000 mgr.y (mgr.24491) 265 : cluster [DBG] pgmap v395: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 925 KiB/s wr, 21 op/s 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:54.090373+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-20T19:24:55.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.420802+0000 mon.a (mon.0) 1305 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.421763+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: audit 2024-09-20T19:24:54.427810+0000 mon.a (mon.0) 1307 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:55 smithi042 bash[32075]: cluster 2024-09-20T19:24:55.092414+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-20T19:24:55.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:55.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:55.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:55.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:55.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:56.033 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:56.123 INFO:tasks.rados.rados.0.smithi042.stdout:868: done (2 left) 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:869: done (1 left) 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:870: done (0 left) 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:871: rollback oid 44 current snap is 100 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 60 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:872: read oid 19 snap 63 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:872: expect (ObjNum 202 snap 59 seq_num 202) 2024-09-20T19:24:56.124 INFO:tasks.rados.rados.0.smithi042.stdout:873: snap_remove snap 93 2024-09-20T19:24:56.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:56.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:56.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:56.306 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:24:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:24:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:24:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:56.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:56.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:56.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:56.584 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:24:56.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:56.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:56.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:56.949 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:56.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:56.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:57.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:57.141 INFO:tasks.rados.rados.0.smithi042.stdout:873: done (2 left) 2024-09-20T19:24:57.141 INFO:tasks.rados.rados.0.smithi042.stdout:874: rollback oid 21 current snap is 100 2024-09-20T19:24:57.141 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 88 2024-09-20T19:24:57.141 INFO:tasks.rados.rados.0.smithi042.stdout:871: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:24:57.141 INFO:tasks.rados.rados.0.smithi042.stdout:872: expect (ObjNum 202 snap 59 seq_num 202) 2024-09-20T19:24:57.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:57.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:57.296 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:57.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:871: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 374 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:871: done (2 left) 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:872: done (1 left) 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:875: snap_create 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:874: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:874: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:874: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:24:57.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 421 (ObjNum 274 snap 82 seq_num 274) dirty exists 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:57 smithi162 bash[24557]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[32075]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.028614+0000 mgr.y (mgr.24491) 266 : cluster [DBG] pgmap v398: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 279 MiB data, 916 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.123997+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.899273+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: cluster 2024-09-20T19:24:56.900543+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: audit 2024-09-20T19:24:57.070125+0000 mon.a (mon.0) 1310 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:57 smithi042 bash[22706]: audit 2024-09-20T19:24:57.072397+0000 mon.a (mon.0) 1311 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:24:57.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:57.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:57.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:57.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:57.727 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:57.749 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:57.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:57.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:57.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:24:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:24:58.082 INFO:tasks.rados.rados.0.smithi042.stdout:874: done (1 left) 2024-09-20T19:24:58.082 INFO:tasks.rados.rados.0.smithi042.stdout:875: done (0 left) 2024-09-20T19:24:58.082 INFO:tasks.rados.rados.0.smithi042.stdout:876: read oid 28 snap -1 2024-09-20T19:24:58.082 INFO:tasks.rados.rados.0.smithi042.stdout:876: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:58.082 INFO:tasks.rados.rados.0.smithi042.stdout:877: delete oid 39 current snap is 101 2024-09-20T19:24:58.090 INFO:tasks.rados.rados.0.smithi042.stdout:876: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:24:58.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:58.179 INFO:tasks.rados.rados.0.smithi042.stdout:876: done (1 left) 2024-09-20T19:24:58.179 INFO:tasks.rados.rados.0.smithi042.stdout:877: done (0 left) 2024-09-20T19:24:58.179 INFO:tasks.rados.rados.0.smithi042.stdout:878: write oid 35 current snap is 101 2024-09-20T19:24:58.179 INFO:tasks.rados.rados.0.smithi042.stdout:878: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-20T19:24:58.184 INFO:tasks.rados.rados.0.smithi042.stdout:878: writing smithi04268048-35 from 511012 to 1060169 tid 1 2024-09-20T19:24:58.192 INFO:tasks.rados.rados.0.smithi042.stdout:878: writing smithi04268048-35 from 1775251 to 2326011 tid 2 2024-09-20T19:24:58.198 INFO:tasks.rados.rados.0.smithi042.stdout:878: writing smithi04268048-35 from 3095195 to 3311012 tid 3 2024-09-20T19:24:58.198 INFO:tasks.rados.rados.0.smithi042.stdout:879: read oid 32 snap 92 2024-09-20T19:24:58.199 INFO:tasks.rados.rados.0.smithi042.stdout:879: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:24:58.199 INFO:tasks.rados.rados.0.smithi042.stdout:880: snap_remove snap 77 2024-09-20T19:24:58.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:58.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:58.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:58.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[22706]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[22706]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[32075]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[32075]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:58 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.385 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:24:58.386 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:24:58.396 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:58 smithi162 bash[24557]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:58 smithi162 bash[24557]: cluster 2024-09-20T19:24:57.139985+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-20T19:24:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:58 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:58 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.081104+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-20T19:24:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:58.405 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:24:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:24:58.425 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:24:58.425 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:24:58.453 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:24:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:24:58.474 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:24:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:24:58.497 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:24:58.497 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:24:58.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:58.518 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:24:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:24:58.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:58.541 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:24:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:24:58.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.642 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stderr:no change 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout:moved 0 / 474 (0%) 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout:avg 59.25 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout:stddev 5.35607 -> 5.35607 (expected baseline 7.20026) 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout:min osd.2 with 52 -> 52 pgs (0.877637 -> 0.877637 * mean) 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout:max osd.3 with 70 -> 70 pgs (1.18143 -> 1.18143 * mean) 2024-09-20T19:24:58.728 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:24:58.729 INFO:teuthology.orchestra.run.smithi042.stdout:oload 1000 2024-09-20T19:24:58.729 INFO:teuthology.orchestra.run.smithi042.stdout:max_change 3 2024-09-20T19:24:58.729 INFO:teuthology.orchestra.run.smithi042.stdout:max_change_osds 4 2024-09-20T19:24:58.729 INFO:teuthology.orchestra.run.smithi042.stdout:average_utilization 0.0013 2024-09-20T19:24:58.729 INFO:teuthology.orchestra.run.smithi042.stdout:overload_utilization 0.0130 2024-09-20T19:24:58.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:58.772 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:58.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:58.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:24:59.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:59.022 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:59.125 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:59.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:24:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:24:59.153 INFO:tasks.rados.rados.0.smithi042.stdout:880: done (2 left) 2024-09-20T19:24:59.153 INFO:tasks.rados.rados.0.smithi042.stdout:881: snap_create 2024-09-20T19:24:59.160 INFO:tasks.rados.rados.0.smithi042.stdout:878: finishing write tid 1 to smithi04268048-35 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: finishing write tid 2 to smithi04268048-35 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: finishing write tid 3 to smithi04268048-35 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: finishing write tid 4 to smithi04268048-35 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: finishing write tid 5 to smithi04268048-35 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: oid 35 updating version 0 to 318 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: oid 35 updating version 318 to 319 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: oid 35 updating version 319 to 321 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: oid 35 version 321 is already newer than 320 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 321 (ObjNum 316 snap 101 seq_num 316) dirty exists 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:878: left oid 35 (ObjNum 316 snap 101 seq_num 316) 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:879: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:24:59.161 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:24:59.212 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 928839 2024-09-20T19:24:59.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:59.268 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:59.276 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2321442 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[32075]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[32075]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[22706]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:24:59 smithi042 bash[22706]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:59.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:24:59.400 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:24:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:24:59.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:59 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:59 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.029495+0000 mgr.y (mgr.24491) 267 : cluster [DBG] pgmap v401: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:24:59.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:59 smithi162 bash[24557]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:24:59 smithi162 bash[24557]: audit 2024-09-20T19:24:58.364287+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.? v1:172.21.15.42:0/1958052866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:24:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:24:59.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:59.524 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:24:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:24:59.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:59.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:24:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:24:59.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:59.782 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:24:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:24:59.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:24:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:24:59.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:00.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:00.043 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:00.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:878: done (2 left) 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:879: done (1 left) 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:881: done (0 left) 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:882: rollback oid 43 current snap is 102 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 88 2024-09-20T19:25:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:883: delete oid 9 current snap is 102 2024-09-20T19:25:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:25:00.170 INFO:tasks.rados.rados.0.smithi042.stdout:882: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:25:00.171 INFO:tasks.rados.rados.0.smithi042.stdout:882: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:25:00.171 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 211 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:25:00.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:882: done (1 left) 2024-09-20T19:25:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:883: done (0 left) 2024-09-20T19:25:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:884: write (excl) oid 13 current snap is 102 2024-09-20T19:25:00.174 INFO:tasks.rados.rados.0.smithi042.stdout:884: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-20T19:25:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:00.181 INFO:tasks.rados.rados.0.smithi042.stdout:884: writing smithi04268048-13 from 544626 to 1244281 tid 1 2024-09-20T19:25:00.188 INFO:tasks.rados.rados.0.smithi042.stdout:884: writing smithi04268048-13 from 1892215 to 2446644 tid 2 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:884: writing smithi04268048-13 from 3150041 to 3344626 tid 3 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:885: read oid 50 snap 88 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:885: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:886: read oid 9 snap -1 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:886: expect deleted 2024-09-20T19:25:00.194 INFO:tasks.rados.rados.0.smithi042.stdout:887: delete oid 36 current snap is 102 2024-09-20T19:25:00.195 INFO:tasks.rados.rados.0.smithi042.stdout:884: finishing write tid 1 to smithi04268048-13 2024-09-20T19:25:00.195 INFO:tasks.rados.rados.0.smithi042.stdout:884: finishing write tid 2 to smithi04268048-13 2024-09-20T19:25:00.196 INFO:tasks.rados.rados.0.smithi042.stdout:884: finishing write tid 3 to smithi04268048-13 2024-09-20T19:25:00.196 INFO:tasks.rados.rados.0.smithi042.stdout:884: finishing write tid 4 to smithi04268048-13 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: finishing write tid 5 to smithi04268048-13 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: oid 13 updating version 0 to 308 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: oid 13 updating version 308 to 311 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: oid 13 version 311 is already newer than 309 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: oid 13 version 311 is already newer than 310 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 311 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: left oid 13 (ObjNum 318 snap 102 seq_num 318) 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:884: done (3 left) 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:887: done (2 left) 2024-09-20T19:25:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:888: snap_create 2024-09-20T19:25:00.201 INFO:tasks.rados.rados.0.smithi042.stdout:885: expect (ObjNum 137 snap 36 seq_num 137) 2024-09-20T19:25:00.201 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:00.238 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-20T19:25:00.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-20T19:25:00.240 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1827918 2024-09-20T19:25:00.249 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1579250 2024-09-20T19:25:00.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[22706]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.392 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:00 smithi042 bash[32075]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:00.425 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:00.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:58.727009+0000 mgr.y (mgr.24491) 268 : audit [DBG] from='client.25043 v1:172.21.15.42:0/3344315749' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.953164+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:58.954088+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: cluster 2024-09-20T19:24:59.152567+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.184228+0000 mgr.y (mgr.24491) 269 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:00.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.462605+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.42:0/1012498116' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:00 smithi162 bash[24557]: audit 2024-09-20T19:24:59.463051+0000 mon.a (mon.0) 1316 : audit [INF] from='client.25049 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:00.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:00.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:00.910 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:00.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:01.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:01.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:01.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:01.185 INFO:tasks.rados.rados.0.smithi042.stdout:885: done (2 left) 2024-09-20T19:25:01.185 INFO:tasks.rados.rados.0.smithi042.stdout:886: done (1 left) 2024-09-20T19:25:01.185 INFO:tasks.rados.rados.0.smithi042.stdout:888: done (0 left) 2024-09-20T19:25:01.185 INFO:tasks.rados.rados.0.smithi042.stdout:889: write (excl) oid 45 current snap is 103 2024-09-20T19:25:01.185 INFO:tasks.rados.rados.0.smithi042.stdout:889: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-20T19:25:01.197 INFO:tasks.rados.rados.0.smithi042.stdout:889: writing smithi04268048-45 from 578240 to 1144746 tid 1 2024-09-20T19:25:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:01.202 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.203 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[32075]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.204 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:01 smithi042 bash[22706]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:01.211 INFO:tasks.rados.rados.0.smithi042.stdout:889: writing smithi04268048-45 from 1609179 to 2167277 tid 2 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:889: writing smithi04268048-45 from 2804887 to 3356637 tid 3 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:890: read oid 34 snap -1 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:890: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:891: snap_create 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:889: finishing write tid 1 to smithi04268048-45 2024-09-20T19:25:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:889: finishing write tid 2 to smithi04268048-45 2024-09-20T19:25:01.229 INFO:tasks.rados.rados.0.smithi042.stdout:889: finishing write tid 3 to smithi04268048-45 2024-09-20T19:25:01.229 INFO:tasks.rados.rados.0.smithi042.stdout:889: finishing write tid 4 to smithi04268048-45 2024-09-20T19:25:01.229 INFO:tasks.rados.rados.0.smithi042.stdout:889: finishing write tid 5 to smithi04268048-45 2024-09-20T19:25:01.229 INFO:tasks.rados.rados.0.smithi042.stdout:889: oid 45 updating version 0 to 387 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:889: oid 45 updating version 387 to 389 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:889: oid 45 version 389 is already newer than 386 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:889: oid 45 version 389 is already newer than 388 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 389 (ObjNum 320 snap 103 seq_num 320) dirty exists 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:889: left oid 45 (ObjNum 320 snap 103 seq_num 320) 2024-09-20T19:25:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:890: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:01.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:01.309 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2230742 2024-09-20T19:25:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:01.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:01.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:01.485 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.030429+0000 mgr.y (mgr.24491) 270 : cluster [DBG] pgmap v404: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 302 MiB data, 954 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-20T19:25:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.148907+0000 mon.a (mon.0) 1317 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:01.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: audit 2024-09-20T19:25:00.151097+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25049 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:01.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:01 smithi162 bash[24557]: cluster 2024-09-20T19:25:00.158801+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-20T19:25:01.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:01.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:01.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:01.720 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:01.823 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:01.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:01.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:01.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:02.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:02.195 INFO:tasks.rados.rados.0.smithi042.stdout:889: done (2 left) 2024-09-20T19:25:02.195 INFO:tasks.rados.rados.0.smithi042.stdout:890: done (1 left) 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:891: done (0 left) 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:892: copy_from oid 47 from oid 3 current snap is 104 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:893: read oid 20 snap -1 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:893: expect deleted 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:894: write oid 18 current snap is 104 2024-09-20T19:25:02.196 INFO:tasks.rados.rados.0.smithi042.stdout:894: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-20T19:25:02.212 INFO:tasks.rados.rados.0.smithi042.stdout:894: writing smithi04268048-18 from 595047 to 1236802 tid 1 2024-09-20T19:25:02.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:02.226 INFO:tasks.rados.rados.0.smithi042.stdout:894: writing smithi04268048-18 from 1667661 to 2085770 tid 2 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:894: writing smithi04268048-18 from 2548663 to 3311685 tid 3 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:895: rollback oid 8 current snap is 104 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 97 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:894: finishing write tid 1 to smithi04268048-18 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:892: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:892: finishing copy_from to smithi04268048-47 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 414 (ObjNum 229 snap 67 seq_num 229) dirty exists 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:892: done (3 left) 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:893: done (2 left) 2024-09-20T19:25:02.245 INFO:tasks.rados.rados.0.smithi042.stdout:896: write oid 44 current snap is 104 2024-09-20T19:25:02.246 INFO:tasks.rados.rados.0.smithi042.stdout:894: finishing write tid 2 to smithi04268048-18 2024-09-20T19:25:02.246 INFO:tasks.rados.rados.0.smithi042.stdout:896: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-20T19:25:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:02.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:02.250 INFO:tasks.rados.rados.0.smithi042.stdout:896: writing smithi04268048-44 from 611854 to 1328858 tid 1 2024-09-20T19:25:02.258 INFO:tasks.rados.rados.0.smithi042.stdout:896: writing smithi04268048-44 from 2126143 to 2687910 tid 2 2024-09-20T19:25:02.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:896: writing smithi04268048-44 from 3259733 to 3411854 tid 3 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:897: copy_from oid 47 from oid 48 current snap is 104 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:895: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 306 (ObjNum 33 snap 0 seq_num 1414926016) dirty dne 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:894: finishing write tid 3 to smithi04268048-18 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:894: finishing write tid 4 to smithi04268048-18 2024-09-20T19:25:02.263 INFO:tasks.rados.rados.0.smithi042.stdout:894: finishing write tid 5 to smithi04268048-18 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: oid 18 updating version 0 to 433 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: oid 18 updating version 433 to 434 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: oid 18 updating version 434 to 436 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: oid 18 version 436 is already newer than 435 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 436 (ObjNum 321 snap 104 seq_num 321) dirty exists 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: left oid 18 (ObjNum 321 snap 104 seq_num 321) 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:896: finishing write tid 1 to smithi04268048-44 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:896: finishing write tid 2 to smithi04268048-44 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:894: done (3 left) 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:895: done (2 left) 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:898: write oid 3 current snap is 104 2024-09-20T19:25:02.264 INFO:tasks.rados.rados.0.smithi042.stdout:898: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-20T19:25:02.271 INFO:tasks.rados.rados.0.smithi042.stdout:898: writing smithi04268048-3 from 628661 to 1420914 tid 1 2024-09-20T19:25:02.278 INFO:tasks.rados.rados.0.smithi042.stdout:898: writing smithi04268048-3 from 2184625 to 2606403 tid 2 2024-09-20T19:25:02.283 INFO:tasks.rados.rados.0.smithi042.stdout:898: writing smithi04268048-3 from 3403509 to 3428661 tid 3 2024-09-20T19:25:02.283 INFO:tasks.rados.rados.0.smithi042.stdout:899: write (excl) oid 38 current snap is 104 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: finishing write tid 3 to smithi04268048-44 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: finishing write tid 4 to smithi04268048-44 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: finishing write tid 5 to smithi04268048-44 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: oid 44 updating version 0 to 392 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: oid 44 version 392 is already newer than 390 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: oid 44 updating version 392 to 393 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: oid 44 version 393 is already newer than 391 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 393 (ObjNum 322 snap 104 seq_num 322) dirty exists 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:896: left oid 44 (ObjNum 322 snap 104 seq_num 322) 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:897: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:897: finishing copy_from to smithi04268048-47 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 344 (ObjNum 256 snap 74 seq_num 256) dirty exists 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:898: finishing write tid 1 to smithi04268048-3 2024-09-20T19:25:02.284 INFO:tasks.rados.rados.0.smithi042.stdout:898: finishing write tid 2 to smithi04268048-3 2024-09-20T19:25:02.285 INFO:tasks.rados.rados.0.smithi042.stdout:899: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-20T19:25:02.289 INFO:tasks.rados.rados.0.smithi042.stdout:899: writing smithi04268048-38 from 645468 to 1112970 tid 1 2024-09-20T19:25:02.296 INFO:tasks.rados.rados.0.smithi042.stdout:899: writing smithi04268048-38 from 1726754 to 2292190 tid 2 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:899: writing smithi04268048-38 from 2798226 to 3395064 tid 3 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:896: done (3 left) 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:897: done (2 left) 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:900: read oid 35 snap 88 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:900: expect (ObjNum 286 snap 88 seq_num 286) 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:901: delete oid 33 current snap is 104 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:898: finishing write tid 3 to smithi04268048-3 2024-09-20T19:25:02.303 INFO:tasks.rados.rados.0.smithi042.stdout:898: finishing write tid 4 to smithi04268048-3 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: finishing write tid 5 to smithi04268048-3 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: oid 3 updating version 0 to 418 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: oid 3 version 418 is already newer than 416 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: oid 3 version 418 is already newer than 415 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: oid 3 version 418 is already newer than 417 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 418 (ObjNum 323 snap 104 seq_num 323) dirty exists 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:898: left oid 3 (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:899: finishing write tid 1 to smithi04268048-38 2024-09-20T19:25:02.304 INFO:tasks.rados.rados.0.smithi042.stdout:899: finishing write tid 2 to smithi04268048-38 2024-09-20T19:25:02.307 INFO:tasks.rados.rados.0.smithi042.stdout:898: done (3 left) 2024-09-20T19:25:02.307 INFO:tasks.rados.rados.0.smithi042.stdout:901: done (2 left) 2024-09-20T19:25:02.307 INFO:tasks.rados.rados.0.smithi042.stdout:902: snap_create 2024-09-20T19:25:02.308 INFO:tasks.rados.rados.0.smithi042.stdout:899: finishing write tid 3 to smithi04268048-38 2024-09-20T19:25:02.309 INFO:tasks.rados.rados.0.smithi042.stdout:899: finishing write tid 4 to smithi04268048-38 2024-09-20T19:25:02.309 INFO:tasks.rados.rados.0.smithi042.stdout:900: expect (ObjNum 286 snap 88 seq_num 286) 2024-09-20T19:25:02.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.359 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:02 smithi162 bash[24557]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.360 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:02 smithi162 bash[24557]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: finishing write tid 5 to smithi04268048-38 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: oid 38 updating version 0 to 377 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: oid 38 updating version 377 to 378 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: oid 38 version 378 is already newer than 376 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: oid 38 updating version 378 to 379 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 379 (ObjNum 324 snap 104 seq_num 324) dirty exists 2024-09-20T19:25:02.445 INFO:tasks.rados.rados.0.smithi042.stdout:899: left oid 38 (ObjNum 324 snap 104 seq_num 324) 2024-09-20T19:25:02.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:02.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:02.621 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:02 smithi042 bash[22706]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.621 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:02 smithi042 bash[22706]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.622 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:02 smithi042 bash[32075]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.622 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:02 smithi042 bash[32075]: cluster 2024-09-20T19:25:01.181052+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-20T19:25:02.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:02.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:02.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:02.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:02.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:03.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:03.079 INFO:tasks.rados.rados.0.smithi042.stdout:899: done (2 left) 2024-09-20T19:25:03.079 INFO:tasks.rados.rados.0.smithi042.stdout:900: done (1 left) 2024-09-20T19:25:03.080 INFO:tasks.rados.rados.0.smithi042.stdout:902: done (0 left) 2024-09-20T19:25:03.080 INFO:tasks.rados.rados.0.smithi042.stdout:903: snap_create 2024-09-20T19:25:03.118 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:25:03.132 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:03.134 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:25:03.134 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:25:03.134 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:25:03.134 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:25:03.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:03.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:03.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:03.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:03.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:03 smithi162 bash[24557]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:03.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:03.676 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[22706]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:02.032225+0000 mgr.y (mgr.24491) 271 : cluster [DBG] pgmap v407: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:02.194236+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:03 smithi042 bash[32075]: cluster 2024-09-20T19:25:03.079770+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-20T19:25:03.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:03.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:03.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:03.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:03.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:04.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:04.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:04.081 INFO:tasks.rados.rados.0.smithi042.stdout:903: done (0 left) 2024-09-20T19:25:04.081 INFO:tasks.rados.rados.0.smithi042.stdout:904: read oid 40 snap -1 2024-09-20T19:25:04.082 INFO:tasks.rados.rados.0.smithi042.stdout:904: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:25:04.082 INFO:tasks.rados.rados.0.smithi042.stdout:905: snap_create 2024-09-20T19:25:04.092 INFO:tasks.rados.rados.0.smithi042.stdout:904: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:25:04.137 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T19:25:04.143 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1694422 2024-09-20T19:25:04.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:04.197 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:04.300 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:04.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:04.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:04.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:04.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:04.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:04.687 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:04.817 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:04.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:04.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:04.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:05.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:904: done (1 left) 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:905: done (0 left) 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:906: read oid 9 snap 100 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:906: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:907: copy_from oid 1 from oid 13 current snap is 107 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:908: write (excl) oid 29 current snap is 107 2024-09-20T19:25:05.093 INFO:tasks.rados.rados.0.smithi042.stdout:908: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-20T19:25:05.099 INFO:tasks.rados.rados.0.smithi042.stdout:908: writing smithi04268048-29 from 679082 to 1297082 tid 1 2024-09-20T19:25:05.107 INFO:tasks.rados.rados.0.smithi042.stdout:908: writing smithi04268048-29 from 1843718 to 2412823 tid 2 2024-09-20T19:25:05.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:05.116 INFO:tasks.rados.rados.0.smithi042.stdout:908: writing smithi04268048-29 from 2853072 to 3479082 tid 3 2024-09-20T19:25:05.116 INFO:tasks.rados.rados.0.smithi042.stdout:909: write oid 19 current snap is 107 2024-09-20T19:25:05.116 INFO:tasks.rados.rados.0.smithi042.stdout:909: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-20T19:25:05.123 INFO:tasks.rados.rados.0.smithi042.stdout:909: writing smithi04268048-19 from 695889 to 1105491 tid 1 2024-09-20T19:25:05.132 INFO:tasks.rados.rados.0.smithi042.stdout:909: writing smithi04268048-19 from 1618553 to 2331316 tid 2 2024-09-20T19:25:05.138 INFO:tasks.rados.rados.0.smithi042.stdout:909: writing smithi04268048-19 from 2996848 to 3495889 tid 3 2024-09-20T19:25:05.138 INFO:tasks.rados.rados.0.smithi042.stdout:910: read oid 17 snap 92 2024-09-20T19:25:05.138 INFO:tasks.rados.rados.0.smithi042.stdout:910: expect (ObjNum 278 snap 84 seq_num 278) 2024-09-20T19:25:05.138 INFO:tasks.rados.rados.0.smithi042.stdout:911: write oid 49 current snap is 107 2024-09-20T19:25:05.138 INFO:tasks.rados.rados.0.smithi042.stdout:911: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-20T19:25:05.146 INFO:tasks.rados.rados.0.smithi042.stdout:911: writing smithi04268048-49 from 712696 to 1197547 tid 1 2024-09-20T19:25:05.153 INFO:tasks.rados.rados.0.smithi042.stdout:911: writing smithi04268048-49 from 1960682 to 2533456 tid 2 2024-09-20T19:25:05.159 INFO:tasks.rados.rados.0.smithi042.stdout:911: writing smithi04268048-49 from 3307918 to 3512696 tid 3 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:912: snap_create 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:908: finishing write tid 1 to smithi04268048-29 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:908: finishing write tid 2 to smithi04268048-29 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:907: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:906: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:25:05.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:05.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:05.232 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:05.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: finishing write tid 3 to smithi04268048-29 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: finishing write tid 4 to smithi04268048-29 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: finishing write tid 5 to smithi04268048-29 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: oid 29 updating version 0 to 382 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: oid 29 version 382 is already newer than 381 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: oid 29 version 382 is already newer than 380 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: oid 29 updating version 382 to 383 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 383 (ObjNum 326 snap 107 seq_num 326) dirty exists 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:908: left oid 29 (ObjNum 326 snap 107 seq_num 326) 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:909: finishing write tid 1 to smithi04268048-19 2024-09-20T19:25:05.268 INFO:tasks.rados.rados.0.smithi042.stdout:907: finishing copy_from to smithi04268048-1 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 311 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: finishing write tid 2 to smithi04268048-19 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: finishing write tid 3 to smithi04268048-19 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: finishing write tid 4 to smithi04268048-19 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: finishing write tid 5 to smithi04268048-19 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: oid 19 updating version 0 to 325 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: oid 19 updating version 325 to 326 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: oid 19 updating version 326 to 328 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: oid 19 version 328 is already newer than 327 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 328 (ObjNum 327 snap 107 seq_num 327) dirty exists 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:909: left oid 19 (ObjNum 327 snap 107 seq_num 327) 2024-09-20T19:25:05.269 INFO:tasks.rados.rados.0.smithi042.stdout:910: expect (ObjNum 278 snap 84 seq_num 278) 2024-09-20T19:25:05.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:05 smithi162 bash[24557]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:05 smithi162 bash[24557]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:05 smithi162 bash[24557]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:05 smithi162 bash[24557]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.342 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:05.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[22706]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[22706]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[22706]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[22706]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[32075]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[32075]: cluster 2024-09-20T19:25:04.033018+0000 mgr.y (mgr.24491) 272 : cluster [DBG] pgmap v410: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[32075]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:05 smithi042 bash[32075]: cluster 2024-09-20T19:25:04.081485+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-20T19:25:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: finishing write tid 1 to smithi04268048-49 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: finishing write tid 2 to smithi04268048-49 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: finishing write tid 3 to smithi04268048-49 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: finishing write tid 4 to smithi04268048-49 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: finishing write tid 5 to smithi04268048-49 2024-09-20T19:25:05.396 INFO:tasks.rados.rados.0.smithi042.stdout:911: oid 49 updating version 0 to 398 2024-09-20T19:25:05.397 INFO:tasks.rados.rados.0.smithi042.stdout:911: oid 49 version 398 is already newer than 397 2024-09-20T19:25:05.397 INFO:tasks.rados.rados.0.smithi042.stdout:911: oid 49 version 398 is already newer than 395 2024-09-20T19:25:05.397 INFO:tasks.rados.rados.0.smithi042.stdout:911: oid 49 version 398 is already newer than 396 2024-09-20T19:25:05.397 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 398 (ObjNum 328 snap 107 seq_num 328) dirty exists 2024-09-20T19:25:05.397 INFO:tasks.rados.rados.0.smithi042.stdout:911: left oid 49 (ObjNum 328 snap 107 seq_num 328) 2024-09-20T19:25:05.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:05.594 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:05.623 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:05.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:05.639 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:05.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:05.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:05.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:05.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:05.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:05.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:06.017 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:906: done (6 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:907: done (5 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:908: done (4 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:909: done (3 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:910: done (2 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:911: done (1 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:912: done (0 left) 2024-09-20T19:25:06.111 INFO:tasks.rados.rados.0.smithi042.stdout:913: delete oid 6 current snap is 108 2024-09-20T19:25:06.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:06.121 INFO:tasks.rados.rados.0.smithi042.stdout:913: done (0 left) 2024-09-20T19:25:06.122 INFO:tasks.rados.rados.0.smithi042.stdout:914: copy_from oid 22 from oid 10 current snap is 108 2024-09-20T19:25:06.122 INFO:tasks.rados.rados.0.smithi042.stdout:915: read oid 18 snap 104 2024-09-20T19:25:06.122 INFO:tasks.rados.rados.0.smithi042.stdout:915: expect (ObjNum 321 snap 104 seq_num 321) 2024-09-20T19:25:06.122 INFO:tasks.rados.rados.0.smithi042.stdout:916: write (excl) oid 48 current snap is 108 2024-09-20T19:25:06.122 INFO:tasks.rados.rados.0.smithi042.stdout:916: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-20T19:25:06.130 INFO:tasks.rados.rados.0.smithi042.stdout:916: writing smithi04268048-48 from 746310 to 1381659 tid 1 2024-09-20T19:25:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:916: writing smithi04268048-48 from 2077646 to 2654089 tid 2 2024-09-20T19:25:06.148 INFO:tasks.rados.rados.0.smithi042.stdout:916: writing smithi04268048-48 from 3362764 to 3546310 tid 3 2024-09-20T19:25:06.148 INFO:tasks.rados.rados.0.smithi042.stdout:917: write (excl) oid 15 current snap is 108 2024-09-20T19:25:06.148 INFO:tasks.rados.rados.0.smithi042.stdout:917: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-20T19:25:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:06.156 INFO:tasks.rados.rados.0.smithi042.stdout:917: writing smithi04268048-15 from 763117 to 1473715 tid 1 2024-09-20T19:25:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:06.164 INFO:tasks.rados.rados.0.smithi042.stdout:917: writing smithi04268048-15 from 2136128 to 2856229 tid 2 2024-09-20T19:25:06.168 INFO:tasks.rados.rados.0.smithi042.stdout:917: writing smithi04268048-15 from 3390187 to 3563117 tid 3 2024-09-20T19:25:06.169 INFO:tasks.rados.rados.0.smithi042.stdout:918: snap_remove snap 99 2024-09-20T19:25:06.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:06.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:06.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:06.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:06 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:06 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:06 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:06 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.402 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:06.418 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:06.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:06 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:06 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.087080+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-20T19:25:06.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:06.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:06.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:06.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:06.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:06.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:07.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:07.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:07.116 INFO:tasks.rados.rados.0.smithi042.stdout:918: done (4 left) 2024-09-20T19:25:07.116 INFO:tasks.rados.rados.0.smithi042.stdout:919: write oid 45 current snap is 108 2024-09-20T19:25:07.116 INFO:tasks.rados.rados.0.smithi042.stdout:919: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-20T19:25:07.122 INFO:tasks.rados.rados.0.smithi042.stdout:919: writing smithi04268048-45 from 779924 to 1565771 tid 1 2024-09-20T19:25:07.130 INFO:tasks.rados.rados.0.smithi042.stdout:919: writing smithi04268048-45 from 2078257 to 2658369 tid 2 2024-09-20T19:25:07.136 INFO:tasks.rados.rados.0.smithi042.stdout:919: writing smithi04268048-45 from 3301257 to 3579924 tid 3 2024-09-20T19:25:07.136 INFO:tasks.rados.rados.0.smithi042.stdout:920: snap_remove snap 60 2024-09-20T19:25:07.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:07.196 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:07.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:07 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.349557+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:05.350486+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.033632+0000 mgr.y (mgr.24491) 273 : cluster [DBG] pgmap v413: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 295 MiB data, 936 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.109474+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-20T19:25:07.430 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:07.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:moved 26 / 474 (5.48523%) 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:avg 59.25 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:stddev 5.35607 -> 6.41775 (expected baseline 7.20026) 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:min osd.2 with 52 -> 59 pgs (0.877637 -> 0.995781 * mean) 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:max osd.3 with 70 -> 58 pgs (1.18143 -> 0.978903 * mean) 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:oload 110 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:max_change 3 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:max_change_osds 4 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:average_utilization 0.0014 2024-09-20T19:25:07.502 INFO:teuthology.orchestra.run.smithi042.stdout:overload_utilization 0.0015 2024-09-20T19:25:07.503 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 weight 1.0000 -> 0.7862 2024-09-20T19:25:07.503 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 weight 1.0000 -> 0.8060 2024-09-20T19:25:07.503 INFO:teuthology.orchestra.run.smithi042.stderr:no change 2024-09-20T19:25:07.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:07.675 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:07.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:07.729 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:07 smithi042 bash[22958]: debug 2024-09-20T19:25:07.496+0000 7f1705029640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-20T19:25:07.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:07.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:07.865 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:07.968 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:07.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:08.082 INFO:tasks.rados.rados.0.smithi042.stdout:920: done (5 left) 2024-09-20T19:25:08.082 INFO:tasks.rados.rados.0.smithi042.stdout:921: copy_from oid 12 from oid 14 current snap is 108 2024-09-20T19:25:08.086 INFO:tasks.rados.rados.0.smithi042.stdout:922: delete oid 25 current snap is 108 2024-09-20T19:25:08.086 INFO:tasks.rados.rados.0.smithi042.stdout:915: expect (ObjNum 321 snap 104 seq_num 321) 2024-09-20T19:25:08.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.127 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:08.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:08.287 INFO:tasks.rados.rados.0.smithi042.stdout:916: finishing write tid 1 to smithi04268048-48 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:914: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:914: finishing copy_from to smithi04268048-22 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 375 (ObjNum 312 snap 99 seq_num 312) dirty exists 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: finishing write tid 2 to smithi04268048-48 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: finishing write tid 3 to smithi04268048-48 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: finishing write tid 4 to smithi04268048-48 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: finishing write tid 5 to smithi04268048-48 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: oid 48 updating version 0 to 438 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: oid 48 updating version 438 to 440 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: oid 48 updating version 440 to 441 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: oid 48 version 441 is already newer than 439 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 441 (ObjNum 330 snap 108 seq_num 330) dirty exists 2024-09-20T19:25:08.288 INFO:tasks.rados.rados.0.smithi042.stdout:916: left oid 48 (ObjNum 330 snap 108 seq_num 330) 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: finishing write tid 1 to smithi04268048-15 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: finishing write tid 2 to smithi04268048-15 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: finishing write tid 3 to smithi04268048-15 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: finishing write tid 4 to smithi04268048-15 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: finishing write tid 5 to smithi04268048-15 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: oid 15 updating version 0 to 317 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: oid 15 version 317 is already newer than 316 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: oid 15 version 317 is already newer than 314 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: oid 15 version 317 is already newer than 315 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 317 (ObjNum 331 snap 108 seq_num 331) dirty exists 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:917: left oid 15 (ObjNum 331 snap 108 seq_num 331) 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:919: finishing write tid 1 to smithi04268048-45 2024-09-20T19:25:08.289 INFO:tasks.rados.rados.0.smithi042.stdout:919: finishing write tid 2 to smithi04268048-45 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: finishing write tid 3 to smithi04268048-45 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: finishing write tid 4 to smithi04268048-45 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: finishing write tid 5 to smithi04268048-45 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: oid 45 updating version 0 to 408 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: oid 45 version 408 is already newer than 406 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: oid 45 version 408 is already newer than 407 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: oid 45 updating version 408 to 409 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 409 (ObjNum 332 snap 108 seq_num 332) dirty exists 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:919: left oid 45 (ObjNum 332 snap 108 seq_num 332) 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:921: finishing copy_from to smithi04268048-12 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:921: got expected ENOENT (src dne) 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:921: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:914: done (6 left) 2024-09-20T19:25:08.290 INFO:tasks.rados.rados.0.smithi042.stdout:915: done (5 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:916: done (4 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:917: done (3 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:919: done (2 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:921: done (1 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:922: done (0 left) 2024-09-20T19:25:08.291 INFO:tasks.rados.rados.0.smithi042.stdout:923: snap_create 2024-09-20T19:25:08.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:08.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[22706]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:08 smithi042 bash[32075]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.386 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:08.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:08.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.810017+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:06.811187+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:07.109760+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: audit 2024-09-20T19:25:07.499247+0000 mgr.y (mgr.24491) 274 : audit [DBG] from='client.25055 v1:172.21.15.42:0/3528604442' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:08 smithi162 bash[24557]: cluster 2024-09-20T19:25:08.081544+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-20T19:25:08.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:08.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:08.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:08.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:08.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:08.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:08.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:09.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:09.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:09.131 INFO:tasks.rados.rados.0.smithi042.stdout:923: done (0 left) 2024-09-20T19:25:09.131 INFO:tasks.rados.rados.0.smithi042.stdout:924: delete oid 37 current snap is 109 2024-09-20T19:25:09.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:09.152 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:09 smithi162 bash[24557]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.152 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:09 smithi162 bash[24557]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.155 INFO:tasks.rados.rados.0.smithi042.stdout:924: done (0 left) 2024-09-20T19:25:09.155 INFO:tasks.rados.rados.0.smithi042.stdout:925: read oid 28 snap 96 2024-09-20T19:25:09.155 INFO:tasks.rados.rados.0.smithi042.stdout:925: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:25:09.155 INFO:tasks.rados.rados.0.smithi042.stdout:926: write (excl) oid 12 current snap is 109 2024-09-20T19:25:09.155 INFO:tasks.rados.rados.0.smithi042.stdout:926: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-20T19:25:09.165 INFO:tasks.rados.rados.0.smithi042.stdout:926: writing smithi04268048-12 from 430345 to 1158292 tid 1 2024-09-20T19:25:09.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:09.176 INFO:tasks.rados.rados.0.smithi042.stdout:926: writing smithi04268048-12 from 1570056 to 2297495 tid 2 2024-09-20T19:25:09.185 INFO:tasks.rados.rados.0.smithi042.stdout:926: writing smithi04268048-12 from 2699879 to 3169768 tid 3 2024-09-20T19:25:09.185 INFO:tasks.rados.rados.0.smithi042.stdout:927: delete oid 25 current snap is 109 2024-09-20T19:25:09.185 INFO:tasks.rados.rados.0.smithi042.stdout:925: expect (ObjNum 257 snap 75 seq_num 257) 2024-09-20T19:25:09.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.293 INFO:tasks.rados.rados.0.smithi042.stdout:926: finishing write tid 1 to smithi04268048-12 2024-09-20T19:25:09.293 INFO:tasks.rados.rados.0.smithi042.stdout:926: finishing write tid 2 to smithi04268048-12 2024-09-20T19:25:09.293 INFO:tasks.rados.rados.0.smithi042.stdout:926: finishing write tid 3 to smithi04268048-12 2024-09-20T19:25:09.293 INFO:tasks.rados.rados.0.smithi042.stdout:926: finishing write tid 4 to smithi04268048-12 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: finishing write tid 5 to smithi04268048-12 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: oid 12 updating version 0 to 219 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: oid 12 version 219 is already newer than 218 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: oid 12 version 219 is already newer than 217 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: oid 12 version 219 is already newer than 216 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 219 (ObjNum 335 snap 109 seq_num 335) dirty exists 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: left oid 12 (ObjNum 335 snap 109 seq_num 335) 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:925: done (2 left) 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:926: done (1 left) 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:927: done (0 left) 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:928: write oid 8 current snap is 109 2024-09-20T19:25:09.294 INFO:tasks.rados.rados.0.smithi042.stdout:928: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-20T19:25:09.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:09.301 INFO:tasks.rados.rados.0.smithi042.stdout:928: writing smithi04268048-8 from 463959 to 942404 tid 1 2024-09-20T19:25:09.304 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-20T19:25:09.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-20T19:25:09.320 INFO:tasks.rados.rados.0.smithi042.stdout:928: writing smithi04268048-8 from 1570667 to 2301775 tid 2 2024-09-20T19:25:09.337 INFO:tasks.rados.rados.0.smithi042.stdout:928: writing smithi04268048-8 from 3038372 to 3530805 tid 3 2024-09-20T19:25:09.337 INFO:tasks.rados.rados.0.smithi042.stdout:929: snap_remove snap 100 2024-09-20T19:25:09.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:09.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:09 smithi042 bash[22706]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:09 smithi042 bash[22706]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:09 smithi042 bash[32075]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.410 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:09 smithi042 bash[32075]: cluster 2024-09-20T19:25:08.034148+0000 mgr.y (mgr.24491) 275 : cluster [DBG] pgmap v416: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:09.425 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:09.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:09.547 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:09.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:09.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.677 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:09.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.781 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:09.781 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:09.806 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:09.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:09.937 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:09.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:10.040 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:10.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:929: done (1 left) 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:930: write oid 33 current snap is 109 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:928: finishing write tid 1 to smithi04268048-8 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:928: finishing write tid 2 to smithi04268048-8 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:928: finishing write tid 3 to smithi04268048-8 2024-09-20T19:25:10.149 INFO:tasks.rados.rados.0.smithi042.stdout:930: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-20T19:25:10.161 INFO:tasks.rados.rados.0.smithi042.stdout:930: writing smithi04268048-33 from 480766 to 1034460 tid 1 2024-09-20T19:25:10.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:10.177 INFO:tasks.rados.rados.0.smithi042.stdout:930: writing smithi04268048-33 from 1629149 to 2220268 tid 2 2024-09-20T19:25:10.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:10 smithi162 bash[24557]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:10 smithi162 bash[24557]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:10 smithi162 bash[24557]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:10 smithi162 bash[24557]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.190 INFO:tasks.rados.rados.0.smithi042.stdout:930: writing smithi04268048-33 from 2782148 to 3202206 tid 3 2024-09-20T19:25:10.190 INFO:tasks.rados.rados.0.smithi042.stdout:931: read oid 47 snap -1 2024-09-20T19:25:10.190 INFO:tasks.rados.rados.0.smithi042.stdout:931: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:25:10.190 INFO:tasks.rados.rados.0.smithi042.stdout:932: snap_remove snap 96 2024-09-20T19:25:10.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[22706]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[22706]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[22706]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[22706]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[32075]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.267 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[32075]: cluster 2024-09-20T19:25:09.130859+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-20T19:25:10.268 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[32075]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.268 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:10 smithi042 bash[32075]: audit 2024-09-20T19:25:09.194587+0000 mgr.y (mgr.24491) 276 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:10.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:10.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:10.419 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:25:10.420 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:25:10.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:25:10.437 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:25:10.454 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:25:10.454 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:25:10.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:10.472 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:25:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:25:10.490 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:25:10.491 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:25:10.509 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:25:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:25:10.530 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:25:10.530 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:25:10.550 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:25:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:25:10.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:10.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:10.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:10.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:10.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:10.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:10.984 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:10.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:11.087 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:11.113 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:11.155 INFO:tasks.rados.rados.0.smithi042.stdout:932: done (3 left) 2024-09-20T19:25:11.155 INFO:tasks.rados.rados.0.smithi042.stdout:933: snap_remove snap 104 2024-09-20T19:25:11.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:11.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[22706]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.358 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:11 smithi042 bash[32075]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:11.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:11.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: cluster 2024-09-20T19:25:10.034964+0000 mgr.y (mgr.24491) 277 : cluster [DBG] pgmap v419: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 319 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 6.3 MiB/s wr, 27 op/s 2024-09-20T19:25:11.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: cluster 2024-09-20T19:25:10.149115+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-20T19:25:11.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:11 smithi162 bash[24557]: audit 2024-09-20T19:25:10.401978+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.42:0/4194643639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:11.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:11.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:11.637 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:11.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:11.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:11.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:11.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:12.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:12.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.162 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:933: done (3 left) 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:934: read oid 25 snap -1 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:934: expect deleted 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:935: read oid 35 snap 108 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:935: expect (ObjNum 316 snap 101 seq_num 316) 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:936: rollback oid 6 current snap is 109 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 85 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:937: delete oid 49 current snap is 109 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:928: finishing write tid 4 to smithi04268048-8 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:928: finishing write tid 5 to smithi04268048-8 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:928: oid 8 updating version 0 to 322 2024-09-20T19:25:12.181 INFO:tasks.rados.rados.0.smithi042.stdout:928: oid 8 version 322 is already newer than 321 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:928: oid 8 version 322 is already newer than 319 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:928: oid 8 version 322 is already newer than 320 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 322 (ObjNum 337 snap 109 seq_num 337) dirty exists 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:928: left oid 8 (ObjNum 337 snap 109 seq_num 337) 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: finishing write tid 1 to smithi04268048-33 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: finishing write tid 2 to smithi04268048-33 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: finishing write tid 3 to smithi04268048-33 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: finishing write tid 4 to smithi04268048-33 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: finishing write tid 5 to smithi04268048-33 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: oid 33 updating version 0 to 198 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: oid 33 updating version 198 to 199 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: oid 33 version 199 is already newer than 197 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:930: oid 33 updating version 199 to 200 2024-09-20T19:25:12.182 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 200 (ObjNum 338 snap 109 seq_num 338) dirty exists 2024-09-20T19:25:12.183 INFO:tasks.rados.rados.0.smithi042.stdout:930: left oid 33 (ObjNum 338 snap 109 seq_num 338) 2024-09-20T19:25:12.183 INFO:tasks.rados.rados.0.smithi042.stdout:931: expect (ObjNum 256 snap 74 seq_num 256) 2024-09-20T19:25:12.198 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1243053 2024-09-20T19:25:12.251 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1383528 2024-09-20T19:25:12.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:12.293 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:936: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:936: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 324 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:928: done (6 left) 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:930: done (5 left) 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:931: done (4 left) 2024-09-20T19:25:12.299 INFO:tasks.rados.rados.0.smithi042.stdout:936: done (3 left) 2024-09-20T19:25:12.300 INFO:tasks.rados.rados.0.smithi042.stdout:937: done (2 left) 2024-09-20T19:25:12.300 INFO:tasks.rados.rados.0.smithi042.stdout:938: write (excl) oid 22 current snap is 109 2024-09-20T19:25:12.300 INFO:tasks.rados.rados.0.smithi042.stdout:938: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-20T19:25:12.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:12 smithi162 bash[24557]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:12 smithi162 bash[24557]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:12 smithi162 bash[24557]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:12 smithi162 bash[24557]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.306 INFO:tasks.rados.rados.0.smithi042.stdout:938: writing smithi04268048-22 from 514380 to 1218572 tid 1 2024-09-20T19:25:12.311 INFO:tasks.rados.rados.0.smithi042.stdout:938: writing smithi04268048-22 from 1629760 to 2108195 tid 2 2024-09-20T19:25:12.316 INFO:tasks.rados.rados.0.smithi042.stdout:938: writing smithi04268048-22 from 2604288 to 3046890 tid 3 2024-09-20T19:25:12.317 INFO:tasks.rados.rados.0.smithi042.stdout:934: done (2 left) 2024-09-20T19:25:12.317 INFO:tasks.rados.rados.0.smithi042.stdout:939: write (excl) oid 9 current snap is 109 2024-09-20T19:25:12.317 INFO:tasks.rados.rados.0.smithi042.stdout:939: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-20T19:25:12.324 INFO:tasks.rados.rados.0.smithi042.stdout:939: writing smithi04268048-9 from 531187 to 1310628 tid 1 2024-09-20T19:25:12.333 INFO:tasks.rados.rados.0.smithi042.stdout:939: writing smithi04268048-9 from 2088242 to 2826688 tid 2 2024-09-20T19:25:12.339 INFO:tasks.rados.rados.0.smithi042.stdout:939: writing smithi04268048-9 from 3431711 to 3731187 tid 3 2024-09-20T19:25:12.339 INFO:tasks.rados.rados.0.smithi042.stdout:940: read oid 3 snap -1 2024-09-20T19:25:12.339 INFO:tasks.rados.rados.0.smithi042.stdout:940: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:12.339 INFO:tasks.rados.rados.0.smithi042.stdout:941: snap_create 2024-09-20T19:25:12.339 INFO:tasks.rados.rados.0.smithi042.stdout:935: expect (ObjNum 316 snap 101 seq_num 316) 2024-09-20T19:25:12.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:12.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:12.426 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:12.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[32075]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[32075]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[32075]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[32075]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[22706]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[22706]: cluster 2024-09-20T19:25:11.156117+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-20T19:25:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[22706]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:12 smithi042 bash[22706]: audit 2024-09-20T19:25:12.101585+0000 mon.a (mon.0) 1332 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:12.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:12.607 INFO:tasks.rados.rados.0.smithi042.stdout:938: finishing write tid 1 to smithi04268048-22 2024-09-20T19:25:12.607 INFO:tasks.rados.rados.0.smithi042.stdout:938: finishing write tid 2 to smithi04268048-22 2024-09-20T19:25:12.607 INFO:tasks.rados.rados.0.smithi042.stdout:938: finishing write tid 3 to smithi04268048-22 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: finishing write tid 4 to smithi04268048-22 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: finishing write tid 5 to smithi04268048-22 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: oid 22 updating version 0 to 383 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: oid 22 updating version 383 to 384 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: oid 22 version 384 is already newer than 382 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: oid 22 version 384 is already newer than 381 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 384 (ObjNum 340 snap 109 seq_num 340) dirty exists 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:938: left oid 22 (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: finishing write tid 1 to smithi04268048-9 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: finishing write tid 2 to smithi04268048-9 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: finishing write tid 3 to smithi04268048-9 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: finishing write tid 4 to smithi04268048-9 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: finishing write tid 5 to smithi04268048-9 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: oid 9 updating version 0 to 462 2024-09-20T19:25:12.608 INFO:tasks.rados.rados.0.smithi042.stdout:939: oid 9 version 462 is already newer than 460 2024-09-20T19:25:12.609 INFO:tasks.rados.rados.0.smithi042.stdout:939: oid 9 version 462 is already newer than 459 2024-09-20T19:25:12.609 INFO:tasks.rados.rados.0.smithi042.stdout:939: oid 9 version 462 is already newer than 461 2024-09-20T19:25:12.609 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 462 (ObjNum 341 snap 109 seq_num 341) dirty exists 2024-09-20T19:25:12.609 INFO:tasks.rados.rados.0.smithi042.stdout:939: left oid 9 (ObjNum 341 snap 109 seq_num 341) 2024-09-20T19:25:12.609 INFO:tasks.rados.rados.0.smithi042.stdout:940: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:12.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:12.686 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3068184 2024-09-20T19:25:12.696 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:12.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:12.911 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:12.938 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:12.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:13.041 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:13.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:13.086 INFO:tasks.rados.rados.0.smithi042.stdout:935: done (4 left) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:938: done (3 left) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:939: done (2 left) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:940: done (1 left) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:941: done (0 left) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:942: read oid 45 snap -1 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:942: expect (ObjNum 332 snap 108 seq_num 332) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:943: read oid 40 snap -1 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:943: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:944: read oid 10 snap -1 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:944: expect (ObjNum 312 snap 99 seq_num 312) 2024-09-20T19:25:13.087 INFO:tasks.rados.rados.0.smithi042.stdout:945: read oid 22 snap -1 2024-09-20T19:25:13.088 INFO:tasks.rados.rados.0.smithi042.stdout:945: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:25:13.088 INFO:tasks.rados.rados.0.smithi042.stdout:946: copy_from oid 30 from oid 44 current snap is 110 2024-09-20T19:25:13.088 INFO:tasks.rados.rados.0.smithi042.stdout:947: write (excl) oid 26 current snap is 110 2024-09-20T19:25:13.088 INFO:tasks.rados.rados.0.smithi042.stdout:947: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-20T19:25:13.094 INFO:tasks.rados.rados.0.smithi042.stdout:947: writing smithi04268048-26 from 547994 to 1002684 tid 1 2024-09-20T19:25:13.103 INFO:tasks.rados.rados.0.smithi042.stdout:947: writing smithi04268048-26 from 1746724 to 2228828 tid 2 2024-09-20T19:25:13.110 INFO:tasks.rados.rados.0.smithi042.stdout:947: writing smithi04268048-26 from 2659134 to 3124280 tid 3 2024-09-20T19:25:13.115 INFO:tasks.rados.rados.0.smithi042.stdout:947: writing smithi04268048-26 from 3605976 to 3747994 tid 4 2024-09-20T19:25:13.115 INFO:tasks.rados.rados.0.smithi042.stdout:948: write oid 43 current snap is 110 2024-09-20T19:25:13.115 INFO:tasks.rados.rados.0.smithi042.stdout:948: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-20T19:25:13.124 INFO:tasks.rados.rados.0.smithi042.stdout:948: writing smithi04268048-43 from 564801 to 1211093 tid 1 2024-09-20T19:25:13.133 INFO:tasks.rados.rados.0.smithi042.stdout:948: writing smithi04268048-43 from 1921559 to 2663674 tid 2 2024-09-20T19:25:13.140 INFO:tasks.rados.rados.0.smithi042.stdout:948: writing smithi04268048-43 from 3202910 to 3764801 tid 3 2024-09-20T19:25:13.141 INFO:tasks.rados.rados.0.smithi042.stdout:949: write (excl) oid 50 current snap is 110 2024-09-20T19:25:13.141 INFO:tasks.rados.rados.0.smithi042.stdout:949: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-20T19:25:13.148 INFO:tasks.rados.rados.0.smithi042.stdout:949: writing smithi04268048-50 from 581608 to 1303149 tid 1 2024-09-20T19:25:13.154 INFO:tasks.rados.rados.0.smithi042.stdout:949: writing smithi04268048-50 from 1863688 to 2349461 tid 2 2024-09-20T19:25:13.163 INFO:tasks.rados.rados.0.smithi042.stdout:949: writing smithi04268048-50 from 3113980 to 3718023 tid 3 2024-09-20T19:25:13.163 INFO:tasks.rados.rados.0.smithi042.stdout:950: snap_remove snap 105 2024-09-20T19:25:13.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.178 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[32075]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:13 smithi042 bash[22706]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:13.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:13.301 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:25:13.301 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:25:13.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:13.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:13.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:12.036654+0000 mgr.y (mgr.24491) 278 : cluster [DBG] pgmap v422: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:13.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:12.173699+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-20T19:25:13.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:13 smithi162 bash[24557]: cluster 2024-09-20T19:25:13.083325+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-20T19:25:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:13.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:13.575 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:13.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:13.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:13.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:13.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:13.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:13.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:14.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:14.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T19:25:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:25:14.181 INFO:tasks.rados.rados.0.smithi042.stdout:950: done (8 left) 2024-09-20T19:25:14.181 INFO:tasks.rados.rados.0.smithi042.stdout:951: write oid 9 current snap is 110 2024-09-20T19:25:14.181 INFO:tasks.rados.rados.0.smithi042.stdout:951: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-20T19:25:14.185 INFO:tasks.rados.rados.0.smithi042.stdout:951: writing smithi04268048-9 from 598415 to 1395205 tid 1 2024-09-20T19:25:14.193 INFO:tasks.rados.rados.0.smithi042.stdout:951: writing smithi04268048-9 from 1922170 to 2667954 tid 2 2024-09-20T19:25:14.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:14.200 INFO:tasks.rados.rados.0.smithi042.stdout:951: writing smithi04268048-9 from 3141403 to 3556718 tid 3 2024-09-20T19:25:14.200 INFO:tasks.rados.rados.0.smithi042.stdout:942: expect (ObjNum 332 snap 108 seq_num 332) 2024-09-20T19:25:14.200 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 81438 2024-09-20T19:25:14.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:14 smithi042 bash[32075]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:14 smithi042 bash[32075]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.210 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:14 smithi042 bash[22706]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.211 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:14 smithi042 bash[22706]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.222 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:14.283 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:14.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:14.425 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3445766 2024-09-20T19:25:14.426 INFO:tasks.rados.rados.0.smithi042.stdout:943: expect (ObjNum 242 snap 70 seq_num 242) 2024-09-20T19:25:14.453 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:14.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:14.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:14 smithi162 bash[24557]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:14 smithi162 bash[24557]: audit 2024-09-20T19:25:13.512291+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:25:14.479 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:14.513 INFO:tasks.rados.rados.0.smithi042.stdout:944: expect (ObjNum 312 snap 99 seq_num 312) 2024-09-20T19:25:14.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:14.584 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1204085 2024-09-20T19:25:14.603 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:14.700 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 1 to smithi04268048-26 2024-09-20T19:25:14.700 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 2 to smithi04268048-26 2024-09-20T19:25:14.700 INFO:tasks.rados.rados.0.smithi042.stdout:946: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:25:14.701 INFO:tasks.rados.rados.0.smithi042.stdout:945: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:25:14.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:14.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:25:14.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:14.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:14.931 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3182750 2024-09-20T19:25:14.941 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 3 to smithi04268048-26 2024-09-20T19:25:14.941 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 4 to smithi04268048-26 2024-09-20T19:25:14.941 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 5 to smithi04268048-26 2024-09-20T19:25:14.941 INFO:tasks.rados.rados.0.smithi042.stdout:947: finishing write tid 6 to smithi04268048-26 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: oid 26 updating version 0 to 257 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: oid 26 updating version 257 to 260 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: oid 26 version 260 is already newer than 259 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: oid 26 version 260 is already newer than 258 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: oid 26 updating version 260 to 261 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 261 (ObjNum 342 snap 110 seq_num 342) dirty exists 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:947: left oid 26 (ObjNum 342 snap 110 seq_num 342) 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:946: finishing copy_from to smithi04268048-30 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 393 (ObjNum 322 snap 104 seq_num 322) dirty exists 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:948: finishing write tid 1 to smithi04268048-43 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:948: finishing write tid 2 to smithi04268048-43 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:948: finishing write tid 3 to smithi04268048-43 2024-09-20T19:25:14.942 INFO:tasks.rados.rados.0.smithi042.stdout:948: finishing write tid 4 to smithi04268048-43 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: finishing write tid 5 to smithi04268048-43 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: oid 43 updating version 0 to 223 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: oid 43 version 223 is already newer than 222 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: oid 43 version 223 is already newer than 221 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: oid 43 updating version 223 to 224 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 224 (ObjNum 343 snap 110 seq_num 343) dirty exists 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:948: left oid 43 (ObjNum 343 snap 110 seq_num 343) 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: finishing write tid 1 to smithi04268048-50 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: finishing write tid 2 to smithi04268048-50 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: finishing write tid 3 to smithi04268048-50 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: finishing write tid 4 to smithi04268048-50 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: finishing write tid 5 to smithi04268048-50 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: oid 50 updating version 0 to 464 2024-09-20T19:25:14.943 INFO:tasks.rados.rados.0.smithi042.stdout:949: oid 50 updating version 464 to 466 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:949: oid 50 updating version 466 to 467 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:949: oid 50 version 467 is already newer than 465 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 467 (ObjNum 344 snap 110 seq_num 344) dirty exists 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:949: left oid 50 (ObjNum 344 snap 110 seq_num 344) 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: finishing write tid 1 to smithi04268048-9 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: finishing write tid 2 to smithi04268048-9 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: finishing write tid 3 to smithi04268048-9 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: finishing write tid 4 to smithi04268048-9 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: finishing write tid 5 to smithi04268048-9 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: oid 9 updating version 0 to 473 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: oid 9 updating version 473 to 476 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: oid 9 version 476 is already newer than 475 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: oid 9 version 476 is already newer than 474 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 476 (ObjNum 345 snap 110 seq_num 345) dirty exists 2024-09-20T19:25:14.944 INFO:tasks.rados.rados.0.smithi042.stdout:951: left oid 9 (ObjNum 345 snap 110 seq_num 345) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:942: done (8 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:943: done (7 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:944: done (6 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:945: done (5 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:946: done (4 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:947: done (3 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:948: done (2 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:949: done (1 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:951: done (0 left) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:952: rollback oid 50 current snap is 110 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 97 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:953: read oid 7 snap -1 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:953: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:14.945 INFO:tasks.rados.rados.0.smithi042.stdout:954: snap_create 2024-09-20T19:25:14.946 INFO:tasks.rados.rados.0.smithi042.stdout:953: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:14.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:14.989 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1867668 2024-09-20T19:25:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:14.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:15.009 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1488788 2024-09-20T19:25:15.029 INFO:tasks.rados.rados.0.smithi042.stdout:952: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:25:15.029 INFO:tasks.rados.rados.0.smithi042.stdout:952: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:25:15.029 INFO:tasks.rados.rados.0.smithi042.stdout:952: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:25:15.029 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 479 (ObjNum 297 snap 94 seq_num 297) dirty exists 2024-09-20T19:25:15.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:15.179 INFO:tasks.rados.rados.0.smithi042.stdout:952: done (2 left) 2024-09-20T19:25:15.179 INFO:tasks.rados.rados.0.smithi042.stdout:953: done (1 left) 2024-09-20T19:25:15.179 INFO:tasks.rados.rados.0.smithi042.stdout:954: done (0 left) 2024-09-20T19:25:15.179 INFO:tasks.rados.rados.0.smithi042.stdout:955: snap_create 2024-09-20T19:25:15.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.241 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:15.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:15.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:15.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:15 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.037404+0000 mgr.y (mgr.24491) 279 : cluster [DBG] pgmap v425: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T19:25:15.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.162395+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:15.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: audit 2024-09-20T19:25:14.170303+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? v1:172.21.15.42:0/2193106485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:25:15.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:15 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.175129+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-20T19:25:15.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:15.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:15.629 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:15.732 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:15.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:15.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:15.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:15.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:16.123 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:16.150 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:16.178 INFO:tasks.rados.rados.0.smithi042.stdout:955: done (0 left) 2024-09-20T19:25:16.178 INFO:tasks.rados.rados.0.smithi042.stdout:956: write oid 15 current snap is 112 2024-09-20T19:25:16.178 INFO:tasks.rados.rados.0.smithi042.stdout:956: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-20T19:25:16.185 INFO:tasks.rados.rados.0.smithi042.stdout:956: writing smithi04268048-15 from 615222 to 1087261 tid 1 2024-09-20T19:25:16.190 INFO:tasks.rados.rados.0.smithi042.stdout:956: writing smithi04268048-15 from 1580652 to 2070094 tid 2 2024-09-20T19:25:16.198 INFO:tasks.rados.rados.0.smithi042.stdout:956: writing smithi04268048-15 from 2768826 to 3395413 tid 3 2024-09-20T19:25:16.198 INFO:tasks.rados.rados.0.smithi042.stdout:957: write oid 31 current snap is 112 2024-09-20T19:25:16.198 INFO:tasks.rados.rados.0.smithi042.stdout:957: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-20T19:25:16.205 INFO:tasks.rados.rados.0.smithi042.stdout:957: writing smithi04268048-31 from 632029 to 1179317 tid 1 2024-09-20T19:25:16.211 INFO:tasks.rados.rados.0.smithi042.stdout:957: writing smithi04268048-31 from 1639134 to 2388587 tid 2 2024-09-20T19:25:16.216 INFO:tasks.rados.rados.0.smithi042.stdout:957: writing smithi04268048-31 from 2796249 to 3234108 tid 3 2024-09-20T19:25:16.220 INFO:tasks.rados.rados.0.smithi042.stdout:957: writing smithi04268048-31 from 3653524 to 3832029 tid 4 2024-09-20T19:25:16.220 INFO:tasks.rados.rados.0.smithi042.stdout:958: read oid 46 snap -1 2024-09-20T19:25:16.221 INFO:tasks.rados.rados.0.smithi042.stdout:958: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:25:16.221 INFO:tasks.rados.rados.0.smithi042.stdout:959: write oid 33 current snap is 112 2024-09-20T19:25:16.221 INFO:tasks.rados.rados.0.smithi042.stdout:956: finishing write tid 1 to smithi04268048-15 2024-09-20T19:25:16.221 INFO:tasks.rados.rados.0.smithi042.stdout:959: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-20T19:25:16.227 INFO:tasks.rados.rados.0.smithi042.stdout:959: writing smithi04268048-33 from 648836 to 1271373 tid 1 2024-09-20T19:25:16.234 INFO:tasks.rados.rados.0.smithi042.stdout:959: writing smithi04268048-33 from 1981263 to 2474374 tid 2 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:959: writing smithi04268048-33 from 3107319 to 3848836 tid 3 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:960: write oid 14 current snap is 112 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: finishing write tid 2 to smithi04268048-15 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: finishing write tid 3 to smithi04268048-15 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: finishing write tid 4 to smithi04268048-15 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: finishing write tid 5 to smithi04268048-15 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: oid 15 updating version 0 to 318 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: oid 15 updating version 318 to 321 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: oid 15 version 321 is already newer than 319 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:956: oid 15 version 321 is already newer than 320 2024-09-20T19:25:16.243 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 321 (ObjNum 346 snap 112 seq_num 346) dirty exists 2024-09-20T19:25:16.244 INFO:tasks.rados.rados.0.smithi042.stdout:956: left oid 15 (ObjNum 346 snap 112 seq_num 346) 2024-09-20T19:25:16.244 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 1 to smithi04268048-31 2024-09-20T19:25:16.244 INFO:tasks.rados.rados.0.smithi042.stdout:960: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-20T19:25:16.250 INFO:tasks.rados.rados.0.smithi042.stdout:960: writing smithi04268048-14 from 665643 to 1363429 tid 1 2024-09-20T19:25:16.254 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:16.258 INFO:tasks.rados.rados.0.smithi042.stdout:960: writing smithi04268048-14 from 2039745 to 2792867 tid 2 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:960: writing smithi04268048-14 from 3534742 to 3865643 tid 3 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:956: done (4 left) 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:961: snap_create 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 2 to smithi04268048-31 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 3 to smithi04268048-31 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 4 to smithi04268048-31 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 5 to smithi04268048-31 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: finishing write tid 6 to smithi04268048-31 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: oid 31 updating version 0 to 325 2024-09-20T19:25:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:957: oid 31 updating version 325 to 329 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:957: oid 31 version 329 is already newer than 326 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:957: oid 31 version 329 is already newer than 328 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:957: oid 31 version 329 is already newer than 327 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 329 (ObjNum 347 snap 112 seq_num 347) dirty exists 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:957: left oid 31 (ObjNum 347 snap 112 seq_num 347) 2024-09-20T19:25:16.266 INFO:tasks.rados.rados.0.smithi042.stdout:958: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:25:16.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:16.336 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1256980 2024-09-20T19:25:16.366 INFO:tasks.rados.rados.0.smithi042.stdout:959: finishing write tid 1 to smithi04268048-33 2024-09-20T19:25:16.366 INFO:tasks.rados.rados.0.smithi042.stdout:959: finishing write tid 2 to smithi04268048-33 2024-09-20T19:25:16.366 INFO:tasks.rados.rados.0.smithi042.stdout:959: finishing write tid 3 to smithi04268048-33 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: finishing write tid 4 to smithi04268048-33 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: finishing write tid 5 to smithi04268048-33 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: oid 33 updating version 0 to 211 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: oid 33 version 211 is already newer than 209 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: oid 33 version 211 is already newer than 208 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: oid 33 version 211 is already newer than 210 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 211 (ObjNum 348 snap 112 seq_num 348) dirty exists 2024-09-20T19:25:16.367 INFO:tasks.rados.rados.0.smithi042.stdout:959: left oid 33 (ObjNum 348 snap 112 seq_num 348) 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: finishing write tid 1 to smithi04268048-14 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: finishing write tid 2 to smithi04268048-14 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: finishing write tid 3 to smithi04268048-14 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: finishing write tid 4 to smithi04268048-14 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: finishing write tid 5 to smithi04268048-14 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: oid 14 updating version 0 to 330 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: oid 14 updating version 330 to 332 2024-09-20T19:25:16.368 INFO:tasks.rados.rados.0.smithi042.stdout:960: oid 14 version 332 is already newer than 331 2024-09-20T19:25:16.369 INFO:tasks.rados.rados.0.smithi042.stdout:960: oid 14 version 332 is already newer than 329 2024-09-20T19:25:16.369 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 332 (ObjNum 349 snap 112 seq_num 349) dirty exists 2024-09-20T19:25:16.369 INFO:tasks.rados.rados.0.smithi042.stdout:960: left oid 14 (ObjNum 349 snap 112 seq_num 349) 2024-09-20T19:25:16.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:16.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:16.428 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[32075]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:16 smithi042 bash[22706]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:16.518 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:16.539 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:16.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.567976+0000 osd.3 (osd.3) 143 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.571001+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.775223+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.b scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:14.776616+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.b scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.159485+0000 osd.0 (osd.0) 143 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.163430+0000 osd.0 (osd.0) 144 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:16 smithi162 bash[24557]: cluster 2024-09-20T19:25:15.179276+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-20T19:25:16.642 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:16.668 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:16.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:16.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:16.929 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:16.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:17.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:25:17.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:17.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.191 INFO:tasks.rados.rados.0.smithi042.stdout:957: done (4 left) 2024-09-20T19:25:17.192 INFO:tasks.rados.rados.0.smithi042.stdout:958: done (3 left) 2024-09-20T19:25:17.192 INFO:tasks.rados.rados.0.smithi042.stdout:959: done (2 left) 2024-09-20T19:25:17.192 INFO:tasks.rados.rados.0.smithi042.stdout:960: done (1 left) 2024-09-20T19:25:17.192 INFO:tasks.rados.rados.0.smithi042.stdout:961: done (0 left) 2024-09-20T19:25:17.192 INFO:tasks.rados.rados.0.smithi042.stdout:962: snap_remove snap 111 2024-09-20T19:25:17.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:17.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:17.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:17 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:17 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:17 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:17 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:17.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:17.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:17.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.577 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.038038+0000 mgr.y (mgr.24491) 280 : cluster [DBG] pgmap v428: 158 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 151 active+clean; 306 MiB data, 966 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:17.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:17 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.178638+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-20T19:25:17.744 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:17.775 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:17.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:17.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:17.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:18.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:18.086 INFO:tasks.rados.rados.0.smithi042.stdout:962: done (0 left) 2024-09-20T19:25:18.087 INFO:tasks.rados.rados.0.smithi042.stdout:963: read oid 34 snap -1 2024-09-20T19:25:18.087 INFO:tasks.rados.rados.0.smithi042.stdout:963: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:18.087 INFO:tasks.rados.rados.0.smithi042.stdout:964: snap_remove snap 109 2024-09-20T19:25:18.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:18.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.277 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.279 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.280 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:18 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.288 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:18.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.414 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:18.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:18.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.374699+0000 osd.6 (osd.6) 105 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.376912+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.434314+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.437029+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.872144+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.874733+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.897842+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.a scrub starts 2024-09-20T19:25:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.530 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:16.899847+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.a scrub ok 2024-09-20T19:25:18.530 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.530 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:17.191583+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-20T19:25:18.530 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.530 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:18 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.085976+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-20T19:25:18.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:18.647 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:18.776 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:18.804 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:18.907 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:18.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:19.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:19.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:moved 3 / 474 (0.632911%) 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:avg 59.25 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:stddev 5.35607 -> 4.94343 (expected baseline 7.20026) 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:min osd.2 with 52 -> 53 pgs (0.877637 -> 0.894515 * mean) 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:max osd.3 with 70 -> 68 pgs (1.18143 -> 1.14768 * mean) 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:oload 110 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:max_change 0.05 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:max_change_osds 4 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:average_utilization 0.0014 2024-09-20T19:25:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:overload_utilization 0.0015 2024-09-20T19:25:19.085 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-20T19:25:19.085 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-20T19:25:19.094 INFO:tasks.rados.rados.0.smithi042.stdout:964: done (1 left) 2024-09-20T19:25:19.094 INFO:tasks.rados.rados.0.smithi042.stdout:965: copy_from oid 5 from oid 30 current snap is 113 2024-09-20T19:25:19.094 INFO:tasks.rados.rados.0.smithi042.stdout:966: write oid 29 current snap is 113 2024-09-20T19:25:19.094 INFO:tasks.rados.rados.0.smithi042.stdout:966: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-20T19:25:19.100 INFO:tasks.rados.rados.0.smithi042.stdout:966: writing smithi04268048-29 from 682450 to 1171838 tid 1 2024-09-20T19:25:19.108 INFO:tasks.rados.rados.0.smithi042.stdout:966: writing smithi04268048-29 from 1814580 to 2311360 tid 2 2024-09-20T19:25:19.117 INFO:tasks.rados.rados.0.smithi042.stdout:966: writing smithi04268048-29 from 2878518 to 3666546 tid 3 2024-09-20T19:25:19.117 INFO:tasks.rados.rados.0.smithi042.stdout:967: snap_create 2024-09-20T19:25:19.117 INFO:tasks.rados.rados.0.smithi042.stdout:963: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:19.142 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:19.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:19.173 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:966: finishing write tid 1 to smithi04268048-29 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:966: finishing write tid 2 to smithi04268048-29 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:965: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:965: finishing copy_from to smithi04268048-5 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 393 (ObjNum 322 snap 104 seq_num 322) dirty exists 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:966: finishing write tid 3 to smithi04268048-29 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:966: finishing write tid 4 to smithi04268048-29 2024-09-20T19:25:19.207 INFO:tasks.rados.rados.0.smithi042.stdout:966: finishing write tid 5 to smithi04268048-29 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:966: oid 29 updating version 0 to 386 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:966: oid 29 updating version 386 to 387 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:966: oid 29 version 387 is already newer than 385 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:966: oid 29 version 387 is already newer than 384 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 387 (ObjNum 350 snap 113 seq_num 350) dirty exists 2024-09-20T19:25:19.208 INFO:tasks.rados.rados.0.smithi042.stdout:966: left oid 29 (ObjNum 350 snap 113 seq_num 350) 2024-09-20T19:25:19.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:19.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:19 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.316 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:19.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:19.436 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.437 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.038591+0000 mgr.y (mgr.24491) 281 : cluster [DBG] pgmap v431: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:18.541815+0000 mgr.y (mgr.24491) 282 : audit [DBG] from='client.15216 v1:172.21.15.42:0/3936528434' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:18.542800+0000 mon.a (mon.0) 1343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"} : dispatch 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: audit 2024-09-20T19:25:19.083935+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260'}"}]': finished 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.438 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:19 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.089395+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-20T19:25:19.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:19.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:19.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:19.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:19.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:19.840 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:19.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:19.971 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-20T19:25:19.971 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-20T19:25:19.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:19.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:20.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:963: done (3 left) 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:965: done (2 left) 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:966: done (1 left) 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:967: done (0 left) 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:968: write oid 14 current snap is 114 2024-09-20T19:25:20.099 INFO:tasks.rados.rados.0.smithi042.stdout:968: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-20T19:25:20.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:20.111 INFO:tasks.rados.rados.0.smithi042.stdout:968: writing smithi04268048-14 from 699257 to 1263894 tid 1 2024-09-20T19:25:20.125 INFO:tasks.rados.rados.0.smithi042.stdout:968: writing smithi04268048-14 from 1873062 to 2513500 tid 2 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:968: writing smithi04268048-14 from 3305941 to 3899257 tid 3 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:969: copy_from oid 6 from oid 24 current snap is 114 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:968: finishing write tid 1 to smithi04268048-14 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:968: finishing write tid 2 to smithi04268048-14 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:970: read oid 12 snap -1 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:970: expect (ObjNum 335 snap 109 seq_num 335) 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:971: write oid 28 current snap is 114 2024-09-20T19:25:20.140 INFO:tasks.rados.rados.0.smithi042.stdout:971: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-20T19:25:20.152 INFO:tasks.rados.rados.0.smithi042.stdout:971: writing smithi04268048-28 from 716064 to 1355950 tid 1 2024-09-20T19:25:20.160 INFO:tasks.rados.rados.0.smithi042.stdout:971: writing smithi04268048-28 from 1815191 to 2315640 tid 2 2024-09-20T19:25:20.168 INFO:tasks.rados.rados.0.smithi042.stdout:971: writing smithi04268048-28 from 2817011 to 3343936 tid 3 2024-09-20T19:25:20.168 INFO:tasks.rados.rados.0.smithi042.stdout:972: snap_remove snap 102 2024-09-20T19:25:20.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:20.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:20.381 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:20.381 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:20.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.471 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.471 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.471 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.471 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:20 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:20.523 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.523 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.523 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.525 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.425591+0000 osd.1 (osd.1) 147 : cluster [DBG] 7.b scrub starts 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.429554+0000 osd.1 (osd.1) 148 : cluster [DBG] 7.b scrub ok 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.615989+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:25:20.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.620420+0000 osd.3 (osd.3) 146 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.832247+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:18.833851+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: audit 2024-09-20T19:25:19.204182+0000 mgr.y (mgr.24491) 283 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:20.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:20 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.093011+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-20T19:25:20.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:20.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:20.744 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:20.848 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.873 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:20.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:20.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:21.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:21.105 INFO:tasks.rados.rados.0.smithi042.stdout:972: done (4 left) 2024-09-20T19:25:21.105 INFO:tasks.rados.rados.0.smithi042.stdout:973: delete oid 29 current snap is 114 2024-09-20T19:25:21.105 INFO:tasks.rados.rados.0.smithi042.stdout:970: expect (ObjNum 335 snap 109 seq_num 335) 2024-09-20T19:25:21.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:21.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:21.204 INFO:tasks.rados.rados.0.smithi042.stdout:968: finishing write tid 3 to smithi04268048-14 2024-09-20T19:25:21.204 INFO:tasks.rados.rados.0.smithi042.stdout:968: finishing write tid 4 to smithi04268048-14 2024-09-20T19:25:21.204 INFO:tasks.rados.rados.0.smithi042.stdout:968: finishing write tid 5 to smithi04268048-14 2024-09-20T19:25:21.204 INFO:tasks.rados.rados.0.smithi042.stdout:968: oid 14 updating version 0 to 395 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:968: oid 14 version 395 is already newer than 394 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:968: oid 14 updating version 395 to 397 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:968: oid 14 version 397 is already newer than 396 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 397 (ObjNum 351 snap 114 seq_num 351) dirty exists 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:968: left oid 14 (ObjNum 351 snap 114 seq_num 351) 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:971: finishing write tid 1 to smithi04268048-28 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:968: done (4 left) 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:970: done (3 left) 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:973: done (2 left) 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:974: rollback oid 8 current snap is 114 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 103 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:971: finishing write tid 2 to smithi04268048-28 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:971: finishing write tid 3 to smithi04268048-28 2024-09-20T19:25:21.205 INFO:tasks.rados.rados.0.smithi042.stdout:971: finishing write tid 4 to smithi04268048-28 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: finishing write tid 5 to smithi04268048-28 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: oid 28 updating version 0 to 329 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: oid 28 updating version 329 to 331 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: oid 28 updating version 331 to 332 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: oid 28 version 332 is already newer than 330 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 332 (ObjNum 352 snap 114 seq_num 352) dirty exists 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: left oid 28 (ObjNum 352 snap 114 seq_num 352) 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:969: finishing copy_from to smithi04268048-6 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:969: got expected ENOENT (src dne) 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:969: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:969: done (2 left) 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:971: done (1 left) 2024-09-20T19:25:21.206 INFO:tasks.rados.rados.0.smithi042.stdout:975: snap_create 2024-09-20T19:25:21.208 INFO:tasks.rados.rados.0.smithi042.stdout:974: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:25:21.208 INFO:tasks.rados.rados.0.smithi042.stdout:974: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:25:21.208 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 331 (ObjNum 129 snap 0 seq_num 1415337584) dirty dne 2024-09-20T19:25:21.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:21 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.360 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:21.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:21.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:21.618 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:21.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.797042+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.799784+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.833265+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:19.834603+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.039299+0000 mgr.y (mgr.24491) 284 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 334 MiB data, 1019 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.0 MiB/s wr, 21 op/s 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.608670+0000 osd.4 (osd.4) 141 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:25:21.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.610991+0000 osd.4 (osd.4) 142 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:25:21.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:21 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.097032+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-20T19:25:21.645 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:21.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:21.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:21.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:22.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:22.039 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:974: done (1 left) 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:975: done (0 left) 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:976: read oid 32 snap 85 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:976: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:977: write oid 36 current snap is 115 2024-09-20T19:25:22.099 INFO:tasks.rados.rados.0.smithi042.stdout:977: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-20T19:25:22.107 INFO:tasks.rados.rados.0.smithi042.stdout:977: writing smithi04268048-36 from 749678 to 1540062 tid 1 2024-09-20T19:25:22.115 INFO:tasks.rados.rados.0.smithi042.stdout:977: writing smithi04268048-36 from 2332155 to 2836273 tid 2 2024-09-20T19:25:22.121 INFO:tasks.rados.rados.0.smithi042.stdout:977: writing smithi04268048-36 from 3271857 to 3821326 tid 3 2024-09-20T19:25:22.121 INFO:tasks.rados.rados.0.smithi042.stdout:978: write oid 49 current snap is 115 2024-09-20T19:25:22.121 INFO:tasks.rados.rados.0.smithi042.stdout:978: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-20T19:25:22.128 INFO:tasks.rados.rados.0.smithi042.stdout:978: writing smithi04268048-49 from 766485 to 1232118 tid 1 2024-09-20T19:25:22.136 INFO:tasks.rados.rados.0.smithi042.stdout:978: writing smithi04268048-49 from 1990637 to 2638413 tid 2 2024-09-20T19:25:22.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:22.144 INFO:tasks.rados.rados.0.smithi042.stdout:978: writing smithi04268048-49 from 3299280 to 3966485 tid 3 2024-09-20T19:25:22.144 INFO:tasks.rados.rados.0.smithi042.stdout:979: write oid 50 current snap is 115 2024-09-20T19:25:22.144 INFO:tasks.rados.rados.0.smithi042.stdout:979: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-20T19:25:22.151 INFO:tasks.rados.rados.0.smithi042.stdout:979: writing smithi04268048-50 from 783292 to 1324174 tid 1 2024-09-20T19:25:22.158 INFO:tasks.rados.rados.0.smithi042.stdout:979: writing smithi04268048-50 from 1932766 to 2440553 tid 2 2024-09-20T19:25:22.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:22.166 INFO:tasks.rados.rados.0.smithi042.stdout:979: writing smithi04268048-50 from 3210350 to 3782363 tid 3 2024-09-20T19:25:22.166 INFO:tasks.rados.rados.0.smithi042.stdout:980: write (excl) oid 47 current snap is 115 2024-09-20T19:25:22.166 INFO:tasks.rados.rados.0.smithi042.stdout:976: expect (ObjNum 196 snap 59 seq_num 196) 2024-09-20T19:25:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:22.217 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:22.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:22.272 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:22.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:22.321 INFO:tasks.rados.rados.0.smithi042.stdout:977: finishing write tid 1 to smithi04268048-36 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: finishing write tid 2 to smithi04268048-36 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: finishing write tid 3 to smithi04268048-36 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: finishing write tid 4 to smithi04268048-36 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: finishing write tid 5 to smithi04268048-36 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: oid 36 updating version 0 to 314 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: oid 36 updating version 314 to 316 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: oid 36 version 316 is already newer than 315 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: oid 36 version 316 is already newer than 313 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 316 (ObjNum 354 snap 115 seq_num 354) dirty exists 2024-09-20T19:25:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:977: left oid 36 (ObjNum 354 snap 115 seq_num 354) 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: finishing write tid 1 to smithi04268048-49 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: finishing write tid 2 to smithi04268048-49 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: finishing write tid 3 to smithi04268048-49 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: finishing write tid 4 to smithi04268048-49 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: finishing write tid 5 to smithi04268048-49 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: oid 49 updating version 0 to 439 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: oid 49 version 439 is already newer than 438 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: oid 49 updating version 439 to 440 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: oid 49 updating version 440 to 441 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 441 (ObjNum 355 snap 115 seq_num 355) dirty exists 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:978: left oid 49 (ObjNum 355 snap 115 seq_num 355) 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:979: finishing write tid 1 to smithi04268048-50 2024-09-20T19:25:22.323 INFO:tasks.rados.rados.0.smithi042.stdout:979: finishing write tid 2 to smithi04268048-50 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: finishing write tid 3 to smithi04268048-50 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: finishing write tid 4 to smithi04268048-50 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: finishing write tid 5 to smithi04268048-50 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: oid 50 updating version 0 to 493 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: oid 50 updating version 493 to 496 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: oid 50 version 496 is already newer than 495 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: oid 50 version 496 is already newer than 494 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 496 (ObjNum 356 snap 115 seq_num 356) dirty exists 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:979: left oid 50 (ObjNum 356 snap 115 seq_num 356) 2024-09-20T19:25:22.324 INFO:tasks.rados.rados.0.smithi042.stdout:980: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-20T19:25:22.331 INFO:tasks.rados.rados.0.smithi042.stdout:980: writing smithi04268048-47 from 400099 to 1016230 tid 1 2024-09-20T19:25:22.339 INFO:tasks.rados.rados.0.smithi042.stdout:980: writing smithi04268048-47 from 1591248 to 2000099 tid 2 2024-09-20T19:25:22.339 INFO:tasks.rados.rados.0.smithi042.stdout:976: done (4 left) 2024-09-20T19:25:22.339 INFO:tasks.rados.rados.0.smithi042.stdout:977: done (3 left) 2024-09-20T19:25:22.339 INFO:tasks.rados.rados.0.smithi042.stdout:978: done (2 left) 2024-09-20T19:25:22.340 INFO:tasks.rados.rados.0.smithi042.stdout:979: done (1 left) 2024-09-20T19:25:22.340 INFO:tasks.rados.rados.0.smithi042.stdout:981: read oid 2 snap -1 2024-09-20T19:25:22.340 INFO:tasks.rados.rados.0.smithi042.stdout:981: expect (ObjNum 307 snap 99 seq_num 307) 2024-09-20T19:25:22.340 INFO:tasks.rados.rados.0.smithi042.stdout:982: snap_remove snap 108 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.397 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.397 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:22 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:22.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:22.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.334974+0000 osd.7 (osd.7) 113 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.337758+0000 osd.7 (osd.7) 114 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.430577+0000 osd.1 (osd.1) 149 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:25:22.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.432345+0000 osd.1 (osd.1) 150 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.680636+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.682348+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.755883+0000 osd.5 (osd.5) 129 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:20.758453+0000 osd.5 (osd.5) 130 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.116279+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.127622+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:25:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:22 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.099018+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-20T19:25:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:22.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:22.564 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:25:22.564 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:25:22.583 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:25:22.583 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:25:22.602 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:25:22.602 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:25:22.614 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.7.0:63 7.5 e148/112 7:a66d8c87:::smithi04268048-11:11 v 148'61, mlcod=143'57)", 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:17.028336+0000", 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 185.58476723300001, 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023463126000000001, 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.615 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028336+0000", 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028336+0000", 2024-09-20T19:25:22.616 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028337+0000", 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4670000000000001e-06 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028349+0000", 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1596e-05 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028371+0000", 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1807e-05 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.028389+0000", 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8340000000000001e-05 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.029740+0000", 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001351594 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.029793+0000", 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2403e-05 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.051681+0000", 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021888149999999999 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.051799+0000", 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000117769 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:506 7.9 7:9b44618a:::smithi04268048-36:head [copy-from ver 23] snapc 16=[16,15,13,12,11,10,f,a] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:21.017205+0000", 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 181.59589755900001, 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036370779999999998, 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 506 2024-09-20T19:25:22.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017205+0000", 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017205+0000", 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.622 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017210+0000", 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5569999999999997e-06 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017228+0000", 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7793000000000001e-05 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017260+0000", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2165999999999997e-05 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017284+0000", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.423e-05 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017475+0000", 2024-09-20T19:25:22.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019078199999999999 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.017736+0000", 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026109800000000003 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.037380+0000", 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019644399 2024-09-20T19:25:22.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.050666+0000", 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01328587 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.051866+0000", 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011999459999999999 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.626 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.053380+0000", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001514034 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.053494+0000", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000113489 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.053576+0000", 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2416000000000005e-05 2024-09-20T19:25:22.627 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:507 7.9 7:9b44618a:::smithi04268048-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e152)", 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:21.018226+0000", 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 181.594876849, 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032325113000000003, 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.628 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 507 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018226+0000", 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.629 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018226+0000", 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018229+0000", 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7030000000000002e-06 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018240+0000", 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1283e-05 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.630 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018265+0000", 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4994000000000002e-05 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018288+0000", 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3093000000000001e-05 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.631 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018653+0000", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00036493499999999999 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.018759+0000", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000105629 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.050050+0000", 2024-09-20T19:25:22.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.031290879000000001 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.050301+0000", 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00025178499999999998 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.050551+0000", 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024981200000000003 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.633 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:699 7.6 7:6c48c6fe:::smithi04268048-7:head [copy-from ver 34] snapc 1c=[1c,1a,19,10] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:37.155719+0000", 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 165.45738399699999, 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020341227, 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.634 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 699 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155719+0000", 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155719+0000", 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.635 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155720+0000", 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1399999999999995e-07 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155724+0000", 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1189999999999999e-06 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.636 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155733+0000", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1810000000000007e-06 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.155738+0000", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4139999999999996e-06 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.157590+0000", 2024-09-20T19:25:22.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0018527210000000001 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.157679+0000", 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8838000000000006e-05 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.166937+0000", 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0092581269999999997 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.173551+0000", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0066137510000000002 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.173611+0000", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0390999999999998e-05 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.175950+0000", 2024-09-20T19:25:22.639 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0023388150000000002 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.176010+0000", 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0123e-05 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:37.176060+0000", 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9833000000000002e-05 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.640 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1151 7.4 7:2924b69c:::smithi04268048-21:head [rollback 30] snapc 38=[38,37,36,34,33,30,2f,2c] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:29.042529+0000", 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 113.570574264, 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020095115, 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.641 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1151 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042529+0000", 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042529+0000", 2024-09-20T19:25:22.642 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042530+0000", 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0720000000000001e-06 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042535+0000", 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.48e-06 2024-09-20T19:25:22.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042545+0000", 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3659999999999998e-06 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042553+0000", 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1340000000000004e-06 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.644 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042708+0000", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015569999999999999 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042715+0000", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8759999999999999e-06 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.053043+0000", 2024-09-20T19:25:22.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01032748 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.053181+0000", 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013855700000000001 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.053434+0000", 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00025284299999999999 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.055650+0000", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0022162990000000001 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.061485+0000", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0058341290000000004 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062512+0000", 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010275060000000001 2024-09-20T19:25:22.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062575+0000", 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3390000000000001e-05 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062624+0000", 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8282999999999998e-05 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1152 7.4 7:2924b69c:::smithi04268048-21:head [append 0~0] snapc 38=[38,37,36,34,33,30,2f,2c] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:29.042664+0000", 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 113.570438579, 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.02010733, 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.649 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1152 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042664+0000", 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.650 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042664+0000", 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042666+0000", 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1550000000000001e-06 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.651 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042683+0000", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6813999999999999e-05 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042696+0000", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2221999999999999e-05 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042706+0000", 2024-09-20T19:25:22.652 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.093e-05 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042766+0000", 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9404000000000002e-05 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.042769+0000", 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.484e-06 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.054257+0000", 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011487189 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.054391+0000", 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013423999999999999 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.054629+0000", 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023807 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.055659+0000", 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010299739999999999 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.061529+0000", 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0058701059999999999 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062665+0000", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011362290000000001 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062715+0000", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.0148000000000002e-05 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:29.062772+0000", 2024-09-20T19:25:22.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6365e-05 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1244 7.4 7:2a17ba7b:::smithi04268048-18:head [copy-from ver 162] snapc 39=[39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.325347+0000", 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 106.28775598, 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.044968528000000001, 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.657 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1244 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325347+0000", 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325347+0000", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325348+0000", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.536e-06 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325354+0000", 2024-09-20T19:25:22.659 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6679999999999998e-06 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325365+0000", 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.062e-05 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325373+0000", 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.7810000000000005e-06 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327331+0000", 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0019581139999999999 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327431+0000", 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100111 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.358862+0000", 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.031430756999999997 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.367083+0000", 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0082212729999999994 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.367220+0000", 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013679400000000001 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370214+0000", 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002994895 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370258+0000", 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3996999999999999e-05 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370315+0000", 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6981999999999998e-05 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1245 7.4 7:2a17ba7b:::smithi04268048-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.325386+0000", 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 106.28771656000001, 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037559007999999998, 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.664 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1245 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325386+0000", 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.665 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325386+0000", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325387+0000", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6199999999999997e-07 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325391+0000", 2024-09-20T19:25:22.666 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8720000000000004e-06 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325398+0000", 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0299999999999996e-06 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.325404+0000", 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9030000000000001e-06 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.328415+0000", 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0030109410000000001 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.328461+0000", 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6541e-05 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.362797+0000", 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.034335782000000002 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.362865+0000", 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8236000000000002e-05 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.362945+0000", 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9940999999999999e-05 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1576 7.4 7:23ed6a02:::smithi04268048-25:head [copy-from ver 104] snapc 44=[44,43,42,41,3e,3d] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:55.071648+0000", 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 87.541455377000005, 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.049634655, 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.670 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1576 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071648+0000", 2024-09-20T19:25:22.671 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071648+0000", 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071653+0000", 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.0470000000000001e-06 2024-09-20T19:25:22.672 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071670+0000", 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7306999999999999e-05 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071702+0000", 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1636e-05 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.673 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071723+0000", 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1679999999999999e-05 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072204+0000", 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048065299999999999 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072441+0000", 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023696200000000001 2024-09-20T19:25:22.674 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.109091+0000", 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036649982999999997 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.119699+0000", 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010607742999999999 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.675 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.119776+0000", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.7866000000000003e-05 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.121145+0000", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001368604 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.121230+0000", 2024-09-20T19:25:22.676 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5069999999999997e-05 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.121282+0000", 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2104000000000001e-05 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.677 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1577 7.4 7:23ed6a02:::smithi04268048-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e246)", 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:55.071777+0000", 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 87.541325827999998, 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.047811940999999997, 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1577 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.678 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071777+0000", 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071777+0000", 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.679 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071779+0000", 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3800000000000001e-06 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071790+0000", 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.078e-05 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071812+0000", 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1733999999999999e-05 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.680 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071829+0000", 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7215999999999999e-05 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.076279+0000", 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044496029999999999 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.681 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.076386+0000", 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107666 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.119268+0000", 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.042881720999999998 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.119420+0000", 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015217299999999999 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.682 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.119589+0000", 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016866799999999999 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1578 7.6 7:6c48c6fe:::smithi04268048-7:head [copy-from ver 250] snapc 44=[44,43,42,41,3e,3d] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:55.071890+0000", 2024-09-20T19:25:22.683 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 87.541212548000004, 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.039092665999999998, 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1578 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.684 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071890+0000", 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071890+0000", 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.685 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071893+0000", 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6359999999999998e-06 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071905+0000", 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2119999999999999e-05 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.686 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071927+0000", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1759999999999998e-05 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071943+0000", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6447e-05 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072070+0000", 2024-09-20T19:25:22.687 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000126303 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072182+0000", 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000112199 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.097235+0000", 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.025052873 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.108486+0000", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011251667999999999 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.110601+0000", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002114576 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.110712+0000", 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000110717 2024-09-20T19:25:22.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.110859+0000", 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014687599999999999 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.110983+0000", 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012449100000000001 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.690 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1579 7.6 7:6c48c6fe:::smithi04268048-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e246)", 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:55.071993+0000", 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 87.541109634999998, 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036301514999999999, 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.691 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1579 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071993+0000", 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071993+0000", 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.071996+0000", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6089999999999999e-06 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072005+0000", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4469999999999995e-06 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072026+0000", 2024-09-20T19:25:22.693 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0735999999999999e-05 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072042+0000", 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5800000000000001e-05 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072808+0000", 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00076570499999999999 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.694 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.072934+0000", 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000126406 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.107938+0000", 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.035003588000000002 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.695 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.108079+0000", 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000141485 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.108295+0000", 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000215739 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.696 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1600 7.4 7:2a17ba7b:::smithi04268048-18:head [append 0~0] snapc 46=[46,45,44,42,41,3e,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:58.056415+0000", 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 84.556688105999996, 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.027623660000000001, 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1600 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.697 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056415+0000", 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056415+0000", 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.698 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056419+0000", 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9369999999999997e-06 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056431+0000", 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2459999999999999e-05 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056462+0000", 2024-09-20T19:25:22.699 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0620000000000002e-05 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056480+0000", 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7960000000000001e-05 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056595+0000", 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000115591 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.700 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056606+0000", 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0183000000000001e-05 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.062219+0000", 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.005613626 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.701 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.062469+0000", 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00025020100000000003 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.063007+0000", 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00053771299999999999 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.082389+0000", 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019381589000000001 2024-09-20T19:25:22.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.083669+0000", 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012800369999999999 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.083839+0000", 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000170058 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.083938+0000", 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9693999999999998e-05 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.084038+0000", 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9990999999999999e-05 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.704 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1601 7.4 7:2a17ba7b:::smithi04268048-18:head [rollback 42] snapc 46=[46,45,44,42,41,3e,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:58.056518+0000", 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 84.556584779999994, 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.033490151000000003, 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1601 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056518+0000", 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056518+0000", 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056519+0000", 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.243e-06 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056525+0000", 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1599999999999997e-06 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056538+0000", 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3689999999999999e-05 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056550+0000", 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2103000000000001e-05 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056762+0000", 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021158599999999999 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.709 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056826+0000", 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4278999999999999e-05 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.065053+0000", 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0082267799999999995 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.065292+0000", 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023878199999999999 2024-09-20T19:25:22.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.065760+0000", 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000468159 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.083755+0000", 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017994649000000001 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.084135+0000", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00038016599999999998 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.089874+0000", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0057391869999999998 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.089957+0000", 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2856999999999996e-05 2024-09-20T19:25:22.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.090008+0000", 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.151e-05 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1602 7.4 7:2a17ba7b:::smithi04268048-18:head [append 0~0] snapc 46=[46,45,44,42,41,3e,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:58.056633+0000", 2024-09-20T19:25:22.713 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 84.556470054000002, 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.033877149000000002, 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1602 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.714 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056633+0000", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056633+0000", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056634+0000", 2024-09-20T19:25:22.715 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3570000000000001e-06 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056643+0000", 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9230000000000008e-06 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056665+0000", 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2062999999999999e-05 2024-09-20T19:25:22.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056682+0000", 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.641e-05 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056940+0000", 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00025809200000000002 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.056958+0000", 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7946000000000001e-05 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.082487+0000", 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.025529083000000001 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.082711+0000", 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0002242 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.083013+0000", 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000302343 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.089852+0000", 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0068384689999999998 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.090016+0000", 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016379599999999999 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.090392+0000", 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00037667000000000001 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.720 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.090439+0000", 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7212000000000001e-05 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:58.090510+0000", 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0585000000000003e-05 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2054 7.f 7:f6310c05:::smithi04268048-37:head [copy-from ver 297] snapc 5a=[5a,59,57,55,52,51,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-20T19:25:22.721 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:36.073813+0000", 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 46.539290254000001, 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.034799512999999997, 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2054 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073813+0000", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073813+0000", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073817+0000", 2024-09-20T19:25:22.723 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7700000000000001e-06 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073836+0000", 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8810000000000001e-05 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073865+0000", 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9234000000000002e-05 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073889+0000", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3283000000000001e-05 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.074002+0000", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000113603 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.074011+0000", 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5750000000000001e-06 2024-09-20T19:25:22.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.077551+0000", 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0035401550000000001 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.077727+0000", 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017568099999999999 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.726 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.096841+0000", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019114576000000001 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.104973+0000", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0081313189999999997 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.105147+0000", 2024-09-20T19:25:22.727 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017460300000000001 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.108520+0000", 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0033730309999999999 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.108559+0000", 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9066000000000002e-05 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.108612+0000", 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2806999999999998e-05 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2055 7.f 7:f6310c05:::smithi04268048-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e287)", 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:36.073955+0000", 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 46.539147991, 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028844496000000001, 2024-09-20T19:25:22.729 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2055 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073955+0000", 2024-09-20T19:25:22.730 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073955+0000", 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073957+0000", 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.176e-06 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073967+0000", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5270000000000004e-06 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.073986+0000", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9709999999999999e-05 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.074004+0000", 2024-09-20T19:25:22.732 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7586999999999999e-05 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.074089+0000", 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.4967000000000002e-05 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.074098+0000", 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7930000000000003e-06 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.077994+0000", 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0038958859999999999 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.078158+0000", 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000164711 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.102648+0000", 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.024489802000000001 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.102720+0000", 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.1709000000000001e-05 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:36.102799+0000", 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9628000000000004e-05 2024-09-20T19:25:22.735 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2115 7.f 7:f6310c05:::smithi04268048-37:head [copy-from ver 324] snapc 5f=[5f,5e,5d,5a,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:46.083594+0000", 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 36.529509142000002, 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.040151315999999999, 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.736 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2115 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083594+0000", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083594+0000", 2024-09-20T19:25:22.737 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083599+0000", 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9139999999999999e-06 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083618+0000", 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9692999999999999e-05 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083665+0000", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7102999999999999e-05 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083690+0000", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4083e-05 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083877+0000", 2024-09-20T19:25:22.739 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018754199999999999 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083898+0000", 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1233000000000002e-05 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.093672+0000", 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0097732919999999994 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.093936+0000", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026463799999999998 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.111216+0000", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017280119 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.121600+0000", 2024-09-20T19:25:22.741 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010383607 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.123123+0000", 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001522799 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.123555+0000", 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00043199699999999999 2024-09-20T19:25:22.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.123657+0000", 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010198 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.123745+0000", 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8315999999999999e-05 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.743 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2116 7.f 7:f6310c05:::smithi04268048-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:46.083754+0000", 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 36.529348489, 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037710201999999998, 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2116 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083754+0000", 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.745 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083754+0000", 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083757+0000", 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3460000000000001e-06 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083767+0000", 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0023999999999999e-05 2024-09-20T19:25:22.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083788+0000", 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1690000000000001e-05 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.083810+0000", 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1460000000000001e-05 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.084059+0000", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024913399999999998 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.084077+0000", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7626999999999999e-05 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.095889+0000", 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011811877 2024-09-20T19:25:22.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.096064+0000", 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017520599999999999 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.121089+0000", 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.025025162 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.749 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.121275+0000", 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018595100000000001 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.121465+0000", 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018972499999999999 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.750 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2403 7.f 7:f6310c05:::smithi04268048-37:head [stat,delete] snapc 6e=[6e,6d,6c,6b,6a,69,68,67,66,63,62,5e,5d,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:09.132795+0000", 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 13.480307881, 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023004809000000001, 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:22.751 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2403 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132795+0000", 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132795+0000", 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:22.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132801+0000", 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9449999999999998e-06 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132819+0000", 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7810999999999999e-05 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132862+0000", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2854e-05 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.132882+0000", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0276e-05 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.133109+0000", 2024-09-20T19:25:22.754 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022728599999999999 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.133138+0000", 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9173e-05 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.147836+0000", 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014698048 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.148020+0000", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018319699999999999 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.150287+0000", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0022673649999999999 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.152891+0000", 2024-09-20T19:25:22.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0026038530000000002 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.153085+0000", 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019367899999999999 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.155659+0000", 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00257413 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.155718+0000", 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9219999999999999e-05 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:09.155800+0000", 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1971999999999994e-05 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.758 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:22.759 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:22.759 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:25:22.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:22.760 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:25:22.760 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:25:22.760 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:25:22.761 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:25:22.761 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:25:22.761 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:25:22.762 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:25:22.762 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:25:22.762 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:25:22.762 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:25:22.763 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:22.789 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:22.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:22.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:22.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:23.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:23.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:982: done (2 left) 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:983: read oid 35 snap -1 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:983: expect (ObjNum 316 snap 101 seq_num 316) 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:984: read oid 3 snap -1 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:984: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:985: rollback oid 23 current snap is 115 2024-09-20T19:25:23.095 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 103 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: finishing write tid 1 to smithi04268048-47 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:986: rollback oid 19 current snap is 115 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 103 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:987: copy_from oid 15 from oid 43 current snap is 115 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:988: rollback oid 48 current snap is 115 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 103 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:989: copy_from oid 10 from oid 17 current snap is 115 2024-09-20T19:25:23.096 INFO:tasks.rados.rados.0.smithi042.stdout:990: snap_remove snap 114 2024-09-20T19:25:23.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:23.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:23.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:23.294 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.296 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:23 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:23.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:23.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:23.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.329601+0000 osd.7 (osd.7) 115 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.331874+0000 osd.7 (osd.7) 116 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.380307+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:21.382454+0000 osd.6 (osd.6) 108 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.040390+0000 mgr.y (mgr.24491) 285 : cluster [DBG] pgmap v437: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.109550+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.b scrub starts 2024-09-20T19:25:23.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.114690+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.b scrub ok 2024-09-20T19:25:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: audit 2024-09-20T19:25:22.544700+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.42:0/700822685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:23 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.093392+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-20T19:25:23.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:23.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:23.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:23.831 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:23.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:23.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:23.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:24.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:24.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:990: done (9 left) 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:991: copy_from oid 12 from oid 34 current snap is 115 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: finishing write tid 2 to smithi04268048-47 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: finishing write tid 3 to smithi04268048-47 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: finishing write tid 4 to smithi04268048-47 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: oid 47 updating version 0 to 499 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: oid 47 version 499 is already newer than 498 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: oid 47 updating version 499 to 500 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 500 (ObjNum 357 snap 115 seq_num 357) dirty exists 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: left oid 47 (ObjNum 357 snap 115 seq_num 357) 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:980: done (9 left) 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:992: copy_from oid 14 from oid 46 current snap is 115 2024-09-20T19:25:24.096 INFO:tasks.rados.rados.0.smithi042.stdout:981: expect (ObjNum 307 snap 99 seq_num 307) 2024-09-20T19:25:24.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:24.219 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:24.233 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:24.290 INFO:tasks.rados.rados.0.smithi042.stdout:983: expect (ObjNum 316 snap 101 seq_num 316) 2024-09-20T19:25:24.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:24.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:24 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.348 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:24.451 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:24.453 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3120563 2024-09-20T19:25:24.461 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.461 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.461 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.461 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.461 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.369855+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:22.371399+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:25:24.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.157606+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.161048+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:24 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.094649+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-20T19:25:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:24.545 INFO:tasks.rados.rados.0.smithi042.stdout:986: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:25:24.545 INFO:tasks.rados.rados.0.smithi042.stdout:984: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:24.545 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:24.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:24.599 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:24.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:24.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:25:24.662 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:24.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:24.714 INFO:tasks.rados.rados.0.smithi042.stdout:989: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:25:24.714 INFO:tasks.rados.rados.0.smithi042.stdout:985: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:25:24.714 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 370 (ObjNum 33 snap 0 seq_num 1413021408) dirty dne 2024-09-20T19:25:24.714 INFO:tasks.rados.rados.0.smithi042.stdout:986: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 334 (ObjNum 33 snap 0 seq_num 1415983472) dirty dne 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:987: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:981: done (9 left) 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:983: done (8 left) 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:984: done (7 left) 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:985: done (6 left) 2024-09-20T19:25:24.715 INFO:tasks.rados.rados.0.smithi042.stdout:986: done (5 left) 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:993: read oid 7 snap -1 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:993: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:994: copy_from oid 38 from oid 33 current snap is 115 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:988: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:988: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:988: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 496 (ObjNum 256 snap 74 seq_num 256) dirty exists 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:989: finishing copy_from to smithi04268048-10 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 407 (ObjNum 297 snap 94 seq_num 297) dirty exists 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:987: finishing copy_from to smithi04268048-15 2024-09-20T19:25:24.716 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 224 (ObjNum 343 snap 110 seq_num 343) dirty exists 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:992: finishing copy_from racing read to smithi04268048-14 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:992: finishing copy_from to smithi04268048-14 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 337 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:991: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:991: finishing copy_from to smithi04268048-12 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:987: done (6 left) 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:988: done (5 left) 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:989: done (4 left) 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:991: done (3 left) 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:992: done (2 left) 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:995: copy_from oid 31 from oid 1 current snap is 115 2024-09-20T19:25:24.717 INFO:tasks.rados.rados.0.smithi042.stdout:996: rollback oid 30 current snap is 115 2024-09-20T19:25:24.718 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 63 2024-09-20T19:25:24.718 INFO:tasks.rados.rados.0.smithi042.stdout:997: snap_remove snap 97 2024-09-20T19:25:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:24.840 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:24.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:24.961 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:24.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:25.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.093 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:25.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:25.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:25.250 INFO:tasks.rados.rados.0.smithi042.stdout:997: done (4 left) 2024-09-20T19:25:25.250 INFO:tasks.rados.rados.0.smithi042.stdout:998: write (excl) oid 23 current snap is 115 2024-09-20T19:25:25.250 INFO:tasks.rados.rados.0.smithi042.stdout:998: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-20T19:25:25.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:25.264 INFO:tasks.rados.rados.0.smithi042.stdout:998: writing smithi04268048-23 from 416906 to 824639 tid 1 2024-09-20T19:25:25.278 INFO:tasks.rados.rados.0.smithi042.stdout:998: writing smithi04268048-23 from 1366083 to 1877539 tid 2 2024-09-20T19:25:25.278 INFO:tasks.rados.rados.0.smithi042.stdout:999: delete oid 9 current snap is 115 2024-09-20T19:25:25.282 INFO:tasks.rados.rados.0.smithi042.stdout:993: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:25.282 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:25.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:25.390 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 914825 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:996: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:996: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 503 (ObjNum 129 snap 0 seq_num 1414870176) dirty dne 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:995: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:995: finishing copy_from to smithi04268048-31 2024-09-20T19:25:25.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 311 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:994: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:994: finishing copy_from to smithi04268048-38 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 211 (ObjNum 348 snap 112 seq_num 348) dirty exists 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:993: done (5 left) 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:994: done (4 left) 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:995: done (3 left) 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:996: done (2 left) 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:999: done (1 left) 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:1000: copy_from oid 11 from oid 21 current snap is 115 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:998: finishing write tid 1 to smithi04268048-23 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:998: finishing write tid 2 to smithi04268048-23 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:998: finishing write tid 3 to smithi04268048-23 2024-09-20T19:25:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:998: finishing write tid 4 to smithi04268048-23 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:998: oid 23 updating version 0 to 421 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:998: oid 23 version 421 is already newer than 420 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:998: oid 23 version 421 is already newer than 419 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 421 (ObjNum 358 snap 115 seq_num 358) dirty exists 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:998: left oid 23 (ObjNum 358 snap 115 seq_num 358) 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:998: done (1 left) 2024-09-20T19:25:25.415 INFO:tasks.rados.rados.0.smithi042.stdout:1001: snap_create 2024-09-20T19:25:25.429 INFO:tasks.rados.rados.0.smithi042.stdout:1000: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:25:25.430 INFO:tasks.rados.rados.0.smithi042.stdout:1000: finishing copy_from to smithi04268048-11 2024-09-20T19:25:25.430 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 421 (ObjNum 274 snap 82 seq_num 274) dirty exists 2024-09-20T19:25:25.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:25.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[32075]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:25 smithi042 bash[22706]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.514 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.444518+0000 osd.1 (osd.1) 151 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:23.446574+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.041146+0000 mgr.y (mgr.24491) 286 : cluster [DBG] pgmap v440: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s, 5 objects/s recovering 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.529900+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:25 smithi162 bash[24557]: cluster 2024-09-20T19:25:24.530475+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:25.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.644 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:25.747 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:25.776 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:25.879 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:25.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:26.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:26.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:26.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:26.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:26.258 INFO:tasks.rados.rados.0.smithi042.stdout:1000: done (1 left) 2024-09-20T19:25:26.258 INFO:tasks.rados.rados.0.smithi042.stdout:1001: done (0 left) 2024-09-20T19:25:26.258 INFO:tasks.rados.rados.0.smithi042.stdout:1002: write (excl) oid 32 current snap is 116 2024-09-20T19:25:26.258 INFO:tasks.rados.rados.0.smithi042.stdout:1002: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-20T19:25:26.263 INFO:tasks.rados.rados.0.smithi042.stdout:1002: writing smithi04268048-32 from 450520 to 1008751 tid 1 2024-09-20T19:25:26.269 INFO:tasks.rados.rados.0.smithi042.stdout:1002: writing smithi04268048-32 from 1483047 to 1998172 tid 2 2024-09-20T19:25:26.269 INFO:tasks.rados.rados.0.smithi042.stdout:1003: read oid 34 snap -1 2024-09-20T19:25:26.269 INFO:tasks.rados.rados.0.smithi042.stdout:1003: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:26.269 INFO:tasks.rados.rados.0.smithi042.stdout:1004: write (excl) oid 33 current snap is 116 2024-09-20T19:25:26.269 INFO:tasks.rados.rados.0.smithi042.stdout:1004: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-20T19:25:26.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:26.275 INFO:tasks.rados.rados.0.smithi042.stdout:1004: writing smithi04268048-33 from 467327 to 1100807 tid 1 2024-09-20T19:25:26.281 INFO:tasks.rados.rados.0.smithi042.stdout:1004: writing smithi04268048-33 from 1825176 to 2067327 tid 2 2024-09-20T19:25:26.281 INFO:tasks.rados.rados.0.smithi042.stdout:1005: read oid 39 snap 113 2024-09-20T19:25:26.281 INFO:tasks.rados.rados.0.smithi042.stdout:1005: expect deleted 2024-09-20T19:25:26.281 INFO:tasks.rados.rados.0.smithi042.stdout:1006: copy_from oid 10 from oid 36 current snap is 116 2024-09-20T19:25:26.282 INFO:tasks.rados.rados.0.smithi042.stdout:1007: write (excl) oid 49 current snap is 116 2024-09-20T19:25:26.282 INFO:tasks.rados.rados.0.smithi042.stdout:1003: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:26 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.297 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:26.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:26.343 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2177718 2024-09-20T19:25:26.350 INFO:tasks.rados.rados.0.smithi042.stdout:1002: finishing write tid 1 to smithi04268048-32 2024-09-20T19:25:26.350 INFO:tasks.rados.rados.0.smithi042.stdout:1002: finishing write tid 2 to smithi04268048-32 2024-09-20T19:25:26.350 INFO:tasks.rados.rados.0.smithi042.stdout:1007: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-20T19:25:26.356 INFO:tasks.rados.rados.0.smithi042.stdout:1007: writing smithi04268048-49 from 484134 to 1192863 tid 1 2024-09-20T19:25:26.362 INFO:tasks.rados.rados.0.smithi042.stdout:1007: writing smithi04268048-49 from 1883658 to 2084134 tid 2 2024-09-20T19:25:26.362 INFO:tasks.rados.rados.0.smithi042.stdout:1003: done (5 left) 2024-09-20T19:25:26.362 INFO:tasks.rados.rados.0.smithi042.stdout:1008: read oid 19 snap 92 2024-09-20T19:25:26.362 INFO:tasks.rados.rados.0.smithi042.stdout:1008: expect deleted 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1009: copy_from oid 37 from oid 31 current snap is 116 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: finishing write tid 3 to smithi04268048-32 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: finishing write tid 4 to smithi04268048-32 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: oid 32 updating version 0 to 451 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: oid 32 version 451 is already newer than 449 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: oid 32 version 451 is already newer than 450 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 451 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1002: left oid 32 (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1004: finishing write tid 1 to smithi04268048-33 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1004: finishing write tid 2 to smithi04268048-33 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1004: finishing write tid 3 to smithi04268048-33 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1004: finishing write tid 4 to smithi04268048-33 2024-09-20T19:25:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:1004: oid 33 updating version 0 to 223 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1004: oid 33 updating version 223 to 224 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1004: oid 33 updating version 224 to 225 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 225 (ObjNum 361 snap 116 seq_num 361) dirty exists 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1004: left oid 33 (ObjNum 361 snap 116 seq_num 361) 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1006: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1006: finishing copy_from to smithi04268048-10 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 316 (ObjNum 354 snap 115 seq_num 354) dirty exists 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1007: finishing write tid 1 to smithi04268048-49 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1002: done (6 left) 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1004: done (5 left) 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1005: done (4 left) 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1006: done (3 left) 2024-09-20T19:25:26.364 INFO:tasks.rados.rados.0.smithi042.stdout:1010: snap_remove snap 91 2024-09-20T19:25:26.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:26.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:26.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:26.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:26.647 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:25:26.657 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:26.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.248363+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.697198+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:26 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.699692+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:25:26.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:26.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:26.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:26.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:26.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:27.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:27.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:27.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:27.262 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:25:27.273 INFO:tasks.rados.rados.0.smithi042.stdout:1010: done (3 left) 2024-09-20T19:25:27.273 INFO:tasks.rados.rados.0.smithi042.stdout:1011: write (excl) oid 10 current snap is 116 2024-09-20T19:25:27.273 INFO:tasks.rados.rados.0.smithi042.stdout:1011: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-20T19:25:27.280 INFO:tasks.rados.rados.0.smithi042.stdout:1011: writing smithi04268048-10 from 500941 to 1284919 tid 1 2024-09-20T19:25:27.286 INFO:tasks.rados.rados.0.smithi042.stdout:1011: writing smithi04268048-10 from 1942140 to 2100941 tid 2 2024-09-20T19:25:27.286 INFO:tasks.rados.rados.0.smithi042.stdout:1012: write oid 43 current snap is 116 2024-09-20T19:25:27.286 INFO:tasks.rados.rados.0.smithi042.stdout:1012: seq_num 364 ranges {517748=459227,1600622=406110} 2024-09-20T19:25:27.293 INFO:tasks.rados.rados.0.smithi042.stdout:1012: writing smithi04268048-43 from 517748 to 976975 tid 1 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1012: writing smithi04268048-43 from 1600622 to 2006732 tid 2 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1013: snap_create 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: finishing write tid 2 to smithi04268048-49 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: finishing write tid 3 to smithi04268048-49 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: finishing write tid 4 to smithi04268048-49 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: oid 49 updating version 0 to 454 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: oid 49 version 454 is already newer than 453 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: oid 49 updating version 454 to 455 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 455 (ObjNum 362 snap 116 seq_num 362) dirty exists 2024-09-20T19:25:27.301 INFO:tasks.rados.rados.0.smithi042.stdout:1007: left oid 49 (ObjNum 362 snap 116 seq_num 362) 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1009: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1009: finishing copy_from to smithi04268048-37 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 311 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: finishing write tid 1 to smithi04268048-10 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: finishing write tid 2 to smithi04268048-10 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: finishing write tid 3 to smithi04268048-10 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1012: finishing write tid 1 to smithi04268048-43 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: finishing write tid 4 to smithi04268048-10 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: oid 10 updating version 0 to 408 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: oid 10 updating version 408 to 409 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: oid 10 updating version 409 to 410 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 410 (ObjNum 363 snap 116 seq_num 363) dirty exists 2024-09-20T19:25:27.302 INFO:tasks.rados.rados.0.smithi042.stdout:1011: left oid 10 (ObjNum 363 snap 116 seq_num 363) 2024-09-20T19:25:27.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.303 INFO:tasks.rados.rados.0.smithi042.stdout:1012: finishing write tid 2 to smithi04268048-43 2024-09-20T19:25:27.304 INFO:tasks.rados.rados.0.smithi042.stdout:1012: finishing write tid 3 to smithi04268048-43 2024-09-20T19:25:27.304 INFO:tasks.rados.rados.0.smithi042.stdout:1012: finishing write tid 4 to smithi04268048-43 2024-09-20T19:25:27.304 INFO:tasks.rados.rados.0.smithi042.stdout:1012: oid 43 updating version 0 to 316 2024-09-20T19:25:27.304 INFO:tasks.rados.rados.0.smithi042.stdout:1012: oid 43 version 316 is already newer than 315 2024-09-20T19:25:27.305 INFO:tasks.rados.rados.0.smithi042.stdout:1012: oid 43 version 316 is already newer than 314 2024-09-20T19:25:27.305 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 316 (ObjNum 364 snap 116 seq_num 364) dirty exists 2024-09-20T19:25:27.305 INFO:tasks.rados.rados.0.smithi042.stdout:1012: left oid 43 (ObjNum 364 snap 116 seq_num 364) 2024-09-20T19:25:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:27.430 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:27.448 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:27.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[22706]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.495 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.496 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:27 smithi042 bash[32075]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.676875+0000 osd.2 (osd.2) 131 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:25.679630+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.041918+0000 mgr.y (mgr.24491) 287 : cluster [DBG] pgmap v443: 158 pgs: 1 active+undersized+remapped, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.257399+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.378869+0000 osd.6 (osd.6) 109 : cluster [DBG] 4.a scrub starts 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.386902+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2024-09-20T19:25:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: audit 2024-09-20T19:25:26.639606+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:27 smithi162 bash[24557]: audit 2024-09-20T19:25:27.131936+0000 mon.a (mon.0) 1354 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:27.573 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stderr:no change 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:moved 0 / 474 (0%) 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:avg 59.25 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:stddev 4.94343 -> 4.94343 (expected baseline 7.20026) 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:min osd.2 with 53 -> 53 pgs (0.894515 -> 0.894515 * mean) 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:25:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:oload 1000 2024-09-20T19:25:27.599 INFO:teuthology.orchestra.run.smithi042.stdout:max_change 3 2024-09-20T19:25:27.599 INFO:teuthology.orchestra.run.smithi042.stdout:max_change_osds 4 2024-09-20T19:25:27.599 INFO:teuthology.orchestra.run.smithi042.stdout:average_utilization 0.0016 2024-09-20T19:25:27.599 INFO:teuthology.orchestra.run.smithi042.stdout:overload_utilization 0.0157 2024-09-20T19:25:27.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:27.703 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:27.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:27.936 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:27.963 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:27.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:28.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1007: done (5 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1008: done (4 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1009: done (3 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1011: done (2 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1012: done (1 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1013: done (0 left) 2024-09-20T19:25:28.094 INFO:tasks.rados.rados.0.smithi042.stdout:1014: write (excl) oid 28 current snap is 117 2024-09-20T19:25:28.095 INFO:tasks.rados.rados.0.smithi042.stdout:1014: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-20T19:25:28.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:28.106 INFO:tasks.rados.rados.0.smithi042.stdout:1014: writing smithi04268048-28 from 534555 to 1185384 tid 1 2024-09-20T19:25:28.116 INFO:tasks.rados.rados.0.smithi042.stdout:1014: writing smithi04268048-28 from 1659104 to 2134555 tid 2 2024-09-20T19:25:28.116 INFO:tasks.rados.rados.0.smithi042.stdout:1015: rollback oid 20 current snap is 117 2024-09-20T19:25:28.116 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 20 to 112 2024-09-20T19:25:28.116 INFO:tasks.rados.rados.0.smithi042.stdout:1016: write (excl) oid 1 current snap is 117 2024-09-20T19:25:28.117 INFO:tasks.rados.rados.0.smithi042.stdout:1016: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-20T19:25:28.130 INFO:tasks.rados.rados.0.smithi042.stdout:1016: writing smithi04268048-1 from 551362 to 1277440 tid 1 2024-09-20T19:25:28.138 INFO:tasks.rados.rados.0.smithi042.stdout:1016: writing smithi04268048-1 from 1717586 to 2127365 tid 2 2024-09-20T19:25:28.138 INFO:tasks.rados.rados.0.smithi042.stdout:1017: rollback oid 30 current snap is 117 2024-09-20T19:25:28.138 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 112 2024-09-20T19:25:28.138 INFO:tasks.rados.rados.0.smithi042.stdout:1015: finishing rollback tid 1 to smithi04268048-20 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1015: done (3 left) 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1018: delete oid 21 current snap is 117 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: finishing write tid 1 to smithi04268048-28 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: finishing write tid 2 to smithi04268048-28 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: finishing write tid 3 to smithi04268048-28 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: finishing write tid 4 to smithi04268048-28 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: oid 28 updating version 0 to 337 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: oid 28 version 337 is already newer than 335 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: oid 28 version 337 is already newer than 336 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 337 (ObjNum 365 snap 117 seq_num 365) dirty exists 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1014: left oid 28 (ObjNum 365 snap 117 seq_num 365) 2024-09-20T19:25:28.139 INFO:tasks.rados.rados.0.smithi042.stdout:1016: finishing write tid 1 to smithi04268048-1 2024-09-20T19:25:28.141 INFO:tasks.rados.rados.0.smithi042.stdout:1016: finishing write tid 2 to smithi04268048-1 2024-09-20T19:25:28.141 INFO:tasks.rados.rados.0.smithi042.stdout:1016: finishing write tid 3 to smithi04268048-1 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1016: finishing write tid 4 to smithi04268048-1 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1016: oid 1 updating version 0 to 337 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1016: oid 1 version 337 is already newer than 336 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1016: oid 1 version 337 is already newer than 335 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 337 (ObjNum 366 snap 117 seq_num 366) dirty exists 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1016: left oid 1 (ObjNum 366 snap 117 seq_num 366) 2024-09-20T19:25:28.142 INFO:tasks.rados.rados.0.smithi042.stdout:1017: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1017: finishing rollback tid 2 to smithi04268048-30 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 523 (ObjNum 322 snap 104 seq_num 322) dirty exists 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1014: done (3 left) 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1016: done (2 left) 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1017: done (1 left) 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1018: done (0 left) 2024-09-20T19:25:28.143 INFO:tasks.rados.rados.0.smithi042.stdout:1019: snap_create 2024-09-20T19:25:28.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:28.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:28.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:28.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:28 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:28.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:28.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.789062+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.d scrub starts 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:26.791235+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.d scrub ok 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.253746+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.253782+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: audit 2024-09-20T19:25:27.263150+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? v1:172.21.15.42:0/1630949818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.267570+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-20T19:25:28.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.294959+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.296690+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.480386+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.482609+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: audit 2024-09-20T19:25:27.598036+0000 mgr.y (mgr.24491) 288 : audit [DBG] from='client.15234 v1:172.21.15.42:0/2709420463' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.629777+0000 osd.3 (osd.3) 151 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:25:28.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.631197+0000 osd.3 (osd.3) 152 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.746227+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:25:28.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:27.747330+0000 osd.5 (osd.5) 136 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:25:28.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:28 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.093588+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-20T19:25:28.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:28.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:28.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:28.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.744 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:28.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:28.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:28.978 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:29.008 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:1019: done (0 left) 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:1020: rollback oid 48 current snap is 118 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 88 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:1021: rollback oid 2 current snap is 118 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 106 2024-09-20T19:25:29.107 INFO:tasks.rados.rados.0.smithi042.stdout:1022: write (excl) oid 45 current snap is 118 2024-09-20T19:25:29.108 INFO:tasks.rados.rados.0.smithi042.stdout:1022: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-20T19:25:29.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:29.119 INFO:tasks.rados.rados.0.smithi042.stdout:1022: writing smithi04268048-45 from 584976 to 1061552 tid 1 2024-09-20T19:25:29.132 INFO:tasks.rados.rados.0.smithi042.stdout:1022: writing smithi04268048-45 from 1834550 to 2184976 tid 2 2024-09-20T19:25:29.132 INFO:tasks.rados.rados.0.smithi042.stdout:1023: write oid 16 current snap is 118 2024-09-20T19:25:29.132 INFO:tasks.rados.rados.0.smithi042.stdout:1023: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-20T19:25:29.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:29.143 INFO:tasks.rados.rados.0.smithi042.stdout:1023: writing smithi04268048-16 from 601783 to 1153608 tid 1 2024-09-20T19:25:29.154 INFO:tasks.rados.rados.0.smithi042.stdout:1023: writing smithi04268048-16 from 1776679 to 2201783 tid 2 2024-09-20T19:25:29.154 INFO:tasks.rados.rados.0.smithi042.stdout:1024: read oid 3 snap -1 2024-09-20T19:25:29.154 INFO:tasks.rados.rados.0.smithi042.stdout:1024: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1025: write oid 43 current snap is 118 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1021: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1021: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1020: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1021: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 231 (ObjNum 307 snap 99 seq_num 307) dirty exists 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1020: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1020: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 529 (ObjNum 256 snap 74 seq_num 256) dirty exists 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1022: finishing write tid 1 to smithi04268048-45 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1022: finishing write tid 2 to smithi04268048-45 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1022: finishing write tid 3 to smithi04268048-45 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1022: finishing write tid 4 to smithi04268048-45 2024-09-20T19:25:29.155 INFO:tasks.rados.rados.0.smithi042.stdout:1022: oid 45 updating version 0 to 465 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:1022: oid 45 updating version 465 to 467 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:1022: oid 45 version 467 is already newer than 466 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 467 (ObjNum 368 snap 118 seq_num 368) dirty exists 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:1022: left oid 45 (ObjNum 368 snap 118 seq_num 368) 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:1023: finishing write tid 1 to smithi04268048-16 2024-09-20T19:25:29.156 INFO:tasks.rados.rados.0.smithi042.stdout:1025: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-20T19:25:29.168 INFO:tasks.rados.rados.0.smithi042.stdout:1025: writing smithi04268048-43 from 618590 to 1245664 tid 1 2024-09-20T19:25:29.179 INFO:tasks.rados.rados.0.smithi042.stdout:1025: writing smithi04268048-43 from 1835161 to 2218590 tid 2 2024-09-20T19:25:29.179 INFO:tasks.rados.rados.0.smithi042.stdout:1020: done (5 left) 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1021: done (4 left) 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1022: done (3 left) 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1026: read oid 15 snap -1 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1026: expect (ObjNum 343 snap 110 seq_num 343) 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1027: read oid 4 snap -1 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1027: expect (ObjNum 266 snap 80 seq_num 266) 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1028: write (excl) oid 10 current snap is 118 2024-09-20T19:25:29.180 INFO:tasks.rados.rados.0.smithi042.stdout:1028: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-20T19:25:29.192 INFO:tasks.rados.rados.0.smithi042.stdout:1028: writing smithi04268048-10 from 635397 to 1337720 tid 1 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1028: writing smithi04268048-10 from 1893643 to 2235397 tid 2 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1029: delete oid 2 current snap is 118 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: finishing write tid 2 to smithi04268048-16 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: finishing write tid 3 to smithi04268048-16 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: finishing write tid 4 to smithi04268048-16 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: oid 16 updating version 0 to 323 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: oid 16 version 323 is already newer than 322 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:1023: oid 16 updating version 323 to 324 2024-09-20T19:25:29.200 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 324 (ObjNum 369 snap 118 seq_num 369) dirty exists 2024-09-20T19:25:29.201 INFO:tasks.rados.rados.0.smithi042.stdout:1023: left oid 16 (ObjNum 369 snap 118 seq_num 369) 2024-09-20T19:25:29.201 INFO:tasks.rados.rados.0.smithi042.stdout:1024: expect (ObjNum 323 snap 104 seq_num 323) 2024-09-20T19:25:29.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:29.268 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:29.320 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:29.371 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:29.372 INFO:tasks.rados.rados.0.smithi042.stdout:1025: finishing write tid 1 to smithi04268048-43 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1023: done (6 left) 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1024: done (5 left) 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1029: done (4 left) 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1030: write oid 35 current snap is 118 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: finishing write tid 2 to smithi04268048-43 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: finishing write tid 3 to smithi04268048-43 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: finishing write tid 4 to smithi04268048-43 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: oid 43 updating version 0 to 318 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: oid 43 updating version 318 to 319 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: oid 43 version 319 is already newer than 317 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 319 (ObjNum 370 snap 118 seq_num 370) dirty exists 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1025: left oid 43 (ObjNum 370 snap 118 seq_num 370) 2024-09-20T19:25:29.373 INFO:tasks.rados.rados.0.smithi042.stdout:1030: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-20T19:25:29.379 INFO:tasks.rados.rados.0.smithi042.stdout:1030: writing smithi04268048-35 from 669011 to 1238185 tid 1 2024-09-20T19:25:29.386 INFO:tasks.rados.rados.0.smithi042.stdout:1030: writing smithi04268048-35 from 2010607 to 2269011 tid 2 2024-09-20T19:25:29.386 INFO:tasks.rados.rados.0.smithi042.stdout:1025: done (4 left) 2024-09-20T19:25:29.386 INFO:tasks.rados.rados.0.smithi042.stdout:1031: copy_from oid 42 from oid 5 current snap is 118 2024-09-20T19:25:29.386 INFO:tasks.rados.rados.0.smithi042.stdout:1027: expect (ObjNum 266 snap 80 seq_num 266) 2024-09-20T19:25:29.400 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:29.435 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:29.483 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:29.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:29.514 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:29 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:29.531 INFO:tasks.rados.rados.0.smithi042.stdout:1028: finishing write tid 1 to smithi04268048-10 2024-09-20T19:25:29.531 INFO:tasks.rados.rados.0.smithi042.stdout:1026: expect (ObjNum 343 snap 110 seq_num 343) 2024-09-20T19:25:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:29.540 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1124129 2024-09-20T19:25:29.589 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:29.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:29.657 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1366849 2024-09-20T19:25:29.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.042460+0000 mgr.y (mgr.24491) 289 : cluster [DBG] pgmap v446: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.126652+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.f scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.132128+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.f scrub ok 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.317786+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.320464+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.468878+0000 osd.4 (osd.4) 145 : cluster [DBG] 2.f scrub starts 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.472183+0000 osd.4 (osd.4) 146 : cluster [DBG] 2.f scrub ok 2024-09-20T19:25:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:29 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.106276+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-20T19:25:29.707 INFO:tasks.rados.rados.0.smithi042.stdout:1028: finishing write tid 2 to smithi04268048-10 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: finishing write tid 3 to smithi04268048-10 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: finishing write tid 4 to smithi04268048-10 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: oid 10 updating version 0 to 411 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: oid 10 updating version 411 to 413 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: oid 10 version 413 is already newer than 412 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 413 (ObjNum 371 snap 118 seq_num 371) dirty exists 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: left oid 10 (ObjNum 371 snap 118 seq_num 371) 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1026: done (4 left) 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1027: done (3 left) 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1028: done (2 left) 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1032: delete oid 20 current snap is 118 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1030: finishing write tid 1 to smithi04268048-35 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1030: finishing write tid 2 to smithi04268048-35 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1030: finishing write tid 3 to smithi04268048-35 2024-09-20T19:25:29.708 INFO:tasks.rados.rados.0.smithi042.stdout:1030: finishing write tid 4 to smithi04268048-35 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1030: oid 35 updating version 0 to 324 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1030: oid 35 version 324 is already newer than 323 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1030: oid 35 version 324 is already newer than 322 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 324 (ObjNum 373 snap 118 seq_num 373) dirty exists 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1030: left oid 35 (ObjNum 373 snap 118 seq_num 373) 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1031: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:1031: finishing copy_from to smithi04268048-42 2024-09-20T19:25:29.709 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 393 (ObjNum 322 snap 104 seq_num 322) dirty exists 2024-09-20T19:25:29.710 INFO:tasks.rados.rados.0.smithi042.stdout:1030: done (2 left) 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1031: done (1 left) 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1032: done (0 left) 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1033: read oid 23 snap 113 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1033: expect deleted 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1034: write (excl) oid 20 current snap is 118 2024-09-20T19:25:29.711 INFO:tasks.rados.rados.0.smithi042.stdout:1034: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-20T19:25:29.720 INFO:tasks.rados.rados.0.smithi042.stdout:1034: writing smithi04268048-20 from 702625 to 1422297 tid 1 2024-09-20T19:25:29.725 INFO:tasks.rados.rados.0.smithi042.stdout:1034: writing smithi04268048-20 from 2127571 to 2302625 tid 2 2024-09-20T19:25:29.725 INFO:tasks.rados.rados.0.smithi042.stdout:1035: snap_remove snap 115 2024-09-20T19:25:29.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:29.788 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:29.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:29.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:29.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:30.021 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:30.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:30.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:30.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:30.298 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:25:30.302 INFO:tasks.rados.rados.0.smithi042.stdout:1035: done (2 left) 2024-09-20T19:25:30.302 INFO:tasks.rados.rados.0.smithi042.stdout:1036: write oid 8 current snap is 118 2024-09-20T19:25:30.302 INFO:tasks.rados.rados.0.smithi042.stdout:1036: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-20T19:25:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:30.311 INFO:tasks.rados.rados.0.smithi042.stdout:1036: writing smithi04268048-8 from 719432 to 1514353 tid 1 2024-09-20T19:25:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1036: writing smithi04268048-8 from 2186053 to 2319432 tid 2 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: finishing write tid 1 to smithi04268048-20 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: finishing write tid 2 to smithi04268048-20 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: finishing write tid 3 to smithi04268048-20 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: finishing write tid 4 to smithi04268048-20 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: oid 20 updating version 0 to 85 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: oid 20 version 85 is already newer than 83 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:1034: oid 20 version 85 is already newer than 84 2024-09-20T19:25:30.318 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 85 (ObjNum 375 snap 118 seq_num 375) dirty exists 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1034: left oid 20 (ObjNum 375 snap 118 seq_num 375) 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1033: done (2 left) 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1034: done (1 left) 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1037: copy_from oid 49 from oid 37 current snap is 118 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1038: read oid 12 snap -1 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1038: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1039: copy_from oid 18 from oid 9 current snap is 118 2024-09-20T19:25:30.319 INFO:tasks.rados.rados.0.smithi042.stdout:1040: write oid 26 current snap is 118 2024-09-20T19:25:30.320 INFO:tasks.rados.rados.0.smithi042.stdout:1040: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-20T19:25:30.326 INFO:tasks.rados.rados.0.smithi042.stdout:1040: writing smithi04268048-26 from 736239 to 1206409 tid 1 2024-09-20T19:25:30.335 INFO:tasks.rados.rados.0.smithi042.stdout:1040: writing smithi04268048-26 from 1728182 to 2299964 tid 2 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1041: snap_create 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: finishing write tid 1 to smithi04268048-8 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: finishing write tid 2 to smithi04268048-8 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1039: finishing copy_from to smithi04268048-18 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1039: got expected ENOENT (src dne) 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1039: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: finishing write tid 3 to smithi04268048-8 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: finishing write tid 4 to smithi04268048-8 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: oid 8 updating version 0 to 342 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: oid 8 version 342 is already newer than 340 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: oid 8 version 342 is already newer than 341 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 342 (ObjNum 376 snap 118 seq_num 376) dirty exists 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1036: left oid 8 (ObjNum 376 snap 118 seq_num 376) 2024-09-20T19:25:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1038: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:25:30.341 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1852677 2024-09-20T19:25:30.413 INFO:tasks.rados.rados.0.smithi042.stdout:1040: finishing write tid 1 to smithi04268048-26 2024-09-20T19:25:30.413 INFO:tasks.rados.rados.0.smithi042.stdout:1040: finishing write tid 2 to smithi04268048-26 2024-09-20T19:25:30.413 INFO:tasks.rados.rados.0.smithi042.stdout:1040: finishing write tid 3 to smithi04268048-26 2024-09-20T19:25:30.413 INFO:tasks.rados.rados.0.smithi042.stdout:1040: finishing write tid 4 to smithi04268048-26 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1040: oid 26 updating version 0 to 264 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1040: oid 26 version 264 is already newer than 262 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1040: oid 26 version 264 is already newer than 263 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 264 (ObjNum 377 snap 118 seq_num 377) dirty exists 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1040: left oid 26 (ObjNum 377 snap 118 seq_num 377) 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1037: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:1037: finishing copy_from to smithi04268048-49 2024-09-20T19:25:30.414 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 311 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:30.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:30.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:30.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[22706]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:30 smithi042 bash[32075]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.759290+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:28.761051+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: audit 2024-09-20T19:25:29.212255+0000 mgr.y (mgr.24491) 290 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.420373+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:30 smithi162 bash[24557]: cluster 2024-09-20T19:25:29.422418+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:25:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:30.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.688 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:30.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.819 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:30.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:30.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:30.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:31.049 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:31.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:31.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:31.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:31.294 INFO:tasks.rados.rados.0.smithi042.stdout:1036: done (5 left) 2024-09-20T19:25:31.294 INFO:tasks.rados.rados.0.smithi042.stdout:1037: done (4 left) 2024-09-20T19:25:31.294 INFO:tasks.rados.rados.0.smithi042.stdout:1038: done (3 left) 2024-09-20T19:25:31.294 INFO:tasks.rados.rados.0.smithi042.stdout:1039: done (2 left) 2024-09-20T19:25:31.294 INFO:tasks.rados.rados.0.smithi042.stdout:1040: done (1 left) 2024-09-20T19:25:31.295 INFO:tasks.rados.rados.0.smithi042.stdout:1041: done (0 left) 2024-09-20T19:25:31.295 INFO:tasks.rados.rados.0.smithi042.stdout:1042: read oid 7 snap -1 2024-09-20T19:25:31.295 INFO:tasks.rados.rados.0.smithi042.stdout:1042: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:31.295 INFO:tasks.rados.rados.0.smithi042.stdout:1043: snap_create 2024-09-20T19:25:31.304 INFO:tasks.rados.rados.0.smithi042.stdout:1042: expect (ObjNum 259 snap 76 seq_num 259) 2024-09-20T19:25:31.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:31.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:31.347 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:31.368 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:25:31.368 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:25:31.368 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:25:31.368 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[22706]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.380 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.381 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:31 smithi042 bash[32075]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.439 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:31.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.042977+0000 mgr.y (mgr.24491) 291 : cluster [DBG] pgmap v449: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 382 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 32 op/s 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.078304+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.079826+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:31 smithi162 bash[24557]: cluster 2024-09-20T19:25:30.296527+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-20T19:25:31.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:31.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:31.690 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:31.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:31.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:31.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:31.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:32.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:32.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:32.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:25:32.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:32.296 INFO:tasks.rados.rados.0.smithi042.stdout:1042: done (1 left) 2024-09-20T19:25:32.296 INFO:tasks.rados.rados.0.smithi042.stdout:1043: done (0 left) 2024-09-20T19:25:32.297 INFO:tasks.rados.rados.0.smithi042.stdout:1044: delete oid 2 current snap is 120 2024-09-20T19:25:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1044: done (0 left) 2024-09-20T19:25:32.306 INFO:tasks.rados.rados.0.smithi042.stdout:1045: snap_create 2024-09-20T19:25:32.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:32.358 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:32 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:32.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.038879+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:25:32.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.041083+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:25:32.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.293419+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-20T19:25:32.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.662844+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:25:32.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:32 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.665235+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:25:32.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:32.592 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:32.617 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:32.720 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:32.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:32.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:32.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:33.007 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:33.101 INFO:tasks.rados.rados.0.smithi042.stdout:1045: done (0 left) 2024-09-20T19:25:33.234 INFO:tasks.rados.rados.0.smithi042.stdout:1046: write (excl) oid 19 current snap is 121 2024-09-20T19:25:33.234 INFO:tasks.rados.rados.0.smithi042.stdout:1046: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-20T19:25:33.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: writing smithi04268048-19 from 769853 to 1390521 tid 1 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: writing smithi04268048-19 from 1845146 to 2369853 tid 2 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1047: snap_create 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: finishing write tid 1 to smithi04268048-19 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: finishing write tid 2 to smithi04268048-19 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: finishing write tid 3 to smithi04268048-19 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: finishing write tid 4 to smithi04268048-19 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: oid 19 updating version 0 to 339 2024-09-20T19:25:33.236 INFO:tasks.rados.rados.0.smithi042.stdout:1046: oid 19 updating version 339 to 340 2024-09-20T19:25:33.237 INFO:tasks.rados.rados.0.smithi042.stdout:1046: oid 19 version 340 is already newer than 338 2024-09-20T19:25:33.237 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 340 (ObjNum 379 snap 121 seq_num 379) dirty exists 2024-09-20T19:25:33.237 INFO:tasks.rados.rados.0.smithi042.stdout:1046: left oid 19 (ObjNum 379 snap 121 seq_num 379) 2024-09-20T19:25:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:33.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:33.391 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:33.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.509 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.510 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:33 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:33.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:33.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.620502+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.e scrub starts 2024-09-20T19:25:33.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.622600+0000 osd.2 (osd.2) 134 : cluster [DBG] 2.e scrub ok 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.990682+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:31.993506+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.044608+0000 mgr.y (mgr.24491) 292 : cluster [DBG] pgmap v452: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.293406+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.542943+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.544987+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:33 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.098448+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-20T19:25:33.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:33.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:33.779 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:25:33.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:25:33.779 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-20T19:25:33.779 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-20T19:25:33.780 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-20T19:25:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:33.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:33.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:34.007 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:25:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:25:34.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.026 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:25:34.026 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:25:34.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:34.042 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:25:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:25:34.063 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:25:34.063 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:25:34.080 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:25:34.080 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:25:34.099 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:25:34.099 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:25:34.112 INFO:tasks.rados.rados.0.smithi042.stdout:1046: done (1 left) 2024-09-20T19:25:34.112 INFO:tasks.rados.rados.0.smithi042.stdout:1047: done (0 left) 2024-09-20T19:25:34.112 INFO:tasks.rados.rados.0.smithi042.stdout:1048: rollback oid 24 current snap is 122 2024-09-20T19:25:34.112 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 120 2024-09-20T19:25:34.112 INFO:tasks.rados.rados.0.smithi042.stdout:1049: snap_create 2024-09-20T19:25:34.114 INFO:tasks.rados.rados.0.smithi042.stdout:1048: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:25:34.114 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 198 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T19:25:34.122 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:25:34.122 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:25:34.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.144 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:25:34.145 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:25:34.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:34.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:34.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:34.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:34.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:34 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:34.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:34.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:34.683 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:34.786 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.964592+0000 osd.0 (osd.0) 159 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:32.967068+0000 osd.0 (osd.0) 160 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.390786+0000 osd.4 (osd.4) 147 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:33.397431+0000 osd.4 (osd.4) 148 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: audit 2024-09-20T19:25:33.987613+0000 mon.a (mon.0) 1365 : audit [DBG] from='client.? v1:172.21.15.42:0/3074022331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:34 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.107776+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-20T19:25:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:34.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:34.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:35.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:35.070 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:35.112 INFO:tasks.rados.rados.0.smithi042.stdout:1048: done (1 left) 2024-09-20T19:25:35.112 INFO:tasks.rados.rados.0.smithi042.stdout:1049: done (0 left) 2024-09-20T19:25:35.112 INFO:tasks.rados.rados.0.smithi042.stdout:1050: write (excl) oid 47 current snap is 123 2024-09-20T19:25:35.112 INFO:tasks.rados.rados.0.smithi042.stdout:1050: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-20T19:25:35.120 INFO:tasks.rados.rados.0.smithi042.stdout:1050: writing smithi04268048-47 from 786660 to 1482577 tid 1 2024-09-20T19:25:35.125 INFO:tasks.rados.rados.0.smithi042.stdout:1050: writing smithi04268048-47 from 1903628 to 2339090 tid 2 2024-09-20T19:25:35.125 INFO:tasks.rados.rados.0.smithi042.stdout:1051: delete oid 38 current snap is 123 2024-09-20T19:25:35.137 INFO:tasks.rados.rados.0.smithi042.stdout:1050: finishing write tid 1 to smithi04268048-47 2024-09-20T19:25:35.140 INFO:tasks.rados.rados.0.smithi042.stdout:1051: done (1 left) 2024-09-20T19:25:35.140 INFO:tasks.rados.rados.0.smithi042.stdout:1052: rollback oid 43 current snap is 123 2024-09-20T19:25:35.140 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 107 2024-09-20T19:25:35.141 INFO:tasks.rados.rados.0.smithi042.stdout:1053: write (excl) oid 49 current snap is 123 2024-09-20T19:25:35.141 INFO:tasks.rados.rados.0.smithi042.stdout:1053: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-20T19:25:35.145 INFO:tasks.rados.rados.0.smithi042.stdout:1053: writing smithi04268048-49 from 420274 to 983042 tid 1 2024-09-20T19:25:35.155 INFO:tasks.rados.rados.0.smithi042.stdout:1053: writing smithi04268048-49 from 1620592 to 2059723 tid 2 2024-09-20T19:25:35.155 INFO:tasks.rados.rados.0.smithi042.stdout:1054: read oid 6 snap 85 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1054: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1055: rollback oid 32 current snap is 123 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 121 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: finishing write tid 2 to smithi04268048-47 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1056: write (excl) oid 46 current snap is 123 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1052: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: finishing write tid 3 to smithi04268048-47 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: finishing write tid 4 to smithi04268048-47 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: oid 47 updating version 0 to 529 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: oid 47 version 529 is already newer than 528 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: oid 47 version 529 is already newer than 527 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 529 (ObjNum 380 snap 123 seq_num 380) dirty exists 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1050: left oid 47 (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:25:35.156 INFO:tasks.rados.rados.0.smithi042.stdout:1052: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:25:35.157 INFO:tasks.rados.rados.0.smithi042.stdout:1052: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:25:35.157 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 322 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:25:35.157 INFO:tasks.rados.rados.0.smithi042.stdout:1053: finishing write tid 1 to smithi04268048-49 2024-09-20T19:25:35.157 INFO:tasks.rados.rados.0.smithi042.stdout:1056: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-20T19:25:35.164 INFO:tasks.rados.rados.0.smithi042.stdout:1056: writing smithi04268048-46 from 437081 to 1075098 tid 1 2024-09-20T19:25:35.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:1056: writing smithi04268048-46 from 1679074 to 2261863 tid 2 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:1050: done (5 left) 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:1052: done (4 left) 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:1057: rollback oid 8 current snap is 123 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 119 2024-09-20T19:25:35.177 INFO:tasks.rados.rados.0.smithi042.stdout:1053: finishing write tid 2 to smithi04268048-49 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:1053: finishing write tid 3 to smithi04268048-49 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:1053: finishing write tid 4 to smithi04268048-49 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:1053: oid 49 updating version 0 to 478 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:1053: oid 49 version 478 is already newer than 477 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:1053: oid 49 updating version 478 to 479 2024-09-20T19:25:35.178 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 479 (ObjNum 382 snap 123 seq_num 382) dirty exists 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:1053: left oid 49 (ObjNum 382 snap 123 seq_num 382) 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:1055: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:1055: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:1055: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 483 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:25:35.179 INFO:tasks.rados.rados.0.smithi042.stdout:1056: finishing write tid 1 to smithi04268048-46 2024-09-20T19:25:35.180 INFO:tasks.rados.rados.0.smithi042.stdout:1054: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:25:35.180 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 57441 2024-09-20T19:25:35.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:35.251 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:35.294 INFO:tasks.rados.rados.0.smithi042.stdout:1056: finishing write tid 2 to smithi04268048-46 2024-09-20T19:25:35.294 INFO:tasks.rados.rados.0.smithi042.stdout:1053: done (4 left) 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1054: done (3 left) 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1055: done (2 left) 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1058: write (excl) oid 35 current snap is 123 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: finishing write tid 3 to smithi04268048-46 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1057: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1057: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: finishing write tid 4 to smithi04268048-46 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: oid 46 updating version 0 to 485 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: oid 46 updating version 485 to 486 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: oid 46 updating version 486 to 487 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 487 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1056: left oid 46 (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:1057: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:25:35.295 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 352 (ObjNum 376 snap 118 seq_num 376) dirty exists 2024-09-20T19:25:35.296 INFO:tasks.rados.rados.0.smithi042.stdout:1058: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-20T19:25:35.301 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:35.309 INFO:tasks.rados.rados.0.smithi042.stdout:1058: writing smithi04268048-35 from 453888 to 1167154 tid 1 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1058: writing smithi04268048-35 from 1737556 to 2180356 tid 2 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1056: done (2 left) 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1057: done (1 left) 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1059: write oid 40 current snap is 123 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1058: finishing write tid 1 to smithi04268048-35 2024-09-20T19:25:35.321 INFO:tasks.rados.rados.0.smithi042.stdout:1059: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-20T19:25:35.326 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:35.329 INFO:tasks.rados.rados.0.smithi042.stdout:1059: writing smithi04268048-40 from 470695 to 1259210 tid 1 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1059: writing smithi04268048-40 from 1679685 to 2266143 tid 2 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1060: read oid 47 snap -1 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1060: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1061: rollback oid 19 current snap is 123 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 110 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1058: finishing write tid 2 to smithi04268048-35 2024-09-20T19:25:35.336 INFO:tasks.rados.rados.0.smithi042.stdout:1058: finishing write tid 3 to smithi04268048-35 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: finishing write tid 4 to smithi04268048-35 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: oid 35 updating version 0 to 325 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: oid 35 updating version 325 to 326 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: oid 35 updating version 326 to 327 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 327 (ObjNum 384 snap 123 seq_num 384) dirty exists 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: left oid 35 (ObjNum 384 snap 123 seq_num 384) 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1058: done (3 left) 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1062: read oid 5 snap -1 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1062: expect (ObjNum 322 snap 104 seq_num 322) 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1063: delete oid 24 current snap is 123 2024-09-20T19:25:35.337 INFO:tasks.rados.rados.0.smithi042.stdout:1059: finishing write tid 1 to smithi04268048-40 2024-09-20T19:25:35.339 INFO:tasks.rados.rados.0.smithi042.stdout:1062: expect (ObjNum 322 snap 104 seq_num 322) 2024-09-20T19:25:35.340 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:35.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1061: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1061: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1061: finishing rollback tid 2 to smithi04268048-19 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 343 (ObjNum 327 snap 107 seq_num 327) dirty exists 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: finishing write tid 2 to smithi04268048-40 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: finishing write tid 3 to smithi04268048-40 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: finishing write tid 4 to smithi04268048-40 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: oid 40 updating version 0 to 417 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: oid 40 version 417 is already newer than 416 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:1059: oid 40 version 417 is already newer than 415 2024-09-20T19:25:35.432 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 417 (ObjNum 385 snap 123 seq_num 385) dirty exists 2024-09-20T19:25:35.433 INFO:tasks.rados.rados.0.smithi042.stdout:1059: left oid 40 (ObjNum 385 snap 123 seq_num 385) 2024-09-20T19:25:35.433 INFO:tasks.rados.rados.0.smithi042.stdout:1060: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:25:35.438 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1544469 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:35 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:35.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:35.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1059: done (4 left) 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1060: done (3 left) 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1061: done (2 left) 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1062: done (1 left) 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1063: done (0 left) 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1064: write oid 21 current snap is 123 2024-09-20T19:25:35.561 INFO:tasks.rados.rados.0.smithi042.stdout:1064: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-20T19:25:35.567 INFO:tasks.rados.rados.0.smithi042.stdout:1064: writing smithi04268048-21 from 504309 to 1043322 tid 1 2024-09-20T19:25:35.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.045100+0000 mgr.y (mgr.24491) 293 : cluster [DBG] pgmap v455: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.4 MiB/s wr, 31 op/s 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.310321+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.c scrub starts 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.311605+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.c scrub ok 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.355024+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:34.359256+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:35 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.112697+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-20T19:25:35.575 INFO:tasks.rados.rados.0.smithi042.stdout:1064: writing smithi04268048-21 from 1796649 to 2386776 tid 2 2024-09-20T19:25:35.576 INFO:tasks.rados.rados.0.smithi042.stdout:1065: snap_remove snap 110 2024-09-20T19:25:35.581 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:35.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:35.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:35.839 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:35.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:35.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:35.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:36.070 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:36.098 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:36.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:36.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:1065: done (1 left) 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:1066: rollback oid 22 current snap is 123 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 112 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:1067: read oid 18 snap -1 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:1067: expect (ObjNum 321 snap 104 seq_num 321) 2024-09-20T19:25:36.325 INFO:tasks.rados.rados.0.smithi042.stdout:1068: snap_remove snap 113 2024-09-20T19:25:36.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.348 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.348 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:36.349 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:36 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:36.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.491 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.292970+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.294846+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.723148+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:36 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.724669+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-09-20T19:25:36.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:36.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:36.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:36.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:36.853 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:36.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:36.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:37.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:37.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:25:37.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:37.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:37.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:37.260 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:37.317 INFO:tasks.rados.rados.0.smithi042.stdout:1068: done (3 left) 2024-09-20T19:25:37.318 INFO:tasks.rados.rados.0.smithi042.stdout:1069: read oid 24 snap -1 2024-09-20T19:25:37.318 INFO:tasks.rados.rados.0.smithi042.stdout:1069: expect deleted 2024-09-20T19:25:37.318 INFO:tasks.rados.rados.0.smithi042.stdout:1070: delete oid 25 current snap is 123 2024-09-20T19:25:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:25:37.318 INFO:tasks.rados.rados.0.smithi042.stdout:1064: finishing write tid 1 to smithi04268048-21 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: finishing write tid 2 to smithi04268048-21 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: finishing write tid 3 to smithi04268048-21 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: finishing write tid 4 to smithi04268048-21 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: oid 21 updating version 0 to 534 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: oid 21 updating version 534 to 535 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: oid 21 updating version 535 to 536 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 536 (ObjNum 387 snap 123 seq_num 387) dirty exists 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1064: left oid 21 (ObjNum 387 snap 123 seq_num 387) 2024-09-20T19:25:37.319 INFO:tasks.rados.rados.0.smithi042.stdout:1067: expect (ObjNum 321 snap 104 seq_num 321) 2024-09-20T19:25:37.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:37.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:37.390 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:37.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:37.494 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2229219 2024-09-20T19:25:37.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:37.527 INFO:tasks.rados.rados.0.smithi042.stdout:1066: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:25:37.527 INFO:tasks.rados.rados.0.smithi042.stdout:1066: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:25:37.527 INFO:tasks.rados.rados.0.smithi042.stdout:1066: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 387 (ObjNum 340 snap 109 seq_num 340) dirty exists 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1064: done (4 left) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1066: done (3 left) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1067: done (2 left) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1070: done (1 left) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1071: read oid 40 snap -1 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1071: expect (ObjNum 385 snap 123 seq_num 385) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1072: rollback oid 42 current snap is 123 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 88 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1069: done (2 left) 2024-09-20T19:25:37.528 INFO:tasks.rados.rados.0.smithi042.stdout:1073: delete oid 5 current snap is 123 2024-09-20T19:25:37.530 INFO:tasks.rados.rados.0.smithi042.stdout:1073: done (2 left) 2024-09-20T19:25:37.531 INFO:tasks.rados.rados.0.smithi042.stdout:1074: delete oid 33 current snap is 123 2024-09-20T19:25:37.532 INFO:tasks.rados.rados.0.smithi042.stdout:1072: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:25:37.533 INFO:tasks.rados.rados.0.smithi042.stdout:1072: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:25:37.533 INFO:tasks.rados.rados.0.smithi042.stdout:1072: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:25:37.533 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 541 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:25:37.535 INFO:tasks.rados.rados.0.smithi042.stdout:1072: done (2 left) 2024-09-20T19:25:37.536 INFO:tasks.rados.rados.0.smithi042.stdout:1074: done (1 left) 2024-09-20T19:25:37.536 INFO:tasks.rados.rados.0.smithi042.stdout:1075: snap_remove snap 120 2024-09-20T19:25:37.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:37.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[32075]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:37 smithi042 bash[22706]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.638 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:37.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.634268+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-20T19:25:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:35.635703+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-20T19:25:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.000876+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.003171+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.045834+0000 mgr.y (mgr.24491) 294 : cluster [DBG] pgmap v458: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 396 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.321062+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.712745+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.714317+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.230418+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.42:0/2433809340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.230885+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:37.313017+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: audit 2024-09-20T19:25:37.314445+0000 mon.a (mon.0) 1371 : audit [INF] from='client.25103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:37 smithi162 bash[24557]: cluster 2024-09-20T19:25:37.318317+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-20T19:25:37.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:37.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:37.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:37.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:38.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:38.105 INFO:tasks.rados.rados.0.smithi042.stdout:1075: done (1 left) 2024-09-20T19:25:38.105 INFO:tasks.rados.rados.0.smithi042.stdout:1076: delete oid 17 current snap is 123 2024-09-20T19:25:38.105 INFO:tasks.rados.rados.0.smithi042.stdout:1071: expect (ObjNum 385 snap 123 seq_num 385) 2024-09-20T19:25:38.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:38.154 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:38.246 INFO:tasks.rados.rados.0.smithi042.stdout:1071: done (1 left) 2024-09-20T19:25:38.246 INFO:tasks.rados.rados.0.smithi042.stdout:1076: done (0 left) 2024-09-20T19:25:38.246 INFO:tasks.rados.rados.0.smithi042.stdout:1077: read oid 5 snap -1 2024-09-20T19:25:38.246 INFO:tasks.rados.rados.0.smithi042.stdout:1077: expect deleted 2024-09-20T19:25:38.246 INFO:tasks.rados.rados.0.smithi042.stdout:1078: snap_remove snap 85 2024-09-20T19:25:38.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:38.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:38.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:38.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.394 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:38 smithi162 bash[24557]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[22706]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.953520+0000 osd.0 (osd.0) 163 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:25:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:36.955120+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:25:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:38 smithi042 bash[32075]: cluster 2024-09-20T19:25:38.104459+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-20T19:25:38.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:38.537 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:38.617 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:38.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:38.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:38.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:38.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:38.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:38.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:39.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:39.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:39.131 INFO:tasks.rados.rados.0.smithi042.stdout:1078: done (1 left) 2024-09-20T19:25:39.131 INFO:tasks.rados.rados.0.smithi042.stdout:1079: write oid 4 current snap is 123 2024-09-20T19:25:39.131 INFO:tasks.rados.rados.0.smithi042.stdout:1079: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-20T19:25:39.137 INFO:tasks.rados.rados.0.smithi042.stdout:1079: writing smithi04268048-4 from 588344 to 1219955 tid 1 2024-09-20T19:25:39.144 INFO:tasks.rados.rados.0.smithi042.stdout:1079: writing smithi04268048-4 from 1689059 to 2430182 tid 2 2024-09-20T19:25:39.145 INFO:tasks.rados.rados.0.smithi042.stdout:1080: copy_from oid 23 from oid 6 current snap is 123 2024-09-20T19:25:39.145 INFO:tasks.rados.rados.0.smithi042.stdout:1077: done (2 left) 2024-09-20T19:25:39.145 INFO:tasks.rados.rados.0.smithi042.stdout:1081: snap_create 2024-09-20T19:25:39.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:39.151 INFO:tasks.rados.rados.0.smithi042.stdout:1079: finishing write tid 1 to smithi04268048-4 2024-09-20T19:25:39.153 INFO:tasks.rados.rados.0.smithi042.stdout:1079: finishing write tid 2 to smithi04268048-4 2024-09-20T19:25:39.153 INFO:tasks.rados.rados.0.smithi042.stdout:1079: finishing write tid 3 to smithi04268048-4 2024-09-20T19:25:39.153 INFO:tasks.rados.rados.0.smithi042.stdout:1079: finishing write tid 4 to smithi04268048-4 2024-09-20T19:25:39.154 INFO:tasks.rados.rados.0.smithi042.stdout:1079: oid 4 updating version 0 to 544 2024-09-20T19:25:39.154 INFO:tasks.rados.rados.0.smithi042.stdout:1079: oid 4 updating version 544 to 545 2024-09-20T19:25:39.154 INFO:tasks.rados.rados.0.smithi042.stdout:1079: oid 4 version 545 is already newer than 543 2024-09-20T19:25:39.154 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 545 (ObjNum 392 snap 123 seq_num 392) dirty exists 2024-09-20T19:25:39.154 INFO:tasks.rados.rados.0.smithi042.stdout:1079: left oid 4 (ObjNum 392 snap 123 seq_num 392) 2024-09-20T19:25:39.164 INFO:tasks.rados.rados.0.smithi042.stdout:1080: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:25:39.167 INFO:tasks.rados.rados.0.smithi042.stdout:1080: finishing copy_from to smithi04268048-23 2024-09-20T19:25:39.167 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 324 (ObjNum 156 snap 45 seq_num 156) dirty exists 2024-09-20T19:25:39.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:39.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:39.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:39.302 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:39.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[32075]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[32075]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[32075]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[32075]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[22706]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[22706]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[22706]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:39 smithi042 bash[22706]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.432 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:39.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:39.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:39 smithi162 bash[24557]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:39 smithi162 bash[24557]: cluster 2024-09-20T19:25:38.046603+0000 mgr.y (mgr.24491) 295 : cluster [DBG] pgmap v461: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:39.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:39 smithi162 bash[24557]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:39 smithi162 bash[24557]: cluster 2024-09-20T19:25:39.116599+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-20T19:25:39.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:39.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:39.685 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:39.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:39.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:39.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:39.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:40.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:1079: done (2 left) 2024-09-20T19:25:40.120 INFO:tasks.rados.rados.0.smithi042.stdout:1080: done (1 left) 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1081: done (0 left) 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1082: read oid 9 snap -1 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1082: expect deleted 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1083: read oid 26 snap -1 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1083: expect (ObjNum 377 snap 118 seq_num 377) 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1084: write (excl) oid 4 current snap is 124 2024-09-20T19:25:40.121 INFO:tasks.rados.rados.0.smithi042.stdout:1084: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-20T19:25:40.141 INFO:tasks.rados.rados.0.smithi042.stdout:1084: writing smithi04268048-4 from 605151 to 1312011 tid 1 2024-09-20T19:25:40.161 INFO:tasks.rados.rados.0.smithi042.stdout:1084: writing smithi04268048-4 from 2031188 to 2605151 tid 2 2024-09-20T19:25:40.162 INFO:tasks.rados.rados.0.smithi042.stdout:1085: rollback oid 41 current snap is 124 2024-09-20T19:25:40.162 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 119 2024-09-20T19:25:40.162 INFO:tasks.rados.rados.0.smithi042.stdout:1083: expect (ObjNum 377 snap 118 seq_num 377) 2024-09-20T19:25:40.162 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:40.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:25:40.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:40.229 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:40.291 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 737767 2024-09-20T19:25:40.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.329 INFO:tasks.rados.rados.0.smithi042.stdout:1084: finishing write tid 1 to smithi04268048-4 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: finishing write tid 2 to smithi04268048-4 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: finishing write tid 3 to smithi04268048-4 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: finishing write tid 4 to smithi04268048-4 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: oid 4 updating version 0 to 552 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: oid 4 version 552 is already newer than 551 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: oid 4 updating version 552 to 553 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 553 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1084: left oid 4 (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1085: finishing rollback tid 0 to smithi04268048-41 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1085: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:1085: finishing rollback tid 2 to smithi04268048-41 2024-09-20T19:25:40.330 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 255 (ObjNum 292 snap 93 seq_num 292) dirty exists 2024-09-20T19:25:40.331 INFO:tasks.rados.rados.0.smithi042.stdout:1082: done (3 left) 2024-09-20T19:25:40.331 INFO:tasks.rados.rados.0.smithi042.stdout:1083: done (2 left) 2024-09-20T19:25:40.331 INFO:tasks.rados.rados.0.smithi042.stdout:1084: done (1 left) 2024-09-20T19:25:40.331 INFO:tasks.rados.rados.0.smithi042.stdout:1085: done (0 left) 2024-09-20T19:25:40.331 INFO:tasks.rados.rados.0.smithi042.stdout:1086: snap_create 2024-09-20T19:25:40.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:40.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:40.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:40.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:40.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:40.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:40.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[22706]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[22706]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[22706]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[22706]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[32075]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[32075]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[32075]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:40 smithi042 bash[32075]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:40.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:40 smithi162 bash[24557]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:40 smithi162 bash[24557]: audit 2024-09-20T19:25:39.223341+0000 mgr.y (mgr.24491) 296 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:40 smithi162 bash[24557]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:40 smithi162 bash[24557]: cluster 2024-09-20T19:25:40.119307+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-20T19:25:40.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:40.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:40.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:40.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:41.043 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:25:41.043 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:25:41.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:41.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:41.123 INFO:tasks.rados.rados.0.smithi042.stdout:1086: done (0 left) 2024-09-20T19:25:41.123 INFO:tasks.rados.rados.0.smithi042.stdout:1087: copy_from oid 18 from oid 32 current snap is 125 2024-09-20T19:25:41.123 INFO:tasks.rados.rados.0.smithi042.stdout:1088: write oid 35 current snap is 125 2024-09-20T19:25:41.123 INFO:tasks.rados.rados.0.smithi042.stdout:1088: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-20T19:25:41.125 INFO:tasks.rados.rados.0.smithi042.stdout:1088: writing smithi04268048-35 from 621958 to 1404067 tid 1 2024-09-20T19:25:41.133 INFO:tasks.rados.rados.0.smithi042.stdout:1088: writing smithi04268048-35 from 2089670 to 2621958 tid 2 2024-09-20T19:25:41.133 INFO:tasks.rados.rados.0.smithi042.stdout:1089: write oid 44 current snap is 125 2024-09-20T19:25:41.133 INFO:tasks.rados.rados.0.smithi042.stdout:1087: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:25:41.133 INFO:tasks.rados.rados.0.smithi042.stdout:1089: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-20T19:25:41.139 INFO:tasks.rados.rados.0.smithi042.stdout:1089: writing smithi04268048-44 from 638765 to 1096123 tid 1 2024-09-20T19:25:41.146 INFO:tasks.rados.rados.0.smithi042.stdout:1089: writing smithi04268048-44 from 1748152 to 2352955 tid 2 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1090: snap_create 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1087: finishing copy_from to smithi04268048-18 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 483 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: finishing write tid 1 to smithi04268048-35 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: finishing write tid 2 to smithi04268048-35 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: finishing write tid 3 to smithi04268048-35 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: finishing write tid 4 to smithi04268048-35 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: oid 35 updating version 0 to 329 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: oid 35 version 329 is already newer than 328 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: oid 35 updating version 329 to 330 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 330 (ObjNum 394 snap 125 seq_num 394) dirty exists 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1088: left oid 35 (ObjNum 394 snap 125 seq_num 394) 2024-09-20T19:25:41.147 INFO:tasks.rados.rados.0.smithi042.stdout:1089: finishing write tid 1 to smithi04268048-44 2024-09-20T19:25:41.151 INFO:tasks.rados.rados.0.smithi042.stdout:1089: finishing write tid 2 to smithi04268048-44 2024-09-20T19:25:41.151 INFO:tasks.rados.rados.0.smithi042.stdout:1089: finishing write tid 3 to smithi04268048-44 2024-09-20T19:25:41.151 INFO:tasks.rados.rados.0.smithi042.stdout:1089: finishing write tid 4 to smithi04268048-44 2024-09-20T19:25:41.151 INFO:tasks.rados.rados.0.smithi042.stdout:1089: oid 44 updating version 0 to 518 2024-09-20T19:25:41.152 INFO:tasks.rados.rados.0.smithi042.stdout:1089: oid 44 updating version 518 to 519 2024-09-20T19:25:41.152 INFO:tasks.rados.rados.0.smithi042.stdout:1089: oid 44 version 519 is already newer than 517 2024-09-20T19:25:41.152 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 519 (ObjNum 395 snap 125 seq_num 395) dirty exists 2024-09-20T19:25:41.152 INFO:tasks.rados.rados.0.smithi042.stdout:1089: left oid 44 (ObjNum 395 snap 125 seq_num 395) 2024-09-20T19:25:41.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:41.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:41.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:41.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:41.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:41.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:41.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:41 smithi162 bash[24557]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[32075]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: cluster 2024-09-20T19:25:40.047393+0000 mgr.y (mgr.24491) 297 : cluster [DBG] pgmap v464: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 410 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: cluster 2024-09-20T19:25:41.117043+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: audit 2024-09-20T19:25:41.185669+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.42:0/3430041752' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:41 smithi042 bash[22706]: audit 2024-09-20T19:25:41.186019+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25109 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:25:41.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:41.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:41.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:41.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:25:41.877 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:41.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:41.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:42.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:42.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:42.118 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-20T19:25:42.119 INFO:tasks.rados.rados.0.smithi042.stdout:1087: done (3 left) 2024-09-20T19:25:42.119 INFO:tasks.rados.rados.0.smithi042.stdout:1088: done (2 left) 2024-09-20T19:25:42.119 INFO:tasks.rados.rados.0.smithi042.stdout:1089: done (1 left) 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1090: done (0 left) 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1091: rollback oid 27 current snap is 126 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 119 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1092: read oid 44 snap -1 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1092: expect (ObjNum 395 snap 125 seq_num 395) 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1093: copy_from oid 29 from oid 39 current snap is 126 2024-09-20T19:25:42.120 INFO:tasks.rados.rados.0.smithi042.stdout:1094: snap_remove snap 121 2024-09-20T19:25:42.132 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:42.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:42.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:42.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:42.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:42.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:42.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:42.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:42.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:42.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:42.900 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:42.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:43.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:43.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:43.111 INFO:tasks.rados.rados.0.smithi042.stdout:1094: done (3 left) 2024-09-20T19:25:43.111 INFO:tasks.rados.rados.0.smithi042.stdout:1095: delete oid 33 current snap is 126 2024-09-20T19:25:43.111 INFO:tasks.rados.rados.0.smithi042.stdout:1092: expect (ObjNum 395 snap 125 seq_num 395) 2024-09-20T19:25:43.111 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:43.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:43.163 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:43.202 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1093: finishing copy_from to smithi04268048-29 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1093: got expected ENOENT (src dne) 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1093: finishing copy_from racing read to smithi04268048-29 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1091: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 389 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1091: done (3 left) 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1092: done (2 left) 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1093: done (1 left) 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1095: done (0 left) 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1096: copy_from oid 38 from oid 46 current snap is 126 2024-09-20T19:25:43.245 INFO:tasks.rados.rados.0.smithi042.stdout:1097: copy_from oid 48 from oid 42 current snap is 126 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1098: read oid 1 snap 125 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1098: expect (ObjNum 366 snap 117 seq_num 366) 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1099: rollback oid 27 current snap is 126 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 116 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1100: read oid 19 snap -1 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1100: expect (ObjNum 327 snap 107 seq_num 327) 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1101: copy_from oid 33 from oid 10 current snap is 126 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1102: rollback oid 31 current snap is 126 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 119 2024-09-20T19:25:43.246 INFO:tasks.rados.rados.0.smithi042.stdout:1103: delete oid 37 current snap is 126 2024-09-20T19:25:43.248 INFO:tasks.rados.rados.0.smithi042.stdout:1103: done (7 left) 2024-09-20T19:25:43.248 INFO:tasks.rados.rados.0.smithi042.stdout:1104: write oid 2 current snap is 126 2024-09-20T19:25:43.248 INFO:tasks.rados.rados.0.smithi042.stdout:1104: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-20T19:25:43.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:43.278 INFO:tasks.rados.rados.0.smithi042.stdout:1104: writing smithi04268048-2 from 689186 to 1488644 tid 1 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:1104: writing smithi04268048-2 from 1923598 to 2675728 tid 2 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:1105: read oid 21 snap -1 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:1105: expect (ObjNum 387 snap 123 seq_num 387) 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:1106: delete oid 37 current snap is 126 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:1099: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:25:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 389 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:25:43.291 INFO:tasks.rados.rados.0.smithi042.stdout:1100: expect (ObjNum 327 snap 107 seq_num 327) 2024-09-20T19:25:43.291 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:43.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:43 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.341 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2327182 2024-09-20T19:25:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.376 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:43.393 INFO:tasks.rados.rados.0.smithi042.stdout:1099: done (9 left) 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:1100: done (8 left) 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:1106: done (7 left) 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:1107: copy_from oid 26 from oid 47 current snap is 126 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:1108: snap_create 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:1098: expect (ObjNum 366 snap 117 seq_num 366) 2024-09-20T19:25:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:43.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:43.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:42.049135+0000 mgr.y (mgr.24491) 298 : cluster [DBG] pgmap v467: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: audit 2024-09-20T19:25:42.115789+0000 mon.a (mon.0) 1378 : audit [INF] from='client.25109 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:42.119315+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: audit 2024-09-20T19:25:42.167964+0000 mon.a (mon.0) 1380 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:43 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.105535+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-20T19:25:43.432 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:1102: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:1102: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:1102: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 368 (ObjNum 318 snap 102 seq_num 318) dirty exists 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:1096: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:25:43.498 INFO:tasks.rados.rados.0.smithi042.stdout:1101: finishing copy_from racing read to smithi04268048-33 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1096: finishing copy_from to smithi04268048-38 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 487 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1101: finishing copy_from to smithi04268048-33 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 413 (ObjNum 371 snap 118 seq_num 371) dirty exists 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1097: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1104: finishing write tid 1 to smithi04268048-2 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1097: finishing copy_from to smithi04268048-48 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 541 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:25:43.499 INFO:tasks.rados.rados.0.smithi042.stdout:1105: expect (ObjNum 387 snap 123 seq_num 387) 2024-09-20T19:25:43.500 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1395189 2024-09-20T19:25:43.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:43.544 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 201594 2024-09-20T19:25:43.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:43.570 INFO:tasks.rados.rados.0.smithi042.stdout:1104: finishing write tid 2 to smithi04268048-2 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: finishing write tid 3 to smithi04268048-2 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: finishing write tid 4 to smithi04268048-2 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: oid 2 updating version 0 to 414 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: oid 2 updating version 414 to 416 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: oid 2 version 416 is already newer than 415 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 416 (ObjNum 398 snap 126 seq_num 398) dirty exists 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1104: left oid 2 (ObjNum 398 snap 126 seq_num 398) 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1107: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:1107: finishing copy_from to smithi04268048-26 2024-09-20T19:25:43.571 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 529 (ObjNum 380 snap 123 seq_num 380) dirty exists 2024-09-20T19:25:43.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:43.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:43.787 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:43.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:43.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:43.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:44.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:44.124 INFO:tasks.rados.rados.0.smithi042.stdout:1096: done (8 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1097: done (7 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1098: done (6 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1101: done (5 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1102: done (4 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1104: done (3 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1105: done (2 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1107: done (1 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1108: done (0 left) 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1109: write (excl) oid 19 current snap is 127 2024-09-20T19:25:44.125 INFO:tasks.rados.rados.0.smithi042.stdout:1109: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-20T19:25:44.133 INFO:tasks.rados.rados.0.smithi042.stdout:1109: writing smithi04268048-19 from 722800 to 1272756 tid 1 2024-09-20T19:25:44.143 INFO:tasks.rados.rados.0.smithi042.stdout:1109: writing smithi04268048-19 from 2040562 to 2722800 tid 2 2024-09-20T19:25:44.143 INFO:tasks.rados.rados.0.smithi042.stdout:1110: snap_remove snap 118 2024-09-20T19:25:44.177 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:44.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:44.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:44.461 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:44.564 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:44.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:44.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:44.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:44.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:44.929 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:25:44.929 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:25:44.947 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:25:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:25:44.949 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:44.968 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:25:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:25:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:44.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:44.986 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:25:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:25:45.007 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:25:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:25:45.028 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:25:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:25:45.049 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:25:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:25:45.070 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:25:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:25:45.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1110: done (1 left) 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1111: write oid 1 current snap is 127 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1109: finishing write tid 1 to smithi04268048-19 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1109: finishing write tid 2 to smithi04268048-19 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1109: finishing write tid 3 to smithi04268048-19 2024-09-20T19:25:45.134 INFO:tasks.rados.rados.0.smithi042.stdout:1111: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-20T19:25:45.142 INFO:tasks.rados.rados.0.smithi042.stdout:1111: writing smithi04268048-1 from 739607 to 1364812 tid 1 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1111: writing smithi04268048-1 from 1982691 to 2482148 tid 2 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1112: snap_create 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1109: finishing write tid 4 to smithi04268048-19 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1109: oid 19 updating version 0 to 346 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1109: oid 19 version 346 is already newer than 344 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1109: oid 19 version 346 is already newer than 345 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 346 (ObjNum 400 snap 127 seq_num 400) dirty exists 2024-09-20T19:25:45.149 INFO:tasks.rados.rados.0.smithi042.stdout:1109: left oid 19 (ObjNum 400 snap 127 seq_num 400) 2024-09-20T19:25:45.161 INFO:tasks.rados.rados.0.smithi042.stdout:1111: finishing write tid 1 to smithi04268048-1 2024-09-20T19:25:45.161 INFO:tasks.rados.rados.0.smithi042.stdout:1111: finishing write tid 2 to smithi04268048-1 2024-09-20T19:25:45.161 INFO:tasks.rados.rados.0.smithi042.stdout:1111: finishing write tid 3 to smithi04268048-1 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:1111: finishing write tid 4 to smithi04268048-1 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:1111: oid 1 updating version 0 to 372 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:1111: oid 1 version 372 is already newer than 370 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:1111: oid 1 version 372 is already newer than 371 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 372 (ObjNum 401 snap 127 seq_num 401) dirty exists 2024-09-20T19:25:45.162 INFO:tasks.rados.rados.0.smithi042.stdout:1111: left oid 1 (ObjNum 401 snap 127 seq_num 401) 2024-09-20T19:25:45.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:45.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:45.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[22706]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.335 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.336 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:45 smithi042 bash[32075]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.347 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:45.451 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:45.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.787838+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:43.790627+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:44.049909+0000 mgr.y (mgr.24491) 299 : cluster [DBG] pgmap v470: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: cluster 2024-09-20T19:25:44.124208+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:45 smithi162 bash[24557]: audit 2024-09-20T19:25:44.911366+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.42:0/4146701868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:45.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:45.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:45.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:45.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:45.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:45.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:46.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:46.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:46.150 INFO:tasks.rados.rados.0.smithi042.stdout:1109: done (2 left) 2024-09-20T19:25:46.151 INFO:tasks.rados.rados.0.smithi042.stdout:1111: done (1 left) 2024-09-20T19:25:46.151 INFO:tasks.rados.rados.0.smithi042.stdout:1112: done (0 left) 2024-09-20T19:25:46.151 INFO:tasks.rados.rados.0.smithi042.stdout:1113: delete oid 26 current snap is 128 2024-09-20T19:25:46.155 INFO:tasks.rados.rados.0.smithi042.stdout:1113: done (0 left) 2024-09-20T19:25:46.155 INFO:tasks.rados.rados.0.smithi042.stdout:1114: delete oid 5 current snap is 128 2024-09-20T19:25:46.160 INFO:tasks.rados.rados.0.smithi042.stdout:1114: done (0 left) 2024-09-20T19:25:46.160 INFO:tasks.rados.rados.0.smithi042.stdout:1115: rollback oid 35 current snap is 128 2024-09-20T19:25:46.160 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 92 2024-09-20T19:25:46.160 INFO:tasks.rados.rados.0.smithi042.stdout:1116: write (excl) oid 21 current snap is 128 2024-09-20T19:25:46.160 INFO:tasks.rados.rados.0.smithi042.stdout:1116: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-20T19:25:46.175 INFO:tasks.rados.rados.0.smithi042.stdout:1116: writing smithi04268048-21 from 790028 to 1357333 tid 1 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1116: writing smithi04268048-21 from 1874490 to 2637627 tid 2 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1117: copy_from oid 28 from oid 9 current snap is 128 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1115: finishing rollback tid 0 to smithi04268048-35 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1118: write oid 15 current snap is 128 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1115: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:25:46.190 INFO:tasks.rados.rados.0.smithi042.stdout:1118: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-20T19:25:46.201 INFO:tasks.rados.rados.0.smithi042.stdout:1118: writing smithi04268048-15 from 406835 to 1049389 tid 1 2024-09-20T19:25:46.212 INFO:tasks.rados.rados.0.smithi042.stdout:1118: writing smithi04268048-15 from 1532972 to 2039767 tid 2 2024-09-20T19:25:46.221 INFO:tasks.rados.rados.0.smithi042.stdout:1118: writing smithi04268048-15 from 2772312 to 2806835 tid 3 2024-09-20T19:25:46.221 INFO:tasks.rados.rados.0.smithi042.stdout:1119: delete oid 44 current snap is 128 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1115: finishing rollback tid 2 to smithi04268048-35 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 333 (ObjNum 291 snap 91 seq_num 291) dirty exists 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: finishing write tid 1 to smithi04268048-21 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1117: finishing copy_from to smithi04268048-28 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1117: got expected ENOENT (src dne) 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1117: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: finishing write tid 2 to smithi04268048-21 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: finishing write tid 3 to smithi04268048-21 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: finishing write tid 4 to smithi04268048-21 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: oid 21 updating version 0 to 569 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: oid 21 version 569 is already newer than 568 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:1116: oid 21 version 569 is already newer than 567 2024-09-20T19:25:46.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 569 (ObjNum 404 snap 128 seq_num 404) dirty exists 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1116: left oid 21 (ObjNum 404 snap 128 seq_num 404) 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1118: finishing write tid 1 to smithi04268048-15 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1118: finishing write tid 2 to smithi04268048-15 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1115: done (4 left) 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1116: done (3 left) 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1117: done (2 left) 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1119: done (1 left) 2024-09-20T19:25:46.223 INFO:tasks.rados.rados.0.smithi042.stdout:1120: read oid 14 snap -1 2024-09-20T19:25:46.224 INFO:tasks.rados.rados.0.smithi042.stdout:1120: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:25:46.224 INFO:tasks.rados.rados.0.smithi042.stdout:1121: snap_remove snap 126 2024-09-20T19:25:46.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:46.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:46 smithi042 bash[32075]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.243 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:46 smithi042 bash[32075]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.243 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:46 smithi042 bash[22706]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.243 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:46 smithi042 bash[22706]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.266 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:46.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:46.399 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:46.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:46.501 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:46.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:46 smithi162 bash[24557]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:46 smithi162 bash[24557]: cluster 2024-09-20T19:25:45.134306+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-20T19:25:46.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:46.656 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:46.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:46.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:46.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:46.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:46.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:47.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:47.048 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:47.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:47.153 INFO:tasks.rados.rados.0.smithi042.stdout:1121: done (2 left) 2024-09-20T19:25:47.153 INFO:tasks.rados.rados.0.smithi042.stdout:1122: read oid 23 snap -1 2024-09-20T19:25:47.153 INFO:tasks.rados.rados.0.smithi042.stdout:1122: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:25:47.153 INFO:tasks.rados.rados.0.smithi042.stdout:1123: write (excl) oid 20 current snap is 128 2024-09-20T19:25:47.153 INFO:tasks.rados.rados.0.smithi042.stdout:1123: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-20T19:25:47.158 INFO:tasks.rados.rados.0.smithi042.stdout:1123: writing smithi04268048-20 from 440449 to 1233501 tid 1 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1123: writing smithi04268048-20 from 1933583 to 2444047 tid 2 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: finishing write tid 3 to smithi04268048-15 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: finishing write tid 4 to smithi04268048-15 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: finishing write tid 5 to smithi04268048-15 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: oid 15 updating version 0 to 326 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: oid 15 updating version 326 to 328 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: oid 15 version 328 is already newer than 325 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: oid 15 version 328 is already newer than 327 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 328 (ObjNum 405 snap 128 seq_num 405) dirty exists 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1118: left oid 15 (ObjNum 405 snap 128 seq_num 405) 2024-09-20T19:25:47.168 INFO:tasks.rados.rados.0.smithi042.stdout:1120: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:25:47.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:47.208 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:47.225 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[22706]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.225 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[22706]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.225 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[22706]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.251 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 812732 2024-09-20T19:25:47.281 INFO:tasks.rados.rados.0.smithi042.stdout:1123: finishing write tid 1 to smithi04268048-20 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: finishing write tid 2 to smithi04268048-20 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: finishing write tid 3 to smithi04268048-20 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: finishing write tid 4 to smithi04268048-20 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: oid 20 updating version 0 to 87 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: oid 20 updating version 87 to 88 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: oid 20 updating version 88 to 89 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 89 (ObjNum 407 snap 128 seq_num 407) dirty exists 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1123: left oid 20 (ObjNum 407 snap 128 seq_num 407) 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:1122: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:25:47.282 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 378404 2024-09-20T19:25:47.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:47.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[22706]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.297 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[32075]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[32075]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[32075]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:47 smithi042 bash[32075]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.310 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:47.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:47.369 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 642624 2024-09-20T19:25:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:1118: done (3 left) 2024-09-20T19:25:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:1120: done (2 left) 2024-09-20T19:25:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:1122: done (1 left) 2024-09-20T19:25:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:1123: done (0 left) 2024-09-20T19:25:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:1124: snap_remove snap 103 2024-09-20T19:25:47.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:47.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:47 smithi162 bash[24557]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:47 smithi162 bash[24557]: cluster 2024-09-20T19:25:46.050644+0000 mgr.y (mgr.24491) 300 : cluster [DBG] pgmap v473: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 414 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:47 smithi162 bash[24557]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:47 smithi162 bash[24557]: cluster 2024-09-20T19:25:46.150681+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-20T19:25:47.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:47.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:47.704 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:47.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:47.835 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:47.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:47.966 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:25:47.966 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:25:47.967 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-20T19:25:47.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T19:25:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:47.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:48.021 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:25:48.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:48.112 INFO:tasks.rados.rados.0.smithi042.stdout:1124: done (0 left) 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1125: read oid 6 snap 101 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1125: expect (ObjNum 301 snap 94 seq_num 301) 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1126: read oid 39 snap -1 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1126: expect deleted 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1127: write (excl) oid 33 current snap is 128 2024-09-20T19:25:48.113 INFO:tasks.rados.rados.0.smithi042.stdout:1127: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-20T19:25:48.125 INFO:tasks.rados.rados.0.smithi042.stdout:1127: writing smithi04268048-33 from 457256 to 925557 tid 1 2024-09-20T19:25:48.147 INFO:tasks.rados.rados.0.smithi042.stdout:1127: writing smithi04268048-33 from 1592065 to 2362540 tid 2 2024-09-20T19:25:48.153 INFO:tasks.rados.rados.0.smithi042.stdout:1127: writing smithi04268048-33 from 2854581 to 2857256 tid 3 2024-09-20T19:25:48.153 INFO:tasks.rados.rados.0.smithi042.stdout:1128: snap_create 2024-09-20T19:25:48.153 INFO:tasks.rados.rados.0.smithi042.stdout:1125: expect (ObjNum 301 snap 94 seq_num 301) 2024-09-20T19:25:48.154 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:48.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[22706]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[22706]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[22706]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[22706]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[32075]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[32075]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[32075]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:48 smithi042 bash[32075]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:48.300 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:48.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: finishing write tid 1 to smithi04268048-33 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: finishing write tid 2 to smithi04268048-33 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: finishing write tid 3 to smithi04268048-33 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: finishing write tid 4 to smithi04268048-33 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: finishing write tid 5 to smithi04268048-33 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: oid 33 updating version 0 to 419 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: oid 33 updating version 419 to 420 2024-09-20T19:25:48.375 INFO:tasks.rados.rados.0.smithi042.stdout:1127: oid 33 version 420 is already newer than 418 2024-09-20T19:25:48.376 INFO:tasks.rados.rados.0.smithi042.stdout:1127: oid 33 version 420 is already newer than 417 2024-09-20T19:25:48.376 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 420 (ObjNum 408 snap 128 seq_num 408) dirty exists 2024-09-20T19:25:48.376 INFO:tasks.rados.rados.0.smithi042.stdout:1127: left oid 33 (ObjNum 408 snap 128 seq_num 408) 2024-09-20T19:25:48.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:48.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:48 smithi162 bash[24557]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:48 smithi162 bash[24557]: cluster 2024-09-20T19:25:47.147453+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-20T19:25:48.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:48 smithi162 bash[24557]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:48 smithi162 bash[24557]: cluster 2024-09-20T19:25:48.103536+0000 mon.a (mon.0) 1386 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-20T19:25:48.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:48.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:48.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:48.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:48.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:49.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:49.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.110 INFO:tasks.rados.rados.0.smithi042.stdout:1125: done (3 left) 2024-09-20T19:25:49.110 INFO:tasks.rados.rados.0.smithi042.stdout:1126: done (2 left) 2024-09-20T19:25:49.110 INFO:tasks.rados.rados.0.smithi042.stdout:1127: done (1 left) 2024-09-20T19:25:49.111 INFO:tasks.rados.rados.0.smithi042.stdout:1128: done (0 left) 2024-09-20T19:25:49.111 INFO:tasks.rados.rados.0.smithi042.stdout:1129: copy_from oid 26 from oid 17 current snap is 129 2024-09-20T19:25:49.111 INFO:tasks.rados.rados.0.smithi042.stdout:1130: delete oid 30 current snap is 129 2024-09-20T19:25:49.117 INFO:tasks.rados.rados.0.smithi042.stdout:1129: finishing copy_from to smithi04268048-26 2024-09-20T19:25:49.117 INFO:tasks.rados.rados.0.smithi042.stdout:1129: got expected ENOENT (src dne) 2024-09-20T19:25:49.118 INFO:tasks.rados.rados.0.smithi042.stdout:1129: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:25:49.118 INFO:tasks.rados.rados.0.smithi042.stdout:1129: done (1 left) 2024-09-20T19:25:49.118 INFO:tasks.rados.rados.0.smithi042.stdout:1130: done (0 left) 2024-09-20T19:25:49.118 INFO:tasks.rados.rados.0.smithi042.stdout:1131: write oid 30 current snap is 129 2024-09-20T19:25:49.118 INFO:tasks.rados.rados.0.smithi042.stdout:1131: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-20T19:25:49.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.136 INFO:tasks.rados.rados.0.smithi042.stdout:1131: writing smithi04268048-30 from 490870 to 1109669 tid 1 2024-09-20T19:25:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:49.157 INFO:tasks.rados.rados.0.smithi042.stdout:1131: writing smithi04268048-30 from 1592676 to 2366820 tid 2 2024-09-20T19:25:49.166 INFO:tasks.rados.rados.0.smithi042.stdout:1131: writing smithi04268048-30 from 2793074 to 2890870 tid 3 2024-09-20T19:25:49.166 INFO:tasks.rados.rados.0.smithi042.stdout:1132: delete oid 33 current snap is 129 2024-09-20T19:25:49.166 INFO:tasks.rados.rados.0.smithi042.stdout:1131: finishing write tid 1 to smithi04268048-30 2024-09-20T19:25:49.167 INFO:tasks.rados.rados.0.smithi042.stdout:1131: finishing write tid 2 to smithi04268048-30 2024-09-20T19:25:49.171 INFO:tasks.rados.rados.0.smithi042.stdout:1132: done (1 left) 2024-09-20T19:25:49.171 INFO:tasks.rados.rados.0.smithi042.stdout:1133: rollback oid 19 current snap is 129 2024-09-20T19:25:49.172 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 88 2024-09-20T19:25:49.172 INFO:tasks.rados.rados.0.smithi042.stdout:1134: snap_remove snap 117 2024-09-20T19:25:49.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:49.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.395 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[32075]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[32075]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[32075]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[32075]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[22706]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[22706]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[22706]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:49 smithi042 bash[22706]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:49.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:49.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:49 smithi162 bash[24557]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:49 smithi162 bash[24557]: cluster 2024-09-20T19:25:48.051151+0000 mgr.y (mgr.24491) 301 : cluster [DBG] pgmap v476: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:49 smithi162 bash[24557]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:49 smithi162 bash[24557]: cluster 2024-09-20T19:25:49.108926+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:18:09.279678+0000", 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 460.29160971599998, 2024-09-20T19:25:49.573 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021480112999999999, 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 43 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279678+0000", 2024-09-20T19:25:49.574 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279678+0000", 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279681+0000", 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4020000000000002e-06 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.575 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279693+0000", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2138e-05 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279736+0000", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2583999999999998e-05 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279752+0000", 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6483000000000001e-05 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279869+0000", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000117008 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.279879+0000", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0074e-05 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.283836+0000", 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.003956996 2024-09-20T19:25:49.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.283865+0000", 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8586e-05 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.300919+0000", 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017054004000000001 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.578 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:18:09.301158+0000", 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023883800000000001 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.24490.0:85 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:19:53.991741+0000", 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 355.57954647999998, 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019959687, 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.579 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.24490", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.162:0/1964141317", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 85 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:53.991741+0000", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.580 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:53.991741+0000", 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:53.991745+0000", 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4460000000000003e-06 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:53.991763+0000", 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7456999999999999e-05 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.581 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:54.010947+0000", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019184459000000001 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:54.010977+0000", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9819999999999999e-05 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:54.011252+0000", 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027510799999999998 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.582 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:54.011467+0000", 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021518500000000001 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:19:54.011700+0000", 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000233212 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:242 7.b 7:d579898a:::smithi04268048-35:head [copy-from ver 22] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-20T19:25:49.583 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 238.749996353, 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016001677999999998, 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 242 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.584 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821293+0000", 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0899999999999999e-06 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.585 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821299+0000", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5559999999999998e-06 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821310+0000", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1832e-05 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821318+0000", 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0879999999999995e-06 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821402+0000", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3906000000000006e-05 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821501+0000", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8999999999999994e-05 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.826058+0000", 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045566590000000002 2024-09-20T19:25:49.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.832133+0000", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060751090000000004 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.836775+0000", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0046420899999999998 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837179+0000", 2024-09-20T19:25:49.588 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00040368899999999999 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837240+0000", 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0934999999999998e-05 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837293+0000", 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2723999999999999e-05 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:478 7.4 e148/112 7:24603624:::smithi04268048-48:head v 148'31, mlcod=136'27)", 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:17.076976+0000", 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 212.494311321, 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010830374, 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.076976+0000", 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.076976+0000", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.076977+0000", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.372e-06 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.079982+0000", 2024-09-20T19:25:49.591 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.003004647 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.080018+0000", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6263000000000002e-05 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.080038+0000", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0132e-05 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.080212+0000", 2024-09-20T19:25:49.592 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000173303 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.080271+0000", 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9862999999999999e-05 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.087701+0000", 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0074292589999999997 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.593 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.087806+0000", 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000105535 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:506 7.9 e152/112 7:9b44618a:::smithi04268048-36:head v 152'28, mlcod=136'24)", 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:21.037624+0000", 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 208.53366292199999, 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010522002000000001, 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.037624+0000", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.037624+0000", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.595 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.037628+0000", 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1470000000000003e-06 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.041785+0000", 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0041564549999999999 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.041817+0000", 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2416999999999998e-05 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.596 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.041836+0000", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9130000000000001e-05 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.041997+0000", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000160545 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.042092+0000", 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5278999999999994e-05 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.597 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.048005+0000", 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0059128469999999997 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:21.048146+0000", 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000141182 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.0.0:296 7.4 e198/112 7:23ed6a02:::smithi04268048-25:1b v 198'114, mlcod=198'110)", 2024-09-20T19:25:49.598 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:07.082329+0000", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 162.48895870499999, 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01275812, 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082329+0000", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082329+0000", 2024-09-20T19:25:49.599 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082331+0000", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8990000000000001e-06 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082346+0000", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4698999999999999e-05 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082369+0000", 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2442999999999998e-05 2024-09-20T19:25:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082379+0000", 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0292000000000001e-05 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082491+0000", 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000111733 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.601 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082542+0000", 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1051000000000003e-05 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095011+0000", 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012469223 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095087+0000", 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5779999999999996e-05 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.0.0:297 7.4 e198/112 7:2b212c05:::smithi04268048-30:22 v 198'116, mlcod=198'110)", 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:07.082406+0000", 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 162.488880897, 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012734752, 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082406+0000", 2024-09-20T19:25:49.603 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082406+0000", 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082407+0000", 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8599999999999998e-07 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.604 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082441+0000", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4255999999999999e-05 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082454+0000", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2971e-05 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.082460+0000", 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.161e-06 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083210+0000", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00074921299999999998 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083251+0000", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1730000000000002e-05 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095094+0000", 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011842915000000001 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095141+0000", 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6919999999999998e-05 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:974 7.4 e198/112 7:24603624:::smithi04268048-48:head v 198'118, mlcod=198'110)", 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:07.083848+0000", 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 162.487439338, 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011354408, 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.607 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083848+0000", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083848+0000", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.608 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083850+0000", 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0870000000000001e-06 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083861+0000", 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1199000000000001e-05 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083882+0000", 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0814000000000001e-05 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.609 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.083894+0000", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2286999999999999e-05 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.094151+0000", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010256692 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.094173+0000", 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1702e-05 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095157+0000", 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0009845050000000001 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:07.095202+0000", 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5121999999999999e-05 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.611 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:1517 7.0 e243/112 7:060c6a05:::smithi04268048-38:head v 243'183, mlcod=239'179)", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:52.320300+0000", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 117.25098758599999, 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012787892, 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.320300+0000", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.612 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.320300+0000", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.320300+0000", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3000000000000001e-07 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.326373+0000", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060728650000000002 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.613 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.326402+0000", 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8809000000000001e-05 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.326423+0000", 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1494000000000001e-05 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.326621+0000", 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019779899999999999 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.614 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.326680+0000", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9364999999999998e-05 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.332975+0000", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0062940110000000004 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:52.333088+0000", 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000113019 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2075 7.b 7:d579898a:::smithi04268048-35:head [write 490837~718873,stat] snapc 5c=[5b,5a,59,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:41.097595+0000", 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 68.473692381000006, 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012876453, 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2075 2024-09-20T19:25:49.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.097595+0000", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.097595+0000", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.097598+0000", 2024-09-20T19:25:49.617 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.96e-06 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098016+0000", 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00041852799999999999 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098034+0000", 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.751e-05 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.618 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098043+0000", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.901e-06 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098173+0000", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000130304 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098296+0000", 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012300000000000001 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.098540+0000", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024368400000000001 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.101021+0000", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002481296 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.108348+0000", 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0073272379999999998 2024-09-20T19:25:49.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.110266+0000", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001917897 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.110373+0000", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000106822 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:41.110471+0000", 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8313000000000006e-05 2024-09-20T19:25:49.621 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2132 7.d e297/112 7:baa4e37b:::smithi04268048-17:head v 297'157, mlcod=296'153)", 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 63.318499150999997, 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017849924999999999, 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8000000000000002e-07 2024-09-20T19:25:49.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259424+0000", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0066358739999999999 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259460+0000", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.5748999999999999e-05 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259482+0000", 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1800000000000001e-05 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.624 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259597+0000", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000114894 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259636+0000", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9044000000000001e-05 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.270493+0000", 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010857148 2024-09-20T19:25:49.625 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.270638+0000", 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000145136 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.4.0:290 7.1 e334/333 7:892846d2:::smithi04268048-15:71 v 334'295, mlcod=329'291)", 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.461768571, 2024-09-20T19:25:49.626 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015268339000000001, 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.627 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109520+0000", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.824e-06 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109538+0000", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7889000000000001e-05 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109559+0000", 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0863e-05 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.628 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109583+0000", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3574000000000002e-05 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109844+0000", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026147399999999998 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109908+0000", 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4127000000000005e-05 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.629 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124676+0000", 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014767656000000001 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124787+0000", 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000110932 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.630 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "pg_update_log_missing(7.d epoch 334/112 rep_tid 522 entries 334'201 (0'0) error 7:b90bdf9e:::smithi04268048-23:head by client.14904.0:2586 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 327'198)", 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.46101298, 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014371523000000001, 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.631 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110275+0000", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7999999999999996e-07 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110284+0000", 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.816e-06 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.632 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110305+0000", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0985e-05 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110320+0000", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5965e-05 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.120126+0000", 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0098058480000000007 2024-09-20T19:25:49.633 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124646+0000", 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045194290000000002 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2594 7.0 e334/112 7:03a4d37b:::smithi04268048-10:head v 334'309, mlcod=332'305)", 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.116111+0000", 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.455176584, 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010418966999999999, 2024-09-20T19:25:49.634 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.116111+0000", 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.116111+0000", 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.635 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.116112+0000", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.697e-06 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.122815+0000", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0067026439999999998 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.122849+0000", 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3779999999999998e-05 2024-09-20T19:25:49.636 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.122872+0000", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2870999999999998e-05 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.122954+0000", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2594000000000002e-05 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.122998+0000", 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4196999999999997e-05 2024-09-20T19:25:49.637 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.126434+0000", 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0034352139999999998 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.126530+0000", 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5970000000000004e-05 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.638 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2589 7.1 e334/333 7:892846d2:::smithi04268048-15:head v 334'297, mlcod=329'291)", 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.446533555999999, 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013064749000000001, 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.639 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124757+0000", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.157e-06 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133124+0000", 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0083669580000000007 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.640 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133149+0000", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4864000000000001e-05 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133161+0000", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2856e-05 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133235+0000", 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3444999999999999e-05 2024-09-20T19:25:49.641 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133259+0000", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3730000000000001e-05 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.137670+0000", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044115889999999996 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.137818+0000", 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014815 2024-09-20T19:25:49.642 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2604 7.0 e335/112 7:060c6a05:::smithi04268048-38:head v 335'313, mlcod=334'309)", 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 24.833783662999998, 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013697485000000001, 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.643 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0600000000000001e-07 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745768+0000", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0082640609999999996 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745786+0000", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7507e-05 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745793+0000", 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3640000000000001e-06 2024-09-20T19:25:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745860+0000", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7523999999999999e-05 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745880+0000", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9174999999999999e-05 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.751155+0000", 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0052757890000000003 2024-09-20T19:25:49.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.751201+0000", 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5658999999999998e-05 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2633 7.0 e337/112 7:03a4d37b:::smithi04268048-10:head v 337'317, mlcod=335'313)", 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:25:49.647 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 23.273246774, 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013586387, 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298041+0000", 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8800000000000002e-07 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305314+0000", 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0072731929999999998 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305347+0000", 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3139000000000002e-05 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305362+0000", 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4559000000000001e-05 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305477+0000", 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011479799999999999 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.650 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305508+0000", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1517000000000003e-05 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.311557+0000", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060486180000000004 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.311627+0000", 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9875000000000004e-05 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.651 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2896 7.b 7:d579898a:::smithi04268048-35:head [append 0~0] snapc 81=[81,80,7f,7e,7d,7c,79,77,76,72,6d,6c,69,67,5e,5a,41] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:46.161698+0000", 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.409589253, 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010717735000000001, 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:49.652 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2896 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161698+0000", 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161698+0000", 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.653 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161703+0000", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1800000000000004e-06 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161718+0000", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4863e-05 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161754+0000", 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.5771e-05 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.654 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161770+0000", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5977e-05 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161960+0000", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019016899999999999 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161982+0000", 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2096e-05 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.166408+0000", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044262909999999997 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.166668+0000", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000259881 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.167286+0000", 2024-09-20T19:25:49.656 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00061737000000000001 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172003+0000", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0047173500000000004 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172069+0000", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.5989999999999997e-05 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172183+0000", 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000114286 2024-09-20T19:25:49.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172268+0000", 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5291999999999996e-05 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172416+0000", 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000147219 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.658 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2897 7.b 7:d579898a:::smithi04268048-35:head [rollback 5e] snapc 81=[81,80,7f,7e,7d,7c,79,77,76,72,6d,6c,69,67,5e,5a,41] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:46.161805+0000", 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.4094827130000001, 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012759494, 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2897 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.659 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161805+0000", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161805+0000", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161806+0000", 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1999999999999999e-06 2024-09-20T19:25:49.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161811+0000", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9880000000000004e-06 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161820+0000", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3030000000000005e-06 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161828+0000", 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0600000000000008e-06 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162166+0000", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00033821699999999998 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162179+0000", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2323e-05 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.168554+0000", 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.006375538 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.168844+0000", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000289945 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.169408+0000", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00056419599999999999 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172152+0000", 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0027438369999999998 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172444+0000", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029218499999999998 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.174375+0000", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0019306429999999999 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.174479+0000", 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000103736 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.174564+0000", 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5322999999999996e-05 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2898 7.b 7:d579898a:::smithi04268048-35:head [append 0~0] snapc 81=[81,80,7f,7e,7d,7c,79,77,76,72,6d,6c,69,67,5e,5a,41] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.409379903, 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013430002, 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:25:49.665 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2898 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:25:49.666 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161909+0000", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.192e-06 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161913+0000", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7770000000000002e-06 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161929+0000", 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5282999999999999e-05 2024-09-20T19:25:49.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161940+0000", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1268e-05 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162265+0000", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000324691 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162284+0000", 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9853000000000001e-05 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.668 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.170752+0000", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0084678630000000008 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.171031+0000", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027857499999999998 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.171512+0000", 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048082700000000002 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172465+0000", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00095308600000000002 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172548+0000", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3375999999999996e-05 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175140+0000", 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0025914649999999998 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.670 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175247+0000", 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107666 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175337+0000", 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0080000000000005e-05 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:49.671 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:25:49.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:49.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.829 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:49.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:49.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:50.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:50.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:25:50.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[22706]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[22706]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[22706]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[22706]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.183 INFO:tasks.rados.rados.0.smithi042.stdout:1134: done (2 left) 2024-09-20T19:25:50.183 INFO:tasks.rados.rados.0.smithi042.stdout:1135: snap_remove snap 107 2024-09-20T19:25:50.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:50.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:50.315 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:50.326 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[32075]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.326 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[32075]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.327 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[32075]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.327 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:50 smithi042 bash[32075]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:50.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:25:50.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:50.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:50 smithi162 bash[24557]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:50 smithi162 bash[24557]: audit 2024-09-20T19:25:49.228294+0000 mgr.y (mgr.24491) 302 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:25:50.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:50 smithi162 bash[24557]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:50 smithi162 bash[24557]: audit 2024-09-20T19:25:49.765801+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:25:50.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:50.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:50.600 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:50.703 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:50.729 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:50.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:50.859 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:50.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:50.991 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:50.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:51.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1135: done (2 left) 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1136: write oid 26 current snap is 129 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: finishing write tid 3 to smithi04268048-30 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: finishing write tid 4 to smithi04268048-30 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: finishing write tid 5 to smithi04268048-30 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: oid 30 updating version 0 to 587 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: oid 30 version 587 is already newer than 584 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: oid 30 version 587 is already newer than 586 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: oid 30 version 587 is already newer than 585 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 587 (ObjNum 410 snap 129 seq_num 410) dirty exists 2024-09-20T19:25:51.188 INFO:tasks.rados.rados.0.smithi042.stdout:1131: left oid 30 (ObjNum 410 snap 129 seq_num 410) 2024-09-20T19:25:51.189 INFO:tasks.rados.rados.0.smithi042.stdout:1133: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:25:51.189 INFO:tasks.rados.rados.0.smithi042.stdout:1133: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:25:51.189 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 348 (ObjNum 129 snap 0 seq_num 1415346496) dirty dne 2024-09-20T19:25:51.189 INFO:tasks.rados.rados.0.smithi042.stdout:1136: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-20T19:25:51.190 INFO:tasks.rados.rados.0.smithi042.stdout:1136: writing smithi04268048-26 from 524484 to 1010134 tid 1 2024-09-20T19:25:51.197 INFO:tasks.rados.rados.0.smithi042.stdout:1136: writing smithi04268048-26 from 1425993 to 2087453 tid 2 2024-09-20T19:25:51.202 INFO:tasks.rados.rados.0.smithi042.stdout:1136: writing smithi04268048-26 from 2847920 to 2924484 tid 3 2024-09-20T19:25:51.202 INFO:tasks.rados.rados.0.smithi042.stdout:1131: done (2 left) 2024-09-20T19:25:51.202 INFO:tasks.rados.rados.0.smithi042.stdout:1133: done (1 left) 2024-09-20T19:25:51.202 INFO:tasks.rados.rados.0.smithi042.stdout:1137: write (excl) oid 2 current snap is 129 2024-09-20T19:25:51.202 INFO:tasks.rados.rados.0.smithi042.stdout:1137: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-20T19:25:51.208 INFO:tasks.rados.rados.0.smithi042.stdout:1137: writing smithi04268048-2 from 541291 to 1102190 tid 1 2024-09-20T19:25:51.216 INFO:tasks.rados.rados.0.smithi042.stdout:1137: writing smithi04268048-2 from 1884475 to 2405946 tid 2 2024-09-20T19:25:51.216 INFO:tasks.rados.rados.0.smithi042.stdout:1138: rollback oid 25 current snap is 129 2024-09-20T19:25:51.216 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 106 2024-09-20T19:25:51.216 INFO:tasks.rados.rados.0.smithi042.stdout:1136: finishing write tid 1 to smithi04268048-26 2024-09-20T19:25:51.216 INFO:tasks.rados.rados.0.smithi042.stdout:1139: write oid 39 current snap is 129 2024-09-20T19:25:51.217 INFO:tasks.rados.rados.0.smithi042.stdout:1139: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-20T19:25:51.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:51.223 INFO:tasks.rados.rados.0.smithi042.stdout:1139: writing smithi04268048-39 from 558098 to 1194246 tid 1 2024-09-20T19:25:51.229 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[32075]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.230 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.231 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:51 smithi042 bash[22706]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.231 INFO:tasks.rados.rados.0.smithi042.stdout:1139: writing smithi04268048-39 from 1826604 to 2491733 tid 2 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1140: copy_from oid 10 from oid 4 current snap is 129 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: finishing write tid 2 to smithi04268048-26 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: finishing write tid 3 to smithi04268048-26 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: finishing write tid 4 to smithi04268048-26 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: finishing write tid 5 to smithi04268048-26 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: oid 26 updating version 0 to 534 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: oid 26 version 534 is already newer than 531 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: oid 26 version 534 is already newer than 533 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:1136: oid 26 version 534 is already newer than 532 2024-09-20T19:25:51.232 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 534 (ObjNum 412 snap 129 seq_num 412) dirty exists 2024-09-20T19:25:51.233 INFO:tasks.rados.rados.0.smithi042.stdout:1136: left oid 26 (ObjNum 412 snap 129 seq_num 412) 2024-09-20T19:25:51.233 INFO:tasks.rados.rados.0.smithi042.stdout:1137: finishing write tid 1 to smithi04268048-2 2024-09-20T19:25:51.233 INFO:tasks.rados.rados.0.smithi042.stdout:1136: done (4 left) 2024-09-20T19:25:51.233 INFO:tasks.rados.rados.0.smithi042.stdout:1141: snap_remove snap 88 2024-09-20T19:25:51.241 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:51.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:51.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.051801+0000 mgr.y (mgr.24491) 303 : cluster [DBG] pgmap v479: 158 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 150 active+clean; 432 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.165886+0000 mon.a (mon.0) 1389 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: audit 2024-09-20T19:25:50.172863+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? v1:172.21.15.42:0/2597192666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:51 smithi162 bash[24557]: cluster 2024-09-20T19:25:50.183515+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-20T19:25:51.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:51.605 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:51.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:51.754 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:51.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:51.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:51.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:52.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:52.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:52.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:52.207 INFO:tasks.rados.rados.0.smithi042.stdout:1141: done (4 left) 2024-09-20T19:25:52.207 INFO:tasks.rados.rados.0.smithi042.stdout:1142: write (excl) oid 44 current snap is 129 2024-09-20T19:25:52.207 INFO:tasks.rados.rados.0.smithi042.stdout:1142: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-20T19:25:52.210 INFO:tasks.rados.rados.0.smithi042.stdout:1142: writing smithi04268048-44 from 574905 to 1286302 tid 1 2024-09-20T19:25:52.218 INFO:tasks.rados.rados.0.smithi042.stdout:1142: writing smithi04268048-44 from 1885086 to 2410226 tid 2 2024-09-20T19:25:52.222 INFO:tasks.rados.rados.0.smithi042.stdout:1142: writing smithi04268048-44 from 2930189 to 2974905 tid 3 2024-09-20T19:25:52.223 INFO:tasks.rados.rados.0.smithi042.stdout:1143: read oid 19 snap -1 2024-09-20T19:25:52.223 INFO:tasks.rados.rados.0.smithi042.stdout:1143: expect deleted 2024-09-20T19:25:52.223 INFO:tasks.rados.rados.0.smithi042.stdout:1144: write (excl) oid 30 current snap is 129 2024-09-20T19:25:52.223 INFO:tasks.rados.rados.0.smithi042.stdout:1144: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-20T19:25:52.232 INFO:tasks.rados.rados.0.smithi042.stdout:1144: writing smithi04268048-30 from 591712 to 1378358 tid 1 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1144: writing smithi04268048-30 from 1943568 to 2612366 tid 2 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1145: copy_from oid 7 from oid 18 current snap is 129 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1138: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1138: finishing rollback tid 2 to smithi04268048-25 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 605 (ObjNum 302 snap 95 seq_num 302) dirty exists 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1137: finishing write tid 2 to smithi04268048-2 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1137: finishing write tid 3 to smithi04268048-2 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1137: finishing write tid 4 to smithi04268048-2 2024-09-20T19:25:52.242 INFO:tasks.rados.rados.0.smithi042.stdout:1137: oid 2 updating version 0 to 422 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1137: oid 2 updating version 422 to 423 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1137: oid 2 updating version 423 to 424 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 424 (ObjNum 413 snap 129 seq_num 413) dirty exists 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1137: left oid 2 (ObjNum 413 snap 129 seq_num 413) 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1139: finishing write tid 1 to smithi04268048-39 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1137: done (7 left) 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1138: done (6 left) 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1146: write oid 35 current snap is 129 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1139: finishing write tid 2 to smithi04268048-39 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1139: finishing write tid 3 to smithi04268048-39 2024-09-20T19:25:52.243 INFO:tasks.rados.rados.0.smithi042.stdout:1146: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-20T19:25:52.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:52.248 INFO:tasks.rados.rados.0.smithi042.stdout:1146: writing smithi04268048-35 from 608519 to 1070414 tid 1 2024-09-20T19:25:52.256 INFO:tasks.rados.rados.0.smithi042.stdout:1146: writing smithi04268048-35 from 1602050 to 2130859 tid 2 2024-09-20T19:25:52.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:52 smithi162 bash[24557]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:52 smithi162 bash[24557]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1146: writing smithi04268048-35 from 2585035 to 3008519 tid 3 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1147: delete oid 13 current snap is 129 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1139: finishing write tid 4 to smithi04268048-39 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1139: oid 39 updating version 0 to 427 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1139: oid 39 version 427 is already newer than 426 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1139: oid 39 version 427 is already newer than 425 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 427 (ObjNum 414 snap 129 seq_num 414) dirty exists 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1139: left oid 39 (ObjNum 414 snap 129 seq_num 414) 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1140: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:25:52.262 INFO:tasks.rados.rados.0.smithi042.stdout:1140: finishing copy_from to smithi04268048-10 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 553 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: finishing write tid 1 to smithi04268048-44 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: finishing write tid 2 to smithi04268048-44 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: finishing write tid 3 to smithi04268048-44 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: finishing write tid 4 to smithi04268048-44 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: finishing write tid 5 to smithi04268048-44 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: oid 44 updating version 0 to 579 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: oid 44 version 579 is already newer than 578 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: oid 44 updating version 579 to 580 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: oid 44 updating version 580 to 581 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 581 (ObjNum 415 snap 129 seq_num 415) dirty exists 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1142: left oid 44 (ObjNum 415 snap 129 seq_num 415) 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1144: finishing write tid 1 to smithi04268048-30 2024-09-20T19:25:52.263 INFO:tasks.rados.rados.0.smithi042.stdout:1144: finishing write tid 2 to smithi04268048-30 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: finishing write tid 3 to smithi04268048-30 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: finishing write tid 4 to smithi04268048-30 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: oid 30 updating version 0 to 616 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: oid 30 version 616 is already newer than 614 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: oid 30 version 616 is already newer than 615 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 616 (ObjNum 416 snap 129 seq_num 416) dirty exists 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1144: left oid 30 (ObjNum 416 snap 129 seq_num 416) 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1146: finishing write tid 1 to smithi04268048-35 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1145: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1145: finishing copy_from to smithi04268048-7 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 483 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:25:52.264 INFO:tasks.rados.rados.0.smithi042.stdout:1146: finishing write tid 2 to smithi04268048-35 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1139: done (7 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1140: done (6 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1142: done (5 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1143: done (4 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1144: done (3 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1145: done (2 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1147: done (1 left) 2024-09-20T19:25:52.265 INFO:tasks.rados.rados.0.smithi042.stdout:1148: snap_create 2024-09-20T19:25:52.266 INFO:tasks.rados.rados.0.smithi042.stdout:1146: finishing write tid 3 to smithi04268048-35 2024-09-20T19:25:52.266 INFO:tasks.rados.rados.0.smithi042.stdout:1146: finishing write tid 4 to smithi04268048-35 2024-09-20T19:25:52.266 INFO:tasks.rados.rados.0.smithi042.stdout:1146: finishing write tid 5 to smithi04268048-35 2024-09-20T19:25:52.266 INFO:tasks.rados.rados.0.smithi042.stdout:1146: oid 35 updating version 0 to 337 2024-09-20T19:25:52.266 INFO:tasks.rados.rados.0.smithi042.stdout:1146: oid 35 version 337 is already newer than 334 2024-09-20T19:25:52.267 INFO:tasks.rados.rados.0.smithi042.stdout:1146: oid 35 version 337 is already newer than 336 2024-09-20T19:25:52.267 INFO:tasks.rados.rados.0.smithi042.stdout:1146: oid 35 version 337 is already newer than 335 2024-09-20T19:25:52.267 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 337 (ObjNum 417 snap 129 seq_num 417) dirty exists 2024-09-20T19:25:52.267 INFO:tasks.rados.rados.0.smithi042.stdout:1146: left oid 35 (ObjNum 417 snap 129 seq_num 417) 2024-09-20T19:25:52.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:52.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:52.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:52.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:52.514 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:52 smithi042 bash[32075]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.514 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:52 smithi042 bash[32075]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:52 smithi042 bash[22706]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:52 smithi042 bash[22706]: cluster 2024-09-20T19:25:51.178639+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-20T19:25:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:52.634 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:52.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:52.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:52.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:53.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:53.046 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:53.094 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:25:53.108 INFO:tasks.rados.rados.0.smithi042.stdout:1146: done (1 left) 2024-09-20T19:25:53.108 INFO:tasks.rados.rados.0.smithi042.stdout:1148: done (0 left) 2024-09-20T19:25:53.108 INFO:tasks.rados.rados.0.smithi042.stdout:1149: snap_remove snap 124 2024-09-20T19:25:53.149 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:53.171 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:53.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:53.284 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.285 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:53 smithi162 bash[24557]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:53.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:53.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:53.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[22706]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:52.053441+0000 mgr.y (mgr.24491) 304 : cluster [DBG] pgmap v482: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:52.201306+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:53 smithi042 bash[32075]: cluster 2024-09-20T19:25:53.106473+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-20T19:25:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:53.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:53.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:53.628 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:53.629 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:53.661 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:53.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:53.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:53.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:53.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:53.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:53.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:54.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:54.113 INFO:tasks.rados.rados.0.smithi042.stdout:1149: done (0 left) 2024-09-20T19:25:54.113 INFO:tasks.rados.rados.0.smithi042.stdout:1150: read oid 26 snap -1 2024-09-20T19:25:54.113 INFO:tasks.rados.rados.0.smithi042.stdout:1150: expect (ObjNum 412 snap 129 seq_num 412) 2024-09-20T19:25:54.114 INFO:tasks.rados.rados.0.smithi042.stdout:1151: snap_remove snap 116 2024-09-20T19:25:54.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.197 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:54.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:54.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:54.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:54.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:54.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:54.687 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:54.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:54.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:54.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:54.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:55.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.111 INFO:tasks.rados.rados.0.smithi042.stdout:1151: done (1 left) 2024-09-20T19:25:55.111 INFO:tasks.rados.rados.0.smithi042.stdout:1152: copy_from oid 34 from oid 38 current snap is 130 2024-09-20T19:25:55.111 INFO:tasks.rados.rados.0.smithi042.stdout:1153: delete oid 16 current snap is 130 2024-09-20T19:25:55.111 INFO:tasks.rados.rados.0.smithi042.stdout:1150: expect (ObjNum 412 snap 129 seq_num 412) 2024-09-20T19:25:55.111 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.119 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:55 smithi162 bash[24557]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2207717 2024-09-20T19:25:55.173 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1152: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1152: finishing copy_from to smithi04268048-34 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 487 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1150: done (2 left) 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1152: done (1 left) 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1153: done (0 left) 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1154: copy_from oid 18 from oid 12 current snap is 130 2024-09-20T19:25:55.205 INFO:tasks.rados.rados.0.smithi042.stdout:1155: read oid 45 snap -1 2024-09-20T19:25:55.206 INFO:tasks.rados.rados.0.smithi042.stdout:1155: expect (ObjNum 368 snap 118 seq_num 368) 2024-09-20T19:25:55.206 INFO:tasks.rados.rados.0.smithi042.stdout:1156: read oid 1 snap -1 2024-09-20T19:25:55.206 INFO:tasks.rados.rados.0.smithi042.stdout:1156: expect (ObjNum 401 snap 127 seq_num 401) 2024-09-20T19:25:55.206 INFO:tasks.rados.rados.0.smithi042.stdout:1157: delete oid 27 current snap is 130 2024-09-20T19:25:55.208 INFO:tasks.rados.rados.0.smithi042.stdout:1157: done (3 left) 2024-09-20T19:25:55.208 INFO:tasks.rados.rados.0.smithi042.stdout:1158: rollback oid 3 current snap is 130 2024-09-20T19:25:55.209 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 129 2024-09-20T19:25:55.209 INFO:tasks.rados.rados.0.smithi042.stdout:1159: write oid 20 current snap is 130 2024-09-20T19:25:55.209 INFO:tasks.rados.rados.0.smithi042.stdout:1159: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-20T19:25:55.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.216 INFO:tasks.rados.rados.0.smithi042.stdout:1159: writing smithi04268048-20 from 675747 to 1154991 tid 1 2024-09-20T19:25:55.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[22706]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: cluster 2024-09-20T19:25:54.054219+0000 mgr.y (mgr.24491) 305 : cluster [DBG] pgmap v485: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: cluster 2024-09-20T19:25:54.112204+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:55 smithi042 bash[32075]: audit 2024-09-20T19:25:54.858140+0000 mon.a (mon.0) 1396 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:25:55.225 INFO:tasks.rados.rados.0.smithi042.stdout:1159: writing smithi04268048-20 from 1835978 to 2372125 tid 2 2024-09-20T19:25:55.225 INFO:tasks.rados.rados.0.smithi042.stdout:1160: write oid 44 current snap is 130 2024-09-20T19:25:55.225 INFO:tasks.rados.rados.0.smithi042.stdout:1160: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-20T19:25:55.232 INFO:tasks.rados.rados.0.smithi042.stdout:1160: writing smithi04268048-44 from 692554 to 1247047 tid 1 2024-09-20T19:25:55.236 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:55.241 INFO:tasks.rados.rados.0.smithi042.stdout:1160: writing smithi04268048-44 from 1778107 to 2457912 tid 2 2024-09-20T19:25:55.245 INFO:tasks.rados.rados.0.smithi042.stdout:1160: writing smithi04268048-44 from 2889444 to 3092554 tid 3 2024-09-20T19:25:55.245 INFO:tasks.rados.rados.0.smithi042.stdout:1161: snap_create 2024-09-20T19:25:55.246 INFO:tasks.rados.rados.0.smithi042.stdout:1155: expect (ObjNum 368 snap 118 seq_num 368) 2024-09-20T19:25:55.246 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:55.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:55.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:55.399 INFO:tasks.rados.rados.0.smithi042.stdout:1154: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:25:55.399 INFO:tasks.rados.rados.0.smithi042.stdout:1159: finishing write tid 1 to smithi04268048-20 2024-09-20T19:25:55.400 INFO:tasks.rados.rados.0.smithi042.stdout:1154: finishing copy_from to smithi04268048-18 2024-09-20T19:25:55.400 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:25:55.400 INFO:tasks.rados.rados.0.smithi042.stdout:1156: expect (ObjNum 401 snap 127 seq_num 401) 2024-09-20T19:25:55.400 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:25:55.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:55.590 INFO:tasks.rados.rados.0.smithi042.stdout:1158: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1158: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1158: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 425 (ObjNum 323 snap 104 seq_num 323) dirty exists 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: finishing write tid 2 to smithi04268048-20 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: finishing write tid 3 to smithi04268048-20 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: finishing write tid 4 to smithi04268048-20 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: oid 20 updating version 0 to 95 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: oid 20 version 95 is already newer than 93 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: oid 20 version 95 is already newer than 94 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 95 (ObjNum 421 snap 130 seq_num 421) dirty exists 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1159: left oid 20 (ObjNum 421 snap 130 seq_num 421) 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1160: finishing write tid 1 to smithi04268048-44 2024-09-20T19:25:55.591 INFO:tasks.rados.rados.0.smithi042.stdout:1160: finishing write tid 2 to smithi04268048-44 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: finishing write tid 3 to smithi04268048-44 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: finishing write tid 4 to smithi04268048-44 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: finishing write tid 5 to smithi04268048-44 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: oid 44 updating version 0 to 597 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: oid 44 version 597 is already newer than 596 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: oid 44 version 597 is already newer than 595 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: oid 44 updating version 597 to 598 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 598 (ObjNum 422 snap 130 seq_num 422) dirty exists 2024-09-20T19:25:55.592 INFO:tasks.rados.rados.0.smithi042.stdout:1160: left oid 44 (ObjNum 422 snap 130 seq_num 422) 2024-09-20T19:25:55.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:55.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:55.625 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:25:55.626 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:25:55.654 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:25:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:25:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:25:55.673 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:25:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:25:55.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:25:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:25:55.711 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:25:55.711 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:25:55.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.731 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:25:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:25:55.742 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:55.751 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:25:55.751 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:25:55.771 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:25:55.771 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:25:55.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:55.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 366, 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "49cb5aae-7783-11ef-baf5-efdc52797490", 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-20T19:07:00.852407+0000", 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-20T19:25:55.104218+0000", 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "last_up_change": "2024-09-20T19:17:34.145990+0000", 2024-09-20T19:25:55.944 INFO:teuthology.orchestra.run.smithi042.stdout: "last_in_change": "2024-09-20T19:17:00.522021+0000", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_num": 5806080, 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_set": [ 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nodeep-scrub", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "noscrub", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "pglog_hardlimit", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "purged_snapdirs", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "recovery_deletes", 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "sortbitwise" 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_version": 18, 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T19:25:55.945 INFO:teuthology.orchestra.run.smithi042.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_snapshot": "", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_max": 7, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "max_osd": 8, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "require_min_compat_client": "luminous", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "min_compat_client": "luminous", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "require_osd_release": "squid", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "allow_crimson": false, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "pools": [ 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 1, 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": ".mgr", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:12:46.311971+0000", 2024-09-20T19:25:55.946 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 11, 2024-09-20T19:25:55.947 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 11, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 11, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 11, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 11, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "261", 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.948 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "259", 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.949 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.950 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "options": { 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_max": 32, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_min": 1 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": {} 2024-09-20T19:25:55.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2.1800000667572021, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2.1800000667572021, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.952 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 2, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": ".rgw.root", 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:05.791948+0000", 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.953 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "71", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.955 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.956 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.957 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.5, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.5, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.5, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.5, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 3, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "datapool", 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:06.729842+0000", 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 8193, 2024-09-20T19:25:55.958 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 3, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 3, 2024-09-20T19:25:55.959 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 3, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 3, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 3, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "79", 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.960 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 2, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 79, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.961 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.962 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "rbd": {} 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2.6500000953674316, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2.6500000953674316, 2024-09-20T19:25:55.963 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 4, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.log", 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:07.353972+0000", 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.964 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.965 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "73", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:25:55.966 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.967 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.968 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2.25, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2.25, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2.25, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2.25, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 5, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.control", 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:09.310321+0000", 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.970 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.971 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "75", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.972 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.973 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.974 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.25, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.25, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.25, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.25, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 6, 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "default.rgw.meta", 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:18:11.438339+0000", 2024-09-20T19:25:55.975 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 1, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool", 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 32, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 32, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 32, 2024-09-20T19:25:55.976 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 32, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 32, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "77", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.977 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.979 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "options": { 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_bias": 4 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: "rgw": {} 2024-09-20T19:25:55.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 1.75, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 1.75, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 1.75, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 1.75, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 7, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_name": "unique_pool_0", 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "create_time": "2024-09-20T19:21:28.700742+0000", 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "flags": 8193, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "type": 1, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 3, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "min_size": 2, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_rule": 0, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "is_stretch_pool": false, 2024-09-20T19:25:55.982 INFO:teuthology.orchestra.run.smithi042.stdout: "object_hash": 2, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_autoscale_mode": "off", 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num": 16, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num": 16, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_placement_num_target": 16, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_target": 16, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_num_pending": 16, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "last_pg_merge_meta": { 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "source_pgid": "0.0", 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "ready_epoch": 0, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_started": 0, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "last_epoch_clean": 0, 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "source_version": "0'0", 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: "target_version": "0'0" 2024-09-20T19:25:55.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "last_change": "366", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend": "0", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "auid": 0, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_mode": "selfmanaged", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_seq": 131, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "snap_epoch": 366, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "pool_snaps": [], 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps": "[]", 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_bytes": 0, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "quota_max_objects": 0, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "tiers": [], 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "tier_of": -1, 2024-09-20T19:25:55.984 INFO:teuthology.orchestra.run.smithi042.stdout: "read_tier": -1, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "write_tier": -1, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_mode": "none", 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_bytes": 0, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "target_max_objects": 0, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_flush_age": 0, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "cache_min_evict_age": 0, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profile": "", 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_params": { 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "none" 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_period": 0, 2024-09-20T19:25:55.985 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_count": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "use_gmt_hitset": true, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "hit_set_search_last_n": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "grade_table": [], 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "stripe_width": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "expected_num_objects": 0, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "fast_read": false, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "options": {}, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "application_metadata": { 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "rados": {} 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "read_balance": { 2024-09-20T19:25:55.986 INFO:teuthology.orchestra.run.smithi042.stdout: "score_type": "Fair distribution", 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "score_acting": 2, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "score_stable": 2, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "optimal_score": 1, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_acting": 2, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "raw_score_stable": 2, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity_weighted": 1, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity": 1, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "osds": [ 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 0, 2024-09-20T19:25:55.987 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "fb59ab36-8bbc-4e10-bc0f-b0cca84b317b", 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0.95001220703125, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 8, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 112, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6801", 2024-09-20T19:25:55.988 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6802", 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6804", 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6803", 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2265043130 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6801/2265043130", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6802/2265043130", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6804/2265043130", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6803/2265043130", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 1, 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d", 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:55.991 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 13, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 330, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6805", 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6806", 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.993 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6808", 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6807", 2024-09-20T19:25:55.994 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 3513258758 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6805/3513258758", 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6806/3513258758", 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6808/3513258758", 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6807/3513258758", 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 2, 2024-09-20T19:25:55.995 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "12b15f1c-ba47-49cb-aefe-d99ff84d926d", 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 19, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 123, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.996 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6809", 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6810", 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6812", 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6811", 2024-09-20T19:25:55.998 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2620956901 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6809/2620956901", 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6810/2620956901", 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6812/2620956901", 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6811/2620956901", 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:55.999 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 3, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "f0b69ee0-b93b-4ab6-8a35-783b18b9c73e", 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0.95001220703125, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 28, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 261, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.000 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6813", 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6814", 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6816", 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.002 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6815", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1507333597 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6813/1507333597", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.42:6814/1507333597", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.42:6816/1507333597", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.42:6815/1507333597", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 4, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "90c5738d-14a9-415f-9249-eac0455635ad", 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 35, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 333, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:56.004 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6800", 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6801", 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.005 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6803", 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6802", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2898101120 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6800/2898101120", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6801/2898101120", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6803/2898101120", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6802/2898101120", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 5, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "e7648e52-3260-47ec-8778-35d4fa70ee27", 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 44, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 261, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6804", 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6805", 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6807", 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6806", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 4269525575 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6804/4269525575", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6805/4269525575", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4269525575", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6806/4269525575", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.011 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 6, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "34b6c8df-9cb5-4fcb-955f-69fd6520465a", 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 54, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 261, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6808", 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6809", 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6811", 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6810", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1785075253 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6808/1785075253", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6809/1785075253", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6811/1785075253", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6810/1785075253", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 7, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "uuid": "6046a58c-ccf6-41ce-9888-b0c7a771205e", 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "up": 1, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "in": 1, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 1, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_affinity": 1, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_begin": 0, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "last_clean_end": 0, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "up_from": 64, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "up_thru": 261, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "down_at": 0, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "lost_at": 0, 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6812", 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addrs": { 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6813", 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addrs": { 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6815", 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addrs": { 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.162:6814", 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 1845916231 2024-09-20T19:25:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.162:6812/1845916231", 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_addr": "172.21.15.162:6813/1845916231", 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1845916231", 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "heartbeat_front_addr": "172.21.15.162:6814/1845916231", 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "state": [ 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "exists", 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "up" 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_xinfo": [ 2024-09-20T19:25:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 0, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:25:43.790636+0000", 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 1, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:24:35.098862+0000", 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 2, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:24:36.836661+0000", 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 3, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:25:06.811192+0000", 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 4, 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:24:56.900549+0000", 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 5, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:25:24.530481+0000", 2024-09-20T19:25:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 6, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:25:05.350490+0000", 2024-09-20T19:25:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": 7, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "down_stamp": "0.000000", 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_probability": 0, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "laggy_interval": 0, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "features": 4540719139924082687, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "old_weight": 0, 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "last_purged_snaps_scrub": "2024-09-20T19:24:58.954093+0000", 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "dead_epoch": 0 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap": [ 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "2.8", 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "osds": [ 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: 2, 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: 3, 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap_items": [ 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "2.5", 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 7, 2024-09-20T19:25:56.026 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.7", 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.027 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.9", 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 5 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "pgid": "4.1f", 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "mappings": [ 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "from": 1, 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: "to": 2 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.028 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_upmap_primaries": [], 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "pg_temp": [], 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "primary_temp": [], 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "blocklist": { 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/1743761075": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3575489582": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/2546869704": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/1361717385": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/2117630454": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/409595637": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/2206296983": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:25:56.029 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3240017151": "2024-09-21T19:08:07.715726+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:6800/3764180564": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3774737654": "2024-09-21T19:07:29.716210+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3982292550": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/2784207196": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/3505349280": "2024-09-21T19:18:55.853462+0000", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "172.21.15.42:0/368716713": "2024-09-21T19:18:55.853462+0000" 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "range_blocklist": {}, 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "erasure_code_profiles": { 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "default": { 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "crush-failure-domain": "osd", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "k": "2", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "m": "1", 2024-09-20T19:25:56.030 INFO:teuthology.orchestra.run.smithi042.stdout: "plugin": "jerasure", 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "technique": "reed_sol_van" 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_snaps_queue": [ 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 7, 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "snaps": [ 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 90, 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 105, 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 109, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 116, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 5 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 122, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 2 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.032 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 126, 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 128, 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "new_removed_snaps": [ 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "pool": 7, 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "snaps": [ 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "begin": 118, 2024-09-20T19:25:56.033 INFO:teuthology.orchestra.run.smithi042.stdout: "length": 1 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "new_purged_snaps": [], 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_node_flags": {}, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "device_class_flags": {}, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": { 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode_enabled": false, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_bucket_count": 0, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "degraded_stretch_mode": 0, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "recovering_stretch_mode": 0, 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode_bucket": 0 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:25:56.034 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:25:56.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1154: done (6 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1155: done (5 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1156: done (4 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1158: done (3 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1159: done (2 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1160: done (1 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1161: done (0 left) 2024-09-20T19:25:56.130 INFO:tasks.rados.rados.0.smithi042.stdout:1162: snap_create 2024-09-20T19:25:56.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:56.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.168 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:56 smithi162 bash[24557]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.182 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:56.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:56.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:56.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:25:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.427 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[22706]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.109398+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.594866+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.42:0/3192054816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:25:56.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.624283+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.625105+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.630214+0000 mon.a (mon.0) 1400 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:55.941584+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? v1:172.21.15.42:0/2861083829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:56 smithi042 bash[32075]: audit 2024-09-20T19:25:56.107720+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]} : dispatch 2024-09-20T19:25:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:56.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:56.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:56.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:56.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:56.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:56.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:56.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:56.962 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:56.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:57.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:57.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:57.135 INFO:tasks.rados.rados.0.smithi042.stdout:1162: done (0 left) 2024-09-20T19:25:57.135 INFO:tasks.rados.rados.0.smithi042.stdout:1163: delete oid 40 current snap is 132 2024-09-20T19:25:57.178 INFO:tasks.rados.rados.0.smithi042.stdout:1163: done (0 left) 2024-09-20T19:25:57.178 INFO:tasks.rados.rados.0.smithi042.stdout:1164: snap_remove snap 92 2024-09-20T19:25:57.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:57.208 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.208 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.210 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.210 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.210 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[22706]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.210 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.211 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.212 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:57 smithi042 bash[32075]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:57.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:57.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.996661+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:55.997096+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:56.054695+0000 mgr.y (mgr.24491) 306 : cluster [DBG] pgmap v488: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 448 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: audit 2024-09-20T19:25:56.125789+0000 mon.a (mon.0) 1403 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:57 smithi162 bash[24557]: cluster 2024-09-20T19:25:56.128871+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-20T19:25:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:57.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:57.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:57.580 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:57.608 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:57.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:57.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:57.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:57.868 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:25:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:57.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:25:57.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:57.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:58.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:58.114 INFO:tasks.rados.rados.0.smithi042.stdout:1164: done (0 left) 2024-09-20T19:25:58.114 INFO:tasks.rados.rados.0.smithi042.stdout:1165: snap_create 2024-09-20T19:25:58.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:58.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:58.252 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.252 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:58 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:25:58.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:58.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: cluster 2024-09-20T19:25:57.135123+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: audit 2024-09-20T19:25:57.182049+0000 mon.a (mon.0) 1406 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:25:58.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:58 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.113475+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-20T19:25:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:58.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:58.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:58.555 INFO:tasks.thrashosds.thrasher:j is {'epoch': 366, 'fsid': '49cb5aae-7783-11ef-baf5-efdc52797490', 'created': '2024-09-20T19:07:00.852407+0000', 'modified': '2024-09-20T19:25:55.104218+0000', 'last_up_change': '2024-09-20T19:17:34.145990+0000', 'last_in_change': '2024-09-20T19:17:00.522021+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T19:12:46.311971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '261', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '259', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T19:18:05.791948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-20T19:18:06.729842+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 79, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T19:18:07.353972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T19:18:09.310321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T19:18:11.438339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T19:21:28.700742+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '366', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 131, 'snap_epoch': 366, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'fb59ab36-8bbc-4e10-bc0f-b0cca84b317b', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6801', 'nonce': 2265043130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6802', 'nonce': 2265043130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6804', 'nonce': 2265043130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6803', 'nonce': 2265043130}]}, 'public_addr': '172.21.15.42:6801/2265043130', 'cluster_addr': '172.21.15.42:6802/2265043130', 'heartbeat_back_addr': '172.21.15.42:6804/2265043130', 'heartbeat_front_addr': '172.21.15.42:6803/2265043130', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6805', 'nonce': 3513258758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6806', 'nonce': 3513258758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6808', 'nonce': 3513258758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6807', 'nonce': 3513258758}]}, 'public_addr': '172.21.15.42:6805/3513258758', 'cluster_addr': '172.21.15.42:6806/3513258758', 'heartbeat_back_addr': '172.21.15.42:6808/3513258758', 'heartbeat_front_addr': '172.21.15.42:6807/3513258758', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '12b15f1c-ba47-49cb-aefe-d99ff84d926d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6809', 'nonce': 2620956901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6810', 'nonce': 2620956901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6812', 'nonce': 2620956901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6811', 'nonce': 2620956901}]}, 'public_addr': '172.21.15.42:6809/2620956901', 'cluster_addr': '172.21.15.42:6810/2620956901', 'heartbeat_back_addr': '172.21.15.42:6812/2620956901', 'heartbeat_front_addr': '172.21.15.42:6811/2620956901', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f0b69ee0-b93b-4ab6-8a35-783b18b9c73e', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6813', 'nonce': 1507333597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6814', 'nonce': 1507333597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6816', 'nonce': 1507333597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.42:6815', 'nonce': 1507333597}]}, 'public_addr': '172.21.15.42:6813/1507333597', 'cluster_addr': '172.21.15.42:6814/1507333597', 'heartbeat_back_addr': '172.21.15.42:6816/1507333597', 'heartbeat_front_addr': '172.21.15.42:6815/1507333597', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '90c5738d-14a9-415f-9249-eac0455635ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 2898101120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 2898101120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 2898101120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 2898101120}]}, 'public_addr': '172.21.15.162:6800/2898101120', 'cluster_addr': '172.21.15.162:6801/2898101120', 'heartbeat_back_addr': '172.21.15.162:6803/2898101120', 'heartbeat_front_addr': '172.21.15.162:6802/2898101120', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e7648e52-3260-47ec-8778-35d4fa70ee27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 4269525575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4269525575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4269525575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 4269525575}]}, 'public_addr': '172.21.15.162:6804/4269525575', 'cluster_addr': '172.21.15.162:6805/4269525575', 'heartbeat_back_addr': '172.21.15.162:6807/4269525575', 'heartbeat_front_addr': '172.21.15.162:6806/4269525575', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34b6c8df-9cb5-4fcb-955f-69fd6520465a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 1785075253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1785075253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1785075253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 1785075253}]}, 'public_addr': '172.21.15.162:6808/1785075253', 'cluster_addr': '172.21.15.162:6809/1785075253', 'heartbeat_back_addr': '172.21.15.162:6811/1785075253', 'heartbeat_front_addr': '172.21.15.162:6810/1785075253', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6046a58c-ccf6-41ce-9888-b0c7a771205e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 64, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 1845916231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1845916231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1845916231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 1845916231}]}, 'public_addr': '172.21.15.162:6812/1845916231', 'cluster_addr': '172.21.15.162:6813/1845916231', 'heartbeat_back_addr': '172.21.15.162:6815/1845916231', 'heartbeat_front_addr': '172.21.15.162:6814/1845916231', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:25:43.790636+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:24:35.098862+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:24:36.836661+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:25:06.811192+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:24:56.900549+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:25:24.530481+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:25:05.350490+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T19:24:58.954093+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.8', 'osds': [2, 3, 1]}], 'pg_upmap_items': [{'pgid': '2.5', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.42:0/1743761075': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3575489582': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:6800/2546869704': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:0/1361717385': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/2117630454': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/409595637': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:6800/2206296983': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:0/3240017151': '2024-09-21T19:08:07.715726+0000', '172.21.15.42:6800/3764180564': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3774737654': '2024-09-21T19:07:29.716210+0000', '172.21.15.42:0/3982292550': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/2784207196': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/3505349280': '2024-09-21T19:18:55.853462+0000', '172.21.15.42:0/368716713': '2024-09-21T19:18:55.853462+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 90, 'length': 1}, {'begin': 105, 'length': 1}, {'begin': 109, 'length': 1}, {'begin': 116, 'length': 5}, {'begin': 122, 'length': 2}, {'begin': 126, 'length': 1}, {'begin': 128, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 118, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T19:25:58.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:25:58.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:58.662 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:25:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:25:58.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:58.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:58.842 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:25:58.842 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:25:58.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:58.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:58.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:59.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:59.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:25:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:1165: done (0 left) 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:1166: rollback oid 39 current snap is 133 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 125 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:1167: copy_from oid 28 from oid 41 current snap is 133 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:1168: write oid 31 current snap is 133 2024-09-20T19:25:59.119 INFO:tasks.rados.rados.0.smithi042.stdout:1168: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-20T19:25:59.141 INFO:tasks.rados.rados.0.smithi042.stdout:1168: writing smithi04268048-31 from 726168 to 1431159 tid 1 2024-09-20T19:25:59.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1168: writing smithi04268048-31 from 1895071 to 2578545 tid 2 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1169: read oid 38 snap -1 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1169: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1170: write oid 6 current snap is 133 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1166: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1166: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 429 (ObjNum 177 snap 0 seq_num 1415899648) dirty dne 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1167: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1168: finishing write tid 1 to smithi04268048-31 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:1167: finishing copy_from to smithi04268048-28 2024-09-20T19:25:59.157 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 255 (ObjNum 292 snap 93 seq_num 292) dirty exists 2024-09-20T19:25:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1170: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-20T19:25:59.175 INFO:tasks.rados.rados.0.smithi042.stdout:1170: writing smithi04268048-6 from 742975 to 1523215 tid 1 2024-09-20T19:25:59.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:59.182 INFO:tasks.rados.rados.0.smithi042.stdout:1170: writing smithi04268048-6 from 1953553 to 2380685 tid 2 2024-09-20T19:25:59.186 INFO:tasks.rados.rados.0.smithi042.stdout:1170: writing smithi04268048-6 from 2971713 to 3142975 tid 3 2024-09-20T19:25:59.187 INFO:tasks.rados.rados.0.smithi042.stdout:1166: done (4 left) 2024-09-20T19:25:59.187 INFO:tasks.rados.rados.0.smithi042.stdout:1167: done (3 left) 2024-09-20T19:25:59.187 INFO:tasks.rados.rados.0.smithi042.stdout:1171: snap_remove snap 123 2024-09-20T19:25:59.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:59.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[32075]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.293 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[32075]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.294 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.294 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.294 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[22706]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.294 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:25:59 smithi042 bash[22706]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:25:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:25:59.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:59 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:59 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.055368+0000 mgr.y (mgr.24491) 307 : cluster [DBG] pgmap v491: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:25:59.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:59 smithi162 bash[24557]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:25:59 smithi162 bash[24557]: cluster 2024-09-20T19:25:59.119139+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-20T19:25:59.308 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:59.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:59.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:25:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:25:59.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:25:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:25:59.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:25:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:25:59.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:25:59.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:59.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:25:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:25:59.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:59.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:25:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:59.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:00.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:00.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:00.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:1171: done (3 left) 2024-09-20T19:26:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:1172: write oid 9 current snap is 133 2024-09-20T19:26:00.173 INFO:tasks.rados.rados.0.smithi042.stdout:1172: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-20T19:26:00.177 INFO:tasks.rados.rados.0.smithi042.stdout:1172: writing smithi04268048-9 from 759782 to 1331624 tid 1 2024-09-20T19:26:00.185 INFO:tasks.rados.rados.0.smithi042.stdout:1172: writing smithi04268048-9 from 2012035 to 2699178 tid 2 2024-09-20T19:26:00.196 INFO:tasks.rados.rados.0.smithi042.stdout:1168: finishing write tid 2 to smithi04268048-31 2024-09-20T19:26:00.196 INFO:tasks.rados.rados.0.smithi042.stdout:1168: finishing write tid 3 to smithi04268048-31 2024-09-20T19:26:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:1169: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:26:00.197 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1094608 2024-09-20T19:26:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:00.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:00.230 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:00.264 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 666080 2024-09-20T19:26:00.266 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:1168: finishing write tid 4 to smithi04268048-31 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:1168: oid 31 updating version 0 to 391 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:1168: oid 31 updating version 391 to 392 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:1168: oid 31 version 392 is already newer than 390 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 392 (ObjNum 424 snap 133 seq_num 424) dirty exists 2024-09-20T19:26:00.296 INFO:tasks.rados.rados.0.smithi042.stdout:1168: left oid 31 (ObjNum 424 snap 133 seq_num 424) 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: finishing write tid 1 to smithi04268048-6 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: finishing write tid 2 to smithi04268048-6 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: finishing write tid 3 to smithi04268048-6 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: finishing write tid 4 to smithi04268048-6 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: finishing write tid 5 to smithi04268048-6 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: oid 6 updating version 0 to 396 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: oid 6 updating version 396 to 397 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: oid 6 version 397 is already newer than 394 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: oid 6 version 397 is already newer than 395 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 397 (ObjNum 425 snap 133 seq_num 425) dirty exists 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1170: left oid 6 (ObjNum 425 snap 133 seq_num 425) 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1172: finishing write tid 1 to smithi04268048-9 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1172: finishing write tid 2 to smithi04268048-9 2024-09-20T19:26:00.297 INFO:tasks.rados.rados.0.smithi042.stdout:1172: finishing write tid 3 to smithi04268048-9 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1172: finishing write tid 4 to smithi04268048-9 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1172: oid 9 updating version 0 to 595 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1172: oid 9 updating version 595 to 596 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1172: oid 9 updating version 596 to 597 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 597 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1172: left oid 9 (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1168: done (3 left) 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1169: done (2 left) 2024-09-20T19:26:00.298 INFO:tasks.rados.rados.0.smithi042.stdout:1170: done (1 left) 2024-09-20T19:26:00.299 INFO:tasks.rados.rados.0.smithi042.stdout:1172: done (0 left) 2024-09-20T19:26:00.299 INFO:tasks.rados.rados.0.smithi042.stdout:1173: delete oid 6 current snap is 133 2024-09-20T19:26:00.301 INFO:tasks.rados.rados.0.smithi042.stdout:1173: done (0 left) 2024-09-20T19:26:00.301 INFO:tasks.rados.rados.0.smithi042.stdout:1174: write oid 29 current snap is 133 2024-09-20T19:26:00.301 INFO:tasks.rados.rados.0.smithi042.stdout:1174: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-20T19:26:00.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:00 smithi162 bash[24557]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.311 INFO:tasks.rados.rados.0.smithi042.stdout:1174: writing smithi04268048-29 from 793396 to 1515736 tid 1 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1174: writing smithi04268048-29 from 2128999 to 2819811 tid 2 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1174: finishing write tid 1 to smithi04268048-29 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1175: read oid 16 snap 129 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1175: expect (ObjNum 369 snap 118 seq_num 369) 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1176: write oid 39 current snap is 133 2024-09-20T19:26:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:1176: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-20T19:26:00.330 INFO:tasks.rados.rados.0.smithi042.stdout:1176: writing smithi04268048-39 from 410203 to 1207792 tid 1 2024-09-20T19:26:00.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.336 INFO:tasks.rados.rados.0.smithi042.stdout:1176: writing smithi04268048-39 from 1787481 to 2221951 tid 2 2024-09-20T19:26:00.342 INFO:tasks.rados.rados.0.smithi042.stdout:1176: writing smithi04268048-39 from 2681405 to 3210203 tid 3 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1177: read oid 14 snap -1 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1177: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1178: read oid 35 snap -1 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1178: expect (ObjNum 417 snap 129 seq_num 417) 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1179: copy_from oid 44 from oid 9 current snap is 133 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: finishing write tid 2 to smithi04268048-29 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: finishing write tid 3 to smithi04268048-29 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: finishing write tid 4 to smithi04268048-29 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: oid 29 updating version 0 to 555 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: oid 29 updating version 555 to 557 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: oid 29 version 557 is already newer than 556 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 557 (ObjNum 428 snap 133 seq_num 428) dirty exists 2024-09-20T19:26:00.343 INFO:tasks.rados.rados.0.smithi042.stdout:1174: left oid 29 (ObjNum 428 snap 133 seq_num 428) 2024-09-20T19:26:00.344 INFO:tasks.rados.rados.0.smithi042.stdout:1175: expect (ObjNum 369 snap 118 seq_num 369) 2024-09-20T19:26:00.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:00.394 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:1176: finishing write tid 1 to smithi04268048-39 2024-09-20T19:26:00.418 INFO:tasks.rados.rados.0.smithi042.stdout:1176: finishing write tid 2 to smithi04268048-39 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1176: finishing write tid 3 to smithi04268048-39 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1174: done (5 left) 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1175: done (4 left) 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1180: read oid 31 snap -1 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1180: expect (ObjNum 424 snap 133 seq_num 424) 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1181: snap_create 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1176: finishing write tid 4 to smithi04268048-39 2024-09-20T19:26:00.419 INFO:tasks.rados.rados.0.smithi042.stdout:1178: expect (ObjNum 417 snap 129 seq_num 417) 2024-09-20T19:26:00.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[32075]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.249227+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: cluster 2024-09-20T19:25:58.249736+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:00 smithi042 bash[22706]: audit 2024-09-20T19:25:59.239684+0000 mgr.y (mgr.24491) 308 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:00.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.490 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:00.572 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1744026 2024-09-20T19:26:00.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: finishing write tid 5 to smithi04268048-39 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: oid 39 updating version 0 to 430 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: oid 39 updating version 430 to 433 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: oid 39 version 433 is already newer than 431 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: oid 39 version 433 is already newer than 432 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 433 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1176: left oid 39 (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:1177: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:26:00.609 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:00.656 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2413830 2024-09-20T19:26:00.673 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:00.711 INFO:tasks.rados.rados.0.smithi042.stdout:1179: finishing copy_from racing read to smithi04268048-44 2024-09-20T19:26:00.711 INFO:tasks.rados.rados.0.smithi042.stdout:1179: finishing copy_from to smithi04268048-44 2024-09-20T19:26:00.711 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 597 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:00.711 INFO:tasks.rados.rados.0.smithi042.stdout:1180: expect (ObjNum 424 snap 133 seq_num 424) 2024-09-20T19:26:00.711 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:00.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:00.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:00.804 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3031602 2024-09-20T19:26:00.811 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 230277 2024-09-20T19:26:00.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:00.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:01.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:01.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:01.140 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1176: done (5 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1177: done (4 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1178: done (3 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1179: done (2 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1180: done (1 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1181: done (0 left) 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1182: rollback oid 27 current snap is 134 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 128 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1183: write oid 11 current snap is 134 2024-09-20T19:26:01.177 INFO:tasks.rados.rados.0.smithi042.stdout:1183: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-20T19:26:01.178 INFO:tasks.rados.rados.0.smithi042.stdout:1183: writing smithi04268048-11 from 427010 to 899848 tid 1 2024-09-20T19:26:01.187 INFO:tasks.rados.rados.0.smithi042.stdout:1183: writing smithi04268048-11 from 1329610 to 2024091 tid 2 2024-09-20T19:26:01.198 INFO:tasks.rados.rados.0.smithi042.stdout:1183: writing smithi04268048-11 from 2708828 to 3227010 tid 3 2024-09-20T19:26:01.198 INFO:tasks.rados.rados.0.smithi042.stdout:1184: copy_from oid 3 from oid 32 current snap is 134 2024-09-20T19:26:01.198 INFO:tasks.rados.rados.0.smithi042.stdout:1182: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:26:01.198 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 389 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:26:01.199 INFO:tasks.rados.rados.0.smithi042.stdout:1183: finishing write tid 1 to smithi04268048-11 2024-09-20T19:26:01.199 INFO:tasks.rados.rados.0.smithi042.stdout:1183: finishing write tid 2 to smithi04268048-11 2024-09-20T19:26:01.199 INFO:tasks.rados.rados.0.smithi042.stdout:1182: done (2 left) 2024-09-20T19:26:01.199 INFO:tasks.rados.rados.0.smithi042.stdout:1185: delete oid 37 current snap is 134 2024-09-20T19:26:01.201 INFO:tasks.rados.rados.0.smithi042.stdout:1185: done (2 left) 2024-09-20T19:26:01.201 INFO:tasks.rados.rados.0.smithi042.stdout:1186: snap_create 2024-09-20T19:26:01.203 INFO:tasks.rados.rados.0.smithi042.stdout:1183: finishing write tid 3 to smithi04268048-11 2024-09-20T19:26:01.203 INFO:tasks.rados.rados.0.smithi042.stdout:1183: finishing write tid 4 to smithi04268048-11 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: finishing write tid 5 to smithi04268048-11 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: oid 11 updating version 0 to 602 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: oid 11 updating version 602 to 603 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: oid 11 version 603 is already newer than 601 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: oid 11 version 603 is already newer than 600 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 603 (ObjNum 430 snap 134 seq_num 430) dirty exists 2024-09-20T19:26:01.204 INFO:tasks.rados.rados.0.smithi042.stdout:1183: left oid 11 (ObjNum 430 snap 134 seq_num 430) 2024-09-20T19:26:01.211 INFO:tasks.rados.rados.0.smithi042.stdout:1184: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:26:01.215 INFO:tasks.rados.rados.0.smithi042.stdout:1184: finishing copy_from to smithi04268048-3 2024-09-20T19:26:01.215 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 483 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:26:01.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:01.269 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:01.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:01.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:01 smithi162 bash[24557]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:01 smithi162 bash[24557]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:01 smithi162 bash[24557]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:01 smithi162 bash[24557]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:01.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[32075]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[32075]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[32075]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[32075]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[22706]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[22706]: cluster 2024-09-20T19:26:00.056102+0000 mgr.y (mgr.24491) 309 : cluster [DBG] pgmap v494: 158 pgs: 1 peering, 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 148 active+clean; 454 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[22706]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:01 smithi042 bash[22706]: cluster 2024-09-20T19:26:00.167640+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-20T19:26:01.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:01.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:01.648 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:01.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:01.873 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:01.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:01.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:01.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:02.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:02.185 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:1183: done (2 left) 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:1184: done (1 left) 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:1186: done (0 left) 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:1187: rollback oid 30 current snap is 135 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 129 2024-09-20T19:26:02.186 INFO:tasks.rados.rados.0.smithi042.stdout:1188: snap_create 2024-09-20T19:26:02.191 INFO:tasks.rados.rados.0.smithi042.stdout:1187: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:26:02.192 INFO:tasks.rados.rados.0.smithi042.stdout:1187: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:26:02.192 INFO:tasks.rados.rados.0.smithi042.stdout:1187: finishing rollback tid 2 to smithi04268048-30 2024-09-20T19:26:02.192 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 652 (ObjNum 416 snap 129 seq_num 416) dirty exists 2024-09-20T19:26:02.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.275 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:02 smithi162 bash[24557]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:02.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.423 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[32075]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.168755+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.558997+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: cluster 2024-09-20T19:26:01.560480+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: audit 2024-09-20T19:26:01.862027+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.42:0/241267077' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.425 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.425 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:02 smithi042 bash[22706]: audit 2024-09-20T19:26:01.862519+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25139 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:26:02.438 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:02.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:02.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:02.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:02.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:02.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:02.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:02.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:03.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:03.116 INFO:tasks.rados.rados.0.smithi042.stdout:1187: done (1 left) 2024-09-20T19:26:03.116 INFO:tasks.rados.rados.0.smithi042.stdout:1188: done (0 left) 2024-09-20T19:26:03.116 INFO:tasks.rados.rados.0.smithi042.stdout:1189: read oid 34 snap -1 2024-09-20T19:26:03.116 INFO:tasks.rados.rados.0.smithi042.stdout:1189: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:26:03.116 INFO:tasks.rados.rados.0.smithi042.stdout:1190: write oid 22 current snap is 136 2024-09-20T19:26:03.117 INFO:tasks.rados.rados.0.smithi042.stdout:1190: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-20T19:26:03.128 INFO:tasks.rados.rados.0.smithi042.stdout:1190: writing smithi04268048-22 from 460624 to 1083960 tid 1 2024-09-20T19:26:03.141 INFO:tasks.rados.rados.0.smithi042.stdout:1190: writing smithi04268048-22 from 1846574 to 2544724 tid 2 2024-09-20T19:26:03.147 INFO:tasks.rados.rados.0.smithi042.stdout:1190: writing smithi04268048-22 from 3163674 to 3260624 tid 3 2024-09-20T19:26:03.147 INFO:tasks.rados.rados.0.smithi042.stdout:1191: rollback oid 3 current snap is 136 2024-09-20T19:26:03.147 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 133 2024-09-20T19:26:03.147 INFO:tasks.rados.rados.0.smithi042.stdout:1189: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:26:03.192 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:03.214 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:03.264 INFO:tasks.rados.rados.0.smithi042.stdout:1190: finishing write tid 1 to smithi04268048-22 2024-09-20T19:26:03.264 INFO:tasks.rados.rados.0.smithi042.stdout:1190: finishing write tid 2 to smithi04268048-22 2024-09-20T19:26:03.264 INFO:tasks.rados.rados.0.smithi042.stdout:1189: done (2 left) 2024-09-20T19:26:03.264 INFO:tasks.rados.rados.0.smithi042.stdout:1192: snap_remove snap 106 2024-09-20T19:26:03.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:03.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:03 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.057755+0000 mgr.y (mgr.24491) 310 : cluster [DBG] pgmap v497: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.172454+0000 mon.a (mon.0) 1412 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: audit 2024-09-20T19:26:02.179256+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25139 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:26:03.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.185776+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-20T19:26:03.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:03 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.115185+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-20T19:26:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:03.551 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:03.573 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:03.593 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:03.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:03.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:03.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:03.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:03.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:04.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:04.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:04.201 INFO:tasks.rados.rados.0.smithi042.stdout:1192: done (2 left) 2024-09-20T19:26:04.201 INFO:tasks.rados.rados.0.smithi042.stdout:1193: read oid 25 snap -1 2024-09-20T19:26:04.201 INFO:tasks.rados.rados.0.smithi042.stdout:1193: expect (ObjNum 302 snap 95 seq_num 302) 2024-09-20T19:26:04.201 INFO:tasks.rados.rados.0.smithi042.stdout:1194: read oid 47 snap -1 2024-09-20T19:26:04.201 INFO:tasks.rados.rados.0.smithi042.stdout:1194: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1195: rollback oid 28 current snap is 136 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 122 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1196: copy_from oid 17 from oid 30 current snap is 136 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1197: snap_create 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1191: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1191: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1191: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 486 (ObjNum 323 snap 104 seq_num 323) dirty exists 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1190: finishing write tid 3 to smithi04268048-22 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1190: finishing write tid 4 to smithi04268048-22 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1190: finishing write tid 5 to smithi04268048-22 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1190: oid 22 updating version 0 to 391 2024-09-20T19:26:04.202 INFO:tasks.rados.rados.0.smithi042.stdout:1190: oid 22 version 391 is already newer than 390 2024-09-20T19:26:04.203 INFO:tasks.rados.rados.0.smithi042.stdout:1190: oid 22 version 391 is already newer than 388 2024-09-20T19:26:04.203 INFO:tasks.rados.rados.0.smithi042.stdout:1190: oid 22 version 391 is already newer than 389 2024-09-20T19:26:04.203 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 391 (ObjNum 432 snap 136 seq_num 432) dirty exists 2024-09-20T19:26:04.203 INFO:tasks.rados.rados.0.smithi042.stdout:1190: left oid 22 (ObjNum 432 snap 136 seq_num 432) 2024-09-20T19:26:04.215 INFO:tasks.rados.rados.0.smithi042.stdout:1194: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:26:04.217 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 776922 2024-09-20T19:26:04.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:04.345 INFO:tasks.rados.rados.0.smithi042.stdout:1193: expect (ObjNum 302 snap 95 seq_num 302) 2024-09-20T19:26:04.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.392 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.393 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.394 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:04 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:04.411 INFO:tasks.rados.rados.0.smithi042.stdout:1195: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:26:04.411 INFO:tasks.rados.rados.0.smithi042.stdout:1195: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:26:04.412 INFO:tasks.rados.rados.0.smithi042.stdout:1195: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:26:04.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 351 (ObjNum 365 snap 117 seq_num 365) dirty exists 2024-09-20T19:26:04.412 INFO:tasks.rados.rados.0.smithi042.stdout:1196: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:26:04.412 INFO:tasks.rados.rados.0.smithi042.stdout:1196: finishing copy_from to smithi04268048-17 2024-09-20T19:26:04.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 652 (ObjNum 416 snap 129 seq_num 416) dirty exists 2024-09-20T19:26:04.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:02.985465+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.006811+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.064343+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.e scrub starts 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.523 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:04 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.068229+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.e scrub ok 2024-09-20T19:26:04.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:04.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:04.773 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.803 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:04.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:04.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:05.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:05.065 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:05.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:26:05.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.187 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1190: done (6 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1191: done (5 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1193: done (4 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1194: done (3 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1195: done (2 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1196: done (1 left) 2024-09-20T19:26:05.195 INFO:tasks.rados.rados.0.smithi042.stdout:1197: done (0 left) 2024-09-20T19:26:05.196 INFO:tasks.rados.rados.0.smithi042.stdout:1198: copy_from oid 21 from oid 46 current snap is 137 2024-09-20T19:26:05.196 INFO:tasks.rados.rados.0.smithi042.stdout:1199: snap_create 2024-09-20T19:26:05.210 INFO:tasks.rados.rados.0.smithi042.stdout:1198: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:26:05.211 INFO:tasks.rados.rados.0.smithi042.stdout:1198: finishing copy_from to smithi04268048-21 2024-09-20T19:26:05.211 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 487 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:26:05.291 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:05.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:05.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.313 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[32075]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:05 smithi042 bash[22706]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:05.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:05.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:05.466 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:26:05.469 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:26:05.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.877142+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:26:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:03.880108+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:26:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.058297+0000 mgr.y (mgr.24491) 311 : cluster [DBG] pgmap v500: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 27 B/s, 1 objects/s recovering 2024-09-20T19:26:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:05 smithi162 bash[24557]: cluster 2024-09-20T19:26:04.199575+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-20T19:26:05.565 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:05.650 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:05.651 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:05.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:05.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:05.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:05.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:05.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:05.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:06.049 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:06.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:06.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:06.201 INFO:tasks.rados.rados.0.smithi042.stdout:1198: done (1 left) 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1199: done (0 left) 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1200: read oid 24 snap -1 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1200: expect deleted 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1201: read oid 36 snap -1 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1201: expect (ObjNum 354 snap 115 seq_num 354) 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1202: read oid 4 snap -1 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1202: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:06.202 INFO:tasks.rados.rados.0.smithi042.stdout:1203: snap_create 2024-09-20T19:26:06.205 INFO:tasks.rados.rados.0.smithi042.stdout:1201: expect (ObjNum 354 snap 115 seq_num 354) 2024-09-20T19:26:06.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:06.219 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2388860 2024-09-20T19:26:06.278 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2444455 2024-09-20T19:26:06.292 INFO:tasks.rados.rados.0.smithi042.stdout:1202: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:06.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:06.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:06.349 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:06.409 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:06.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:06.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:06 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:05.195039+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: audit 2024-09-20T19:26:05.456652+0000 mgr.y (mgr.24491) 312 : audit [DBG] from='client.15288 v1:172.21.15.42:0/3640843662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.085153+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:06 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.201819+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-20T19:26:06.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:06.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:06.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:06.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:06.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:06.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:06.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:07.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:07.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:07.204 INFO:tasks.rados.rados.0.smithi042.stdout:1200: done (3 left) 2024-09-20T19:26:07.205 INFO:tasks.rados.rados.0.smithi042.stdout:1201: done (2 left) 2024-09-20T19:26:07.205 INFO:tasks.rados.rados.0.smithi042.stdout:1202: done (1 left) 2024-09-20T19:26:07.205 INFO:tasks.rados.rados.0.smithi042.stdout:1203: done (0 left) 2024-09-20T19:26:07.205 INFO:tasks.rados.rados.0.smithi042.stdout:1204: write oid 12 current snap is 139 2024-09-20T19:26:07.205 INFO:tasks.rados.rados.0.smithi042.stdout:1204: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-20T19:26:07.224 INFO:tasks.rados.rados.0.smithi042.stdout:1204: writing smithi04268048-12 from 477431 to 1176016 tid 1 2024-09-20T19:26:07.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:07.244 INFO:tasks.rados.rados.0.smithi042.stdout:1204: writing smithi04268048-12 from 1905056 to 2346864 tid 2 2024-09-20T19:26:07.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:07.254 INFO:tasks.rados.rados.0.smithi042.stdout:1204: writing smithi04268048-12 from 3074744 to 3277431 tid 3 2024-09-20T19:26:07.254 INFO:tasks.rados.rados.0.smithi042.stdout:1205: read oid 4 snap 128 2024-09-20T19:26:07.254 INFO:tasks.rados.rados.0.smithi042.stdout:1205: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:07.254 INFO:tasks.rados.rados.0.smithi042.stdout:1206: snap_remove snap 101 2024-09-20T19:26:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:07.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:07.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.366 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.367 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:07 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:07.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:07.509 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:07.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.058973+0000 mgr.y (mgr.24491) 313 : cluster [DBG] pgmap v503: 158 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 149 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:07.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.088971+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.673910+0000 osd.5 (osd.5) 139 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.678007+0000 osd.5 (osd.5) 140 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:07 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.204525+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-20T19:26:07.559 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-20T19:26:05.191790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":454720922,"num_objects":401,"num_object_clones":152,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":11,"num_read":3708,"num_read_kb":1901418,"num_write":2227,"num_write_kb":692413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":8797226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6599,"ondisk_log_size":6599,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1311548,"kb_used_data":1032912,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":748649668,"statfs":{"total":767960285184,"available":766617260032,"internally_reserved":0,"allocated":1057701888,"data_stored":1354571172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21022784,"num_objects":-12,"num_object_clones":-12,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12,"num_whiteouts":0,"num_read":60,"num_read_kb":49086,"num_write":34,"num_write_kb":16615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029480"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.164893+0000","last_change":"2024-09-20T19:24:03.086154+0000","last_active":"2024-09-20T19:26:01.164893+0000","last_peered":"2024-09-20T19:26:01.164893+0000","last_clean":"2024-09-20T19:26:01.164893+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:26:01.164893+0000","last_undegraded":"2024-09-20T19:26:01.164893+0000","last_fullsized":"2024-09-20T19:26:01.164893+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:03.085896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.085896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187527+0000","last_change":"2024-09-20T19:25:32.967210+0000","last_active":"2024-09-20T19:26:01.187527+0000","last_peered":"2024-09-20T19:26:01.187527+0000","last_clean":"2024-09-20T19:26:01.187527+0000","last_became_active":"2024-09-20T19:18:07.268694+0000","last_became_peered":"2024-09-20T19:18:07.268694+0000","last_unstale":"2024-09-20T19:26:01.187527+0000","last_undegraded":"2024-09-20T19:26:01.187527+0000","last_fullsized":"2024-09-20T19:26:01.187527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.967081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.967081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:46.384162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"77'15","reported_seq":670,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166534+0000","last_change":"2024-09-20T19:25:18.620564+0000","last_active":"2024-09-20T19:26:01.166534+0000","last_peered":"2024-09-20T19:26:01.166534+0000","last_clean":"2024-09-20T19:26:01.166534+0000","last_became_active":"2024-09-20T19:18:09.289998+0000","last_became_peered":"2024-09-20T19:18:09.289998+0000","last_unstale":"2024-09-20T19:26:01.166534+0000","last_undegraded":"2024-09-20T19:26:01.166534+0000","last_fullsized":"2024-09-20T19:26:01.166534+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:25:18.620430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.620430+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:27.558193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":634,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172208+0000","last_change":"2024-09-20T19:23:50.236531+0000","last_active":"2024-09-20T19:26:01.172208+0000","last_peered":"2024-09-20T19:26:01.172208+0000","last_clean":"2024-09-20T19:26:01.172208+0000","last_became_active":"2024-09-20T19:18:11.317348+0000","last_became_peered":"2024-09-20T19:18:11.317348+0000","last_unstale":"2024-09-20T19:26:01.172208+0000","last_undegraded":"2024-09-20T19:26:01.172208+0000","last_fullsized":"2024-09-20T19:26:01.172208+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.236477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.236477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:43.351372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":649,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.170457+0000","last_change":"2024-09-20T19:25:27.631260+0000","last_active":"2024-09-20T19:26:01.170457+0000","last_peered":"2024-09-20T19:26:01.170457+0000","last_clean":"2024-09-20T19:26:01.170457+0000","last_became_active":"2024-09-20T19:18:07.273524+0000","last_became_peered":"2024-09-20T19:18:07.273524+0000","last_unstale":"2024-09-20T19:26:01.170457+0000","last_undegraded":"2024-09-20T19:26:01.170457+0000","last_fullsized":"2024-09-20T19:26:01.170457+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.631202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.631202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:29.240229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"77'9","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172363+0000","last_change":"2024-09-20T19:23:47.200445+0000","last_active":"2024-09-20T19:26:01.172363+0000","last_peered":"2024-09-20T19:26:01.172363+0000","last_clean":"2024-09-20T19:26:01.172363+0000","last_became_active":"2024-09-20T19:18:09.305971+0000","last_became_peered":"2024-09-20T19:18:09.305971+0000","last_unstale":"2024-09-20T19:26:01.172363+0000","last_undegraded":"2024-09-20T19:26:01.172363+0000","last_fullsized":"2024-09-20T19:26:01.172363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:23:47.200185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.200185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:37.337731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.168121+0000","last_change":"2024-09-20T19:25:23.446713+0000","last_active":"2024-09-20T19:26:01.168121+0000","last_peered":"2024-09-20T19:26:01.168121+0000","last_clean":"2024-09-20T19:26:01.168121+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:26:01.168121+0000","last_undegraded":"2024-09-20T19:26:01.168121+0000","last_fullsized":"2024-09-20T19:26:01.168121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:23.446585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:23.446585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:25.050315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172469+0000","last_change":"2024-09-20T19:24:26.473044+0000","last_active":"2024-09-20T19:26:01.172469+0000","last_peered":"2024-09-20T19:26:01.172469+0000","last_clean":"2024-09-20T19:26:01.172469+0000","last_became_active":"2024-09-20T19:18:13.365786+0000","last_became_peered":"2024-09-20T19:18:13.365786+0000","last_unstale":"2024-09-20T19:26:01.172469+0000","last_undegraded":"2024-09-20T19:26:01.172469+0000","last_fullsized":"2024-09-20T19:26:01.172469+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:26.472975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:26.472975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186481+0000","last_change":"2024-09-20T19:24:46.297180+0000","last_active":"2024-09-20T19:26:01.186481+0000","last_peered":"2024-09-20T19:26:01.186481+0000","last_clean":"2024-09-20T19:26:01.186481+0000","last_became_active":"2024-09-20T19:18:07.274379+0000","last_became_peered":"2024-09-20T19:18:07.274379+0000","last_unstale":"2024-09-20T19:26:01.186481+0000","last_undegraded":"2024-09-20T19:26:01.186481+0000","last_fullsized":"2024-09-20T19:26:01.186481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:46.297121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.297121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:09.281574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":648,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172771+0000","last_change":"2024-09-20T19:23:02.238503+0000","last_active":"2024-09-20T19:26:01.172771+0000","last_peered":"2024-09-20T19:26:01.172771+0000","last_clean":"2024-09-20T19:26:01.172771+0000","last_became_active":"2024-09-20T19:18:09.311202+0000","last_became_peered":"2024-09-20T19:18:09.311202+0000","last_unstale":"2024-09-20T19:26:01.172771+0000","last_undegraded":"2024-09-20T19:26:01.172771+0000","last_fullsized":"2024-09-20T19:26:01.172771+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:23:02.238438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:02.238438+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:10.561092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185105+0000","last_change":"2024-09-20T19:22:08.909960+0000","last_active":"2024-09-20T19:26:01.185105+0000","last_peered":"2024-09-20T19:26:01.185105+0000","last_clean":"2024-09-20T19:26:01.185105+0000","last_became_active":"2024-09-20T19:18:11.329193+0000","last_became_peered":"2024-09-20T19:18:11.329193+0000","last_unstale":"2024-09-20T19:26:01.185105+0000","last_undegraded":"2024-09-20T19:26:01.185105+0000","last_fullsized":"2024-09-20T19:26:01.185105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:08.909835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.909835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:08.482448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":636,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199587+0000","last_change":"2024-09-20T19:23:58.846995+0000","last_active":"2024-09-20T19:26:01.199587+0000","last_peered":"2024-09-20T19:26:01.199587+0000","last_clean":"2024-09-20T19:26:01.199587+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:26:01.199587+0000","last_undegraded":"2024-09-20T19:26:01.199587+0000","last_fullsized":"2024-09-20T19:26:01.199587+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:58.846866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:58.846866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186171+0000","last_change":"2024-09-20T19:24:27.396341+0000","last_active":"2024-09-20T19:26:01.186171+0000","last_peered":"2024-09-20T19:26:01.186171+0000","last_clean":"2024-09-20T19:26:01.186171+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:26:01.186171+0000","last_undegraded":"2024-09-20T19:26:01.186171+0000","last_fullsized":"2024-09-20T19:26:01.186171+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:27.396275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:27.396275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172657+0000","last_change":"2024-09-20T19:24:37.338038+0000","last_active":"2024-09-20T19:26:01.172657+0000","last_peered":"2024-09-20T19:26:01.172657+0000","last_clean":"2024-09-20T19:26:01.172657+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:18:09.299047+0000","last_unstale":"2024-09-20T19:26:01.172657+0000","last_undegraded":"2024-09-20T19:26:01.172657+0000","last_fullsized":"2024-09-20T19:26:01.172657+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:24:37.337852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:37.337852+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:06.598684+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":632,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199573+0000","last_change":"2024-09-20T19:24:44.924986+0000","last_active":"2024-09-20T19:26:01.199573+0000","last_peered":"2024-09-20T19:26:01.199573+0000","last_clean":"2024-09-20T19:26:01.199573+0000","last_became_active":"2024-09-20T19:18:11.332381+0000","last_became_peered":"2024-09-20T19:18:11.332381+0000","last_unstale":"2024-09-20T19:26:01.199573+0000","last_undegraded":"2024-09-20T19:26:01.199573+0000","last_fullsized":"2024-09-20T19:26:01.199573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:44.924865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.924865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166417+0000","last_change":"2024-09-20T19:25:20.158201+0000","last_active":"2024-09-20T19:26:01.166417+0000","last_peered":"2024-09-20T19:26:01.166417+0000","last_clean":"2024-09-20T19:26:01.166417+0000","last_became_active":"2024-09-20T19:25:20.157858+0000","last_became_peered":"2024-09-20T19:25:20.157858+0000","last_unstale":"2024-09-20T19:26:01.166417+0000","last_undegraded":"2024-09-20T19:26:01.166417+0000","last_fullsized":"2024-09-20T19:26:01.166417+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:34.899368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:34.899368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:03.061849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":247,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199082+0000","last_change":"2024-09-20T19:25:20.758590+0000","last_active":"2024-09-20T19:26:01.199082+0000","last_peered":"2024-09-20T19:26:01.199082+0000","last_clean":"2024-09-20T19:26:01.199082+0000","last_became_active":"2024-09-20T19:24:11.093457+0000","last_became_peered":"2024-09-20T19:24:11.093457+0000","last_unstale":"2024-09-20T19:26:01.199082+0000","last_undegraded":"2024-09-20T19:26:01.199082+0000","last_fullsized":"2024-09-20T19:26:01.199082+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:20.758466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.758466+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:38.921598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"371'97","reported_seq":1390,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195123+0000","last_change":"2024-09-20T19:26:00.174253+0000","last_active":"2024-09-20T19:26:01.195123+0000","last_peered":"2024-09-20T19:26:01.195123+0000","last_clean":"2024-09-20T19:26:01.195123+0000","last_became_active":"2024-09-20T19:21:30.254492+0000","last_became_peered":"2024-09-20T19:21:30.254492+0000","last_unstale":"2024-09-20T19:26:01.195123+0000","last_undegraded":"2024-09-20T19:26:01.195123+0000","last_fullsized":"2024-09-20T19:26:01.195123+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'81","last_scrub_stamp":"2024-09-20T19:24:35.790673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.790673+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045235409999999998,"stat_sum":{"num_bytes":5253232,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":11861,"num_write":43,"num_write_kb":11146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.d","version":"77'17","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173437+0000","last_change":"2024-09-20T19:23:55.223430+0000","last_active":"2024-09-20T19:26:01.173437+0000","last_peered":"2024-09-20T19:26:01.173437+0000","last_clean":"2024-09-20T19:26:01.173437+0000","last_became_active":"2024-09-20T19:18:09.312709+0000","last_became_peered":"2024-09-20T19:18:09.312709+0000","last_unstale":"2024-09-20T19:26:01.173437+0000","last_undegraded":"2024-09-20T19:26:01.173437+0000","last_fullsized":"2024-09-20T19:26:01.173437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:23:55.223328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:55.223328+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185813+0000","last_change":"2024-09-20T19:21:28.083655+0000","last_active":"2024-09-20T19:26:01.185813+0000","last_peered":"2024-09-20T19:26:01.185813+0000","last_clean":"2024-09-20T19:26:01.185813+0000","last_became_active":"2024-09-20T19:18:07.266014+0000","last_became_peered":"2024-09-20T19:18:07.266014+0000","last_unstale":"2024-09-20T19:26:01.185813+0000","last_undegraded":"2024-09-20T19:26:01.185813+0000","last_fullsized":"2024-09-20T19:26:01.185813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:28.083532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:28.083532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:07.504323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173431+0000","last_change":"2024-09-20T19:25:34.311682+0000","last_active":"2024-09-20T19:26:01.173431+0000","last_peered":"2024-09-20T19:26:01.173431+0000","last_clean":"2024-09-20T19:26:01.173431+0000","last_became_active":"2024-09-20T19:18:11.310750+0000","last_became_peered":"2024-09-20T19:18:11.310750+0000","last_unstale":"2024-09-20T19:26:01.173431+0000","last_undegraded":"2024-09-20T19:26:01.173431+0000","last_fullsized":"2024-09-20T19:26:01.173431+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:34.311610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:34.311610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:27:02.032011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195938+0000","last_change":"2024-09-20T19:25:18.833923+0000","last_active":"2024-09-20T19:26:01.195938+0000","last_peered":"2024-09-20T19:26:01.195938+0000","last_clean":"2024-09-20T19:26:01.195938+0000","last_became_active":"2024-09-20T19:18:13.353443+0000","last_became_peered":"2024-09-20T19:18:13.353443+0000","last_unstale":"2024-09-20T19:26:01.195938+0000","last_undegraded":"2024-09-20T19:26:01.195938+0000","last_fullsized":"2024-09-20T19:26:01.195938+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:18.833856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.833856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:44.269208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185145+0000","last_change":"2024-09-20T19:25:21.332017+0000","last_active":"2024-09-20T19:26:01.185145+0000","last_peered":"2024-09-20T19:26:01.185145+0000","last_clean":"2024-09-20T19:26:01.185145+0000","last_became_active":"2024-09-20T19:24:11.093089+0000","last_became_peered":"2024-09-20T19:24:11.093089+0000","last_unstale":"2024-09-20T19:26:01.185145+0000","last_undegraded":"2024-09-20T19:26:01.185145+0000","last_fullsized":"2024-09-20T19:26:01.185145+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.331889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.331889+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:37.521314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"372'333","reported_seq":1625,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.202083+0000","last_change":"2024-09-20T19:26:00.183521+0000","last_active":"2024-09-20T19:26:01.202083+0000","last_peered":"2024-09-20T19:26:01.202083+0000","last_clean":"2024-09-20T19:26:01.202083+0000","last_became_active":"2024-09-20T19:21:30.242108+0000","last_became_peered":"2024-09-20T19:21:30.242108+0000","last_unstale":"2024-09-20T19:26:01.202083+0000","last_undegraded":"2024-09-20T19:26:01.202083+0000","last_fullsized":"2024-09-20T19:26:01.202083+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'282","last_scrub_stamp":"2024-09-20T19:25:28.132134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.132134+0000","objects_scrubbed":11,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:37.154844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0076639289999999999,"stat_sum":{"num_bytes":22908040,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":145,"num_read_kb":142236,"num_write":90,"num_write_kb":43093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.c","version":"77'10","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173097+0000","last_change":"2024-09-20T19:23:42.253431+0000","last_active":"2024-09-20T19:26:01.173097+0000","last_peered":"2024-09-20T19:26:01.173097+0000","last_clean":"2024-09-20T19:26:01.173097+0000","last_became_active":"2024-09-20T19:18:09.304471+0000","last_became_peered":"2024-09-20T19:18:09.304471+0000","last_unstale":"2024-09-20T19:26:01.173097+0000","last_undegraded":"2024-09-20T19:26:01.173097+0000","last_fullsized":"2024-09-20T19:26:01.173097+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:42.253363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.253363+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:39.383146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.167115+0000","last_change":"2024-09-20T19:24:40.522582+0000","last_active":"2024-09-20T19:26:01.167115+0000","last_peered":"2024-09-20T19:26:01.167115+0000","last_clean":"2024-09-20T19:26:01.167115+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:26:01.167115+0000","last_undegraded":"2024-09-20T19:26:01.167115+0000","last_fullsized":"2024-09-20T19:26:01.167115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:40.522455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:40.522455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:49.318959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":741,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.197012+0000","last_change":"2024-09-20T19:24:01.760145+0000","last_active":"2024-09-20T19:26:01.197012+0000","last_peered":"2024-09-20T19:26:01.197012+0000","last_clean":"2024-09-20T19:26:01.197012+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:26:01.197012+0000","last_undegraded":"2024-09-20T19:26:01.197012+0000","last_fullsized":"2024-09-20T19:26:01.197012+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:01.760085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:01.760085+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:02.680456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173306+0000","last_change":"2024-09-20T19:24:32.370424+0000","last_active":"2024-09-20T19:26:01.173306+0000","last_peered":"2024-09-20T19:26:01.173306+0000","last_clean":"2024-09-20T19:26:01.173306+0000","last_became_active":"2024-09-20T19:18:13.361177+0000","last_became_peered":"2024-09-20T19:18:13.361177+0000","last_unstale":"2024-09-20T19:26:01.173306+0000","last_undegraded":"2024-09-20T19:26:01.173306+0000","last_fullsized":"2024-09-20T19:26:01.173306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:32.370267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.370267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":240,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199258+0000","last_change":"2024-09-20T19:25:16.899924+0000","last_active":"2024-09-20T19:26:01.199258+0000","last_peered":"2024-09-20T19:26:01.199258+0000","last_clean":"2024-09-20T19:26:01.199258+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:26:01.199258+0000","last_undegraded":"2024-09-20T19:26:01.199258+0000","last_fullsized":"2024-09-20T19:26:01.199258+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:16.899851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.899851+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:25.277276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"371'400","reported_seq":1661,"reported_epoch":372,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:26:01.379635+0000","last_change":"2024-09-20T19:26:01.379635+0000","last_active":"2024-09-20T19:26:01.379635+0000","last_peered":"2024-09-20T19:26:01.379635+0000","last_clean":"2024-09-20T19:26:01.379635+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:26:01.379635+0000","last_undegraded":"2024-09-20T19:26:01.379635+0000","last_fullsized":"2024-09-20T19:26:01.379635+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'237","last_scrub_stamp":"2024-09-20T19:24:35.542766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.542766+0000","objects_scrubbed":12,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.171882752,"stat_sum":{"num_bytes":44570152,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":199,"num_read_kb":156002,"num_write":143,"num_write_kb":45390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5a","length":"1"},{"start":"69","length":"1"},{"start":"74","length":"2"}]},{"pgid":"4.f","version":"77'15","reported_seq":686,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.179298+0000","last_change":"2024-09-20T19:25:57.204153+0000","last_active":"2024-09-20T19:26:01.179298+0000","last_peered":"2024-09-20T19:26:01.179298+0000","last_clean":"2024-09-20T19:26:01.179298+0000","last_became_active":"2024-09-20T19:25:57.183915+0000","last_became_peered":"2024-09-20T19:25:57.183915+0000","last_unstale":"2024-09-20T19:26:01.179298+0000","last_undegraded":"2024-09-20T19:26:01.179298+0000","last_fullsized":"2024-09-20T19:26:01.179298+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":650,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.168800+0000","last_change":"2024-09-20T19:25:35.294974+0000","last_active":"2024-09-20T19:26:01.168800+0000","last_peered":"2024-09-20T19:26:01.168800+0000","last_clean":"2024-09-20T19:26:01.168800+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:26:01.168800+0000","last_undegraded":"2024-09-20T19:26:01.168800+0000","last_fullsized":"2024-09-20T19:26:01.168800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:35.294857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:35.294857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:57.274073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"77'11","reported_seq":739,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173509+0000","last_change":"2024-09-20T19:23:03.226408+0000","last_active":"2024-09-20T19:26:01.173509+0000","last_peered":"2024-09-20T19:26:01.173509+0000","last_clean":"2024-09-20T19:26:01.173509+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:18:11.309941+0000","last_unstale":"2024-09-20T19:26:01.173509+0000","last_undegraded":"2024-09-20T19:26:01.173509+0000","last_fullsized":"2024-09-20T19:26:01.173509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:52.166180+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":636,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199748+0000","last_change":"2024-09-20T19:25:26.791427+0000","last_active":"2024-09-20T19:26:01.199748+0000","last_peered":"2024-09-20T19:26:01.199748+0000","last_clean":"2024-09-20T19:26:01.199748+0000","last_became_active":"2024-09-20T19:18:13.363432+0000","last_became_peered":"2024-09-20T19:18:13.363432+0000","last_unstale":"2024-09-20T19:26:01.199748+0000","last_undegraded":"2024-09-20T19:26:01.199748+0000","last_fullsized":"2024-09-20T19:26:01.199748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:26.791248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:26.791248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:38.154590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"372'244","reported_seq":1522,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.382539+0000","last_change":"2024-09-20T19:26:01.379670+0000","last_active":"2024-09-20T19:26:01.382539+0000","last_peered":"2024-09-20T19:26:01.382539+0000","last_clean":"2024-09-20T19:26:01.382539+0000","last_became_active":"2024-09-20T19:21:30.271055+0000","last_became_peered":"2024-09-20T19:21:30.271055+0000","last_unstale":"2024-09-20T19:26:01.382539+0000","last_undegraded":"2024-09-20T19:26:01.382539+0000","last_fullsized":"2024-09-20T19:26:01.382539+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'149","last_scrub_stamp":"2024-09-20T19:24:44.564761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.564761+0000","objects_scrubbed":9,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.075229911999999996,"stat_sum":{"num_bytes":22214451,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":154,"num_read_kb":122333,"num_write":84,"num_write_kb":34148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"1"}]},{"pgid":"4.e","version":"77'11","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173577+0000","last_change":"2024-09-20T19:22:58.180428+0000","last_active":"2024-09-20T19:26:01.173577+0000","last_peered":"2024-09-20T19:26:01.173577+0000","last_clean":"2024-09-20T19:26:01.173577+0000","last_became_active":"2024-09-20T19:18:09.305074+0000","last_became_peered":"2024-09-20T19:18:09.305074+0000","last_unstale":"2024-09-20T19:26:01.173577+0000","last_undegraded":"2024-09-20T19:26:01.173577+0000","last_fullsized":"2024-09-20T19:26:01.173577+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:58.180287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:58.180287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":525,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.190700+0000","last_change":"2024-09-20T19:25:16.874919+0000","last_active":"2024-09-20T19:26:01.190700+0000","last_peered":"2024-09-20T19:26:01.190700+0000","last_clean":"2024-09-20T19:26:01.190700+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:26:01.190700+0000","last_undegraded":"2024-09-20T19:26:01.190700+0000","last_fullsized":"2024-09-20T19:26:01.190700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:16.874763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.874763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:22.154310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":632,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200223+0000","last_change":"2024-09-20T19:24:38.906807+0000","last_active":"2024-09-20T19:26:01.200223+0000","last_peered":"2024-09-20T19:26:01.200223+0000","last_clean":"2024-09-20T19:26:01.200223+0000","last_became_active":"2024-09-20T19:18:11.324237+0000","last_became_peered":"2024-09-20T19:18:11.324237+0000","last_unstale":"2024-09-20T19:26:01.200223+0000","last_undegraded":"2024-09-20T19:26:01.200223+0000","last_fullsized":"2024-09-20T19:26:01.200223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.906652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.906652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":621,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.171178+0000","last_change":"2024-09-20T19:24:43.030261+0000","last_active":"2024-09-20T19:26:01.171178+0000","last_peered":"2024-09-20T19:26:01.171178+0000","last_clean":"2024-09-20T19:26:01.171178+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:26:01.171178+0000","last_undegraded":"2024-09-20T19:26:01.171178+0000","last_fullsized":"2024-09-20T19:26:01.171178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:43.030133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.030133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:11.014720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":245,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.170889+0000","last_change":"2024-09-20T19:25:14.571140+0000","last_active":"2024-09-20T19:26:01.170889+0000","last_peered":"2024-09-20T19:26:01.170889+0000","last_clean":"2024-09-20T19:26:01.170889+0000","last_became_active":"2024-09-20T19:24:11.088414+0000","last_became_peered":"2024-09-20T19:24:11.088414+0000","last_unstale":"2024-09-20T19:26:01.170889+0000","last_undegraded":"2024-09-20T19:26:01.170889+0000","last_fullsized":"2024-09-20T19:26:01.170889+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:14.571014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:14.571014+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:21.805869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"370'94","reported_seq":1360,"reported_epoch":372,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:26:01.190290+0000","last_change":"2024-09-20T19:26:01.190447+0000","last_active":"2024-09-20T19:26:01.190290+0000","last_peered":"2024-09-20T19:26:01.190290+0000","last_clean":"2024-09-20T19:26:01.190290+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:26:01.190290+0000","last_undegraded":"2024-09-20T19:26:01.190290+0000","last_fullsized":"2024-09-20T19:26:01.190290+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'31","last_scrub_stamp":"2024-09-20T19:23:57.047151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:57.047151+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021557684000000001,"stat_sum":{"num_bytes":6941357,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":19039,"num_write":28,"num_write_kb":12455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"2"},{"start":"77","length":"2"},{"start":"7a","length":"2"}]},{"pgid":"4.1","version":"77'14","reported_seq":659,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172812+0000","last_change":"2024-09-20T19:25:34.359415+0000","last_active":"2024-09-20T19:26:01.172812+0000","last_peered":"2024-09-20T19:26:01.172812+0000","last_clean":"2024-09-20T19:26:01.172812+0000","last_became_active":"2024-09-20T19:18:09.294158+0000","last_became_peered":"2024-09-20T19:18:09.294158+0000","last_unstale":"2024-09-20T19:26:01.172812+0000","last_undegraded":"2024-09-20T19:26:01.172812+0000","last_fullsized":"2024-09-20T19:26:01.172812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:25:34.359269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:34.359269+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:42.874150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":647,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.209080+0000","last_change":"2024-09-20T19:23:47.645931+0000","last_active":"2024-09-20T19:26:01.209080+0000","last_peered":"2024-09-20T19:26:01.209080+0000","last_clean":"2024-09-20T19:26:01.209080+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:26:01.209080+0000","last_undegraded":"2024-09-20T19:26:01.209080+0000","last_fullsized":"2024-09-20T19:26:01.209080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.645876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.645876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:57.645080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"77'11","reported_seq":738,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165966+0000","last_change":"2024-09-20T19:24:32.089906+0000","last_active":"2024-09-20T19:26:01.165966+0000","last_peered":"2024-09-20T19:26:01.165966+0000","last_clean":"2024-09-20T19:26:01.165966+0000","last_became_active":"2024-09-20T19:18:11.313535+0000","last_became_peered":"2024-09-20T19:18:11.313535+0000","last_unstale":"2024-09-20T19:26:01.165966+0000","last_undegraded":"2024-09-20T19:26:01.165966+0000","last_fullsized":"2024-09-20T19:26:01.165966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:32.089810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.089810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":626,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185411+0000","last_change":"2024-09-20T19:25:22.371471+0000","last_active":"2024-09-20T19:26:01.185411+0000","last_peered":"2024-09-20T19:26:01.185411+0000","last_clean":"2024-09-20T19:26:01.185411+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:26:01.185411+0000","last_undegraded":"2024-09-20T19:26:01.185411+0000","last_fullsized":"2024-09-20T19:26:01.185411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:22.371404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:22.371404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:36.781433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":240,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173033+0000","last_change":"2024-09-20T19:25:20.611172+0000","last_active":"2024-09-20T19:26:01.173033+0000","last_peered":"2024-09-20T19:26:01.173033+0000","last_clean":"2024-09-20T19:26:01.173033+0000","last_became_active":"2024-09-20T19:24:11.086665+0000","last_became_peered":"2024-09-20T19:24:11.086665+0000","last_unstale":"2024-09-20T19:26:01.173033+0000","last_undegraded":"2024-09-20T19:26:01.173033+0000","last_fullsized":"2024-09-20T19:26:01.173033+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:20.611001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.611001+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:35.613802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"371'268","reported_seq":1527,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.167817+0000","last_change":"2024-09-20T19:26:00.379466+0000","last_active":"2024-09-20T19:26:01.167817+0000","last_peered":"2024-09-20T19:26:01.167817+0000","last_clean":"2024-09-20T19:26:01.167817+0000","last_became_active":"2024-09-20T19:21:30.256889+0000","last_became_peered":"2024-09-20T19:21:30.256889+0000","last_unstale":"2024-09-20T19:26:01.167817+0000","last_undegraded":"2024-09-20T19:26:01.167817+0000","last_fullsized":"2024-09-20T19:26:01.167817+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'148","last_scrub_stamp":"2024-09-20T19:23:58.044067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:58.044067+0000","objects_scrubbed":3,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.060303407000000003,"stat_sum":{"num_bytes":19047143,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":120,"num_read_kb":98989,"num_write":79,"num_write_kb":27846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"}]},{"pgid":"4.0","version":"77'18","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165711+0000","last_change":"2024-09-20T19:22:00.803489+0000","last_active":"2024-09-20T19:26:01.165711+0000","last_peered":"2024-09-20T19:26:01.165711+0000","last_clean":"2024-09-20T19:26:01.165711+0000","last_became_active":"2024-09-20T19:18:10.116794+0000","last_became_peered":"2024-09-20T19:18:10.116794+0000","last_unstale":"2024-09-20T19:26:01.165711+0000","last_undegraded":"2024-09-20T19:26:01.165711+0000","last_fullsized":"2024-09-20T19:26:01.165711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:22:00.803432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:00.803432+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:07.726819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":643,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.171266+0000","last_change":"2024-09-20T19:23:51.507352+0000","last_active":"2024-09-20T19:26:01.171266+0000","last_peered":"2024-09-20T19:26:01.171266+0000","last_clean":"2024-09-20T19:26:01.171266+0000","last_became_active":"2024-09-20T19:18:07.262174+0000","last_became_peered":"2024-09-20T19:18:07.262174+0000","last_unstale":"2024-09-20T19:26:01.171266+0000","last_undegraded":"2024-09-20T19:26:01.171266+0000","last_fullsized":"2024-09-20T19:26:01.171266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:51.507216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.507216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:39.272529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172427+0000","last_change":"2024-09-20T19:25:27.482748+0000","last_active":"2024-09-20T19:26:01.172427+0000","last_peered":"2024-09-20T19:26:01.172427+0000","last_clean":"2024-09-20T19:26:01.172427+0000","last_became_active":"2024-09-20T19:18:11.347916+0000","last_became_peered":"2024-09-20T19:18:11.347916+0000","last_unstale":"2024-09-20T19:26:01.172427+0000","last_undegraded":"2024-09-20T19:26:01.172427+0000","last_fullsized":"2024-09-20T19:26:01.172427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.482618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.482618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:40.853698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172587+0000","last_change":"2024-09-20T19:23:48.185887+0000","last_active":"2024-09-20T19:26:01.172587+0000","last_peered":"2024-09-20T19:26:01.172587+0000","last_clean":"2024-09-20T19:26:01.172587+0000","last_became_active":"2024-09-20T19:18:13.353864+0000","last_became_peered":"2024-09-20T19:18:13.353864+0000","last_unstale":"2024-09-20T19:26:01.172587+0000","last_undegraded":"2024-09-20T19:26:01.172587+0000","last_fullsized":"2024-09-20T19:26:01.172587+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:42.389709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":342,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.208075+0000","last_change":"2024-09-20T19:25:21.382626+0000","last_active":"2024-09-20T19:26:01.208075+0000","last_peered":"2024-09-20T19:26:01.208075+0000","last_clean":"2024-09-20T19:26:01.208075+0000","last_became_active":"2024-09-20T19:24:11.099054+0000","last_became_peered":"2024-09-20T19:24:11.099054+0000","last_unstale":"2024-09-20T19:26:01.208075+0000","last_undegraded":"2024-09-20T19:26:01.208075+0000","last_fullsized":"2024-09-20T19:26:01.208075+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.382463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.382463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:23.895406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"371'392","reported_seq":1669,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.199142+0000","last_change":"2024-09-20T19:26:00.402562+0000","last_active":"2024-09-20T19:26:01.199142+0000","last_peered":"2024-09-20T19:26:01.199142+0000","last_clean":"2024-09-20T19:26:01.199142+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:26:01.199142+0000","last_undegraded":"2024-09-20T19:26:01.199142+0000","last_fullsized":"2024-09-20T19:26:01.199142+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'118","last_scrub_stamp":"2024-09-20T19:23:04.906510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.906510+0000","objects_scrubbed":10,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.22476400099999999,"stat_sum":{"num_bytes":42991374,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":234,"num_read_kb":183266,"num_write":144,"num_write_kb":59852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5a","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"1"}]},{"pgid":"4.3","version":"77'19","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186129+0000","last_change":"2024-09-20T19:24:39.971994+0000","last_active":"2024-09-20T19:26:01.186129+0000","last_peered":"2024-09-20T19:26:01.186129+0000","last_clean":"2024-09-20T19:26:01.186129+0000","last_became_active":"2024-09-20T19:18:10.113873+0000","last_became_peered":"2024-09-20T19:18:10.113873+0000","last_unstale":"2024-09-20T19:26:01.186129+0000","last_undegraded":"2024-09-20T19:26:01.186129+0000","last_fullsized":"2024-09-20T19:26:01.186129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:24:39.971842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.971842+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":586,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.196656+0000","last_change":"2024-09-20T19:24:42.788736+0000","last_active":"2024-09-20T19:26:01.196656+0000","last_peered":"2024-09-20T19:26:01.196656+0000","last_clean":"2024-09-20T19:26:01.196656+0000","last_became_active":"2024-09-20T19:19:56.998402+0000","last_became_peered":"2024-09-20T19:19:56.998402+0000","last_unstale":"2024-09-20T19:26:01.196656+0000","last_undegraded":"2024-09-20T19:26:01.196656+0000","last_fullsized":"2024-09-20T19:26:01.196656+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.788611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.788611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":639,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207270+0000","last_change":"2024-09-20T19:23:59.649288+0000","last_active":"2024-09-20T19:26:01.207270+0000","last_peered":"2024-09-20T19:26:01.207270+0000","last_clean":"2024-09-20T19:26:01.207270+0000","last_became_active":"2024-09-20T19:18:11.308890+0000","last_became_peered":"2024-09-20T19:18:11.308890+0000","last_unstale":"2024-09-20T19:26:01.207270+0000","last_undegraded":"2024-09-20T19:26:01.207270+0000","last_fullsized":"2024-09-20T19:26:01.207270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:59.649128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:59.649128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":622,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.167873+0000","last_change":"2024-09-20T19:24:34.554389+0000","last_active":"2024-09-20T19:26:01.167873+0000","last_peered":"2024-09-20T19:26:01.167873+0000","last_clean":"2024-09-20T19:26:01.167873+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:26:01.167873+0000","last_undegraded":"2024-09-20T19:26:01.167873+0000","last_fullsized":"2024-09-20T19:26:01.167873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:34.554263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:34.554263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:48.349020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":342,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.208181+0000","last_change":"2024-09-20T19:25:16.377055+0000","last_active":"2024-09-20T19:26:01.208181+0000","last_peered":"2024-09-20T19:26:01.208181+0000","last_clean":"2024-09-20T19:26:01.208181+0000","last_became_active":"2024-09-20T19:24:11.094059+0000","last_became_peered":"2024-09-20T19:24:11.094059+0000","last_unstale":"2024-09-20T19:26:01.208181+0000","last_undegraded":"2024-09-20T19:26:01.208181+0000","last_fullsized":"2024-09-20T19:26:01.208181+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:16.376923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.376923+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:18.998974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"371'339","reported_seq":1683,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172976+0000","last_change":"2024-09-20T19:26:00.179199+0000","last_active":"2024-09-20T19:26:01.172976+0000","last_peered":"2024-09-20T19:26:01.172976+0000","last_clean":"2024-09-20T19:26:01.172976+0000","last_became_active":"2024-09-20T19:25:23.102838+0000","last_became_peered":"2024-09-20T19:25:23.102838+0000","last_unstale":"2024-09-20T19:26:01.172976+0000","last_undegraded":"2024-09-20T19:26:01.172976+0000","last_fullsized":"2024-09-20T19:26:01.172976+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'306","last_scrub_stamp":"2024-09-20T19:25:33.397446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:33.397446+0000","objects_scrubbed":8,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:49.635927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.010854557000000001,"stat_sum":{"num_bytes":19925578,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":123,"num_read_kb":85087,"num_write":108,"num_write_kb":36670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8689117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.2","version":"77'10","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166766+0000","last_change":"2024-09-20T19:23:54.537172+0000","last_active":"2024-09-20T19:26:01.166766+0000","last_peered":"2024-09-20T19:26:01.166766+0000","last_clean":"2024-09-20T19:26:01.166766+0000","last_became_active":"2024-09-20T19:18:09.299630+0000","last_became_peered":"2024-09-20T19:18:09.299630+0000","last_unstale":"2024-09-20T19:26:01.166766+0000","last_undegraded":"2024-09-20T19:26:01.166766+0000","last_fullsized":"2024-09-20T19:26:01.166766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:54.536975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.536975+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:45.281200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.174501+0000","last_change":"2024-09-20T19:22:21.888862+0000","last_active":"2024-09-20T19:26:01.174501+0000","last_peered":"2024-09-20T19:26:01.174501+0000","last_clean":"2024-09-20T19:26:01.174501+0000","last_became_active":"2024-09-20T19:18:07.274240+0000","last_became_peered":"2024-09-20T19:18:07.274240+0000","last_unstale":"2024-09-20T19:26:01.174501+0000","last_undegraded":"2024-09-20T19:26:01.174501+0000","last_fullsized":"2024-09-20T19:26:01.174501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:54.312721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"77'11","reported_seq":739,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185833+0000","last_change":"2024-09-20T19:24:46.015856+0000","last_active":"2024-09-20T19:26:01.185833+0000","last_peered":"2024-09-20T19:26:01.185833+0000","last_clean":"2024-09-20T19:26:01.185833+0000","last_became_active":"2024-09-20T19:18:11.326025+0000","last_became_peered":"2024-09-20T19:18:11.326025+0000","last_unstale":"2024-09-20T19:26:01.185833+0000","last_undegraded":"2024-09-20T19:26:01.185833+0000","last_fullsized":"2024-09-20T19:26:01.185833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:46.015736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.015736+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.184535+0000","last_change":"2024-09-20T19:22:56.797105+0000","last_active":"2024-09-20T19:26:01.184535+0000","last_peered":"2024-09-20T19:26:01.184535+0000","last_clean":"2024-09-20T19:26:01.184535+0000","last_became_active":"2024-09-20T19:18:13.352746+0000","last_became_peered":"2024-09-20T19:18:13.352746+0000","last_unstale":"2024-09-20T19:26:01.184535+0000","last_undegraded":"2024-09-20T19:26:01.184535+0000","last_fullsized":"2024-09-20T19:26:01.184535+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:57.978817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":240,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199216+0000","last_change":"2024-09-20T19:25:19.799966+0000","last_active":"2024-09-20T19:26:01.199216+0000","last_peered":"2024-09-20T19:26:01.199216+0000","last_clean":"2024-09-20T19:26:01.199216+0000","last_became_active":"2024-09-20T19:24:11.090208+0000","last_became_peered":"2024-09-20T19:24:11.090208+0000","last_unstale":"2024-09-20T19:26:01.199216+0000","last_undegraded":"2024-09-20T19:26:01.199216+0000","last_fullsized":"2024-09-20T19:26:01.199216+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:19.799797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:19.799797+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:45.114233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"372'311","reported_seq":1575,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.285549+0000","last_change":"2024-09-20T19:26:01.283619+0000","last_active":"2024-09-20T19:26:01.285549+0000","last_peered":"2024-09-20T19:26:01.285549+0000","last_clean":"2024-09-20T19:26:01.285549+0000","last_became_active":"2024-09-20T19:21:30.245252+0000","last_became_peered":"2024-09-20T19:21:30.245252+0000","last_unstale":"2024-09-20T19:26:01.285549+0000","last_undegraded":"2024-09-20T19:26:01.285549+0000","last_fullsized":"2024-09-20T19:26:01.285549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'119","last_scrub_stamp":"2024-09-20T19:23:46.636892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.636892+0000","objects_scrubbed":4,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.14814961400000001,"stat_sum":{"num_bytes":36251140,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":180,"num_read_kb":123636,"num_write":127,"num_write_kb":43970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5a","length":"1"},{"start":"69","length":"1"},{"start":"74","length":"2"}]},{"pgid":"4.4","version":"77'30","reported_seq":791,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.174330+0000","last_change":"2024-09-20T19:24:44.491558+0000","last_active":"2024-09-20T19:26:01.174330+0000","last_peered":"2024-09-20T19:26:01.174330+0000","last_clean":"2024-09-20T19:26:01.174330+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:26:01.174330+0000","last_undegraded":"2024-09-20T19:26:01.174330+0000","last_fullsized":"2024-09-20T19:26:01.174330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:24:44.491367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.491367+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:55.307149+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":648,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200277+0000","last_change":"2024-09-20T19:25:28.761124+0000","last_active":"2024-09-20T19:26:01.200277+0000","last_peered":"2024-09-20T19:26:01.200277+0000","last_clean":"2024-09-20T19:26:01.200277+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:26:01.200277+0000","last_undegraded":"2024-09-20T19:26:01.200277+0000","last_fullsized":"2024-09-20T19:26:01.200277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:28.761056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.761056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:55.954495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":650,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187231+0000","last_change":"2024-09-20T19:25:36.955182+0000","last_active":"2024-09-20T19:26:01.187231+0000","last_peered":"2024-09-20T19:26:01.187231+0000","last_clean":"2024-09-20T19:26:01.187231+0000","last_became_active":"2024-09-20T19:18:11.312253+0000","last_became_peered":"2024-09-20T19:18:11.312253+0000","last_unstale":"2024-09-20T19:26:01.187231+0000","last_undegraded":"2024-09-20T19:26:01.187231+0000","last_fullsized":"2024-09-20T19:26:01.187231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.955124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.955124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:42.589450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.169680+0000","last_change":"2024-09-20T19:25:36.714380+0000","last_active":"2024-09-20T19:26:01.169680+0000","last_peered":"2024-09-20T19:26:01.169680+0000","last_clean":"2024-09-20T19:26:01.169680+0000","last_became_active":"2024-09-20T19:18:13.365396+0000","last_became_peered":"2024-09-20T19:18:13.365396+0000","last_unstale":"2024-09-20T19:26:01.169680+0000","last_undegraded":"2024-09-20T19:26:01.169680+0000","last_fullsized":"2024-09-20T19:26:01.169680+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.714321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.714321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:42.946883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185372+0000","last_change":"2024-09-20T19:25:20.337902+0000","last_active":"2024-09-20T19:26:01.185372+0000","last_peered":"2024-09-20T19:26:01.185372+0000","last_clean":"2024-09-20T19:26:01.185372+0000","last_became_active":"2024-09-20T19:24:11.088125+0000","last_became_peered":"2024-09-20T19:24:11.088125+0000","last_unstale":"2024-09-20T19:26:01.185372+0000","last_undegraded":"2024-09-20T19:26:01.185372+0000","last_fullsized":"2024-09-20T19:26:01.185372+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:20.337772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.337772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:38.042606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"371'648","reported_seq":1916,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185930+0000","last_change":"2024-09-20T19:26:00.194883+0000","last_active":"2024-09-20T19:26:01.185930+0000","last_peered":"2024-09-20T19:26:01.185930+0000","last_clean":"2024-09-20T19:26:01.185930+0000","last_became_active":"2024-09-20T19:21:30.254497+0000","last_became_peered":"2024-09-20T19:21:30.254497+0000","last_unstale":"2024-09-20T19:26:01.185930+0000","last_undegraded":"2024-09-20T19:26:01.185930+0000","last_fullsized":"2024-09-20T19:26:01.185930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'377","last_scrub_stamp":"2024-09-20T19:24:26.871633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:26.871633+0000","objects_scrubbed":12,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.019581199000000001,"stat_sum":{"num_bytes":55760317,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":201,"num_read_kb":108448,"num_write":207,"num_write_kb":82687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.7","version":"77'13","reported_seq":590,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195428+0000","last_change":"2024-09-20T19:25:25.679709+0000","last_active":"2024-09-20T19:26:01.195428+0000","last_peered":"2024-09-20T19:26:01.195428+0000","last_clean":"2024-09-20T19:26:01.195428+0000","last_became_active":"2024-09-20T19:19:56.999763+0000","last_became_peered":"2024-09-20T19:19:56.999763+0000","last_unstale":"2024-09-20T19:26:01.195428+0000","last_undegraded":"2024-09-20T19:26:01.195428+0000","last_fullsized":"2024-09-20T19:26:01.195428+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:25:25.679635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:25.679635+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:48.299782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":1108,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.167825+0000","last_change":"2024-09-20T19:24:28.560164+0000","last_active":"2024-09-20T19:26:01.167825+0000","last_peered":"2024-09-20T19:26:01.167825+0000","last_clean":"2024-09-20T19:26:01.167825+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:26:01.167825+0000","last_undegraded":"2024-09-20T19:26:01.167825+0000","last_fullsized":"2024-09-20T19:26:01.167825+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:24:28.560018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:28.560018+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:56.286034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":458,"num_read_kb":453,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.191675+0000","last_change":"2024-09-20T19:24:40.807754+0000","last_active":"2024-09-20T19:26:01.191675+0000","last_peered":"2024-09-20T19:26:01.191675+0000","last_clean":"2024-09-20T19:26:01.191675+0000","last_became_active":"2024-09-20T19:18:07.253620+0000","last_became_peered":"2024-09-20T19:18:07.253620+0000","last_unstale":"2024-09-20T19:26:01.191675+0000","last_undegraded":"2024-09-20T19:26:01.191675+0000","last_fullsized":"2024-09-20T19:26:01.191675+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:40.807648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:40.807648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:04.411904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":633,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195253+0000","last_change":"2024-09-20T19:23:47.565593+0000","last_active":"2024-09-20T19:26:01.195253+0000","last_peered":"2024-09-20T19:26:01.195253+0000","last_clean":"2024-09-20T19:26:01.195253+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:26:01.195253+0000","last_undegraded":"2024-09-20T19:26:01.195253+0000","last_fullsized":"2024-09-20T19:26:01.195253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.565537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.565537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:09.741526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":634,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186086+0000","last_change":"2024-09-20T19:24:35.285541+0000","last_active":"2024-09-20T19:26:01.186086+0000","last_peered":"2024-09-20T19:26:01.186086+0000","last_clean":"2024-09-20T19:26:01.186086+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:26:01.186086+0000","last_undegraded":"2024-09-20T19:26:01.186086+0000","last_fullsized":"2024-09-20T19:26:01.186086+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:35.285415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.285415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":232,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.170654+0000","last_change":"2024-09-20T19:24:11.091335+0000","last_active":"2024-09-20T19:26:01.170654+0000","last_peered":"2024-09-20T19:26:01.170654+0000","last_clean":"2024-09-20T19:26:01.170654+0000","last_became_active":"2024-09-20T19:24:11.090709+0000","last_became_peered":"2024-09-20T19:24:11.090709+0000","last_unstale":"2024-09-20T19:26:01.170654+0000","last_undegraded":"2024-09-20T19:26:01.170654+0000","last_fullsized":"2024-09-20T19:26:01.170654+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:26:13.383574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"372'607","reported_seq":1873,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.309274+0000","last_change":"2024-09-20T19:26:01.307154+0000","last_active":"2024-09-20T19:26:01.309274+0000","last_peered":"2024-09-20T19:26:01.309274+0000","last_clean":"2024-09-20T19:26:01.309274+0000","last_became_active":"2024-09-20T19:21:30.249609+0000","last_became_peered":"2024-09-20T19:21:30.249609+0000","last_unstale":"2024-09-20T19:26:01.309274+0000","last_undegraded":"2024-09-20T19:26:01.309274+0000","last_fullsized":"2024-09-20T19:26:01.309274+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'286","last_scrub_stamp":"2024-09-20T19:24:01.346351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:01.346351+0000","objects_scrubbed":10,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.120913067,"stat_sum":{"num_bytes":58005186,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":344,"num_read_kb":237148,"num_write":235,"num_write_kb":98026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5a","length":"1"},{"start":"69","length":"1"}]},{"pgid":"4.6","version":"77'12","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187930+0000","last_change":"2024-09-20T19:25:21.127734+0000","last_active":"2024-09-20T19:26:01.187930+0000","last_peered":"2024-09-20T19:26:01.187930+0000","last_clean":"2024-09-20T19:26:01.187930+0000","last_became_active":"2024-09-20T19:18:09.292135+0000","last_became_peered":"2024-09-20T19:18:09.292135+0000","last_unstale":"2024-09-20T19:26:01.187930+0000","last_undegraded":"2024-09-20T19:26:01.187930+0000","last_fullsized":"2024-09-20T19:26:01.187930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:21.127630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.127630+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:37.782755+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":648,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186471+0000","last_change":"2024-09-20T19:24:39.000905+0000","last_active":"2024-09-20T19:26:01.186471+0000","last_peered":"2024-09-20T19:26:01.186471+0000","last_clean":"2024-09-20T19:26:01.186471+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:18:08.279045+0000","last_unstale":"2024-09-20T19:26:01.186471+0000","last_undegraded":"2024-09-20T19:26:01.186471+0000","last_fullsized":"2024-09-20T19:26:01.186471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:24:39.000765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.000765+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001124304,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185421+0000","last_change":"2024-09-20T19:24:33.379226+0000","last_active":"2024-09-20T19:26:01.185421+0000","last_peered":"2024-09-20T19:26:01.185421+0000","last_clean":"2024-09-20T19:26:01.185421+0000","last_became_active":"2024-09-20T19:18:07.274454+0000","last_became_peered":"2024-09-20T19:18:07.274454+0000","last_unstale":"2024-09-20T19:26:01.185421+0000","last_undegraded":"2024-09-20T19:26:01.185421+0000","last_fullsized":"2024-09-20T19:26:01.185421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:33.379111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:33.379111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":640,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200373+0000","last_change":"2024-09-20T19:25:27.747392+0000","last_active":"2024-09-20T19:26:01.200373+0000","last_peered":"2024-09-20T19:26:01.200373+0000","last_clean":"2024-09-20T19:26:01.200373+0000","last_became_active":"2024-09-20T19:18:11.320691+0000","last_became_peered":"2024-09-20T19:18:11.320691+0000","last_unstale":"2024-09-20T19:26:01.200373+0000","last_undegraded":"2024-09-20T19:26:01.200373+0000","last_fullsized":"2024-09-20T19:26:01.200373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.747335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.747335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:48.050358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173209+0000","last_change":"2024-09-20T19:24:47.513115+0000","last_active":"2024-09-20T19:26:01.173209+0000","last_peered":"2024-09-20T19:26:01.173209+0000","last_clean":"2024-09-20T19:26:01.173209+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:26:01.173209+0000","last_undegraded":"2024-09-20T19:26:01.173209+0000","last_fullsized":"2024-09-20T19:26:01.173209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:47.513039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:47.513039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"371'197","reported_seq":1509,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187359+0000","last_change":"2024-09-20T19:26:00.196162+0000","last_active":"2024-09-20T19:26:01.187359+0000","last_peered":"2024-09-20T19:26:01.187359+0000","last_clean":"2024-09-20T19:26:01.187359+0000","last_became_active":"2024-09-20T19:21:30.244539+0000","last_became_peered":"2024-09-20T19:21:30.244539+0000","last_unstale":"2024-09-20T19:26:01.187359+0000","last_undegraded":"2024-09-20T19:26:01.187359+0000","last_fullsized":"2024-09-20T19:26:01.187359+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'141","last_scrub_stamp":"2024-09-20T19:25:15.163437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:15.163437+0000","objects_scrubbed":4,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:30.126144+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.019499045,"stat_sum":{"num_bytes":19191391,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":121,"num_read_kb":117142,"num_write":62,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.5","version":"77'16","reported_seq":752,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.208447+0000","last_change":"2024-09-20T19:24:43.584229+0000","last_active":"2024-09-20T19:26:01.208447+0000","last_peered":"2024-09-20T19:26:01.208447+0000","last_clean":"2024-09-20T19:26:01.208447+0000","last_became_active":"2024-09-20T19:18:10.114356+0000","last_became_peered":"2024-09-20T19:18:10.114356+0000","last_unstale":"2024-09-20T19:26:01.208447+0000","last_undegraded":"2024-09-20T19:26:01.208447+0000","last_fullsized":"2024-09-20T19:26:01.208447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:24:43.584143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.584143+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":640,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173991+0000","last_change":"2024-09-20T19:24:46.018528+0000","last_active":"2024-09-20T19:26:01.173991+0000","last_peered":"2024-09-20T19:26:01.173991+0000","last_clean":"2024-09-20T19:26:01.173991+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:26:01.173991+0000","last_undegraded":"2024-09-20T19:26:01.173991+0000","last_fullsized":"2024-09-20T19:26:01.173991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:24:46.018399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.018399+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":640,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200454+0000","last_change":"2024-09-20T19:24:37.868287+0000","last_active":"2024-09-20T19:26:01.200454+0000","last_peered":"2024-09-20T19:26:01.200454+0000","last_clean":"2024-09-20T19:26:01.200454+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:26:01.200454+0000","last_undegraded":"2024-09-20T19:26:01.200454+0000","last_fullsized":"2024-09-20T19:26:01.200454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:37.868230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:37.868230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":688,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185550+0000","last_change":"2024-09-20T19:24:09.094725+0000","last_active":"2024-09-20T19:26:01.185550+0000","last_peered":"2024-09-20T19:26:01.185550+0000","last_clean":"2024-09-20T19:26:01.185550+0000","last_became_active":"2024-09-20T19:24:09.093719+0000","last_became_peered":"2024-09-20T19:24:09.093719+0000","last_unstale":"2024-09-20T19:26:01.185550+0000","last_undegraded":"2024-09-20T19:26:01.185550+0000","last_fullsized":"2024-09-20T19:26:01.185550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:23:51.334302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.334302+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:26:12.459756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185892+0000","last_change":"2024-09-20T19:24:30.320332+0000","last_active":"2024-09-20T19:26:01.185892+0000","last_peered":"2024-09-20T19:26:01.185892+0000","last_clean":"2024-09-20T19:26:01.185892+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:26:01.185892+0000","last_undegraded":"2024-09-20T19:26:01.185892+0000","last_fullsized":"2024-09-20T19:26:01.185892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:30.320193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.320193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":628,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200601+0000","last_change":"2024-09-20T19:23:49.765599+0000","last_active":"2024-09-20T19:26:01.200601+0000","last_peered":"2024-09-20T19:26:01.200601+0000","last_clean":"2024-09-20T19:26:01.200601+0000","last_became_active":"2024-09-20T19:18:13.356787+0000","last_became_peered":"2024-09-20T19:18:13.356787+0000","last_unstale":"2024-09-20T19:26:01.200601+0000","last_undegraded":"2024-09-20T19:26:01.200601+0000","last_fullsized":"2024-09-20T19:26:01.200601+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.765546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.765546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"371'254","reported_seq":1554,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185276+0000","last_change":"2024-09-20T19:26:00.179464+0000","last_active":"2024-09-20T19:26:01.185276+0000","last_peered":"2024-09-20T19:26:01.185276+0000","last_clean":"2024-09-20T19:26:01.185276+0000","last_became_active":"2024-09-20T19:21:30.267298+0000","last_became_peered":"2024-09-20T19:21:30.267298+0000","last_unstale":"2024-09-20T19:26:01.185276+0000","last_undegraded":"2024-09-20T19:26:01.185276+0000","last_fullsized":"2024-09-20T19:26:01.185276+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'184","last_scrub_stamp":"2024-09-20T19:24:28.915872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:28.915872+0000","objects_scrubbed":6,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.005211561,"stat_sum":{"num_bytes":17321675,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":132,"num_read_kb":111732,"num_write":79,"num_write_kb":31415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.a","version":"77'19","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207548+0000","last_change":"2024-09-20T19:25:26.387035+0000","last_active":"2024-09-20T19:26:01.207548+0000","last_peered":"2024-09-20T19:26:01.207548+0000","last_clean":"2024-09-20T19:26:01.207548+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:26:01.207548+0000","last_undegraded":"2024-09-20T19:26:01.207548+0000","last_fullsized":"2024-09-20T19:26:01.207548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:25:26.386911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:26.386911+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:42.950670+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.194383+0000","last_change":"2024-09-20T19:22:11.289926+0000","last_active":"2024-09-20T19:26:01.194383+0000","last_peered":"2024-09-20T19:26:01.194383+0000","last_clean":"2024-09-20T19:26:01.194383+0000","last_became_active":"2024-09-20T19:18:07.253064+0000","last_became_peered":"2024-09-20T19:18:07.253064+0000","last_unstale":"2024-09-20T19:26:01.194383+0000","last_undegraded":"2024-09-20T19:26:01.194383+0000","last_fullsized":"2024-09-20T19:26:01.194383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:11.289877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:11.289877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:06.718830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.192391+0000","last_change":"2024-09-20T19:25:14.776667+0000","last_active":"2024-09-20T19:26:01.192391+0000","last_peered":"2024-09-20T19:26:01.192391+0000","last_clean":"2024-09-20T19:26:01.192391+0000","last_became_active":"2024-09-20T19:18:11.326575+0000","last_became_peered":"2024-09-20T19:18:11.326575+0000","last_unstale":"2024-09-20T19:26:01.192391+0000","last_undegraded":"2024-09-20T19:26:01.192391+0000","last_fullsized":"2024-09-20T19:26:01.192391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:14.776619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:14.776619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:32.017831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":618,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185380+0000","last_change":"2024-09-20T19:23:42.246466+0000","last_active":"2024-09-20T19:26:01.185380+0000","last_peered":"2024-09-20T19:26:01.185380+0000","last_clean":"2024-09-20T19:26:01.185380+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:26:01.185380+0000","last_undegraded":"2024-09-20T19:26:01.185380+0000","last_fullsized":"2024-09-20T19:26:01.185380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.246406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.246406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:48.363790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"371'612","reported_seq":1886,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.193546+0000","last_change":"2024-09-20T19:26:00.170036+0000","last_active":"2024-09-20T19:26:01.193546+0000","last_peered":"2024-09-20T19:26:01.193546+0000","last_clean":"2024-09-20T19:26:01.193546+0000","last_became_active":"2024-09-20T19:21:30.237389+0000","last_became_peered":"2024-09-20T19:21:30.237389+0000","last_unstale":"2024-09-20T19:26:01.193546+0000","last_undegraded":"2024-09-20T19:26:01.193546+0000","last_fullsized":"2024-09-20T19:26:01.193546+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'341","last_scrub_stamp":"2024-09-20T19:24:32.860576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.860576+0000","objects_scrubbed":9,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00706326,"stat_sum":{"num_bytes":43312981,"num_objects":20,"num_object_clones":15,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":220,"num_read_kb":183041,"num_write":192,"num_write_kb":74517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"},{"start":"7d","length":"2"},{"start":"80","length":"1"}]},{"pgid":"4.b","version":"77'9","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185972+0000","last_change":"2024-09-20T19:25:22.114771+0000","last_active":"2024-09-20T19:26:01.185972+0000","last_peered":"2024-09-20T19:26:01.185972+0000","last_clean":"2024-09-20T19:26:01.185972+0000","last_became_active":"2024-09-20T19:18:09.292182+0000","last_became_peered":"2024-09-20T19:18:09.292182+0000","last_unstale":"2024-09-20T19:26:01.185972+0000","last_undegraded":"2024-09-20T19:26:01.185972+0000","last_fullsized":"2024-09-20T19:26:01.185972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:25:22.114696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:22.114696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:43.746548+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172294+0000","last_change":"2024-09-20T19:24:38.562224+0000","last_active":"2024-09-20T19:26:01.172294+0000","last_peered":"2024-09-20T19:26:01.172294+0000","last_clean":"2024-09-20T19:26:01.172294+0000","last_became_active":"2024-09-20T19:18:07.267986+0000","last_became_peered":"2024-09-20T19:18:07.267986+0000","last_unstale":"2024-09-20T19:26:01.172294+0000","last_undegraded":"2024-09-20T19:26:01.172294+0000","last_fullsized":"2024-09-20T19:26:01.172294+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.562125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.562125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.191103+0000","last_change":"2024-09-20T19:25:35.635764+0000","last_active":"2024-09-20T19:26:01.191103+0000","last_peered":"2024-09-20T19:26:01.191103+0000","last_clean":"2024-09-20T19:26:01.191103+0000","last_became_active":"2024-09-20T19:18:11.305264+0000","last_became_peered":"2024-09-20T19:18:11.305264+0000","last_unstale":"2024-09-20T19:26:01.191103+0000","last_undegraded":"2024-09-20T19:26:01.191103+0000","last_fullsized":"2024-09-20T19:26:01.191103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:35.635707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:35.635707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:58.208738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":647,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186170+0000","last_change":"2024-09-20T19:25:31.993650+0000","last_active":"2024-09-20T19:26:01.186170+0000","last_peered":"2024-09-20T19:26:01.186170+0000","last_clean":"2024-09-20T19:26:01.186170+0000","last_became_active":"2024-09-20T19:18:13.338148+0000","last_became_peered":"2024-09-20T19:18:13.338148+0000","last_unstale":"2024-09-20T19:26:01.186170+0000","last_undegraded":"2024-09-20T19:26:01.186170+0000","last_fullsized":"2024-09-20T19:26:01.186170+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:25:31.993519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.993519+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:47.996077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"371'189","reported_seq":1488,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.165947+0000","last_change":"2024-09-20T19:26:00.391701+0000","last_active":"2024-09-20T19:26:01.165947+0000","last_peered":"2024-09-20T19:26:01.165947+0000","last_clean":"2024-09-20T19:26:01.165947+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:26:01.165947+0000","last_undegraded":"2024-09-20T19:26:01.165947+0000","last_fullsized":"2024-09-20T19:26:01.165947+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'122","last_scrub_stamp":"2024-09-20T19:25:18.429559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.429559+0000","objects_scrubbed":7,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:18.888098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.22293802600000001,"stat_sum":{"num_bytes":16364896,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":86289,"num_write":65,"num_write_kb":27674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"5"},{"start":"7a","length":"2"}]},{"pgid":"4.8","version":"77'15","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199356+0000","last_change":"2024-09-20T19:22:22.063998+0000","last_active":"2024-09-20T19:26:01.199356+0000","last_peered":"2024-09-20T19:26:01.199356+0000","last_clean":"2024-09-20T19:26:01.199356+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:26:01.199356+0000","last_undegraded":"2024-09-20T19:26:01.199356+0000","last_fullsized":"2024-09-20T19:26:01.199356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:22:22.063871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:22.063871+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:05.837622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195406+0000","last_change":"2024-09-20T19:25:31.622707+0000","last_active":"2024-09-20T19:26:01.195406+0000","last_peered":"2024-09-20T19:26:01.195406+0000","last_clean":"2024-09-20T19:26:01.195406+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:26:01.195406+0000","last_undegraded":"2024-09-20T19:26:01.195406+0000","last_fullsized":"2024-09-20T19:26:01.195406+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:25:31.622605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.622605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:34.484590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"77'11","reported_seq":743,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.184934+0000","last_change":"2024-09-20T19:24:34.334821+0000","last_active":"2024-09-20T19:26:01.184934+0000","last_peered":"2024-09-20T19:26:01.184934+0000","last_clean":"2024-09-20T19:26:01.184934+0000","last_became_active":"2024-09-20T19:18:11.332179+0000","last_became_peered":"2024-09-20T19:18:11.332179+0000","last_unstale":"2024-09-20T19:26:01.184934+0000","last_undegraded":"2024-09-20T19:26:01.184934+0000","last_fullsized":"2024-09-20T19:26:01.184934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:34.334743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:34.334743+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":628,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199394+0000","last_change":"2024-09-20T19:24:32.907835+0000","last_active":"2024-09-20T19:26:01.199394+0000","last_peered":"2024-09-20T19:26:01.199394+0000","last_clean":"2024-09-20T19:26:01.199394+0000","last_became_active":"2024-09-20T19:18:13.492153+0000","last_became_peered":"2024-09-20T19:18:13.492153+0000","last_unstale":"2024-09-20T19:26:01.199394+0000","last_undegraded":"2024-09-20T19:26:01.199394+0000","last_fullsized":"2024-09-20T19:26:01.199394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:32.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"372'246","reported_seq":1527,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:26:01.190540+0000","last_change":"2024-09-20T19:26:01.190540+0000","last_active":"2024-09-20T19:26:01.190540+0000","last_peered":"2024-09-20T19:26:01.190540+0000","last_clean":"2024-09-20T19:26:01.190540+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:26:01.190540+0000","last_undegraded":"2024-09-20T19:26:01.190540+0000","last_fullsized":"2024-09-20T19:26:01.190540+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'137","last_scrub_stamp":"2024-09-20T19:24:02.123477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:02.123477+0000","objects_scrubbed":6,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.192854954,"stat_sum":{"num_bytes":23958163,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":138,"num_read_kb":114181,"num_write":84,"num_write_kb":37406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5a","length":"1"},{"start":"5e","length":"1"},{"start":"69","length":"1"},{"start":"6d","length":"1"},{"start":"74","length":"2"},{"start":"77","length":"2"},{"start":"7a","length":"2"}]},{"pgid":"4.9","version":"77'12","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172647+0000","last_change":"2024-09-20T19:22:57.202081+0000","last_active":"2024-09-20T19:26:01.172647+0000","last_peered":"2024-09-20T19:26:01.172647+0000","last_clean":"2024-09-20T19:26:01.172647+0000","last_became_active":"2024-09-20T19:19:57.008503+0000","last_became_peered":"2024-09-20T19:19:57.008503+0000","last_unstale":"2024-09-20T19:26:01.172647+0000","last_undegraded":"2024-09-20T19:26:01.172647+0000","last_fullsized":"2024-09-20T19:26:01.172647+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:22:57.201945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:57.201945+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173678+0000","last_change":"2024-09-20T19:25:28.472331+0000","last_active":"2024-09-20T19:26:01.173678+0000","last_peered":"2024-09-20T19:26:01.173678+0000","last_clean":"2024-09-20T19:26:01.173678+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:18:07.264732+0000","last_unstale":"2024-09-20T19:26:01.173678+0000","last_undegraded":"2024-09-20T19:26:01.173678+0000","last_fullsized":"2024-09-20T19:26:01.173678+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:25:28.472194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.472194+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:35.904350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.195098+0000","last_change":"2024-09-20T19:25:19.834673+0000","last_active":"2024-09-20T19:26:01.195098+0000","last_peered":"2024-09-20T19:26:01.195098+0000","last_clean":"2024-09-20T19:26:01.195098+0000","last_became_active":"2024-09-20T19:18:11.311626+0000","last_became_peered":"2024-09-20T19:18:11.311626+0000","last_unstale":"2024-09-20T19:26:01.195098+0000","last_undegraded":"2024-09-20T19:26:01.195098+0000","last_fullsized":"2024-09-20T19:26:01.195098+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:19.834608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:19.834608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:38.986282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":629,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165255+0000","last_change":"2024-09-20T19:25:35.724736+0000","last_active":"2024-09-20T19:26:01.165255+0000","last_peered":"2024-09-20T19:26:01.165255+0000","last_clean":"2024-09-20T19:26:01.165255+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:26:01.165255+0000","last_undegraded":"2024-09-20T19:26:01.165255+0000","last_fullsized":"2024-09-20T19:26:01.165255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:35.724674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:35.724674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:43.751370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"77'9","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187798+0000","last_change":"2024-09-20T19:25:23.161173+0000","last_active":"2024-09-20T19:26:01.187798+0000","last_peered":"2024-09-20T19:26:01.187798+0000","last_clean":"2024-09-20T19:26:01.187798+0000","last_became_active":"2024-09-20T19:18:10.109769+0000","last_became_peered":"2024-09-20T19:18:10.109769+0000","last_unstale":"2024-09-20T19:26:01.187798+0000","last_undegraded":"2024-09-20T19:26:01.187798+0000","last_fullsized":"2024-09-20T19:26:01.187798+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:25:23.161060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:23.161060+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:52.534719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.194863+0000","last_change":"2024-09-20T19:24:34.773702+0000","last_active":"2024-09-20T19:26:01.194863+0000","last_peered":"2024-09-20T19:26:01.194863+0000","last_clean":"2024-09-20T19:26:01.194863+0000","last_became_active":"2024-09-20T19:18:07.263237+0000","last_became_peered":"2024-09-20T19:18:07.263237+0000","last_unstale":"2024-09-20T19:26:01.194863+0000","last_undegraded":"2024-09-20T19:26:01.194863+0000","last_fullsized":"2024-09-20T19:26:01.194863+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:34.773576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:34.773576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":625,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173079+0000","last_change":"2024-09-20T19:21:06.023309+0000","last_active":"2024-09-20T19:26:01.173079+0000","last_peered":"2024-09-20T19:26:01.173079+0000","last_clean":"2024-09-20T19:26:01.173079+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:26:01.173079+0000","last_undegraded":"2024-09-20T19:26:01.173079+0000","last_fullsized":"2024-09-20T19:26:01.173079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:06.023257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:06.023257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:58.932657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.190823+0000","last_change":"2024-09-20T19:24:31.849608+0000","last_active":"2024-09-20T19:26:01.190823+0000","last_peered":"2024-09-20T19:26:01.190823+0000","last_clean":"2024-09-20T19:26:01.190823+0000","last_became_active":"2024-09-20T19:18:13.362554+0000","last_became_peered":"2024-09-20T19:18:13.362554+0000","last_unstale":"2024-09-20T19:26:01.190823+0000","last_undegraded":"2024-09-20T19:26:01.190823+0000","last_fullsized":"2024-09-20T19:26:01.190823+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:31.849481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:31.849481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":647,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207673+0000","last_change":"2024-09-20T19:24:38.575951+0000","last_active":"2024-09-20T19:26:01.207673+0000","last_peered":"2024-09-20T19:26:01.207673+0000","last_clean":"2024-09-20T19:26:01.207673+0000","last_became_active":"2024-09-20T19:18:07.272096+0000","last_became_peered":"2024-09-20T19:18:07.272096+0000","last_unstale":"2024-09-20T19:26:01.207673+0000","last_undegraded":"2024-09-20T19:26:01.207673+0000","last_fullsized":"2024-09-20T19:26:01.207673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.575807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.575807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"77'6","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173332+0000","last_change":"2024-09-20T19:24:29.131043+0000","last_active":"2024-09-20T19:26:01.173332+0000","last_peered":"2024-09-20T19:26:01.173332+0000","last_clean":"2024-09-20T19:26:01.173332+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:26:01.173332+0000","last_undegraded":"2024-09-20T19:26:01.173332+0000","last_fullsized":"2024-09-20T19:26:01.173332+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:24:29.130909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:29.130909+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":632,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200320+0000","last_change":"2024-09-20T19:23:50.759135+0000","last_active":"2024-09-20T19:26:01.200320+0000","last_peered":"2024-09-20T19:26:01.200320+0000","last_clean":"2024-09-20T19:26:01.200320+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:26:01.200320+0000","last_undegraded":"2024-09-20T19:26:01.200320+0000","last_fullsized":"2024-09-20T19:26:01.200320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":626,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185940+0000","last_change":"2024-09-20T19:25:27.296760+0000","last_active":"2024-09-20T19:26:01.185940+0000","last_peered":"2024-09-20T19:26:01.185940+0000","last_clean":"2024-09-20T19:26:01.185940+0000","last_became_active":"2024-09-20T19:18:13.497376+0000","last_became_peered":"2024-09-20T19:18:13.497376+0000","last_unstale":"2024-09-20T19:26:01.185940+0000","last_undegraded":"2024-09-20T19:26:01.185940+0000","last_fullsized":"2024-09-20T19:26:01.185940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.296694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.296694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:54.684693+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"77'10","reported_seq":652,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172091+0000","last_change":"2024-09-20T19:23:50.024939+0000","last_active":"2024-09-20T19:26:01.172091+0000","last_peered":"2024-09-20T19:26:01.172091+0000","last_clean":"2024-09-20T19:26:01.172091+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:26:01.172091+0000","last_undegraded":"2024-09-20T19:26:01.172091+0000","last_fullsized":"2024-09-20T19:26:01.172091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:50.024860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.024860+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":648,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199872+0000","last_change":"2024-09-20T19:23:56.858288+0000","last_active":"2024-09-20T19:26:01.199872+0000","last_peered":"2024-09-20T19:26:01.199872+0000","last_clean":"2024-09-20T19:26:01.199872+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:26:01.199872+0000","last_undegraded":"2024-09-20T19:26:01.199872+0000","last_fullsized":"2024-09-20T19:26:01.199872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:56.858227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.858227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"77'11","reported_seq":745,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200061+0000","last_change":"2024-09-20T19:23:53.824722+0000","last_active":"2024-09-20T19:26:01.200061+0000","last_peered":"2024-09-20T19:26:01.200061+0000","last_clean":"2024-09-20T19:26:01.200061+0000","last_became_active":"2024-09-20T19:18:11.322024+0000","last_became_peered":"2024-09-20T19:18:11.322024+0000","last_unstale":"2024-09-20T19:26:01.200061+0000","last_undegraded":"2024-09-20T19:26:01.200061+0000","last_fullsized":"2024-09-20T19:26:01.200061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:53.824647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:53.824647+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186699+0000","last_change":"2024-09-20T19:25:30.079904+0000","last_active":"2024-09-20T19:26:01.186699+0000","last_peered":"2024-09-20T19:26:01.186699+0000","last_clean":"2024-09-20T19:26:01.186699+0000","last_became_active":"2024-09-20T19:18:13.349392+0000","last_became_peered":"2024-09-20T19:18:13.349392+0000","last_unstale":"2024-09-20T19:26:01.186699+0000","last_undegraded":"2024-09-20T19:26:01.186699+0000","last_fullsized":"2024-09-20T19:26:01.186699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:30.079831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:30.079831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:59.531067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'9","reported_seq":652,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200231+0000","last_change":"2024-09-20T19:22:54.923862+0000","last_active":"2024-09-20T19:26:01.200231+0000","last_peered":"2024-09-20T19:26:01.200231+0000","last_clean":"2024-09-20T19:26:01.200231+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:26:01.200231+0000","last_undegraded":"2024-09-20T19:26:01.200231+0000","last_fullsized":"2024-09-20T19:26:01.200231+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:22:54.923725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:54.923725+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:07.783052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.186752+0000","last_change":"2024-09-20T19:22:20.671257+0000","last_active":"2024-09-20T19:26:01.186752+0000","last_peered":"2024-09-20T19:26:01.186752+0000","last_clean":"2024-09-20T19:26:01.186752+0000","last_became_active":"2024-09-20T19:18:07.271829+0000","last_became_peered":"2024-09-20T19:18:07.271829+0000","last_unstale":"2024-09-20T19:26:01.186752+0000","last_undegraded":"2024-09-20T19:26:01.186752+0000","last_fullsized":"2024-09-20T19:26:01.186752+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:20.671196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:20.671196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:05.137599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":738,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.171427+0000","last_change":"2024-09-20T19:25:31.665321+0000","last_active":"2024-09-20T19:26:01.171427+0000","last_peered":"2024-09-20T19:26:01.171427+0000","last_clean":"2024-09-20T19:26:01.171427+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:26:01.171427+0000","last_undegraded":"2024-09-20T19:26:01.171427+0000","last_fullsized":"2024-09-20T19:26:01.171427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:25:31.665239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.665239+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:38.271280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173442+0000","last_change":"2024-09-20T19:24:39.339495+0000","last_active":"2024-09-20T19:26:01.173442+0000","last_peered":"2024-09-20T19:26:01.173442+0000","last_clean":"2024-09-20T19:26:01.173442+0000","last_became_active":"2024-09-20T19:18:13.343275+0000","last_became_peered":"2024-09-20T19:18:13.343275+0000","last_unstale":"2024-09-20T19:26:01.173442+0000","last_undegraded":"2024-09-20T19:26:01.173442+0000","last_fullsized":"2024-09-20T19:26:01.173442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:39.339335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.339335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166436+0000","last_change":"2024-09-20T19:24:39.523128+0000","last_active":"2024-09-20T19:26:01.166436+0000","last_peered":"2024-09-20T19:26:01.166436+0000","last_clean":"2024-09-20T19:26:01.166436+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:26:01.166436+0000","last_undegraded":"2024-09-20T19:26:01.166436+0000","last_fullsized":"2024-09-20T19:26:01.166436+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:24:39.523025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.523025+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:07.695963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":639,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207935+0000","last_change":"2024-09-20T19:24:42.625537+0000","last_active":"2024-09-20T19:26:01.207935+0000","last_peered":"2024-09-20T19:26:01.207935+0000","last_clean":"2024-09-20T19:26:01.207935+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:26:01.207935+0000","last_undegraded":"2024-09-20T19:26:01.207935+0000","last_fullsized":"2024-09-20T19:26:01.207935+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.625450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.625450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":625,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165514+0000","last_change":"2024-09-20T19:21:05.014597+0000","last_active":"2024-09-20T19:26:01.165514+0000","last_peered":"2024-09-20T19:26:01.165514+0000","last_clean":"2024-09-20T19:26:01.165514+0000","last_became_active":"2024-09-20T19:18:11.312558+0000","last_became_peered":"2024-09-20T19:18:11.312558+0000","last_unstale":"2024-09-20T19:26:01.165514+0000","last_undegraded":"2024-09-20T19:26:01.165514+0000","last_fullsized":"2024-09-20T19:26:01.165514+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:59.922784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.184412+0000","last_change":"2024-09-20T19:25:31.041217+0000","last_active":"2024-09-20T19:26:01.184412+0000","last_peered":"2024-09-20T19:26:01.184412+0000","last_clean":"2024-09-20T19:26:01.184412+0000","last_became_active":"2024-09-20T19:18:13.341874+0000","last_became_peered":"2024-09-20T19:18:13.341874+0000","last_unstale":"2024-09-20T19:26:01.184412+0000","last_undegraded":"2024-09-20T19:26:01.184412+0000","last_fullsized":"2024-09-20T19:26:01.184412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:31.041093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.041093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:40.310789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"77'11","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172790+0000","last_change":"2024-09-20T19:24:30.435401+0000","last_active":"2024-09-20T19:26:01.172790+0000","last_peered":"2024-09-20T19:26:01.172790+0000","last_clean":"2024-09-20T19:26:01.172790+0000","last_became_active":"2024-09-20T19:18:09.311317+0000","last_became_peered":"2024-09-20T19:18:09.311317+0000","last_unstale":"2024-09-20T19:26:01.172790+0000","last_undegraded":"2024-09-20T19:26:01.172790+0000","last_fullsized":"2024-09-20T19:26:01.172790+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:30.435230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.435230+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172007+0000","last_change":"2024-09-20T19:25:16.437209+0000","last_active":"2024-09-20T19:26:01.172007+0000","last_peered":"2024-09-20T19:26:01.172007+0000","last_clean":"2024-09-20T19:26:01.172007+0000","last_became_active":"2024-09-20T19:18:07.274902+0000","last_became_peered":"2024-09-20T19:18:07.274902+0000","last_unstale":"2024-09-20T19:26:01.172007+0000","last_undegraded":"2024-09-20T19:26:01.172007+0000","last_fullsized":"2024-09-20T19:26:01.172007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:25:16.437044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.437044+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:33.706117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":641,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165889+0000","last_change":"2024-09-20T19:25:20.158995+0000","last_active":"2024-09-20T19:26:01.165889+0000","last_peered":"2024-09-20T19:26:01.165889+0000","last_clean":"2024-09-20T19:26:01.165889+0000","last_became_active":"2024-09-20T19:25:20.158758+0000","last_became_peered":"2024-09-20T19:25:20.158758+0000","last_unstale":"2024-09-20T19:26:01.165889+0000","last_undegraded":"2024-09-20T19:26:01.165889+0000","last_fullsized":"2024-09-20T19:26:01.165889+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:44.572407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.572407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:43.240234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":621,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165141+0000","last_change":"2024-09-20T19:23:47.003723+0000","last_active":"2024-09-20T19:26:01.165141+0000","last_peered":"2024-09-20T19:26:01.165141+0000","last_clean":"2024-09-20T19:26:01.165141+0000","last_became_active":"2024-09-20T19:18:13.363166+0000","last_became_peered":"2024-09-20T19:18:13.363166+0000","last_unstale":"2024-09-20T19:26:01.165141+0000","last_undegraded":"2024-09-20T19:26:01.165141+0000","last_fullsized":"2024-09-20T19:26:01.165141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.003602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.003602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:47.786668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"77'4","reported_seq":643,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165141+0000","last_change":"2024-09-20T19:22:50.638425+0000","last_active":"2024-09-20T19:26:01.165141+0000","last_peered":"2024-09-20T19:26:01.165141+0000","last_clean":"2024-09-20T19:26:01.165141+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:26:01.165141+0000","last_undegraded":"2024-09-20T19:26:01.165141+0000","last_fullsized":"2024-09-20T19:26:01.165141+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:22:50.638257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:50.638257+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:03.847632+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":640,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.200444+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:26:01.200444+0000","last_peered":"2024-09-20T19:26:01.200444+0000","last_clean":"2024-09-20T19:26:01.200444+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:26:01.200444+0000","last_undegraded":"2024-09-20T19:26:01.200444+0000","last_fullsized":"2024-09-20T19:26:01.200444+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":639,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207210+0000","last_change":"2024-09-20T19:25:32.545063+0000","last_active":"2024-09-20T19:26:01.207210+0000","last_peered":"2024-09-20T19:26:01.207210+0000","last_clean":"2024-09-20T19:26:01.207210+0000","last_became_active":"2024-09-20T19:18:11.337651+0000","last_became_peered":"2024-09-20T19:18:11.337651+0000","last_unstale":"2024-09-20T19:26:01.207210+0000","last_undegraded":"2024-09-20T19:26:01.207210+0000","last_fullsized":"2024-09-20T19:26:01.207210+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.544996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.544996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:52.041734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":618,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185090+0000","last_change":"2024-09-20T19:24:43.313779+0000","last_active":"2024-09-20T19:26:01.185090+0000","last_peered":"2024-09-20T19:26:01.185090+0000","last_clean":"2024-09-20T19:26:01.185090+0000","last_became_active":"2024-09-20T19:18:13.370151+0000","last_became_peered":"2024-09-20T19:18:13.370151+0000","last_unstale":"2024-09-20T19:26:01.185090+0000","last_undegraded":"2024-09-20T19:26:01.185090+0000","last_fullsized":"2024-09-20T19:26:01.185090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:43.313654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.313654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":656,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166125+0000","last_change":"2024-09-20T19:22:08.939934+0000","last_active":"2024-09-20T19:26:01.166125+0000","last_peered":"2024-09-20T19:26:01.166125+0000","last_clean":"2024-09-20T19:26:01.166125+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:26:01.166125+0000","last_undegraded":"2024-09-20T19:26:01.166125+0000","last_fullsized":"2024-09-20T19:26:01.166125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:22:08.939842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:22:08.939842+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:10.686495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":647,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.209024+0000","last_change":"2024-09-20T19:25:29.422580+0000","last_active":"2024-09-20T19:26:01.209024+0000","last_peered":"2024-09-20T19:26:01.209024+0000","last_clean":"2024-09-20T19:26:01.209024+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:26:01.209024+0000","last_undegraded":"2024-09-20T19:26:01.209024+0000","last_fullsized":"2024-09-20T19:26:01.209024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:29.422430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:29.422430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:49.439972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.184736+0000","last_change":"2024-09-20T19:24:39.314669+0000","last_active":"2024-09-20T19:26:01.184736+0000","last_peered":"2024-09-20T19:26:01.184736+0000","last_clean":"2024-09-20T19:26:01.184736+0000","last_became_active":"2024-09-20T19:18:11.333037+0000","last_became_peered":"2024-09-20T19:18:11.333037+0000","last_unstale":"2024-09-20T19:26:01.184736+0000","last_undegraded":"2024-09-20T19:26:01.184736+0000","last_fullsized":"2024-09-20T19:26:01.184736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:39.314605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.314605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.165158+0000","last_change":"2024-09-20T19:25:20.682415+0000","last_active":"2024-09-20T19:26:01.165158+0000","last_peered":"2024-09-20T19:26:01.165158+0000","last_clean":"2024-09-20T19:26:01.165158+0000","last_became_active":"2024-09-20T19:18:13.492664+0000","last_became_peered":"2024-09-20T19:18:13.492664+0000","last_unstale":"2024-09-20T19:26:01.165158+0000","last_undegraded":"2024-09-20T19:26:01.165158+0000","last_fullsized":"2024-09-20T19:26:01.165158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.682353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.682353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:24.003510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":628,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.185881+0000","last_change":"2024-09-20T19:25:28.320608+0000","last_active":"2024-09-20T19:26:01.185881+0000","last_peered":"2024-09-20T19:26:01.185881+0000","last_clean":"2024-09-20T19:26:01.185881+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:26:01.185881+0000","last_undegraded":"2024-09-20T19:26:01.185881+0000","last_fullsized":"2024-09-20T19:26:01.185881+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:25:28.320477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.320477+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:55.042604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":648,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.199522+0000","last_change":"2024-09-20T19:23:11.005669+0000","last_active":"2024-09-20T19:26:01.199522+0000","last_peered":"2024-09-20T19:26:01.199522+0000","last_clean":"2024-09-20T19:26:01.199522+0000","last_became_active":"2024-09-20T19:18:07.258745+0000","last_became_peered":"2024-09-20T19:18:07.258745+0000","last_unstale":"2024-09-20T19:26:01.199522+0000","last_undegraded":"2024-09-20T19:26:01.199522+0000","last_fullsized":"2024-09-20T19:26:01.199522+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:11.005587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:11.005587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187425+0000","last_change":"2024-09-20T19:24:30.887896+0000","last_active":"2024-09-20T19:26:01.187425+0000","last_peered":"2024-09-20T19:26:01.187425+0000","last_clean":"2024-09-20T19:26:01.187425+0000","last_became_active":"2024-09-20T19:18:10.109829+0000","last_became_peered":"2024-09-20T19:18:10.109829+0000","last_unstale":"2024-09-20T19:26:01.187425+0000","last_undegraded":"2024-09-20T19:26:01.187425+0000","last_fullsized":"2024-09-20T19:26:01.187425+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:24:30.887749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.887749+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:45.992418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"77'11","reported_seq":747,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.208945+0000","last_change":"2024-09-20T19:23:51.659148+0000","last_active":"2024-09-20T19:26:01.208945+0000","last_peered":"2024-09-20T19:26:01.208945+0000","last_clean":"2024-09-20T19:26:01.208945+0000","last_became_active":"2024-09-20T19:18:11.335584+0000","last_became_peered":"2024-09-20T19:18:11.335584+0000","last_unstale":"2024-09-20T19:26:01.208945+0000","last_undegraded":"2024-09-20T19:26:01.208945+0000","last_fullsized":"2024-09-20T19:26:01.208945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:51.659044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.659044+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.170204+0000","last_change":"2024-09-20T19:25:20.432414+0000","last_active":"2024-09-20T19:26:01.170204+0000","last_peered":"2024-09-20T19:26:01.170204+0000","last_clean":"2024-09-20T19:26:01.170204+0000","last_became_active":"2024-09-20T19:18:13.356464+0000","last_became_peered":"2024-09-20T19:18:13.356464+0000","last_unstale":"2024-09-20T19:26:01.170204+0000","last_undegraded":"2024-09-20T19:26:01.170204+0000","last_fullsized":"2024-09-20T19:26:01.170204+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.432349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.432349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:27.601529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":650,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.166936+0000","last_change":"2024-09-20T19:24:30.106374+0000","last_active":"2024-09-20T19:26:01.166936+0000","last_peered":"2024-09-20T19:26:01.166936+0000","last_clean":"2024-09-20T19:26:01.166936+0000","last_became_active":"2024-09-20T19:18:07.271611+0000","last_became_peered":"2024-09-20T19:18:07.271611+0000","last_unstale":"2024-09-20T19:26:01.166936+0000","last_undegraded":"2024-09-20T19:26:01.166936+0000","last_fullsized":"2024-09-20T19:26:01.166936+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:24:30.106245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.106245+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":681,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.208930+0000","last_change":"2024-09-20T19:24:03.690277+0000","last_active":"2024-09-20T19:26:01.208930+0000","last_peered":"2024-09-20T19:26:01.208930+0000","last_clean":"2024-09-20T19:26:01.208930+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:26:01.208930+0000","last_undegraded":"2024-09-20T19:26:01.208930+0000","last_fullsized":"2024-09-20T19:26:01.208930+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:03.690144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.690144+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.187118+0000","last_change":"2024-09-20T19:25:36.003332+0000","last_active":"2024-09-20T19:26:01.187118+0000","last_peered":"2024-09-20T19:26:01.187118+0000","last_clean":"2024-09-20T19:26:01.187118+0000","last_became_active":"2024-09-20T19:18:11.327215+0000","last_became_peered":"2024-09-20T19:18:11.327215+0000","last_unstale":"2024-09-20T19:26:01.187118+0000","last_undegraded":"2024-09-20T19:26:01.187118+0000","last_fullsized":"2024-09-20T19:26:01.187118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.003180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.003180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:43.972756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173462+0000","last_change":"2024-09-20T19:24:36.359489+0000","last_active":"2024-09-20T19:26:01.173462+0000","last_peered":"2024-09-20T19:26:01.173462+0000","last_clean":"2024-09-20T19:26:01.173462+0000","last_became_active":"2024-09-20T19:18:13.494084+0000","last_became_peered":"2024-09-20T19:18:13.494084+0000","last_unstale":"2024-09-20T19:26:01.173462+0000","last_undegraded":"2024-09-20T19:26:01.173462+0000","last_fullsized":"2024-09-20T19:26:01.173462+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:36.359306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:36.359306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":647,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.207888+0000","last_change":"2024-09-20T19:23:52.648685+0000","last_active":"2024-09-20T19:26:01.207888+0000","last_peered":"2024-09-20T19:26:01.207888+0000","last_clean":"2024-09-20T19:26:01.207888+0000","last_became_active":"2024-09-20T19:18:07.265893+0000","last_became_peered":"2024-09-20T19:18:07.265893+0000","last_unstale":"2024-09-20T19:26:01.207888+0000","last_undegraded":"2024-09-20T19:26:01.207888+0000","last_fullsized":"2024-09-20T19:26:01.207888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.648578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.648578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"77'15","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.193760+0000","last_change":"2024-09-20T19:23:56.744113+0000","last_active":"2024-09-20T19:26:01.193760+0000","last_peered":"2024-09-20T19:26:01.193760+0000","last_clean":"2024-09-20T19:26:01.193760+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:26:01.193760+0000","last_undegraded":"2024-09-20T19:26:01.193760+0000","last_fullsized":"2024-09-20T19:26:01.193760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:56.744020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.744020+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:01.693345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.171247+0000","last_change":"2024-09-20T19:23:57.510594+0000","last_active":"2024-09-20T19:26:01.171247+0000","last_peered":"2024-09-20T19:26:01.171247+0000","last_clean":"2024-09-20T19:26:01.171247+0000","last_became_active":"2024-09-20T19:18:11.311613+0000","last_became_peered":"2024-09-20T19:18:11.311613+0000","last_unstale":"2024-09-20T19:26:01.171247+0000","last_undegraded":"2024-09-20T19:26:01.171247+0000","last_fullsized":"2024-09-20T19:26:01.171247+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:57.510542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:57.510542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:47.359326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":621,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.172928+0000","last_change":"2024-09-20T19:24:46.978915+0000","last_active":"2024-09-20T19:26:01.172928+0000","last_peered":"2024-09-20T19:26:01.172928+0000","last_clean":"2024-09-20T19:26:01.172928+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:26:01.172928+0000","last_undegraded":"2024-09-20T19:26:01.172928+0000","last_fullsized":"2024-09-20T19:26:01.172928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:46.978796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.978796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:08.353484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":633,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.167275+0000","last_change":"2024-09-20T19:21:09.038005+0000","last_active":"2024-09-20T19:26:01.167275+0000","last_peered":"2024-09-20T19:26:01.167275+0000","last_clean":"2024-09-20T19:26:01.167275+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:26:01.167275+0000","last_undegraded":"2024-09-20T19:26:01.167275+0000","last_fullsized":"2024-09-20T19:26:01.167275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:09.037843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:21:09.037843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:54.864366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":660,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.168011+0000","last_change":"2024-09-20T19:25:25.699764+0000","last_active":"2024-09-20T19:26:01.168011+0000","last_peered":"2024-09-20T19:26:01.168011+0000","last_clean":"2024-09-20T19:26:01.168011+0000","last_became_active":"2024-09-20T19:18:09.298193+0000","last_became_peered":"2024-09-20T19:18:09.298193+0000","last_unstale":"2024-09-20T19:26:01.168011+0000","last_undegraded":"2024-09-20T19:26:01.168011+0000","last_fullsized":"2024-09-20T19:26:01.168011+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:25.699697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:25.699697+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:26:55.331429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":634,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-20T19:26:01.173061+0000","last_change":"2024-09-20T19:24:35.373273+0000","last_active":"2024-09-20T19:26:01.173061+0000","last_peered":"2024-09-20T19:26:01.173061+0000","last_clean":"2024-09-20T19:26:01.173061+0000","last_became_active":"2024-09-20T19:18:11.313785+0000","last_became_peered":"2024-09-20T19:18:11.313785+0000","last_unstale":"2024-09-20T19:26:01.173061+0000","last_undegraded":"2024-09-20T19:26:01.173061+0000","last_fullsized":"2024-09-20T19:26:01.173061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:35.373219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.373219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":454017076,"num_objects":202,"num_object_clones":152,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":11,"num_read":2492,"num_read_kb":1900430,"num_write":1770,"num_write_kb":692072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8689117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027600384,"data_stored":1335867534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5231,"ondisk_log_size":5231,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":458,"num_read_kb":453,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":41,"num_write":21,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907048,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":187496,"kb_used_data":147984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93557656,"statfs":{"total":95995035648,"available":95803039744,"internally_reserved":0,"allocated":151535616,"data_stored":203408075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234099,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145304,"kb_used_data":109956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93599848,"statfs":{"total":95995035648,"available":95846244352,"internally_reserved":0,"allocated":112594944,"data_stored":144832700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561149,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104216,"kb_used_data":68868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93640936,"statfs":{"total":95995035648,"available":95888318464,"internally_reserved":0,"allocated":70520832,"data_stored":83458117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855497,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139808,"kb_used_data":112772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93605344,"statfs":{"total":95995035648,"available":95851872256,"internally_reserved":0,"allocated":115478528,"data_stored":148646745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084435,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206168,"kb_used_data":166660,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93538984,"statfs":{"total":95995035648,"available":95783919616,"internally_reserved":0,"allocated":170659840,"data_stored":214117069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378785,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150880,"kb_used_data":119692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93594272,"statfs":{"total":95995035648,"available":95840534528,"internally_reserved":0,"allocated":122564608,"data_stored":164654029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575021,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172972,"kb_used_data":133464,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93572180,"statfs":{"total":95995035648,"available":95817912320,"internally_reserved":0,"allocated":136667136,"data_stored":166816913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204704,"kb_used_data":173516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93540448,"statfs":{"total":95995035648,"available":95785418752,"internally_reserved":0,"allocated":177680384,"data_stored":228637524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176771072,"data_stored":227452211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132739072,"data_stored":164565422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118378496,"data_stored":162140367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":166133760,"data_stored":211276027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":111575040,"data_stored":146404175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66019328,"data_stored":80616739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":108691456,"data_stored":142582781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":147292160,"data_stored":200829812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:26:07.565 INFO:tasks.thrashosds.thrasher:Setting 2.10 to [5, 4, 7, 1, 2, 0] 2024-09-20T19:26:07.565 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.10', '5', '4', '7', '1', '2', '0'] 2024-09-20T19:26:07.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pg-upmap-items 2.10 5 4 7 1 2 0 2024-09-20T19:26:07.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:07.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:07.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:07.764 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:07.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:07.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:07.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:07.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:08.119 INFO:tasks.rados.rados.0.smithi042.stdout:1206: done (2 left) 2024-09-20T19:26:08.119 INFO:tasks.rados.rados.0.smithi042.stdout:1207: rollback oid 22 current snap is 139 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 137 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: finishing write tid 1 to smithi04268048-12 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: finishing write tid 2 to smithi04268048-12 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: finishing write tid 3 to smithi04268048-12 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: finishing write tid 4 to smithi04268048-12 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: finishing write tid 5 to smithi04268048-12 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: oid 12 updating version 0 to 326 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: oid 12 version 326 is already newer than 324 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: oid 12 version 326 is already newer than 325 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: oid 12 version 326 is already newer than 323 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 326 (ObjNum 433 snap 139 seq_num 433) dirty exists 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1204: left oid 12 (ObjNum 433 snap 139 seq_num 433) 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:1205: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:08.120 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 45694 2024-09-20T19:26:08.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.155 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:08.183 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:08.253 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 988884 2024-09-20T19:26:08.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:08.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:08.295 INFO:tasks.rados.rados.0.smithi042.stdout:1207: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:26:08.295 INFO:tasks.rados.rados.0.smithi042.stdout:1207: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:26:08.295 INFO:tasks.rados.rados.0.smithi042.stdout:1207: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:26:08.295 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 394 (ObjNum 432 snap 136 seq_num 432) dirty exists 2024-09-20T19:26:08.296 INFO:tasks.rados.rados.0.smithi042.stdout:1204: done (2 left) 2024-09-20T19:26:08.296 INFO:tasks.rados.rados.0.smithi042.stdout:1205: done (1 left) 2024-09-20T19:26:08.296 INFO:tasks.rados.rados.0.smithi042.stdout:1207: done (0 left) 2024-09-20T19:26:08.296 INFO:tasks.rados.rados.0.smithi042.stdout:1208: snap_create 2024-09-20T19:26:08.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:08 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:08.518 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.339859+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.340652+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.969510+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:06.971802+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.304408+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.c scrub starts 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.306207+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub ok 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:08 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.117777+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-20T19:26:08.547 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:08.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:08.671 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:08.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:08.783 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:26:08.783 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:26:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:08.822 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:26:08.822 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:26:08.841 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:26:08.842 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:26:08.859 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:26:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:26:08.876 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:26:08.877 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:26:08.897 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:26:08.898 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:26:08.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.918 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:26:08.919 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:26:08.926 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:08.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:08.937 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:26:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:26:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:26:08.956 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:26:09.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:09.056 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:09.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:09.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:09.241 INFO:tasks.rados.rados.0.smithi042.stdout:1208: done (0 left) 2024-09-20T19:26:09.241 INFO:tasks.rados.rados.0.smithi042.stdout:1209: snap_create 2024-09-20T19:26:09.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:09.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:09.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:09.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:09 smithi162 bash[24557]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[32075]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.846740+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.849943+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.937680+0000 osd.0 (osd.0) 171 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:07.943800+0000 osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.059702+0000 mgr.y (mgr.24491) 314 : cluster [DBG] pgmap v506: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.095297+0000 osd.4 (osd.4) 157 : cluster [DBG] 4.d scrub starts 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.098570+0000 osd.4 (osd.4) 158 : cluster [DBG] 4.d scrub ok 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.666374+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.670064+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:09 smithi042 bash[22706]: audit 2024-09-20T19:26:08.803791+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.42:0/2965422436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:09.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:09.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:26:09.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:09.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:09.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:10.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:10.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:10.140 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:10.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:10.251 INFO:tasks.rados.rados.0.smithi042.stdout:1209: done (0 left) 2024-09-20T19:26:10.251 INFO:tasks.rados.rados.0.smithi042.stdout:1210: delete oid 5 current snap is 141 2024-09-20T19:26:10.255 INFO:tasks.rados.rados.0.smithi042.stdout:1210: done (0 left) 2024-09-20T19:26:10.255 INFO:tasks.rados.rados.0.smithi042.stdout:1211: snap_create 2024-09-20T19:26:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:10.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.391 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:10 smithi162 bash[24557]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:10.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:10.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:10.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[22706]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.328348+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.d scrub starts 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:08.329694+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.d scrub ok 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.241126+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:10 smithi042 bash[32075]: audit 2024-09-20T19:26:09.248234+0000 mgr.y (mgr.24491) 315 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:10.667 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:10.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:10.799 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:10.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:10.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:10.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:11.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:11.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:11.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:11.259 INFO:tasks.rados.rados.0.smithi042.stdout:1211: done (0 left) 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1212: read oid 18 snap -1 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1212: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1213: read oid 10 snap -1 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1213: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1214: rollback oid 16 current snap is 142 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 141 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1215: read oid 38 snap -1 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1215: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:26:11.260 INFO:tasks.rados.rados.0.smithi042.stdout:1216: snap_create 2024-09-20T19:26:11.265 INFO:tasks.rados.rados.0.smithi042.stdout:1214: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:26:11.265 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 331 (ObjNum 129 snap 0 seq_num 1414887248) dirty dne 2024-09-20T19:26:11.268 INFO:tasks.rados.rados.0.smithi042.stdout:1212: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:26:11.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:11.317 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:1213: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:11.391 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 279364 2024-09-20T19:26:11.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[32075]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:11 smithi042 bash[22706]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:11.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.831719+0000 osd.3 (osd.3) 163 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:26:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.838482+0000 osd.3 (osd.3) 164 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.934889+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:09.938793+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.060365+0000 mgr.y (mgr.24491) 316 : cluster [DBG] pgmap v509: 158 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 150 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.088692+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.092627+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.248822+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-20T19:26:11.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.361121+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.b scrub starts 2024-09-20T19:26:11.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:11 smithi162 bash[24557]: cluster 2024-09-20T19:26:10.363577+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.b scrub ok 2024-09-20T19:26:11.557 INFO:tasks.rados.rados.0.smithi042.stdout:1215: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:26:11.580 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:11.591 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:11.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:11.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:11.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:11.838 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:11.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:11.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:11.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:12.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:12.098 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:12.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:12.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:12.257 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.257 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.257 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.273 INFO:tasks.rados.rados.0.smithi042.stdout:1212: done (4 left) 2024-09-20T19:26:12.273 INFO:tasks.rados.rados.0.smithi042.stdout:1213: done (3 left) 2024-09-20T19:26:12.273 INFO:tasks.rados.rados.0.smithi042.stdout:1214: done (2 left) 2024-09-20T19:26:12.274 INFO:tasks.rados.rados.0.smithi042.stdout:1215: done (1 left) 2024-09-20T19:26:12.274 INFO:tasks.rados.rados.0.smithi042.stdout:1216: done (0 left) 2024-09-20T19:26:12.274 INFO:tasks.rados.rados.0.smithi042.stdout:1217: copy_from oid 6 from oid 50 current snap is 143 2024-09-20T19:26:12.274 INFO:tasks.rados.rados.0.smithi042.stdout:1218: snap_create 2024-09-20T19:26:12.301 INFO:tasks.rados.rados.0.smithi042.stdout:1217: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:26:12.304 INFO:tasks.rados.rados.0.smithi042.stdout:1217: finishing copy_from to smithi04268048-6 2024-09-20T19:26:12.304 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 496 (ObjNum 356 snap 115 seq_num 356) dirty exists 2024-09-20T19:26:12.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:12.351 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:26:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:12.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:12.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.473 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.474 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:12 smithi162 bash[24557]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:12.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:12.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[22706]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.107043+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.110026+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.259997+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.322365+0000 osd.7 (osd.7) 127 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.324306+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.785643+0000 osd.3 (osd.3) 165 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: cluster 2024-09-20T19:26:11.789328+0000 osd.3 (osd.3) 166 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:26:12.603 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.603 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:12 smithi042 bash[32075]: audit 2024-09-20T19:26:12.207548+0000 mon.a (mon.0) 1424 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:12.611 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:12.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:12.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:12.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:12.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:12.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:12.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:12.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:13.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:13.124 INFO:tasks.rados.rados.0.smithi042.stdout:1217: done (1 left) 2024-09-20T19:26:13.124 INFO:tasks.rados.rados.0.smithi042.stdout:1218: done (0 left) 2024-09-20T19:26:13.125 INFO:tasks.rados.rados.0.smithi042.stdout:1219: write oid 21 current snap is 144 2024-09-20T19:26:13.125 INFO:tasks.rados.rados.0.smithi042.stdout:1219: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-20T19:26:13.131 INFO:tasks.rados.rados.0.smithi042.stdout:1219: writing smithi04268048-21 from 511045 to 1076481 tid 1 2024-09-20T19:26:13.131 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:13.140 INFO:tasks.rados.rados.0.smithi042.stdout:1219: writing smithi04268048-21 from 1622020 to 2067497 tid 2 2024-09-20T19:26:13.148 INFO:tasks.rados.rados.0.smithi042.stdout:1219: writing smithi04268048-21 from 2729590 to 3286445 tid 3 2024-09-20T19:26:13.148 INFO:tasks.rados.rados.0.smithi042.stdout:1220: rollback oid 7 current snap is 144 2024-09-20T19:26:13.148 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 136 2024-09-20T19:26:13.148 INFO:tasks.rados.rados.0.smithi042.stdout:1219: finishing write tid 1 to smithi04268048-21 2024-09-20T19:26:13.149 INFO:tasks.rados.rados.0.smithi042.stdout:1219: finishing write tid 2 to smithi04268048-21 2024-09-20T19:26:13.149 INFO:tasks.rados.rados.0.smithi042.stdout:1221: read oid 45 snap -1 2024-09-20T19:26:13.149 INFO:tasks.rados.rados.0.smithi042.stdout:1221: expect (ObjNum 368 snap 118 seq_num 368) 2024-09-20T19:26:13.149 INFO:tasks.rados.rados.0.smithi042.stdout:1222: write oid 23 current snap is 144 2024-09-20T19:26:13.149 INFO:tasks.rados.rados.0.smithi042.stdout:1222: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-20T19:26:13.156 INFO:tasks.rados.rados.0.smithi042.stdout:1222: writing smithi04268048-23 from 527852 to 1168537 tid 1 2024-09-20T19:26:13.163 INFO:tasks.rados.rados.0.smithi042.stdout:1222: writing smithi04268048-23 from 1680502 to 2269637 tid 2 2024-09-20T19:26:13.169 INFO:tasks.rados.rados.0.smithi042.stdout:1222: writing smithi04268048-23 from 2757013 to 3327852 tid 3 2024-09-20T19:26:13.169 INFO:tasks.rados.rados.0.smithi042.stdout:1223: write (excl) oid 20 current snap is 144 2024-09-20T19:26:13.169 INFO:tasks.rados.rados.0.smithi042.stdout:1223: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-20T19:26:13.177 INFO:tasks.rados.rados.0.smithi042.stdout:1223: writing smithi04268048-20 from 544659 to 1260593 tid 1 2024-09-20T19:26:13.183 INFO:tasks.rados.rados.0.smithi042.stdout:1223: writing smithi04268048-20 from 1738984 to 2188130 tid 2 2024-09-20T19:26:13.189 INFO:tasks.rados.rados.0.smithi042.stdout:1223: writing smithi04268048-20 from 2784436 to 3344659 tid 3 2024-09-20T19:26:13.189 INFO:tasks.rados.rados.0.smithi042.stdout:1224: copy_from oid 17 from oid 26 current snap is 144 2024-09-20T19:26:13.189 INFO:tasks.rados.rados.0.smithi042.stdout:1221: expect (ObjNum 368 snap 118 seq_num 368) 2024-09-20T19:26:13.189 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:13.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:13.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:13.295 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1219: finishing write tid 3 to smithi04268048-21 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1220: finishing rollback tid 0 to smithi04268048-7 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1220: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1219: finishing write tid 4 to smithi04268048-21 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1220: finishing rollback tid 2 to smithi04268048-7 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 537 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1219: finishing write tid 5 to smithi04268048-21 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1219: oid 21 updating version 0 to 676 2024-09-20T19:26:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:1219: oid 21 updating version 676 to 678 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1219: oid 21 updating version 678 to 679 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1219: oid 21 version 679 is already newer than 677 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 679 (ObjNum 435 snap 144 seq_num 435) dirty exists 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1219: left oid 21 (ObjNum 435 snap 144 seq_num 435) 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: finishing write tid 1 to smithi04268048-23 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: finishing write tid 2 to smithi04268048-23 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: finishing write tid 3 to smithi04268048-23 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1219: done (5 left) 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1220: done (4 left) 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1221: done (3 left) 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1225: write (excl) oid 44 current snap is 144 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: finishing write tid 4 to smithi04268048-23 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: finishing write tid 5 to smithi04268048-23 2024-09-20T19:26:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:1222: oid 23 updating version 0 to 655 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1222: oid 23 updating version 655 to 656 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1222: oid 23 version 656 is already newer than 653 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1222: oid 23 version 656 is already newer than 654 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 656 (ObjNum 436 snap 144 seq_num 436) dirty exists 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1222: left oid 23 (ObjNum 436 snap 144 seq_num 436) 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: finishing write tid 1 to smithi04268048-20 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: finishing write tid 2 to smithi04268048-20 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: finishing write tid 3 to smithi04268048-20 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: finishing write tid 4 to smithi04268048-20 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: finishing write tid 5 to smithi04268048-20 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: oid 20 updating version 0 to 100 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: oid 20 updating version 100 to 101 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: oid 20 updating version 101 to 102 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:1223: oid 20 version 102 is already newer than 99 2024-09-20T19:26:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 102 (ObjNum 437 snap 144 seq_num 437) dirty exists 2024-09-20T19:26:13.351 INFO:tasks.rados.rados.0.smithi042.stdout:1223: left oid 20 (ObjNum 437 snap 144 seq_num 437) 2024-09-20T19:26:13.351 INFO:tasks.rados.rados.0.smithi042.stdout:1224: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:26:13.351 INFO:tasks.rados.rados.0.smithi042.stdout:1224: finishing copy_from to smithi04268048-17 2024-09-20T19:26:13.351 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 534 (ObjNum 412 snap 129 seq_num 412) dirty exists 2024-09-20T19:26:13.351 INFO:tasks.rados.rados.0.smithi042.stdout:1225: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-20T19:26:13.356 INFO:tasks.rados.rados.0.smithi042.stdout:1225: writing smithi04268048-44 from 561466 to 1352649 tid 1 2024-09-20T19:26:13.365 INFO:tasks.rados.rados.0.smithi042.stdout:1225: writing smithi04268048-44 from 2081113 to 2673917 tid 2 2024-09-20T19:26:13.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1225: writing smithi04268048-44 from 3095506 to 3361466 tid 3 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1222: done (3 left) 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1223: done (2 left) 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1224: done (1 left) 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1226: rollback oid 22 current snap is 144 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 127 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1225: finishing write tid 1 to smithi04268048-44 2024-09-20T19:26:13.370 INFO:tasks.rados.rados.0.smithi042.stdout:1227: snap_create 2024-09-20T19:26:13.373 INFO:tasks.rados.rados.0.smithi042.stdout:1225: finishing write tid 2 to smithi04268048-44 2024-09-20T19:26:13.374 INFO:tasks.rados.rados.0.smithi042.stdout:1225: finishing write tid 3 to smithi04268048-44 2024-09-20T19:26:13.375 INFO:tasks.rados.rados.0.smithi042.stdout:1225: finishing write tid 4 to smithi04268048-44 2024-09-20T19:26:13.375 INFO:tasks.rados.rados.0.smithi042.stdout:1225: finishing write tid 5 to smithi04268048-44 2024-09-20T19:26:13.375 INFO:tasks.rados.rados.0.smithi042.stdout:1225: oid 44 updating version 0 to 633 2024-09-20T19:26:13.376 INFO:tasks.rados.rados.0.smithi042.stdout:1225: oid 44 version 633 is already newer than 632 2024-09-20T19:26:13.376 INFO:tasks.rados.rados.0.smithi042.stdout:1225: oid 44 updating version 633 to 634 2024-09-20T19:26:13.376 INFO:tasks.rados.rados.0.smithi042.stdout:1225: oid 44 updating version 634 to 635 2024-09-20T19:26:13.376 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 635 (ObjNum 438 snap 144 seq_num 438) dirty exists 2024-09-20T19:26:13.376 INFO:tasks.rados.rados.0.smithi042.stdout:1225: left oid 44 (ObjNum 438 snap 144 seq_num 438) 2024-09-20T19:26:13.378 INFO:tasks.rados.rados.0.smithi042.stdout:1226: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:26:13.378 INFO:tasks.rados.rados.0.smithi042.stdout:1226: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:26:13.378 INFO:tasks.rados.rados.0.smithi042.stdout:1226: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:26:13.378 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 397 (ObjNum 340 snap 109 seq_num 340) dirty exists 2024-09-20T19:26:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:13.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.508 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.508 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:13 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:13.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.061933+0000 mgr.y (mgr.24491) 317 : cluster [DBG] pgmap v512: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.273250+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.365013+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:12.366510+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:26:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:13 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.122965+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-20T19:26:13.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:13.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:13.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:13.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:13.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:13.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:14.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:14.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:14.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:14.176 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:14.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.287 INFO:tasks.rados.rados.0.smithi042.stdout:1225: done (2 left) 2024-09-20T19:26:14.287 INFO:tasks.rados.rados.0.smithi042.stdout:1226: done (1 left) 2024-09-20T19:26:14.287 INFO:tasks.rados.rados.0.smithi042.stdout:1227: done (0 left) 2024-09-20T19:26:14.287 INFO:tasks.rados.rados.0.smithi042.stdout:1228: copy_from oid 19 from oid 31 current snap is 145 2024-09-20T19:26:14.287 INFO:tasks.rados.rados.0.smithi042.stdout:1229: delete oid 24 current snap is 145 2024-09-20T19:26:14.291 INFO:tasks.rados.rados.0.smithi042.stdout:1229: done (1 left) 2024-09-20T19:26:14.291 INFO:tasks.rados.rados.0.smithi042.stdout:1230: write (excl) oid 27 current snap is 145 2024-09-20T19:26:14.291 INFO:tasks.rados.rados.0.smithi042.stdout:1230: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-20T19:26:14.301 INFO:tasks.rados.rados.0.smithi042.stdout:1230: writing smithi04268048-27 from 595080 to 1136761 tid 1 2024-09-20T19:26:14.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:14.314 INFO:tasks.rados.rados.0.smithi042.stdout:1230: writing smithi04268048-27 from 1798077 to 2394550 tid 2 2024-09-20T19:26:14.323 INFO:tasks.rados.rados.0.smithi042.stdout:1230: writing smithi04268048-27 from 3150352 to 3395080 tid 3 2024-09-20T19:26:14.323 INFO:tasks.rados.rados.0.smithi042.stdout:1231: delete oid 46 current snap is 145 2024-09-20T19:26:14.323 INFO:tasks.rados.rados.0.smithi042.stdout:1230: finishing write tid 1 to smithi04268048-27 2024-09-20T19:26:14.323 INFO:tasks.rados.rados.0.smithi042.stdout:1228: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:26:14.324 INFO:tasks.rados.rados.0.smithi042.stdout:1228: finishing copy_from to smithi04268048-19 2024-09-20T19:26:14.324 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 392 (ObjNum 424 snap 133 seq_num 424) dirty exists 2024-09-20T19:26:14.324 INFO:tasks.rados.rados.0.smithi042.stdout:1230: finishing write tid 2 to smithi04268048-27 2024-09-20T19:26:14.327 INFO:tasks.rados.rados.0.smithi042.stdout:1230: finishing write tid 3 to smithi04268048-27 2024-09-20T19:26:14.327 INFO:tasks.rados.rados.0.smithi042.stdout:1228: done (2 left) 2024-09-20T19:26:14.327 INFO:tasks.rados.rados.0.smithi042.stdout:1231: done (1 left) 2024-09-20T19:26:14.327 INFO:tasks.rados.rados.0.smithi042.stdout:1232: read oid 24 snap -1 2024-09-20T19:26:14.327 INFO:tasks.rados.rados.0.smithi042.stdout:1232: expect deleted 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1233: copy_from oid 34 from oid 18 current snap is 145 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1234: read oid 50 snap -1 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1234: expect (ObjNum 356 snap 115 seq_num 356) 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1235: read oid 1 snap -1 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1235: expect (ObjNum 401 snap 127 seq_num 401) 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1236: write (excl) oid 21 current snap is 145 2024-09-20T19:26:14.328 INFO:tasks.rados.rados.0.smithi042.stdout:1236: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-20T19:26:14.337 INFO:tasks.rados.rados.0.smithi042.stdout:1236: writing smithi04268048-21 from 628694 to 1037226 tid 1 2024-09-20T19:26:14.347 INFO:tasks.rados.rados.0.smithi042.stdout:1236: writing smithi04268048-21 from 1515041 to 2115183 tid 2 2024-09-20T19:26:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:1236: writing smithi04268048-21 from 2805198 to 3357310 tid 3 2024-09-20T19:26:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:1237: copy_from oid 12 from oid 39 current snap is 145 2024-09-20T19:26:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:1230: finishing write tid 4 to smithi04268048-27 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1238: copy_from oid 46 from oid 48 current snap is 145 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: finishing write tid 5 to smithi04268048-27 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: oid 27 updating version 0 to 633 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: oid 27 updating version 633 to 634 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: oid 27 updating version 634 to 635 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: oid 27 version 635 is already newer than 632 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 635 (ObjNum 440 snap 145 seq_num 440) dirty exists 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1230: left oid 27 (ObjNum 440 snap 145 seq_num 440) 2024-09-20T19:26:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:1235: expect (ObjNum 401 snap 127 seq_num 401) 2024-09-20T19:26:14.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:14.435 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:14.534 INFO:tasks.rados.rados.0.smithi042.stdout:1234: expect (ObjNum 356 snap 115 seq_num 356) 2024-09-20T19:26:14.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:14 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:14.586 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3806290 2024-09-20T19:26:14.600 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 911628 2024-09-20T19:26:14.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.655 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.398669+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.400308+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.421921+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.424037+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.656125+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:14 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.657525+0000 osd.6 (osd.6) 116 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:26:14.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:14.673 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1409498 2024-09-20T19:26:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:14.737 INFO:tasks.rados.rados.0.smithi042.stdout:1236: finishing write tid 1 to smithi04268048-21 2024-09-20T19:26:14.737 INFO:tasks.rados.rados.0.smithi042.stdout:1233: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:26:14.737 INFO:tasks.rados.rados.0.smithi042.stdout:1233: finishing copy_from to smithi04268048-34 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: finishing write tid 2 to smithi04268048-21 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1230: done (7 left) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1232: done (6 left) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1233: done (5 left) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1234: done (4 left) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1235: done (3 left) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1239: read oid 14 snap -1 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1239: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1240: copy_from oid 26 from oid 9 current snap is 145 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: finishing write tid 3 to smithi04268048-21 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: finishing write tid 4 to smithi04268048-21 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: finishing write tid 5 to smithi04268048-21 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: oid 21 updating version 0 to 683 2024-09-20T19:26:14.738 INFO:tasks.rados.rados.0.smithi042.stdout:1236: oid 21 version 683 is already newer than 681 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1236: oid 21 version 683 is already newer than 682 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1236: oid 21 updating version 683 to 684 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 684 (ObjNum 442 snap 145 seq_num 442) dirty exists 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1236: left oid 21 (ObjNum 442 snap 145 seq_num 442) 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1237: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1237: finishing copy_from to smithi04268048-12 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 433 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1238: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1236: done (4 left) 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1237: done (3 left) 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1241: copy_from oid 16 from oid 47 current snap is 145 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1238: finishing copy_from to smithi04268048-46 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 541 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1238: done (3 left) 2024-09-20T19:26:14.739 INFO:tasks.rados.rados.0.smithi042.stdout:1242: snap_create 2024-09-20T19:26:14.741 INFO:tasks.rados.rados.0.smithi042.stdout:1239: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:26:14.762 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3370911 2024-09-20T19:26:14.765 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:14.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:14.814 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:14.843 INFO:tasks.rados.rados.0.smithi042.stdout:1241: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:26:14.844 INFO:tasks.rados.rados.0.smithi042.stdout:1240: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:26:14.844 INFO:tasks.rados.rados.0.smithi042.stdout:1241: finishing copy_from to smithi04268048-16 2024-09-20T19:26:14.844 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 529 (ObjNum 380 snap 123 seq_num 380) dirty exists 2024-09-20T19:26:14.844 INFO:tasks.rados.rados.0.smithi042.stdout:1240: finishing copy_from to smithi04268048-26 2024-09-20T19:26:14.844 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 597 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:14.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:14.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:15.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:15.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:15.177 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1239: done (3 left) 2024-09-20T19:26:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1240: done (2 left) 2024-09-20T19:26:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1241: done (1 left) 2024-09-20T19:26:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1242: done (0 left) 2024-09-20T19:26:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1243: delete oid 26 current snap is 146 2024-09-20T19:26:15.284 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:26:15.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:15.302 INFO:tasks.rados.rados.0.smithi042.stdout:1243: done (0 left) 2024-09-20T19:26:15.302 INFO:tasks.rados.rados.0.smithi042.stdout:1244: snap_create 2024-09-20T19:26:15.323 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:15.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:15.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.438 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[32075]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:15 smithi042 bash[22706]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.449 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:15.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.927834+0000 osd.0 (osd.0) 175 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:26:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:13.930345+0000 osd.0 (osd.0) 176 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:26:15.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.062668+0000 mgr.y (mgr.24491) 318 : cluster [DBG] pgmap v515: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 328 KiB/s wr, 11 op/s 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.224631+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.229227+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.287218+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.410666+0000 osd.1 (osd.1) 165 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: cluster 2024-09-20T19:26:14.412127+0000 osd.1 (osd.1) 166 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: audit 2024-09-20T19:26:15.034148+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.42:0/2218805886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:15 smithi162 bash[24557]: audit 2024-09-20T19:26:15.034452+0000 mon.a (mon.0) 1428 : audit [INF] from='client.25154 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:15.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:15.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:15.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:15.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:15.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:15.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:15.971 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:15.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:16.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:16.100 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:16.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:16.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:16.302 INFO:teuthology.orchestra.run.smithi042.stderr:set 2.10 pg_upmap_items mapping to [5->4,7->1,2->0] 2024-09-20T19:26:16.311 INFO:tasks.rados.rados.0.smithi042.stdout:1244: done (0 left) 2024-09-20T19:26:16.312 INFO:tasks.rados.rados.0.smithi042.stdout:1245: read oid 7 snap 138 2024-09-20T19:26:16.312 INFO:tasks.rados.rados.0.smithi042.stdout:1245: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:26:16.312 INFO:tasks.rados.rados.0.smithi042.stdout:1246: rollback oid 36 current snap is 147 2024-09-20T19:26:16.312 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 36 to 141 2024-09-20T19:26:16.312 INFO:tasks.rados.rados.0.smithi042.stdout:1247: snap_remove snap 142 2024-09-20T19:26:16.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:16.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:16.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:16.489 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:16.496 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[22706]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.500 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:16 smithi042 bash[32075]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.258171+0000 osd.2 (osd.2) 143 : cluster [DBG] 5.d scrub starts 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.260347+0000 osd.2 (osd.2) 144 : cluster [DBG] 5.d scrub ok 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.276781+0000 mon.a (mon.0) 1429 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:26:16.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.276801+0000 mon.a (mon.0) 1430 : cluster [INF] Cluster is now healthy 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: audit 2024-09-20T19:26:15.278641+0000 mon.a (mon.0) 1431 : audit [INF] from='client.25154 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.282877+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: audit 2024-09-20T19:26:15.372127+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]} : dispatch 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.966272+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:16 smithi162 bash[24557]: cluster 2024-09-20T19:26:15.968300+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:26:16.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:16.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:16.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:16.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:16.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:16.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:16.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:16.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:17.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:17.118 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:17.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:17.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:17.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:17.324 INFO:tasks.rados.rados.0.smithi042.stdout:1247: done (2 left) 2024-09-20T19:26:17.324 INFO:tasks.rados.rados.0.smithi042.stdout:1248: copy_from oid 13 from oid 10 current snap is 147 2024-09-20T19:26:17.328 INFO:tasks.rados.rados.0.smithi042.stdout:1249: read oid 42 snap -1 2024-09-20T19:26:17.328 INFO:tasks.rados.rados.0.smithi042.stdout:1249: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1250: rollback oid 5 current snap is 147 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 134 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1251: delete oid 30 current snap is 147 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1246: finishing rollback tid 0 to smithi04268048-36 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1246: finishing rollback tid 1 to smithi04268048-36 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1246: finishing rollback tid 2 to smithi04268048-36 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 320 (ObjNum 354 snap 115 seq_num 354) dirty exists 2024-09-20T19:26:17.329 INFO:tasks.rados.rados.0.smithi042.stdout:1245: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:26:17.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:17.399 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:17.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:17.435 INFO:tasks.rados.rados.0.smithi042.stdout:1249: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:17.476 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2868044 2024-09-20T19:26:17.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:17.514 INFO:tasks.rados.rados.0.smithi042.stdout:1250: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:26:17.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 398 (ObjNum 33 snap 0 seq_num 1390889016) dirty dne 2024-09-20T19:26:17.514 INFO:tasks.rados.rados.0.smithi042.stdout:1248: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:26:17.514 INFO:tasks.rados.rados.0.smithi042.stdout:1248: finishing copy_from to smithi04268048-13 2024-09-20T19:26:17.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 553 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1245: done (5 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1246: done (4 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1248: done (3 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1249: done (2 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1250: done (1 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1251: done (0 left) 2024-09-20T19:26:17.515 INFO:tasks.rados.rados.0.smithi042.stdout:1252: snap_remove snap 145 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:17 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.063303+0000 mgr.y (mgr.24491) 319 : cluster [DBG] pgmap v518: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 402 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: audit 2024-09-20T19:26:16.302688+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? v1:172.21.15.42:0/401989298' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [5, 4, 7, 1, 2, 0]}]': finished 2024-09-20T19:26:17.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.310913+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.399694+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:16.401260+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.196069+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:17 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.197007+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:17.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.701 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:470 7.e 7:7341b310:::smithi04268048-20:head [write 561367~458699,stat] snapc 14=[14,13,12,11,10,f,e,a] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:17.044529+0000", 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 240.65522848500001, 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013011863, 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 470 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.702 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.044529+0000", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.044529+0000", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.044534+0000", 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3129999999999998e-06 2024-09-20T19:26:17.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.045286+0000", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00075203900000000003 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.045338+0000", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2373000000000002e-05 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.045365+0000", 2024-09-20T19:26:17.704 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7016e-05 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.048884+0000", 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0035189510000000002 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.049104+0000", 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022003499999999999 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.705 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 0,5", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.049608+0000", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000504007 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.054271+0000", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0046627700000000001 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.056668+0000", 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0023966650000000001 2024-09-20T19:26:17.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.057316+0000", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00064814600000000005 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.057456+0000", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014053899999999999 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:17.057540+0000", 2024-09-20T19:26:17.707 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.4009000000000002e-05 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:775 7.8 7:1b5f1bb3:::smithi04268048-46:head [write 500875~467773,stat] snapc 1f=[1f,1e,1d,10] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:22:43.021459+0000", 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 214.67829782800001, 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014089615, 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.708 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 775 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.021459+0000", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.021459+0000", 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.021465+0000", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3630000000000002e-06 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.022239+0000", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00077471500000000004 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.022281+0000", 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1437000000000003e-05 2024-09-20T19:26:17.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.022307+0000", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6061999999999999e-05 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.022423+0000", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000116363 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.022433+0000", 2024-09-20T19:26:17.711 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8169999999999994e-06 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.025114+0000", 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002681162 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.025350+0000", 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023554199999999999 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.712 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.025873+0000", 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00052283899999999999 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.033292+0000", 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.007419223 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.033325+0000", 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3463000000000001e-05 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.713 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.035406+0000", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002080452 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.035476+0000", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0178999999999993e-05 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:22:43.035549+0000", 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.2997999999999995e-05 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.714 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:951 7.8 7:17ea136f:::smithi04268048-32:head [copy-from ver 73] snapc 2b=[2b,2a,29,26,24,22,20,10] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:03.072298+0000", 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 194.62745897299999, 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016025968000000002, 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.715 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 951 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072298+0000", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072298+0000", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072302+0000", 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7510000000000002e-06 2024-09-20T19:26:17.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072313+0000", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1245e-05 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072336+0000", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2693000000000001e-05 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072348+0000", 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2174000000000001e-05 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072430+0000", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1862999999999999e-05 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.072543+0000", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011345600000000001 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.077140+0000", 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045971720000000001 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.718 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.082826+0000", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0056854200000000001 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.087777+0000", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0049510029999999998 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.088230+0000", 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00045323 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.088273+0000", 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2740000000000001e-05 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:03.088324+0000", 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1221000000000003e-05 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.720 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1019 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 43] snapc 2f=[2f,2e,2d,2c,2a,26,24] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 185.51130217599999, 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032249699999999999, 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1019 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188459+0000", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8380000000000004e-06 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188472+0000", 2024-09-20T19:26:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3171e-05 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188489+0000", 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7516e-05 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188503+0000", 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3641e-05 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.723 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188623+0000", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000120104 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188745+0000", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000121839 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.201923+0000", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013178337 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.724 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.210593+0000", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0086699889999999995 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220372+0000", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.009778241 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220589+0000", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021783800000000001 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.725 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220647+0000", 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7151000000000002e-05 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220705+0000", 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8035000000000003e-05 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1020 7.8 7:1695abb3:::smithi04268048-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e203)", 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:12.188531+0000", 2024-09-20T19:26:17.726 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 185.511225634, 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020296245000000001, 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1020 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188531+0000", 2024-09-20T19:26:17.727 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188531+0000", 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188533+0000", 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.257e-06 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188538+0000", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.096e-06 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188560+0000", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2172999999999999e-05 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188568+0000", 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0330000000000008e-06 2024-09-20T19:26:17.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188933+0000", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00036515300000000001 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.189013+0000", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9524000000000006e-05 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.208635+0000", 2024-09-20T19:26:17.730 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019622212 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.208726+0000", 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0618000000000006e-05 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.208828+0000", 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000102179 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1222 7.8 7:17ea136f:::smithi04268048-32:head [copy-from ver 147] snapc 39=[39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 161.375355184, 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.039637587000000002, 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1222 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.732 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324404+0000", 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.73e-06 2024-09-20T19:26:17.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324413+0000", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7429999999999997e-06 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324432+0000", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8723e-05 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324454+0000", 2024-09-20T19:26:17.734 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1727000000000001e-05 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.326930+0000", 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0024762009999999999 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327029+0000", 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9409999999999999e-05 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.735 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.344328+0000", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017298289000000001 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363895+0000", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019566842000000001 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363943+0000", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8529999999999998e-05 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.736 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363959+0000", 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6246e-05 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363999+0000", 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9836000000000003e-05 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.364039+0000", 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0309999999999999e-05 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1223 7.8 7:17ea136f:::smithi04268048-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 161.37527974400001, 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.039355411999999999, 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1223 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324479+0000", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.643e-06 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324523+0000", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4363000000000002e-05 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324553+0000", 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9856999999999999e-05 2024-09-20T19:26:17.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324571+0000", 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8246000000000001e-05 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327203+0000", 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0026315039999999998 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:26:17.741 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327246+0000", 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3305e-05 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363693+0000", 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036446495000000002 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363768+0000", 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5630000000000006e-05 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363833+0000", 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4369000000000007e-05 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1231 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 27] snapc 39=[39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 161.37511992899999, 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.045744076000000002, 2024-09-20T19:26:17.743 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1231 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:26:17.744 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324640+0000", 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4569999999999999e-06 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.745 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324652+0000", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2007e-05 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324678+0000", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6080000000000001e-05 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324710+0000", 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2492e-05 2024-09-20T19:26:17.746 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327276+0000", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002565496 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327356+0000", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0316999999999997e-05 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.357039+0000", 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.029683095 2024-09-20T19:26:17.747 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.364449+0000", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0074094540000000002 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.367262+0000", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0028136179999999999 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370298+0000", 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0030354420000000002 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.748 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370351+0000", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3581000000000001e-05 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370381+0000", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0037000000000001e-05 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1232 7.8 7:1695abb3:::smithi04268048-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:26:17.749 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 161.37502219300001, 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.038895403000000002, 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1232 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:26:17.750 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324737+0000", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8870000000000001e-06 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324746+0000", 2024-09-20T19:26:17.751 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8899999999999996e-06 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324771+0000", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5136000000000001e-05 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324788+0000", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7410000000000001e-05 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327489+0000", 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0027011679999999999 2024-09-20T19:26:17.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327534+0000", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5117000000000002e-05 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363387+0000", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.035852123 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363496+0000", 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000109545 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363630+0000", 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000134127 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1263 7.e 7:7341b310:::smithi04268048-20:head [append 0~0] snapc 3a=[3a,39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:37.093915+0000", 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 160.605841806, 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015436689, 2024-09-20T19:26:17.754 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1263 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.093915+0000", 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.093915+0000", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.093920+0000", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9599999999999999e-06 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.093937+0000", 2024-09-20T19:26:17.756 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7e-05 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.093982+0000", 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4605999999999999e-05 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094010+0000", 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8503000000000001e-05 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.757 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094279+0000", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026827200000000001 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094521+0000", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000242575 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 0,5", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094988+0000", 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00046689000000000002 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.099865+0000", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0048768259999999999 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.106425+0000", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0065601849999999996 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109104+0000", 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0026791710000000002 2024-09-20T19:26:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109210+0000", 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000106028 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109352+0000", 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014167300000000001 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.760 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1264 7.e 7:7341b310:::smithi04268048-20:head [rollback 30] snapc 3a=[3a,39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:37.094128+0000", 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 160.60562918400001, 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015480345, 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1264 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.761 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094128+0000", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094128+0000", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094130+0000", 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3829999999999999e-06 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.762 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094148+0000", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7952999999999999e-05 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.094171+0000", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2900000000000001e-05 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096384+0000", 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0022131410000000001 2024-09-20T19:26:17.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096537+0000", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000152919 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096788+0000", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00025119600000000003 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 0,5", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.097371+0000", 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00058308199999999996 2024-09-20T19:26:17.764 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.103002+0000", 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.005630164 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.108837+0000", 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0058353570000000002 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109416+0000", 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00057895400000000001 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109493+0000", 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.7100000000000004e-05 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109608+0000", 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000115196 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1265 7.e 7:7341b310:::smithi04268048-20:head [append 0~0] snapc 3a=[3a,39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:37.096436+0000", 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 160.60332108700001, 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013419789, 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1265 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.767 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096436+0000", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096436+0000", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096439+0000", 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0900000000000001e-06 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096451+0000", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2383e-05 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096476+0000", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4536999999999999e-05 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.096496+0000", 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0380000000000001e-05 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.098543+0000", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002046782 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.098806+0000", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026254499999999999 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 0,5", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.099281+0000", 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00047562599999999999 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.103023+0000", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.003741341 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.108985+0000", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0059621910000000004 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109666+0000", 2024-09-20T19:26:17.771 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00068146400000000001 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109748+0000", 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1936999999999998e-05 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:37.109856+0000", 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107513 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:1580 7.a e246/112 7:5fdc189b:::smithi04268048-19:head v 246'127, mlcod=243'123)", 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:55.092840+0000", 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 142.60691668999999, 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013288186, 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.092840+0000", 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.773 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.092840+0000", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.092842+0000", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.274e-06 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.102418+0000", 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0095764779999999994 2024-09-20T19:26:17.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.102455+0000", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7057000000000001e-05 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.102475+0000", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9969000000000001e-05 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.102611+0000", 2024-09-20T19:26:17.775 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013604000000000001 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.102644+0000", 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2687000000000001e-05 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.106037+0000", 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0033927850000000002 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.776 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:55.106129+0000", 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1896000000000002e-05 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1842 7.8 7:1517470f:::smithi04268048-49:head [copy-from ver 67] snapc 4d=[4d,4c,4b,49,46,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:12.239342+0000", 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 125.460414777, 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023329918000000002, 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1842 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239342+0000", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239342+0000", 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239346+0000", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7639999999999999e-06 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239358+0000", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2275e-05 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239381+0000", 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2455000000000001e-05 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239390+0000", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5920000000000007e-06 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239482+0000", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1390000000000004e-05 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.239603+0000", 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000120957 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.780 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.247593+0000", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0079898460000000001 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.253628+0000", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060351959999999996 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.257891+0000", 2024-09-20T19:26:17.781 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.004262811 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.262547+0000", 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0046564019999999996 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.262617+0000", 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9758999999999994e-05 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.782 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:12.262672+0000", 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5470999999999998e-05 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2314 7.5 e313/112 7:adc64224:::smithi04268048-47:head v 313'439, mlcod=311'435)", 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:02.214127+0000", 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 75.485629669000005, 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016582630000000001, 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.783 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.214127+0000", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.214127+0000", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.784 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.214128+0000", 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2600000000000002e-07 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.222940+0000", 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0088116170000000008 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.222988+0000", 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8782999999999997e-05 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.785 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.223012+0000", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3612999999999999e-05 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.223199+0000", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000187374 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.223262+0000", 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.2403999999999994e-05 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.786 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.230622+0000", 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0073598709999999996 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:02.230710+0000", 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8442000000000005e-05 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2718 7.8 7:1517470f:::smithi04268048-49:head [copy-from ver 311] snapc 77=[77,76,73,72,70,6d,6c,69,67,5e,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-20T19:26:17.787 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 47.378804891000001, 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032791556999999999, 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2718 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:26:17.788 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320958+0000", 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8930000000000002e-06 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320980+0000", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2240000000000001e-05 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321019+0000", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.871e-05 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321042+0000", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.281e-05 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.790 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321179+0000", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000137317 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321322+0000", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014273899999999999 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.339340+0000", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.018018031 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.791 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.351735+0000", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012394794000000001 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.352003+0000", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000267951 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353665+0000", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016625170000000001 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.792 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353706+0000", 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0837000000000002e-05 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353744+0000", 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7718e-05 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2719 7.8 7:1517470f:::smithi04268048-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e341)", 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 47.378648941999998, 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.029709101000000002, 2024-09-20T19:26:17.793 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2719 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.794 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321110+0000", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1469999999999999e-06 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321120+0000", 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9129999999999998e-06 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.795 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321145+0000", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4833000000000001e-05 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321163+0000", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7773000000000001e-05 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321650+0000", 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048751600000000001 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.796 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321706+0000", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5578999999999998e-05 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350549+0000", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028842689000000001 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350676+0000", 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012729199999999999 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.797 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350817+0000", 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000141359 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2918 7.e 7:7341b310:::smithi04268048-20:head [write 440449~793052,stat] snapc 81=[81,7f,7e,7d,7c,79,77,76,72,6d,6c,69,67,5e,5a,41] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:47.159581+0000", 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 30.540175812000001, 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019665609000000001, 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.798 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2918 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.159581+0000", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.159581+0000", 2024-09-20T19:26:17.799 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.159588+0000", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.1300000000000003e-06 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.160657+0000", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010684920000000001 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.160696+0000", 2024-09-20T19:26:17.800 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9116000000000003e-05 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.160716+0000", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0446999999999999e-05 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.160922+0000", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00020575500000000001 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.161206+0000", 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028353800000000001 2024-09-20T19:26:17.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 0,5", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.161805+0000", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000599232 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.166888+0000", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0050833919999999999 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.172632+0000", 2024-09-20T19:26:17.802 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0057439420000000001 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.179122+0000", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0064899090000000003 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.179186+0000", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3952e-05 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:47.179247+0000", 2024-09-20T19:26:17.803 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0704e-05 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3207 7.8 7:1b5f1bb3:::smithi04268048-46:head [copy-from ver 541] snapc 92=[92,91,90,8f,8e,8d,8c,8b,8a,89,88,87,86,85,84,83,82,81,7f,7c,79,72,41] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.3409661480000001, 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028733953999999999, 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.804 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3207 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358797+0000", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7100000000000004e-06 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358816+0000", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9830000000000002e-05 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358878+0000", 2024-09-20T19:26:17.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1472999999999999e-05 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358903+0000", 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.51e-05 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359022+0000", 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000119246 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359152+0000", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000130075 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.372285+0000", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013132437 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383793+0000", 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011508107 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387396+0000", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0036035619999999998 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387434+0000", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7360000000000001e-05 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387484+0000", 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9815000000000003e-05 2024-09-20T19:26:17.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387525+0000", 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1239000000000003e-05 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3208 7.8 7:1b5f1bb3:::smithi04268048-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.3407997520000001, 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.024280572, 2024-09-20T19:26:17.810 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3208 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358960+0000", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.216e-06 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358968+0000", 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7539999999999995e-06 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358988+0000", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9530000000000001e-05 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359004+0000", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5943e-05 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359429+0000", 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00042558400000000001 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359586+0000", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000156489 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.382994+0000", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023408279000000001 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383137+0000", 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014323499999999999 2024-09-20T19:26:17.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383238+0000", 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100542 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:26:17.815 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:26:17.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:17.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:17.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:17.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:17.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:18.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:18.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:18.124 INFO:tasks.rados.rados.0.smithi042.stdout:1252: done (0 left) 2024-09-20T19:26:18.125 INFO:tasks.rados.rados.0.smithi042.stdout:1253: snap_create 2024-09-20T19:26:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:18.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:18.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:18.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:18 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:18.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:26:18.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.323984+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.882681+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:17.884614+0000 osd.4 (osd.4) 166 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:18 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.124137+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-20T19:26:18.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:18.854 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:18.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:18.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:26:18.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:19.012 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:19.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:19.131 INFO:tasks.rados.rados.0.smithi042.stdout:1253: done (0 left) 2024-09-20T19:26:19.131 INFO:tasks.rados.rados.0.smithi042.stdout:1254: snap_create 2024-09-20T19:26:19.140 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:19.244 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:19.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:19.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:19.396 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:19.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:19 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.064026+0000 mgr.y (mgr.24491) 320 : cluster [DBG] pgmap v521: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.260130+0000 osd.2 (osd.2) 145 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:26:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.291418+0000 osd.2 (osd.2) 146 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.648070+0000 osd.5 (osd.5) 143 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.656700+0000 osd.5 (osd.5) 144 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.900911+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:18.903263+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:19 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.128927+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-20T19:26:19.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:19.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:19.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:19.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:19.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:19.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:20.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:20.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:20.124 INFO:tasks.rados.rados.0.smithi042.stdout:1254: done (0 left) 2024-09-20T19:26:20.125 INFO:tasks.rados.rados.0.smithi042.stdout:1255: read oid 43 snap -1 2024-09-20T19:26:20.125 INFO:tasks.rados.rados.0.smithi042.stdout:1255: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:26:20.125 INFO:tasks.rados.rados.0.smithi042.stdout:1256: delete oid 50 current snap is 149 2024-09-20T19:26:20.132 INFO:tasks.rados.rados.0.smithi042.stdout:1255: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:26:20.139 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1204342 2024-09-20T19:26:20.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 706186 2024-09-20T19:26:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:20.182 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:20.213 INFO:tasks.rados.rados.0.smithi042.stdout:1255: done (1 left) 2024-09-20T19:26:20.213 INFO:tasks.rados.rados.0.smithi042.stdout:1256: done (0 left) 2024-09-20T19:26:20.213 INFO:tasks.rados.rados.0.smithi042.stdout:1257: delete oid 4 current snap is 149 2024-09-20T19:26:20.217 INFO:tasks.rados.rados.0.smithi042.stdout:1257: done (0 left) 2024-09-20T19:26:20.217 INFO:tasks.rados.rados.0.smithi042.stdout:1258: read oid 21 snap 134 2024-09-20T19:26:20.218 INFO:tasks.rados.rados.0.smithi042.stdout:1258: expect (ObjNum 404 snap 128 seq_num 404) 2024-09-20T19:26:20.218 INFO:tasks.rados.rados.0.smithi042.stdout:1259: read oid 2 snap -1 2024-09-20T19:26:20.218 INFO:tasks.rados.rados.0.smithi042.stdout:1259: expect (ObjNum 413 snap 129 seq_num 413) 2024-09-20T19:26:20.218 INFO:tasks.rados.rados.0.smithi042.stdout:1260: write oid 20 current snap is 149 2024-09-20T19:26:20.218 INFO:tasks.rados.rados.0.smithi042.stdout:1260: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-20T19:26:20.225 INFO:tasks.rados.rados.0.smithi042.stdout:1260: writing smithi04268048-20 from 712729 to 1497506 tid 1 2024-09-20T19:26:20.231 INFO:tasks.rados.rados.0.smithi042.stdout:1260: writing smithi04268048-20 from 2091098 to 2558589 tid 2 2024-09-20T19:26:20.237 INFO:tasks.rados.rados.0.smithi042.stdout:1260: writing smithi04268048-20 from 3342313 to 3512729 tid 3 2024-09-20T19:26:20.237 INFO:tasks.rados.rados.0.smithi042.stdout:1261: write oid 3 current snap is 149 2024-09-20T19:26:20.237 INFO:tasks.rados.rados.0.smithi042.stdout:1261: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-20T19:26:20.240 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:26:20.240 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:26:20.244 INFO:tasks.rados.rados.0.smithi042.stdout:1261: writing smithi04268048-3 from 729536 to 1189562 tid 1 2024-09-20T19:26:20.251 INFO:tasks.rados.rados.0.smithi042.stdout:1261: writing smithi04268048-3 from 1749580 to 2360729 tid 2 2024-09-20T19:26:20.257 INFO:tasks.rados.rados.0.smithi042.stdout:1261: writing smithi04268048-3 from 2853383 to 3305833 tid 3 2024-09-20T19:26:20.257 INFO:tasks.rados.rados.0.smithi042.stdout:1262: read oid 13 snap -1 2024-09-20T19:26:20.257 INFO:tasks.rados.rados.0.smithi042.stdout:1262: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:20.257 INFO:tasks.rados.rados.0.smithi042.stdout:1263: delete oid 17 current snap is 149 2024-09-20T19:26:20.258 INFO:tasks.rados.rados.0.smithi042.stdout:1258: expect (ObjNum 404 snap 128 seq_num 404) 2024-09-20T19:26:20.258 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:20.268 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:26:20.269 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:26:20.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:20.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:26:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:26:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:20.304 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:26:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:26:20.326 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:26:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:26:20.349 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:26:20.349 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:26:20.370 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:26:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:26:20.384 INFO:tasks.rados.rados.0.smithi042.stdout:1259: expect (ObjNum 413 snap 129 seq_num 413) 2024-09-20T19:26:20.391 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:26:20.391 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:26:20.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.402 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2210404 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[22706]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:20 smithi042 bash[32075]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.415 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:20.425 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1082778 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1260: finishing write tid 1 to smithi04268048-20 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1260: finishing write tid 2 to smithi04268048-20 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1258: done (5 left) 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1259: done (4 left) 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1263: done (3 left) 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1264: snap_create 2024-09-20T19:26:20.497 INFO:tasks.rados.rados.0.smithi042.stdout:1260: finishing write tid 3 to smithi04268048-20 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: finishing write tid 4 to smithi04268048-20 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: finishing write tid 5 to smithi04268048-20 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: oid 20 updating version 0 to 107 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: oid 20 version 107 is already newer than 106 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: oid 20 updating version 107 to 109 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: oid 20 version 109 is already newer than 108 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 109 (ObjNum 447 snap 149 seq_num 447) dirty exists 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1260: left oid 20 (ObjNum 447 snap 149 seq_num 447) 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1261: finishing write tid 1 to smithi04268048-3 2024-09-20T19:26:20.498 INFO:tasks.rados.rados.0.smithi042.stdout:1261: finishing write tid 2 to smithi04268048-3 2024-09-20T19:26:20.499 INFO:tasks.rados.rados.0.smithi042.stdout:1262: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:20.499 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:20.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:20.552 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:20.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:20.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.223130+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:26:20.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.224542+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: audit 2024-09-20T19:26:19.259454+0000 mgr.y (mgr.24491) 321 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.939430+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:19.940552+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.125148+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:20 smithi162 bash[24557]: audit 2024-09-20T19:26:20.220088+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.42:0/1292654272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: finishing write tid 3 to smithi04268048-3 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: finishing write tid 4 to smithi04268048-3 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: finishing write tid 5 to smithi04268048-3 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: oid 3 updating version 0 to 660 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: oid 3 version 660 is already newer than 659 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: oid 3 version 660 is already newer than 657 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: oid 3 version 660 is already newer than 658 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 660 (ObjNum 448 snap 149 seq_num 448) dirty exists 2024-09-20T19:26:20.663 INFO:tasks.rados.rados.0.smithi042.stdout:1261: left oid 3 (ObjNum 448 snap 149 seq_num 448) 2024-09-20T19:26:20.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:20.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:20.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:20.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:20.932 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:20.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:21.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:21.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:21.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:21.321 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1260: done (3 left) 2024-09-20T19:26:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1261: done (2 left) 2024-09-20T19:26:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1262: done (1 left) 2024-09-20T19:26:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1264: done (0 left) 2024-09-20T19:26:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1265: snap_remove snap 138 2024-09-20T19:26:21.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:21 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:21.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.567 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.567 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.064795+0000 mgr.y (mgr.24491) 322 : cluster [DBG] pgmap v524: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 436 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.3 MiB/s wr, 17 op/s 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.946522+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.e scrub starts 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:21 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.948181+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.e scrub ok 2024-09-20T19:26:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:21.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:21.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:21.817 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:21.949 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:21.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:22.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:22.212 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:22.342 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:22.362 INFO:tasks.rados.rados.0.smithi042.stdout:1265: done (0 left) 2024-09-20T19:26:22.363 INFO:tasks.rados.rados.0.smithi042.stdout:1266: snap_remove snap 122 2024-09-20T19:26:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:26:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:26:22.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:22.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:22 smithi162 bash[24557]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[22706]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.932757+0000 osd.0 (osd.0) 177 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:20.941449+0000 osd.0 (osd.0) 178 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[32075]: cluster 2024-09-20T19:26:21.348530+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-20T19:26:22.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:22.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:22.818 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:26:22.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:26:22.819 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:26:22.819 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T19:26:22.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0 2024-09-20T19:26:22.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:22.892 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:22.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:22.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:23.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:23.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.144 INFO:tasks.rados.rados.0.smithi042.stdout:1266: done (0 left) 2024-09-20T19:26:23.145 INFO:tasks.rados.rados.0.smithi042.stdout:1267: write oid 1 current snap is 150 2024-09-20T19:26:23.145 INFO:tasks.rados.rados.0.smithi042.stdout:1267: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-20T19:26:23.156 INFO:tasks.rados.rados.0.smithi042.stdout:1267: writing smithi04268048-1 from 763150 to 1490027 tid 1 2024-09-20T19:26:23.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:23.170 INFO:tasks.rados.rados.0.smithi042.stdout:1267: writing smithi04268048-1 from 1982897 to 2597715 tid 2 2024-09-20T19:26:23.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:22 smithi042 systemd[1]: Stopping Ceph osd.0 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:26:23.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[35151]: debug 2024-09-20T19:26:22.975+0000 7f917c645640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:26:23.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[35151]: debug 2024-09-20T19:26:22.975+0000 7f917c645640 -1 osd.0 393 *** Got signal Terminated *** 2024-09-20T19:26:23.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:22 smithi042 bash[35151]: debug 2024-09-20T19:26:22.975+0000 7f917c645640 -1 osd.0 393 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:26:23.180 INFO:tasks.rados.rados.0.smithi042.stdout:1267: writing smithi04268048-1 from 3024582 to 3499576 tid 3 2024-09-20T19:26:23.180 INFO:tasks.rados.rados.0.smithi042.stdout:1268: rollback oid 10 current snap is 150 2024-09-20T19:26:23.180 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 147 2024-09-20T19:26:23.180 INFO:tasks.rados.rados.0.smithi042.stdout:1267: finishing write tid 1 to smithi04268048-1 2024-09-20T19:26:23.180 INFO:tasks.rados.rados.0.smithi042.stdout:1267: finishing write tid 2 to smithi04268048-1 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1269: read oid 46 snap -1 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1269: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1270: read oid 6 snap 125 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1270: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1271: write (excl) oid 7 current snap is 150 2024-09-20T19:26:23.181 INFO:tasks.rados.rados.0.smithi042.stdout:1271: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-20T19:26:23.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:26:23.187 INFO:tasks.rados.rados.0.smithi042.stdout:1271: writing smithi04268048-7 from 779957 to 1182083 tid 1 2024-09-20T19:26:23.196 INFO:tasks.rados.rados.0.smithi042.stdout:1271: writing smithi04268048-7 from 1925026 to 2683502 tid 2 2024-09-20T19:26:23.202 INFO:tasks.rados.rados.0.smithi042.stdout:1271: writing smithi04268048-7 from 3335652 to 3579957 tid 3 2024-09-20T19:26:23.202 INFO:tasks.rados.rados.0.smithi042.stdout:1272: write (excl) oid 49 current snap is 150 2024-09-20T19:26:23.202 INFO:tasks.rados.rados.0.smithi042.stdout:1272: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-20T19:26:23.208 INFO:tasks.rados.rados.0.smithi042.stdout:1272: writing smithi04268048-49 from 796764 to 1274139 tid 1 2024-09-20T19:26:23.216 INFO:tasks.rados.rados.0.smithi042.stdout:1272: writing smithi04268048-49 from 1983508 to 2601995 tid 2 2024-09-20T19:26:23.222 INFO:tasks.rados.rados.0.smithi042.stdout:1272: writing smithi04268048-49 from 3363075 to 3596764 tid 3 2024-09-20T19:26:23.222 INFO:tasks.rados.rados.0.smithi042.stdout:1273: snap_remove snap 144 2024-09-20T19:26:23.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:23.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:23.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.543 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.543 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.543 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.543 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.543 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:23 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:23.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:23.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.066619+0000 mgr.y (mgr.24491) 323 : cluster [DBG] pgmap v527: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.167989+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.169574+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.361890+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:22.981606+0000 mon.a (mon.0) 1442 : cluster [INF] osd.0 marked itself down and dead 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.118402+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:23 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.129418+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e394: 8 total, 7 up, 8 in 2024-09-20T19:26:23.686 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:23.692 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:23.692 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:23.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:23.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:23.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:23.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:24.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:24.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:24.135 INFO:tasks.rados.rados.0.smithi042.stdout:1273: done (6 left) 2024-09-20T19:26:24.135 INFO:tasks.rados.rados.0.smithi042.stdout:1274: rollback oid 44 current snap is 150 2024-09-20T19:26:24.135 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 135 2024-09-20T19:26:24.135 INFO:tasks.rados.rados.0.smithi042.stdout:1275: write (excl) oid 3 current snap is 150 2024-09-20T19:26:24.135 INFO:tasks.rados.rados.0.smithi042.stdout:1275: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-20T19:26:24.141 INFO:tasks.rados.rados.0.smithi042.stdout:1275: writing smithi04268048-3 from 413571 to 966195 tid 1 2024-09-20T19:26:24.150 INFO:tasks.rados.rados.0.smithi042.stdout:1275: writing smithi04268048-3 from 1641990 to 2404135 tid 2 2024-09-20T19:26:24.157 INFO:tasks.rados.rados.0.smithi042.stdout:1275: writing smithi04268048-3 from 2990498 to 3415661 tid 3 2024-09-20T19:26:24.158 INFO:tasks.rados.rados.0.smithi042.stdout:1268: finishing rollback tid 0 to smithi04268048-10 2024-09-20T19:26:24.158 INFO:tasks.rados.rados.0.smithi042.stdout:1269: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:24.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:24.221 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2240042 2024-09-20T19:26:24.237 INFO:tasks.rados.rados.0.smithi042.stdout:1268: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:26:24.237 INFO:tasks.rados.rados.0.smithi042.stdout:1268: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:26:24.237 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 560 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:26:24.237 INFO:tasks.rados.rados.0.smithi042.stdout:1267: finishing write tid 3 to smithi04268048-1 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: finishing write tid 4 to smithi04268048-1 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1268: done (7 left) 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1269: done (6 left) 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1276: delete oid 41 current snap is 150 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: finishing write tid 5 to smithi04268048-1 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: oid 1 updating version 0 to 500 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: oid 1 version 500 is already newer than 498 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: oid 1 version 500 is already newer than 497 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: oid 1 version 500 is already newer than 499 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 500 (ObjNum 450 snap 150 seq_num 450) dirty exists 2024-09-20T19:26:24.238 INFO:tasks.rados.rados.0.smithi042.stdout:1267: left oid 1 (ObjNum 450 snap 150 seq_num 450) 2024-09-20T19:26:24.239 INFO:tasks.rados.rados.0.smithi042.stdout:1270: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:26:24.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:24.309 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:24.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: finishing write tid 1 to smithi04268048-49 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: finishing write tid 2 to smithi04268048-49 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: finishing write tid 3 to smithi04268048-49 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: finishing write tid 4 to smithi04268048-49 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1267: done (6 left) 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1270: done (5 left) 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1276: done (4 left) 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1277: copy_from oid 31 from oid 32 current snap is 150 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: finishing write tid 5 to smithi04268048-49 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: oid 49 updating version 0 to 657 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: oid 49 updating version 657 to 659 2024-09-20T19:26:24.353 INFO:tasks.rados.rados.0.smithi042.stdout:1272: oid 49 version 659 is already newer than 658 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1272: oid 49 version 659 is already newer than 656 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 659 (ObjNum 452 snap 150 seq_num 452) dirty exists 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1272: left oid 49 (ObjNum 452 snap 150 seq_num 452) 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: finishing write tid 1 to smithi04268048-3 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: finishing write tid 2 to smithi04268048-3 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: finishing write tid 3 to smithi04268048-3 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: finishing write tid 4 to smithi04268048-3 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: finishing write tid 5 to smithi04268048-3 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: oid 3 updating version 0 to 663 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: oid 3 updating version 663 to 664 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: oid 3 updating version 664 to 665 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: oid 3 version 665 is already newer than 662 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 665 (ObjNum 453 snap 150 seq_num 453) dirty exists 2024-09-20T19:26:24.354 INFO:tasks.rados.rados.0.smithi042.stdout:1275: left oid 3 (ObjNum 453 snap 150 seq_num 453) 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1274: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1274: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1274: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 667 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: finishing write tid 1 to smithi04268048-7 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1272: done (4 left) 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1274: done (3 left) 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1275: done (2 left) 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1278: delete oid 47 current snap is 150 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: finishing write tid 2 to smithi04268048-7 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: finishing write tid 3 to smithi04268048-7 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: finishing write tid 4 to smithi04268048-7 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: finishing write tid 5 to smithi04268048-7 2024-09-20T19:26:24.355 INFO:tasks.rados.rados.0.smithi042.stdout:1271: oid 7 updating version 0 to 601 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1271: oid 7 updating version 601 to 602 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1271: oid 7 version 602 is already newer than 599 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1271: oid 7 version 602 is already newer than 600 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 602 (ObjNum 451 snap 150 seq_num 451) dirty exists 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1271: left oid 7 (ObjNum 451 snap 150 seq_num 451) 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1271: done (2 left) 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1278: done (1 left) 2024-09-20T19:26:24.356 INFO:tasks.rados.rados.0.smithi042.stdout:1279: delete oid 30 current snap is 150 2024-09-20T19:26:24.359 INFO:tasks.rados.rados.0.smithi042.stdout:1279: done (1 left) 2024-09-20T19:26:24.359 INFO:tasks.rados.rados.0.smithi042.stdout:1280: read oid 4 snap -1 2024-09-20T19:26:24.359 INFO:tasks.rados.rados.0.smithi042.stdout:1280: expect deleted 2024-09-20T19:26:24.359 INFO:tasks.rados.rados.0.smithi042.stdout:1281: snap_create 2024-09-20T19:26:24.361 INFO:tasks.rados.rados.0.smithi042.stdout:1277: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:26:24.366 INFO:tasks.rados.rados.0.smithi042.stdout:1277: finishing copy_from to smithi04268048-31 2024-09-20T19:26:24.366 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 483 (ObjNum 360 snap 116 seq_num 360) dirty exists 2024-09-20T19:26:24.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:24.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:24.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:24.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:24.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:24.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:24.842 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:24.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:24.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:25.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:25.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:25.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:25.234 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:25.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:25.349 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[89863]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:26:25.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[32075]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[22706]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1277: done (2 left) 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1280: done (1 left) 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1281: done (0 left) 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1282: read oid 19 snap 112 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1282: expect (ObjNum 327 snap 107 seq_num 327) 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1283: write (excl) oid 29 current snap is 151 2024-09-20T19:26:25.358 INFO:tasks.rados.rados.0.smithi042.stdout:1283: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-20T19:26:25.364 INFO:tasks.rados.rados.0.smithi042.stdout:1283: writing smithi04268048-29 from 480799 to 1050772 tid 1 2024-09-20T19:26:25.366 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:1283: writing smithi04268048-29 from 1475918 to 2245401 tid 2 2024-09-20T19:26:25.378 INFO:tasks.rados.rados.0.smithi042.stdout:1283: writing smithi04268048-29 from 2700190 to 3286794 tid 3 2024-09-20T19:26:25.379 INFO:tasks.rados.rados.0.smithi042.stdout:1284: write (excl) oid 9 current snap is 151 2024-09-20T19:26:25.379 INFO:tasks.rados.rados.0.smithi042.stdout:1284: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-20T19:26:25.385 INFO:tasks.rados.rados.0.smithi042.stdout:1284: writing smithi04268048-9 from 497606 to 1142828 tid 1 2024-09-20T19:26:25.393 INFO:tasks.rados.rados.0.smithi042.stdout:1284: writing smithi04268048-9 from 1934400 to 2563894 tid 2 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:1284: writing smithi04268048-9 from 3243966 to 3697606 tid 3 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:1285: read oid 39 snap -1 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:1285: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:1286: rollback oid 5 current snap is 151 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 147 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:1282: expect (ObjNum 327 snap 107 seq_num 327) 2024-09-20T19:26:25.401 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:25.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:25.493 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:25.506 INFO:tasks.rados.rados.0.smithi042.stdout:1283: finishing write tid 1 to smithi04268048-29 2024-09-20T19:26:25.506 INFO:tasks.rados.rados.0.smithi042.stdout:1283: finishing write tid 2 to smithi04268048-29 2024-09-20T19:26:25.506 INFO:tasks.rados.rados.0.smithi042.stdout:1283: finishing write tid 3 to smithi04268048-29 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: finishing write tid 4 to smithi04268048-29 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: finishing write tid 5 to smithi04268048-29 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: oid 29 updating version 0 to 561 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: oid 29 updating version 561 to 564 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: oid 29 version 564 is already newer than 563 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: oid 29 version 564 is already newer than 562 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 564 (ObjNum 457 snap 151 seq_num 457) dirty exists 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: left oid 29 (ObjNum 457 snap 151 seq_num 457) 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1284: finishing write tid 1 to smithi04268048-9 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1282: done (4 left) 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1283: done (3 left) 2024-09-20T19:26:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:1287: snap_remove snap 148 2024-09-20T19:26:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.784093+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:23.786940+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:24.067356+0000 mgr.y (mgr.24491) 324 : cluster [DBG] pgmap v530: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:25 smithi162 bash[24557]: cluster 2024-09-20T19:26:24.131676+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e395: 8 total, 7 up, 8 in 2024-09-20T19:26:25.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:25.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:25.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:25.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:25.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:25.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:25.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:25.964 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:26:25 smithi042 bash[90019]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:26:25.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:26.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:26.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:26.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:26.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.257 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:26.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:26.372 INFO:tasks.rados.rados.0.smithi042.stdout:1287: done (3 left) 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1288: read oid 16 snap -1 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1288: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1289: write oid 50 current snap is 151 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: finishing write tid 2 to smithi04268048-9 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1286: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 398 (ObjNum 1809 snap 0 seq_num 3047887248) dirty dne 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: finishing write tid 3 to smithi04268048-9 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: finishing write tid 4 to smithi04268048-9 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: finishing write tid 5 to smithi04268048-9 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: oid 9 updating version 0 to 654 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: oid 9 updating version 654 to 656 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: oid 9 version 656 is already newer than 655 2024-09-20T19:26:26.373 INFO:tasks.rados.rados.0.smithi042.stdout:1284: oid 9 version 656 is already newer than 653 2024-09-20T19:26:26.374 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 656 (ObjNum 458 snap 151 seq_num 458) dirty exists 2024-09-20T19:26:26.374 INFO:tasks.rados.rados.0.smithi042.stdout:1284: left oid 9 (ObjNum 458 snap 151 seq_num 458) 2024-09-20T19:26:26.374 INFO:tasks.rados.rados.0.smithi042.stdout:1285: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:26.374 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 260503 2024-09-20T19:26:26.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:26.477 INFO:tasks.rados.rados.0.smithi042.stdout:1288: expect (ObjNum 380 snap 123 seq_num 380) 2024-09-20T19:26:26.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:26 smithi162 bash[24557]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:26.546 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1953901 2024-09-20T19:26:26.591 INFO:tasks.rados.rados.0.smithi042.stdout:1289: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-20T19:26:26.611 INFO:tasks.rados.rados.0.smithi042.stdout:1289: writing smithi04268048-50 from 514413 to 1234884 tid 1 2024-09-20T19:26:26.634 INFO:tasks.rados.rados.0.smithi042.stdout:1289: writing smithi04268048-50 from 1876529 to 2649681 tid 2 2024-09-20T19:26:26.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[22706]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.357048+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e396: 8 total, 7 up, 8 in 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.555244+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:26 smithi042 bash[32075]: cluster 2024-09-20T19:26:25.557569+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1289: writing smithi04268048-50 from 3438683 to 3714413 tid 3 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1284: done (4 left) 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1285: done (3 left) 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1286: done (2 left) 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1288: done (1 left) 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1290: copy_from oid 49 from oid 41 current snap is 151 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1289: finishing write tid 1 to smithi04268048-50 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1289: finishing write tid 2 to smithi04268048-50 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1291: write (excl) oid 37 current snap is 151 2024-09-20T19:26:26.651 INFO:tasks.rados.rados.0.smithi042.stdout:1291: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-20T19:26:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:26.659 INFO:tasks.rados.rados.0.smithi042.stdout:1291: writing smithi04268048-37 from 531220 to 1326940 tid 1 2024-09-20T19:26:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:26.665 INFO:tasks.rados.rados.0.smithi042.stdout:1291: writing smithi04268048-37 from 1935011 to 2451821 tid 2 2024-09-20T19:26:26.672 INFO:tasks.rados.rados.0.smithi042.stdout:1291: writing smithi04268048-37 from 3066106 to 3486526 tid 3 2024-09-20T19:26:26.672 INFO:tasks.rados.rados.0.smithi042.stdout:1292: write (excl) oid 3 current snap is 151 2024-09-20T19:26:26.672 INFO:tasks.rados.rados.0.smithi042.stdout:1290: finishing copy_from to smithi04268048-49 2024-09-20T19:26:26.672 INFO:tasks.rados.rados.0.smithi042.stdout:1290: got expected ENOENT (src dne) 2024-09-20T19:26:26.672 INFO:tasks.rados.rados.0.smithi042.stdout:1292: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-20T19:26:26.677 INFO:tasks.rados.rados.0.smithi042.stdout:1292: writing smithi04268048-3 from 548027 to 1018996 tid 1 2024-09-20T19:26:26.685 INFO:tasks.rados.rados.0.smithi042.stdout:1292: writing smithi04268048-3 from 1593493 to 2370314 tid 2 2024-09-20T19:26:26.693 INFO:tasks.rados.rados.0.smithi042.stdout:1292: writing smithi04268048-3 from 3093529 to 3725221 tid 3 2024-09-20T19:26:26.693 INFO:tasks.rados.rados.0.smithi042.stdout:1293: write (excl) oid 7 current snap is 151 2024-09-20T19:26:26.693 INFO:tasks.rados.rados.0.smithi042.stdout:1290: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:26:26.693 INFO:tasks.rados.rados.0.smithi042.stdout:1289: finishing write tid 3 to smithi04268048-50 2024-09-20T19:26:26.693 INFO:tasks.rados.rados.0.smithi042.stdout:1293: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-20T19:26:26.699 INFO:tasks.rados.rados.0.smithi042.stdout:1293: writing smithi04268048-7 from 564834 to 1111052 tid 1 2024-09-20T19:26:26.705 INFO:tasks.rados.rados.0.smithi042.stdout:1293: writing smithi04268048-7 from 1651975 to 2172454 tid 2 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1293: writing smithi04268048-7 from 2720952 to 3280269 tid 3 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1290: done (4 left) 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1294: write oid 28 current snap is 151 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1289: finishing write tid 4 to smithi04268048-50 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1289: finishing write tid 5 to smithi04268048-50 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1289: oid 50 updating version 0 to 657 2024-09-20T19:26:26.712 INFO:tasks.rados.rados.0.smithi042.stdout:1289: oid 50 updating version 657 to 658 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:1289: oid 50 updating version 658 to 659 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:1289: oid 50 updating version 659 to 660 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 660 (ObjNum 459 snap 151 seq_num 459) dirty exists 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:1289: left oid 50 (ObjNum 459 snap 151 seq_num 459) 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:1291: finishing write tid 1 to smithi04268048-37 2024-09-20T19:26:26.713 INFO:tasks.rados.rados.0.smithi042.stdout:1294: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-20T19:26:26.718 INFO:tasks.rados.rados.0.smithi042.stdout:1294: writing smithi04268048-28 from 581641 to 1203108 tid 1 2024-09-20T19:26:26.727 INFO:tasks.rados.rados.0.smithi042.stdout:1294: writing smithi04268048-28 from 1994104 to 2774594 tid 2 2024-09-20T19:26:26.733 INFO:tasks.rados.rados.0.smithi042.stdout:1294: writing smithi04268048-28 from 3432022 to 3781641 tid 3 2024-09-20T19:26:26.733 INFO:tasks.rados.rados.0.smithi042.stdout:1289: done (4 left) 2024-09-20T19:26:26.733 INFO:tasks.rados.rados.0.smithi042.stdout:1295: snap_remove snap 147 2024-09-20T19:26:26.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:26.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:26.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:26.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:27.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:27.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:27.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:27.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:27.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:27.379 INFO:tasks.rados.rados.0.smithi042.stdout:1295: done (4 left) 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1296: read oid 41 snap -1 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1296: expect deleted 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1297: rollback oid 18 current snap is 151 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 139 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: finishing write tid 2 to smithi04268048-37 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: finishing write tid 3 to smithi04268048-37 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: finishing write tid 4 to smithi04268048-37 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: finishing write tid 5 to smithi04268048-37 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: oid 37 updating version 0 to 402 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: oid 37 version 402 is already newer than 400 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: oid 37 updating version 402 to 403 2024-09-20T19:26:27.380 INFO:tasks.rados.rados.0.smithi042.stdout:1291: oid 37 version 403 is already newer than 401 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 403 (ObjNum 460 snap 151 seq_num 460) dirty exists 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1291: left oid 37 (ObjNum 460 snap 151 seq_num 460) 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: finishing write tid 1 to smithi04268048-3 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: finishing write tid 2 to smithi04268048-3 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1291: done (5 left) 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1298: write oid 6 current snap is 151 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: finishing write tid 3 to smithi04268048-3 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: finishing write tid 4 to smithi04268048-3 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: finishing write tid 5 to smithi04268048-3 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: oid 3 updating version 0 to 668 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: oid 3 updating version 668 to 669 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: oid 3 version 669 is already newer than 666 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: oid 3 version 669 is already newer than 667 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 669 (ObjNum 461 snap 151 seq_num 461) dirty exists 2024-09-20T19:26:27.381 INFO:tasks.rados.rados.0.smithi042.stdout:1292: left oid 3 (ObjNum 461 snap 151 seq_num 461) 2024-09-20T19:26:27.382 INFO:tasks.rados.rados.0.smithi042.stdout:1293: finishing write tid 1 to smithi04268048-7 2024-09-20T19:26:27.382 INFO:tasks.rados.rados.0.smithi042.stdout:1293: finishing write tid 2 to smithi04268048-7 2024-09-20T19:26:27.382 INFO:tasks.rados.rados.0.smithi042.stdout:1293: finishing write tid 3 to smithi04268048-7 2024-09-20T19:26:27.382 INFO:tasks.rados.rados.0.smithi042.stdout:1298: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-20T19:26:27.387 INFO:tasks.rados.rados.0.smithi042.stdout:1298: writing smithi04268048-6 from 598448 to 1011517 tid 1 2024-09-20T19:26:27.398 INFO:tasks.rados.rados.0.smithi042.stdout:1298: writing smithi04268048-6 from 1768939 to 2293087 tid 2 2024-09-20T19:26:27.406 INFO:tasks.rados.rados.0.smithi042.stdout:1298: writing smithi04268048-6 from 2775798 to 3357659 tid 3 2024-09-20T19:26:27.410 INFO:tasks.rados.rados.0.smithi042.stdout:1298: writing smithi04268048-6 from 3781488 to 3798448 tid 4 2024-09-20T19:26:27.410 INFO:tasks.rados.rados.0.smithi042.stdout:1292: done (5 left) 2024-09-20T19:26:27.410 INFO:tasks.rados.rados.0.smithi042.stdout:1299: read oid 31 snap -1 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1299: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1300: snap_create 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: finishing write tid 4 to smithi04268048-7 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: finishing write tid 5 to smithi04268048-7 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: oid 7 updating version 0 to 604 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: oid 7 updating version 604 to 605 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: oid 7 version 605 is already newer than 603 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: oid 7 updating version 605 to 606 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 606 (ObjNum 462 snap 151 seq_num 462) dirty exists 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1293: left oid 7 (ObjNum 462 snap 151 seq_num 462) 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1294: finishing write tid 1 to smithi04268048-28 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1294: finishing write tid 2 to smithi04268048-28 2024-09-20T19:26:27.411 INFO:tasks.rados.rados.0.smithi042.stdout:1294: finishing write tid 3 to smithi04268048-28 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: finishing write tid 4 to smithi04268048-28 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: finishing write tid 5 to smithi04268048-28 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: oid 28 updating version 0 to 396 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: oid 28 version 396 is already newer than 395 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: oid 28 version 396 is already newer than 393 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: oid 28 version 396 is already newer than 394 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 396 (ObjNum 463 snap 151 seq_num 463) dirty exists 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1294: left oid 28 (ObjNum 463 snap 151 seq_num 463) 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1297: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1297: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1297: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 699 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 1 to smithi04268048-6 2024-09-20T19:26:27.412 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 2 to smithi04268048-6 2024-09-20T19:26:27.413 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 3 to smithi04268048-6 2024-09-20T19:26:27.413 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 4 to smithi04268048-6 2024-09-20T19:26:27.413 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 5 to smithi04268048-6 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: finishing write tid 6 to smithi04268048-6 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: oid 6 updating version 0 to 501 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: oid 6 updating version 501 to 503 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: oid 6 updating version 503 to 504 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: oid 6 updating version 504 to 505 2024-09-20T19:26:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:1298: oid 6 version 505 is already newer than 502 2024-09-20T19:26:27.415 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 505 (ObjNum 464 snap 151 seq_num 464) dirty exists 2024-09-20T19:26:27.415 INFO:tasks.rados.rados.0.smithi042.stdout:1298: left oid 6 (ObjNum 464 snap 151 seq_num 464) 2024-09-20T19:26:27.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.420 INFO:tasks.rados.rados.0.smithi042.stdout:1299: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:26:27.435 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1652549 2024-09-20T19:26:27.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:27.507 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1747978 2024-09-20T19:26:27.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:27 smithi162 bash[24557]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:27.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[22706]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.068119+0000 mgr.y (mgr.24491) 325 : cluster [DBG] pgmap v533: 158 pgs: 19 stale+active+clean, 1 active+clean+snaptrim, 138 active+clean; 437 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.370920+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e397: 8 total, 7 up, 8 in 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.585262+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:26.586974+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.098110+0000 osd.2 (osd.2) 151 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.104865+0000 osd.2 (osd.2) 152 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: audit 2024-09-20T19:26:27.242482+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:27 smithi042 bash[32075]: audit 2024-09-20T19:26:27.244745+0000 mon.a (mon.0) 1449 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:27.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:27.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:27.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:28.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:28.080 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:28.127 INFO:tasks.rados.rados.0.smithi042.stdout:1293: done (6 left) 2024-09-20T19:26:28.127 INFO:tasks.rados.rados.0.smithi042.stdout:1294: done (5 left) 2024-09-20T19:26:28.127 INFO:tasks.rados.rados.0.smithi042.stdout:1296: done (4 left) 2024-09-20T19:26:28.127 INFO:tasks.rados.rados.0.smithi042.stdout:1297: done (3 left) 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1298: done (2 left) 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1299: done (1 left) 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1300: done (0 left) 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1301: read oid 23 snap 130 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1301: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:26:28.128 INFO:tasks.rados.rados.0.smithi042.stdout:1302: snap_remove snap 135 2024-09-20T19:26:28.143 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:26:28.143 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:26:28.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:28.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:28.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:28.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:28.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:28.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:28 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:28.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:28.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.379055+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e398: 8 total, 7 up, 8 in 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.542345+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:26:28.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:27.544362+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:26:28.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:28 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.126097+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e399: 8 total, 7 up, 8 in 2024-09-20T19:26:28.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:28.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:28.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:28.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:28.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:29.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:29.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:26:29.132 INFO:tasks.rados.rados.0.smithi042.stdout:1302: done (1 left) 2024-09-20T19:26:29.132 INFO:tasks.rados.rados.0.smithi042.stdout:1303: copy_from oid 20 from oid 17 current snap is 152 2024-09-20T19:26:29.133 INFO:tasks.rados.rados.0.smithi042.stdout:1301: expect (ObjNum 156 snap 45 seq_num 156) 2024-09-20T19:26:29.133 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:29.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:29.221 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 276798 2024-09-20T19:26:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1303: finishing copy_from to smithi04268048-20 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1303: got expected ENOENT (src dne) 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1303: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1301: done (1 left) 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1303: done (0 left) 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1304: read oid 40 snap -1 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1304: expect deleted 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1305: copy_from oid 34 from oid 17 current snap is 152 2024-09-20T19:26:29.262 INFO:tasks.rados.rados.0.smithi042.stdout:1306: read oid 4 snap -1 2024-09-20T19:26:29.263 INFO:tasks.rados.rados.0.smithi042.stdout:1306: expect deleted 2024-09-20T19:26:29.263 INFO:tasks.rados.rados.0.smithi042.stdout:1307: snap_remove snap 143 2024-09-20T19:26:29.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:29.365 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.407 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:29 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.410 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:26:29.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:29.519 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:29.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.068728+0000 mgr.y (mgr.24491) 326 : cluster [DBG] pgmap v536: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:28.372424+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check failed: Degraded data redundancy: 173/1230 objects degraded (14.065%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: audit 2024-09-20T19:26:28.449545+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.122703+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: audit 2024-09-20T19:26:29.124843+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.42:0/2125765921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:29 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.132316+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e400: 8 total, 7 up, 8 in 2024-09-20T19:26:29.622 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:29.643 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:29.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:29.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:29.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:29.907 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:29.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:30.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:30.140 INFO:tasks.rados.rados.0.smithi042.stdout:1307: done (3 left) 2024-09-20T19:26:30.140 INFO:tasks.rados.rados.0.smithi042.stdout:1308: write oid 50 current snap is 152 2024-09-20T19:26:30.140 INFO:tasks.rados.rados.0.smithi042.stdout:1308: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-20T19:26:30.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:30.155 INFO:tasks.rados.rados.0.smithi042.stdout:1308: writing smithi04268048-50 from 615255 to 1103573 tid 1 2024-09-20T19:26:30.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:30.177 INFO:tasks.rados.rados.0.smithi042.stdout:1308: writing smithi04268048-50 from 1827421 to 2611580 tid 2 2024-09-20T19:26:30.193 INFO:tasks.rados.rados.0.smithi042.stdout:1308: writing smithi04268048-50 from 3203221 to 3815255 tid 3 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1309: rollback oid 20 current snap is 152 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 20 to 127 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1305: finishing copy_from to smithi04268048-34 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1305: got expected ENOENT (src dne) 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1305: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1308: finishing write tid 1 to smithi04268048-50 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1308: finishing write tid 2 to smithi04268048-50 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1304: done (4 left) 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1305: done (3 left) 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1306: done (2 left) 2024-09-20T19:26:30.194 INFO:tasks.rados.rados.0.smithi042.stdout:1310: snap_remove snap 131 2024-09-20T19:26:30.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:30.297 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:30.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:30.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:30.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:30 smithi162 bash[24557]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:30.662 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[22706]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: audit 2024-09-20T19:26:29.261148+0000 mgr.y (mgr.24491) 327 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.611841+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.612729+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.869117+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:29.869537+0000 osd.6 (osd.6) 120 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:30 smithi042 bash[32075]: cluster 2024-09-20T19:26:30.135014+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e401: 8 total, 7 up, 8 in 2024-09-20T19:26:30.691 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:30.758 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T19:26:30.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:30.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:26:30.820 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T19:26:30.820 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-20T19:26:30.820 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T19:26:30.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-20T19:26:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:30.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:30.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:31.013 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:26:31.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:31.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:31.141 INFO:tasks.rados.rados.0.smithi042.stdout:1310: done (2 left) 2024-09-20T19:26:31.142 INFO:tasks.rados.rados.0.smithi042.stdout:1311: write (excl) oid 16 current snap is 152 2024-09-20T19:26:31.142 INFO:tasks.rados.rados.0.smithi042.stdout:1309: finishing rollback tid 0 to smithi04268048-20 2024-09-20T19:26:31.142 INFO:tasks.rados.rados.0.smithi042.stdout:1311: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-20T19:26:31.161 INFO:tasks.rados.rados.0.smithi042.stdout:1311: writing smithi04268048-16 from 632062 to 1195629 tid 1 2024-09-20T19:26:31.180 INFO:tasks.rados.rados.0.smithi042.stdout:1311: writing smithi04268048-16 from 1885903 to 2413720 tid 2 2024-09-20T19:26:31.194 INFO:tasks.rados.rados.0.smithi042.stdout:1311: writing smithi04268048-16 from 2830644 to 3551402 tid 3 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1312: delete oid 6 current snap is 152 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1309: finishing rollback tid 1 to smithi04268048-20 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: finishing write tid 3 to smithi04268048-50 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1309: finishing rollback tid 2 to smithi04268048-20 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 115 (ObjNum 375 snap 118 seq_num 375) dirty exists 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: finishing write tid 4 to smithi04268048-50 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: finishing write tid 5 to smithi04268048-50 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: oid 50 updating version 0 to 663 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: oid 50 version 663 is already newer than 662 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: oid 50 updating version 663 to 664 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: oid 50 updating version 664 to 665 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 665 (ObjNum 465 snap 152 seq_num 465) dirty exists 2024-09-20T19:26:31.195 INFO:tasks.rados.rados.0.smithi042.stdout:1308: left oid 50 (ObjNum 465 snap 152 seq_num 465) 2024-09-20T19:26:31.196 INFO:tasks.rados.rados.0.smithi042.stdout:1311: finishing write tid 1 to smithi04268048-16 2024-09-20T19:26:31.196 INFO:tasks.rados.rados.0.smithi042.stdout:1311: finishing write tid 2 to smithi04268048-16 2024-09-20T19:26:31.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:31.200 INFO:tasks.rados.rados.0.smithi042.stdout:1308: done (3 left) 2024-09-20T19:26:31.200 INFO:tasks.rados.rados.0.smithi042.stdout:1309: done (2 left) 2024-09-20T19:26:31.200 INFO:tasks.rados.rados.0.smithi042.stdout:1312: done (1 left) 2024-09-20T19:26:31.201 INFO:tasks.rados.rados.0.smithi042.stdout:1313: rollback oid 3 current snap is 152 2024-09-20T19:26:31.201 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 119 2024-09-20T19:26:31.201 INFO:tasks.rados.rados.0.smithi042.stdout:1314: write oid 18 current snap is 152 2024-09-20T19:26:31.201 INFO:tasks.rados.rados.0.smithi042.stdout:1314: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-20T19:26:31.219 INFO:tasks.rados.rados.0.smithi042.stdout:1314: writing smithi04268048-18 from 665676 to 1379741 tid 1 2024-09-20T19:26:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:31.227 INFO:tasks.rados.rados.0.smithi042.stdout:1314: writing smithi04268048-18 from 1886514 to 2418000 tid 2 2024-09-20T19:26:31.239 INFO:tasks.rados.rados.0.smithi042.stdout:1314: writing smithi04268048-18 from 3169137 to 3865676 tid 3 2024-09-20T19:26:31.239 INFO:tasks.rados.rados.0.smithi042.stdout:1315: rollback oid 2 current snap is 152 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 150 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: finishing write tid 3 to smithi04268048-16 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: finishing write tid 4 to smithi04268048-16 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: finishing write tid 5 to smithi04268048-16 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: oid 16 updating version 0 to 532 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: oid 16 version 532 is already newer than 530 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: oid 16 updating version 532 to 533 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: oid 16 version 533 is already newer than 531 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 533 (ObjNum 466 snap 152 seq_num 466) dirty exists 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: left oid 16 (ObjNum 466 snap 152 seq_num 466) 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1311: done (3 left) 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1316: rollback oid 49 current snap is 152 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 49 to 139 2024-09-20T19:26:31.240 INFO:tasks.rados.rados.0.smithi042.stdout:1313: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:1313: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:1313: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 672 (ObjNum 323 snap 104 seq_num 323) dirty exists 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:1314: finishing write tid 1 to smithi04268048-18 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:1314: finishing write tid 2 to smithi04268048-18 2024-09-20T19:26:31.241 INFO:tasks.rados.rados.0.smithi042.stdout:1313: done (3 left) 2024-09-20T19:26:31.242 INFO:tasks.rados.rados.0.smithi042.stdout:1317: delete oid 45 current snap is 152 2024-09-20T19:26:31.245 INFO:tasks.rados.rados.0.smithi042.stdout:1314: finishing write tid 3 to smithi04268048-18 2024-09-20T19:26:31.246 INFO:tasks.rados.rados.0.smithi042.stdout:1315: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:26:31.246 INFO:tasks.rados.rados.0.smithi042.stdout:1314: finishing write tid 4 to smithi04268048-18 2024-09-20T19:26:31.247 INFO:tasks.rados.rados.0.smithi042.stdout:1315: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:26:31.247 INFO:tasks.rados.rados.0.smithi042.stdout:1315: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:26:31.247 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 437 (ObjNum 413 snap 129 seq_num 413) dirty exists 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: finishing write tid 5 to smithi04268048-18 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: oid 18 updating version 0 to 704 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: oid 18 version 704 is already newer than 702 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: oid 18 version 704 is already newer than 701 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: oid 18 version 704 is already newer than 703 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 704 (ObjNum 468 snap 152 seq_num 468) dirty exists 2024-09-20T19:26:31.248 INFO:tasks.rados.rados.0.smithi042.stdout:1314: left oid 18 (ObjNum 468 snap 152 seq_num 468) 2024-09-20T19:26:31.251 INFO:tasks.rados.rados.0.smithi042.stdout:1316: finishing rollback tid 0 to smithi04268048-49 2024-09-20T19:26:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1316: finishing rollback tid 1 to smithi04268048-49 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1316: finishing rollback tid 2 to smithi04268048-49 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 691 (ObjNum 382 snap 123 seq_num 382) dirty exists 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1314: done (3 left) 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1315: done (2 left) 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1316: done (1 left) 2024-09-20T19:26:31.252 INFO:tasks.rados.rados.0.smithi042.stdout:1317: done (0 left) 2024-09-20T19:26:31.253 INFO:tasks.rados.rados.0.smithi042.stdout:1318: snap_remove snap 140 2024-09-20T19:26:31.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:31.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:31.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[22706]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[22706]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[22706]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[22706]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[32075]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[32075]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[32075]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.467 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:31 smithi042 bash[32075]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:31.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:31.613 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:31.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:31.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:31 smithi162 bash[24557]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:31 smithi162 bash[24557]: cluster 2024-09-20T19:26:30.069525+0000 mgr.y (mgr.24491) 328 : cluster [DBG] pgmap v539: 158 pgs: 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 98 active+clean; 463 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 28 op/s; 173/1230 objects degraded (14.065%) 2024-09-20T19:26:31.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:31 smithi162 bash[24557]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:31 smithi162 bash[24557]: cluster 2024-09-20T19:26:31.138643+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e402: 8 total, 7 up, 8 in 2024-09-20T19:26:31.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:31.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:31.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:31.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:32.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:32.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:32.137 INFO:tasks.rados.rados.0.smithi042.stdout:1318: done (0 left) 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1319: rollback oid 18 current snap is 152 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 134 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1320: read oid 41 snap -1 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1320: expect deleted 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1321: rollback oid 9 current snap is 152 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 134 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1322: rollback oid 2 current snap is 152 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 133 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1323: read oid 1 snap -1 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1323: expect (ObjNum 450 snap 150 seq_num 450) 2024-09-20T19:26:32.138 INFO:tasks.rados.rados.0.smithi042.stdout:1324: snap_remove snap 63 2024-09-20T19:26:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:32.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:32.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:32.371 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:32.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:32.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:32.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:32.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:32.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:32.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:26:32.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:32.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:33.032 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1324: done (5 left) 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1325: read oid 8 snap -1 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1325: expect (ObjNum 376 snap 118 seq_num 376) 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1326: copy_from oid 17 from oid 45 current snap is 152 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1321: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1321: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1321: finishing rollback tid 2 to smithi04268048-9 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 669 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:33.128 INFO:tasks.rados.rados.0.smithi042.stdout:1323: expect (ObjNum 450 snap 150 seq_num 450) 2024-09-20T19:26:33.160 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.168 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.168 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.168 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.168 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[32075]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:33 smithi042 bash[22706]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.179 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:33.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:33.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:33.317 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3400591 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1319: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1319: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1319: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 707 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1322: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1319: done (6 left) 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1320: done (5 left) 2024-09-20T19:26:33.324 INFO:tasks.rados.rados.0.smithi042.stdout:1321: done (4 left) 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:1323: done (3 left) 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:1327: write (excl) oid 7 current snap is 152 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:1322: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:1322: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 440 (ObjNum 413 snap 129 seq_num 413) dirty exists 2024-09-20T19:26:33.325 INFO:tasks.rados.rados.0.smithi042.stdout:1327: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-20T19:26:33.330 INFO:tasks.rados.rados.0.smithi042.stdout:1327: writing smithi04268048-7 from 699290 to 1163853 tid 1 2024-09-20T19:26:33.336 INFO:tasks.rados.rados.0.smithi042.stdout:1327: writing smithi04268048-7 from 1603478 to 2138633 tid 2 2024-09-20T19:26:33.344 INFO:tasks.rados.rados.0.smithi042.stdout:1327: writing smithi04268048-7 from 2823983 to 3589829 tid 3 2024-09-20T19:26:33.344 INFO:tasks.rados.rados.0.smithi042.stdout:1322: done (3 left) 2024-09-20T19:26:33.344 INFO:tasks.rados.rados.0.smithi042.stdout:1328: rollback oid 33 current snap is 152 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 125 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:1326: finishing copy_from to smithi04268048-17 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:1326: got expected ENOENT (src dne) 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:1326: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:1325: expect (ObjNum 376 snap 118 seq_num 376) 2024-09-20T19:26:33.345 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:33.372 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:33.404 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1235230 2024-09-20T19:26:33.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1325: done (3 left) 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1326: done (2 left) 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1329: write oid 29 current snap is 152 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: finishing write tid 1 to smithi04268048-7 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: finishing write tid 2 to smithi04268048-7 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1328: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 421 (ObjNum 177 snap 0 seq_num 1414962704) dirty dne 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: finishing write tid 3 to smithi04268048-7 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: finishing write tid 4 to smithi04268048-7 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: finishing write tid 5 to smithi04268048-7 2024-09-20T19:26:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:1327: oid 7 updating version 0 to 609 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:1327: oid 7 updating version 609 to 610 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:1327: oid 7 version 610 is already newer than 607 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:1327: oid 7 version 610 is already newer than 608 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 610 (ObjNum 470 snap 152 seq_num 470) dirty exists 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:1327: left oid 7 (ObjNum 470 snap 152 seq_num 470) 2024-09-20T19:26:33.418 INFO:tasks.rados.rados.0.smithi042.stdout:1329: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-20T19:26:33.423 INFO:tasks.rados.rados.0.smithi042.stdout:1329: writing smithi04268048-29 from 716097 to 1255909 tid 1 2024-09-20T19:26:33.432 INFO:tasks.rados.rados.0.smithi042.stdout:1329: writing smithi04268048-29 from 1945607 to 2740773 tid 2 2024-09-20T19:26:33.434 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:33.439 INFO:tasks.rados.rados.0.smithi042.stdout:1329: writing smithi04268048-29 from 3535053 to 3916097 tid 3 2024-09-20T19:26:33.439 INFO:tasks.rados.rados.0.smithi042.stdout:1327: done (2 left) 2024-09-20T19:26:33.439 INFO:tasks.rados.rados.0.smithi042.stdout:1328: done (1 left) 2024-09-20T19:26:33.439 INFO:tasks.rados.rados.0.smithi042.stdout:1330: snap_remove snap 128 2024-09-20T19:26:33.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:32.071420+0000 mgr.y (mgr.24491) 329 : cluster [DBG] pgmap v542: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:32.136779+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e403: 8 total, 7 up, 8 in 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:33 smithi162 bash[24557]: cluster 2024-09-20T19:26:33.127809+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e404: 8 total, 7 up, 8 in 2024-09-20T19:26:33.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:33.600 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:26:33.618 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:26:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:26:33.636 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:26:33.636 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:26:33.653 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:26:33.653 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:26:33.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:33.673 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:26:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:26:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:26:33.692 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:26:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:33.710 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:26:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:26:33.730 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:26:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:26:33.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:33.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:33.924 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:33.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:34.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:34.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:34.170 INFO:tasks.rados.rados.0.smithi042.stdout:1329: finishing write tid 1 to smithi04268048-29 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: finishing write tid 2 to smithi04268048-29 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: finishing write tid 3 to smithi04268048-29 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: finishing write tid 4 to smithi04268048-29 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: finishing write tid 5 to smithi04268048-29 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: oid 29 updating version 0 to 565 2024-09-20T19:26:34.171 INFO:tasks.rados.rados.0.smithi042.stdout:1329: oid 29 updating version 565 to 567 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1329: oid 29 updating version 567 to 568 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1329: oid 29 version 568 is already newer than 566 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 568 (ObjNum 471 snap 152 seq_num 471) dirty exists 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1329: left oid 29 (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1329: done (1 left) 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1330: done (0 left) 2024-09-20T19:26:34.172 INFO:tasks.rados.rados.0.smithi042.stdout:1331: snap_create 2024-09-20T19:26:34.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:34.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:34.327 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:34 smithi162 bash[24557]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.327 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:34 smithi162 bash[24557]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:34.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:34 smithi042 bash[32075]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:34 smithi042 bash[32075]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:34 smithi042 bash[22706]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:34 smithi042 bash[22706]: audit 2024-09-20T19:26:33.600272+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.42:0/2477668083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:34.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:34.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:34.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.734 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:26:34.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:34.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:35.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:35.174 INFO:tasks.rados.rados.0.smithi042.stdout:1331: done (0 left) 2024-09-20T19:26:35.174 INFO:tasks.rados.rados.0.smithi042.stdout:1332: rollback oid 38 current snap is 153 2024-09-20T19:26:35.174 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 132 2024-09-20T19:26:35.174 INFO:tasks.rados.rados.0.smithi042.stdout:1333: snap_create 2024-09-20T19:26:35.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:35.178 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[22706]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.178 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[22706]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.178 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[22706]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[22706]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[32075]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[32075]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[32075]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:35 smithi042 bash[32075]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.189 INFO:tasks.rados.rados.0.smithi042.stdout:1332: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:26:35.191 INFO:tasks.rados.rados.0.smithi042.stdout:1332: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:26:35.191 INFO:tasks.rados.rados.0.smithi042.stdout:1332: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:26:35.191 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 571 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:26:35.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:35.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:35.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:35.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:35 smithi162 bash[24557]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:35 smithi162 bash[24557]: cluster 2024-09-20T19:26:34.072476+0000 mgr.y (mgr.24491) 330 : cluster [DBG] pgmap v545: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:35.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:35 smithi162 bash[24557]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:35 smithi162 bash[24557]: cluster 2024-09-20T19:26:34.169329+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e405: 8 total, 7 up, 8 in 2024-09-20T19:26:35.460 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:35.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:35.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:26:35.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:26:35.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:35.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:35.852 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:35.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:35.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:35.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:35.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:36.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:36.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:36.190 INFO:tasks.rados.rados.0.smithi042.stdout:1332: done (1 left) 2024-09-20T19:26:36.190 INFO:tasks.rados.rados.0.smithi042.stdout:1333: done (0 left) 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1334: rollback oid 12 current snap is 154 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 151 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1335: read oid 4 snap -1 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1335: expect deleted 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1336: copy_from oid 45 from oid 14 current snap is 154 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1337: write oid 26 current snap is 154 2024-09-20T19:26:36.191 INFO:tasks.rados.rados.0.smithi042.stdout:1337: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-20T19:26:36.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: writing smithi04268048-26 from 732904 to 1464318 tid 1 2024-09-20T19:26:36.209 INFO:tasks.rados.rados.0.smithi042.stdout:1337: writing smithi04268048-26 from 2120442 to 2659266 tid 2 2024-09-20T19:26:36.216 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:36.217 INFO:tasks.rados.rados.0.smithi042.stdout:1337: writing smithi04268048-26 from 3278829 to 3783572 tid 3 2024-09-20T19:26:36.217 INFO:tasks.rados.rados.0.smithi042.stdout:1338: write oid 47 current snap is 154 2024-09-20T19:26:36.218 INFO:tasks.rados.rados.0.smithi042.stdout:1338: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-20T19:26:36.224 INFO:tasks.rados.rados.0.smithi042.stdout:1338: writing smithi04268048-47 from 749711 to 1156374 tid 1 2024-09-20T19:26:36.233 INFO:tasks.rados.rados.0.smithi042.stdout:1338: writing smithi04268048-47 from 1778924 to 2461406 tid 2 2024-09-20T19:26:36.240 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:36.243 INFO:tasks.rados.rados.0.smithi042.stdout:1338: writing smithi04268048-47 from 3189899 to 3905914 tid 3 2024-09-20T19:26:36.243 INFO:tasks.rados.rados.0.smithi042.stdout:1339: snap_remove snap 119 2024-09-20T19:26:36.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:36.371 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:36.428 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:36 smithi042 bash[32075]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:36 smithi042 bash[32075]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:36 smithi042 bash[22706]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:36 smithi042 bash[22706]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:36.486 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:36 smithi162 bash[24557]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.486 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:36 smithi162 bash[24557]: cluster 2024-09-20T19:26:35.172758+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e406: 8 total, 7 up, 8 in 2024-09-20T19:26:36.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:36.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:36.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:36.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:36.763 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:36.866 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:36.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:36.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:7.d 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:7.5 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:7.4 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:6.d 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:6.9 2024-09-20T19:26:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:6.16 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:6.11 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:6.10 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:6.0 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.e 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.c 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:7.e 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.8 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.5 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.2 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:7.7 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.1e 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:5.1d 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:2.1f 2024-09-20T19:26:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:2.1e 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:7.9 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:2.1 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:5.1b 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:2.19 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:4.1e 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:2.13 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:1.6 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:7.6 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:1.2 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:1.0 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:6.a 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:2.4 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:2.1d 2024-09-20T19:26:36.943 INFO:teuthology.orchestra.run.smithi042.stdout:1.1 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:2.15 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:5.3 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.b 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:2.5 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:2.c 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:2.10 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.19 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:3.1 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.0 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:7.f 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.16 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:6.13 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.3 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.1a 2024-09-20T19:26:36.944 INFO:teuthology.orchestra.run.smithi042.stdout:4.6 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:4.7 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:5.b 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:2.0 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:2.f 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:5.13 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:5.7 2024-09-20T19:26:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:5.15 2024-09-20T19:26:36.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:37.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:37.198 INFO:tasks.rados.rados.0.smithi042.stdout:1339: done (5 left) 2024-09-20T19:26:37.198 INFO:tasks.rados.rados.0.smithi042.stdout:1340: delete oid 18 current snap is 154 2024-09-20T19:26:37.198 INFO:tasks.rados.rados.0.smithi042.stdout:1334: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1334: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1334: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 436 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1337: finishing write tid 1 to smithi04268048-26 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1336: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1337: finishing write tid 2 to smithi04268048-26 2024-09-20T19:26:37.199 INFO:tasks.rados.rados.0.smithi042.stdout:1337: finishing write tid 3 to smithi04268048-26 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: finishing write tid 4 to smithi04268048-26 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: finishing write tid 5 to smithi04268048-26 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: oid 26 updating version 0 to 614 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: oid 26 version 614 is already newer than 612 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: oid 26 version 614 is already newer than 611 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: oid 26 version 614 is already newer than 613 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 614 (ObjNum 472 snap 154 seq_num 472) dirty exists 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1337: left oid 26 (ObjNum 472 snap 154 seq_num 472) 2024-09-20T19:26:37.200 INFO:tasks.rados.rados.0.smithi042.stdout:1336: finishing copy_from to smithi04268048-45 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 337 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: finishing write tid 1 to smithi04268048-47 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: finishing write tid 2 to smithi04268048-47 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: finishing write tid 3 to smithi04268048-47 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: finishing write tid 4 to smithi04268048-47 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: finishing write tid 5 to smithi04268048-47 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: oid 47 updating version 0 to 672 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: oid 47 updating version 672 to 673 2024-09-20T19:26:37.201 INFO:tasks.rados.rados.0.smithi042.stdout:1338: oid 47 version 673 is already newer than 671 2024-09-20T19:26:37.202 INFO:tasks.rados.rados.0.smithi042.stdout:1338: oid 47 version 673 is already newer than 670 2024-09-20T19:26:37.202 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 673 (ObjNum 473 snap 154 seq_num 473) dirty exists 2024-09-20T19:26:37.202 INFO:tasks.rados.rados.0.smithi042.stdout:1338: left oid 47 (ObjNum 473 snap 154 seq_num 473) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1334: done (5 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1335: done (4 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1336: done (3 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1337: done (2 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1338: done (1 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1340: done (0 left) 2024-09-20T19:26:37.209 INFO:tasks.rados.rados.0.smithi042.stdout:1341: write oid 2 current snap is 154 2024-09-20T19:26:37.210 INFO:tasks.rados.rados.0.smithi042.stdout:1341: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-20T19:26:37.217 INFO:tasks.rados.rados.0.smithi042.stdout:1341: writing smithi04268048-2 from 783325 to 1340486 tid 1 2024-09-20T19:26:37.225 INFO:tasks.rados.rados.0.smithi042.stdout:1341: writing smithi04268048-2 from 1779535 to 2465686 tid 2 2024-09-20T19:26:37.233 INFO:tasks.rados.rados.0.smithi042.stdout:1341: writing smithi04268048-2 from 3244745 to 3699657 tid 3 2024-09-20T19:26:37.233 INFO:tasks.rados.rados.0.smithi042.stdout:1342: delete oid 50 current snap is 154 2024-09-20T19:26:37.234 INFO:tasks.rados.rados.0.smithi042.stdout:1341: finishing write tid 1 to smithi04268048-2 2024-09-20T19:26:37.234 INFO:tasks.rados.rados.0.smithi042.stdout:1341: finishing write tid 2 to smithi04268048-2 2024-09-20T19:26:37.240 INFO:tasks.rados.rados.0.smithi042.stdout:1341: finishing write tid 3 to smithi04268048-2 2024-09-20T19:26:37.241 INFO:tasks.rados.rados.0.smithi042.stdout:1341: finishing write tid 4 to smithi04268048-2 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: finishing write tid 5 to smithi04268048-2 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: oid 2 updating version 0 to 443 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: oid 2 updating version 443 to 444 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: oid 2 version 444 is already newer than 442 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: oid 2 version 444 is already newer than 441 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 444 (ObjNum 475 snap 154 seq_num 475) dirty exists 2024-09-20T19:26:37.242 INFO:tasks.rados.rados.0.smithi042.stdout:1341: left oid 2 (ObjNum 475 snap 154 seq_num 475) 2024-09-20T19:26:37.244 INFO:tasks.rados.rados.0.smithi042.stdout:1341: done (1 left) 2024-09-20T19:26:37.244 INFO:tasks.rados.rados.0.smithi042.stdout:1342: done (0 left) 2024-09-20T19:26:37.244 INFO:tasks.rados.rados.0.smithi042.stdout:1343: snap_remove snap 152 2024-09-20T19:26:37.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:37.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:37 smithi162 bash[24557]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:37 smithi162 bash[24557]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:37 smithi162 bash[24557]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:37 smithi162 bash[24557]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:37.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:37.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:37.507 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[32075]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[32075]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[32075]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[32075]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[22706]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[22706]: cluster 2024-09-20T19:26:36.073647+0000 mgr.y (mgr.24491) 331 : cluster [DBG] pgmap v548: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 476 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 179/1263 objects degraded (14.173%) 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[22706]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:37 smithi042 bash[22706]: cluster 2024-09-20T19:26:36.189262+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e407: 8 total, 7 up, 8 in 2024-09-20T19:26:37.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:37.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:26:37.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:37.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:37.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:37.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:37.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:38.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:38.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:38.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 7.d --file /var/log/ceph/exp.7.d.0 2024-09-20T19:26:38.135 INFO:tasks.rados.rados.0.smithi042.stdout:1343: done (0 left) 2024-09-20T19:26:38.135 INFO:tasks.rados.rados.0.smithi042.stdout:1344: read oid 39 snap -1 2024-09-20T19:26:38.135 INFO:tasks.rados.rados.0.smithi042.stdout:1344: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:38.135 INFO:tasks.rados.rados.0.smithi042.stdout:1345: delete oid 31 current snap is 154 2024-09-20T19:26:38.143 INFO:tasks.rados.rados.0.smithi042.stdout:1345: done (1 left) 2024-09-20T19:26:38.143 INFO:tasks.rados.rados.0.smithi042.stdout:1346: write (excl) oid 15 current snap is 154 2024-09-20T19:26:38.144 INFO:tasks.rados.rados.0.smithi042.stdout:1346: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-20T19:26:38.154 INFO:tasks.rados.rados.0.smithi042.stdout:1346: writing smithi04268048-15 from 433746 to 1216654 tid 1 2024-09-20T19:26:38.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:38.160 INFO:tasks.rados.rados.0.smithi042.stdout:1346: writing smithi04268048-15 from 1954981 to 2033746 tid 2 2024-09-20T19:26:38.160 INFO:tasks.rados.rados.0.smithi042.stdout:1347: rollback oid 36 current snap is 154 2024-09-20T19:26:38.160 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 36 to 141 2024-09-20T19:26:38.160 INFO:tasks.rados.rados.0.smithi042.stdout:1344: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:38.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:38.214 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:38.258 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 615355 2024-09-20T19:26:38.284 INFO:tasks.rados.rados.0.smithi042.stdout:1346: finishing write tid 1 to smithi04268048-15 2024-09-20T19:26:38.284 INFO:tasks.rados.rados.0.smithi042.stdout:1347: finishing rollback tid 0 to smithi04268048-36 2024-09-20T19:26:38.284 INFO:tasks.rados.rados.0.smithi042.stdout:1347: finishing rollback tid 1 to smithi04268048-36 2024-09-20T19:26:38.284 INFO:tasks.rados.rados.0.smithi042.stdout:1347: finishing rollback tid 2 to smithi04268048-36 2024-09-20T19:26:38.284 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 556 (ObjNum 354 snap 115 seq_num 354) dirty exists 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: finishing write tid 2 to smithi04268048-15 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1344: done (2 left) 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1347: done (1 left) 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1348: snap_create 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: finishing write tid 3 to smithi04268048-15 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: finishing write tid 4 to smithi04268048-15 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: oid 15 updating version 0 to 535 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: oid 15 updating version 535 to 536 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: oid 15 version 536 is already newer than 534 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 536 (ObjNum 478 snap 154 seq_num 478) dirty exists 2024-09-20T19:26:38.285 INFO:tasks.rados.rados.0.smithi042.stdout:1346: left oid 15 (ObjNum 478 snap 154 seq_num 478) 2024-09-20T19:26:38.286 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.310 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:38.326 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:26:38.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.417 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.417 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.417 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:38 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:38.535 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:38.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:37.197649+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e408: 8 total, 7 up, 8 in 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.123685+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: Degraded data redundancy: 179/1263 objects degraded (14.173%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:38 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.133583+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e409: 8 total, 7 up, 8 in 2024-09-20T19:26:38.567 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:38.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.698 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:38.801 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:38.830 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:38.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:38.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:38.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:39.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:39.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.216 INFO:tasks.rados.rados.0.smithi042.stdout:1346: done (1 left) 2024-09-20T19:26:39.216 INFO:tasks.rados.rados.0.smithi042.stdout:1348: done (0 left) 2024-09-20T19:26:39.216 INFO:tasks.rados.rados.0.smithi042.stdout:1349: rollback oid 44 current snap is 155 2024-09-20T19:26:39.216 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 137 2024-09-20T19:26:39.216 INFO:tasks.rados.rados.0.smithi042.stdout:1350: snap_create 2024-09-20T19:26:39.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.224 INFO:tasks.rados.rados.0.smithi042.stdout:1349: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:26:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:39.226 INFO:tasks.rados.rados.0.smithi042.stdout:1349: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:26:39.226 INFO:tasks.rados.rados.0.smithi042.stdout:1349: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:26:39.226 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 734 (ObjNum 426 snap 133 seq_num 426) dirty exists 2024-09-20T19:26:39.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:39.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:39 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:39 smithi042 bash[22706]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:39 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:39 smithi042 bash[32075]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:39.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:39 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:39 smithi162 bash[24557]: cluster 2024-09-20T19:26:38.074759+0000 mgr.y (mgr.24491) 332 : cluster [DBG] pgmap v551: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:39.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:39.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:39.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:39.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:39.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:39.871 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:39.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:39.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:40.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:40.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:40.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1349: done (1 left) 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1350: done (0 left) 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1351: rollback oid 49 current snap is 156 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 49 to 112 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1352: read oid 21 snap 155 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1352: expect (ObjNum 442 snap 145 seq_num 442) 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1353: copy_from oid 16 from oid 3 current snap is 156 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1354: read oid 24 snap -1 2024-09-20T19:26:40.227 INFO:tasks.rados.rados.0.smithi042.stdout:1354: expect deleted 2024-09-20T19:26:40.228 INFO:tasks.rados.rados.0.smithi042.stdout:1355: copy_from oid 4 from oid 36 current snap is 156 2024-09-20T19:26:40.228 INFO:tasks.rados.rados.0.smithi042.stdout:1356: snap_create 2024-09-20T19:26:40.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:40.241 INFO:tasks.rados.rados.0.smithi042.stdout:1351: finishing rollback tid 0 to smithi04268048-49 2024-09-20T19:26:40.242 INFO:tasks.rados.rados.0.smithi042.stdout:1351: finishing rollback tid 1 to smithi04268048-49 2024-09-20T19:26:40.242 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 737 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:26:40.259 INFO:tasks.rados.rados.0.smithi042.stdout:1352: expect (ObjNum 442 snap 145 seq_num 442) 2024-09-20T19:26:40.261 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:40.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:40 smithi162 bash[24557]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:40 smithi162 bash[24557]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:40 smithi162 bash[24557]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:40 smithi162 bash[24557]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:40.421 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:40.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:40.501 INFO:tasks.rados.rados.0.smithi042.stdout:1353: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:26:40.501 INFO:tasks.rados.rados.0.smithi042.stdout:1353: finishing copy_from to smithi04268048-16 2024-09-20T19:26:40.501 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 672 (ObjNum 323 snap 104 seq_num 323) dirty exists 2024-09-20T19:26:40.501 INFO:tasks.rados.rados.0.smithi042.stdout:1355: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:26:40.502 INFO:tasks.rados.rados.0.smithi042.stdout:1355: finishing copy_from to smithi04268048-4 2024-09-20T19:26:40.502 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 556 (ObjNum 354 snap 115 seq_num 354) dirty exists 2024-09-20T19:26:40.524 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:40.627 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:40.634 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:40.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[32075]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[32075]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[32075]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[32075]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[22706]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[22706]: cluster 2024-09-20T19:26:39.215227+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e410: 8 total, 7 up, 8 in 2024-09-20T19:26:40.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[22706]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:40 smithi042 bash[22706]: audit 2024-09-20T19:26:39.272184+0000 mgr.y (mgr.24491) 333 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:40.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:40.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:40.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:40.912 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:40.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:41.015 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:41.043 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:41.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:41.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:41.231 INFO:tasks.rados.rados.0.smithi042.stdout:1351: done (5 left) 2024-09-20T19:26:41.231 INFO:tasks.rados.rados.0.smithi042.stdout:1352: done (4 left) 2024-09-20T19:26:41.231 INFO:tasks.rados.rados.0.smithi042.stdout:1353: done (3 left) 2024-09-20T19:26:41.231 INFO:tasks.rados.rados.0.smithi042.stdout:1354: done (2 left) 2024-09-20T19:26:41.232 INFO:tasks.rados.rados.0.smithi042.stdout:1355: done (1 left) 2024-09-20T19:26:41.232 INFO:tasks.rados.rados.0.smithi042.stdout:1356: done (0 left) 2024-09-20T19:26:41.232 INFO:tasks.rados.rados.0.smithi042.stdout:1357: rollback oid 26 current snap is 157 2024-09-20T19:26:41.232 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 26 to 130 2024-09-20T19:26:41.232 INFO:tasks.rados.rados.0.smithi042.stdout:1358: delete oid 2 current snap is 157 2024-09-20T19:26:41.242 INFO:tasks.rados.rados.0.smithi042.stdout:1357: finishing rollback tid 0 to smithi04268048-26 2024-09-20T19:26:41.246 INFO:tasks.rados.rados.0.smithi042.stdout:1358: done (1 left) 2024-09-20T19:26:41.246 INFO:tasks.rados.rados.0.smithi042.stdout:1359: snap_create 2024-09-20T19:26:41.248 INFO:tasks.rados.rados.0.smithi042.stdout:1357: finishing rollback tid 1 to smithi04268048-26 2024-09-20T19:26:41.249 INFO:tasks.rados.rados.0.smithi042.stdout:1357: finishing rollback tid 2 to smithi04268048-26 2024-09-20T19:26:41.249 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 617 (ObjNum 412 snap 129 seq_num 412) dirty exists 2024-09-20T19:26:41.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:41.280 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[22706]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.280 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[22706]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[22706]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[22706]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[32075]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[32075]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[32075]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.281 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:41 smithi042 bash[32075]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.296 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:41.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:41 smithi162 bash[24557]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:41 smithi162 bash[24557]: cluster 2024-09-20T19:26:40.075542+0000 mgr.y (mgr.24491) 334 : cluster [DBG] pgmap v554: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 445 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 181/1224 objects degraded (14.788%) 2024-09-20T19:26:41.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:41 smithi162 bash[24557]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:41 smithi162 bash[24557]: cluster 2024-09-20T19:26:40.224791+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e411: 8 total, 7 up, 8 in 2024-09-20T19:26:41.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:41.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:41.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:41.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:41.921 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:41.946 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:41.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:42.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:42.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:42.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:42.245 INFO:tasks.rados.rados.0.smithi042.stdout:1357: done (1 left) 2024-09-20T19:26:42.245 INFO:tasks.rados.rados.0.smithi042.stdout:1359: done (0 left) 2024-09-20T19:26:42.245 INFO:tasks.rados.rados.0.smithi042.stdout:1360: write (excl) oid 36 current snap is 158 2024-09-20T19:26:42.245 INFO:tasks.rados.rados.0.smithi042.stdout:1360: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-20T19:26:42.251 INFO:tasks.rados.rados.0.smithi042.stdout:1360: writing smithi04268048-36 from 467360 to 1117119 tid 1 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:1360: writing smithi04268048-36 from 1671945 to 2067360 tid 2 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:1361: rollback oid 5 current snap is 158 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 146 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:1360: finishing write tid 1 to smithi04268048-36 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:1362: write oid 16 current snap is 158 2024-09-20T19:26:42.257 INFO:tasks.rados.rados.0.smithi042.stdout:1362: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-20T19:26:42.264 INFO:tasks.rados.rados.0.smithi042.stdout:1362: writing smithi04268048-16 from 484167 to 1209175 tid 1 2024-09-20T19:26:42.269 INFO:tasks.rados.rados.0.smithi042.stdout:1362: writing smithi04268048-16 from 1730427 to 2084167 tid 2 2024-09-20T19:26:42.269 INFO:tasks.rados.rados.0.smithi042.stdout:1363: snap_remove snap 149 2024-09-20T19:26:42.313 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:26:42.313 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:26:42.313 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:26:42.313 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:26:42.313 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:26:42.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:42.325 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:42 smithi162 bash[24557]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.325 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:42 smithi162 bash[24557]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:42.445 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:42.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:42 smithi042 bash[32075]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:42 smithi042 bash[32075]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.473 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:42 smithi042 bash[22706]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.473 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:42 smithi042 bash[22706]: cluster 2024-09-20T19:26:41.229890+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e412: 8 total, 7 up, 8 in 2024-09-20T19:26:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:42.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:42.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:42.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:42.732 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:42.837 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:42.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:42.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:42.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:43.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:43.126 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:43.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:43.257 INFO:tasks.rados.rados.0.smithi042.stdout:1363: done (3 left) 2024-09-20T19:26:43.257 INFO:tasks.rados.rados.0.smithi042.stdout:1364: snap_remove snap 129 2024-09-20T19:26:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:43.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:43.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.376 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[22706]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.377 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:43 smithi042 bash[32075]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:43.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:43.509 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.509 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:42.077456+0000 mgr.y (mgr.24491) 335 : cluster [DBG] pgmap v557: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:43.509 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:42.243674+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e413: 8 total, 7 up, 8 in 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: audit 2024-09-20T19:26:42.276223+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: audit 2024-09-20T19:26:42.277195+0000 mon.a (mon.0) 1472 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:43 smithi162 bash[24557]: cluster 2024-09-20T19:26:43.125583+0000 mon.a (mon.0) 1473 : cluster [WRN] Health check update: Degraded data redundancy: 187/1233 objects degraded (15.166%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:43.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:43.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:43.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:43.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:43.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:43.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:43.913 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:43.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:44.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:44.146 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:44.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:44.275 INFO:tasks.rados.rados.0.smithi042.stdout:1364: done (3 left) 2024-09-20T19:26:44.276 INFO:tasks.rados.rados.0.smithi042.stdout:1365: write (excl) oid 2 current snap is 158 2024-09-20T19:26:44.276 INFO:tasks.rados.rados.0.smithi042.stdout:1365: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-20T19:26:44.277 INFO:tasks.rados.rados.0.smithi042.stdout:1365: writing smithi04268048-2 from 500974 to 901231 tid 1 2024-09-20T19:26:44.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:44.285 INFO:tasks.rados.rados.0.smithi042.stdout:1365: writing smithi04268048-2 from 1388909 to 1946078 tid 2 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1366: read oid 4 snap -1 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1366: expect (ObjNum 354 snap 115 seq_num 354) 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1367: write (excl) oid 47 current snap is 158 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: finishing write tid 2 to smithi04268048-36 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: finishing write tid 3 to smithi04268048-36 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: finishing write tid 4 to smithi04268048-36 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: oid 36 updating version 0 to 558 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: oid 36 version 558 is already newer than 557 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: oid 36 updating version 558 to 559 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 559 (ObjNum 480 snap 158 seq_num 480) dirty exists 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1360: left oid 36 (ObjNum 480 snap 158 seq_num 480) 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:1361: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:26:44.286 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 398 (ObjNum 225 snap 0 seq_num 1415937552) dirty dne 2024-09-20T19:26:44.287 INFO:tasks.rados.rados.0.smithi042.stdout:1362: finishing write tid 1 to smithi04268048-16 2024-09-20T19:26:44.287 INFO:tasks.rados.rados.0.smithi042.stdout:1367: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-20T19:26:44.294 INFO:tasks.rados.rados.0.smithi042.stdout:1367: writing smithi04268048-47 from 517781 to 993287 tid 1 2024-09-20T19:26:44.303 INFO:tasks.rados.rados.0.smithi042.stdout:1367: writing smithi04268048-47 from 1731038 to 2117781 tid 2 2024-09-20T19:26:44.304 INFO:tasks.rados.rados.0.smithi042.stdout:1360: done (5 left) 2024-09-20T19:26:44.304 INFO:tasks.rados.rados.0.smithi042.stdout:1361: done (4 left) 2024-09-20T19:26:44.304 INFO:tasks.rados.rados.0.smithi042.stdout:1368: read oid 24 snap 156 2024-09-20T19:26:44.304 INFO:tasks.rados.rados.0.smithi042.stdout:1368: expect deleted 2024-09-20T19:26:44.304 INFO:tasks.rados.rados.0.smithi042.stdout:1369: snap_remove snap 141 2024-09-20T19:26:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:44.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:26:44.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:44.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:44 smithi042 bash[32075]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:44 smithi042 bash[32075]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:44 smithi042 bash[22706]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:44 smithi042 bash[22706]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:44.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:44.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:44 smithi162 bash[24557]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:44 smithi162 bash[24557]: cluster 2024-09-20T19:26:43.255217+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e414: 8 total, 7 up, 8 in 2024-09-20T19:26:44.560 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:44.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:44.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:44.810 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:44.920 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:44.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:44.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:45.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:45.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:45.274 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:26:45.274 INFO:tasks.rados.rados.0.smithi042.stdout:1369: done (5 left) 2024-09-20T19:26:45.275 INFO:tasks.rados.rados.0.smithi042.stdout:1370: write (excl) oid 40 current snap is 158 2024-09-20T19:26:45.275 INFO:tasks.rados.rados.0.smithi042.stdout:1362: finishing write tid 2 to smithi04268048-16 2024-09-20T19:26:45.275 INFO:tasks.rados.rados.0.smithi042.stdout:1370: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-20T19:26:45.281 INFO:tasks.rados.rados.0.smithi042.stdout:1370: writing smithi04268048-40 from 534588 to 1085343 tid 1 2024-09-20T19:26:45.288 INFO:tasks.rados.rados.0.smithi042.stdout:1370: writing smithi04268048-40 from 1789520 to 2134588 tid 2 2024-09-20T19:26:45.288 INFO:tasks.rados.rados.0.smithi042.stdout:1371: copy_from oid 3 from oid 29 current snap is 158 2024-09-20T19:26:45.288 INFO:tasks.rados.rados.0.smithi042.stdout:1362: finishing write tid 3 to smithi04268048-16 2024-09-20T19:26:45.288 INFO:tasks.rados.rados.0.smithi042.stdout:1362: finishing write tid 4 to smithi04268048-16 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1362: oid 16 updating version 0 to 675 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1362: oid 16 version 675 is already newer than 673 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1362: oid 16 version 675 is already newer than 674 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 675 (ObjNum 481 snap 158 seq_num 481) dirty exists 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1362: left oid 16 (ObjNum 481 snap 158 seq_num 481) 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: finishing write tid 1 to smithi04268048-2 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: finishing write tid 2 to smithi04268048-2 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: finishing write tid 3 to smithi04268048-2 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: finishing write tid 4 to smithi04268048-2 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: oid 2 updating version 0 to 447 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: oid 2 version 447 is already newer than 446 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: oid 2 updating version 447 to 448 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 448 (ObjNum 482 snap 158 seq_num 482) dirty exists 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1365: left oid 2 (ObjNum 482 snap 158 seq_num 482) 2024-09-20T19:26:45.289 INFO:tasks.rados.rados.0.smithi042.stdout:1366: expect (ObjNum 354 snap 115 seq_num 354) 2024-09-20T19:26:45.290 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:45.299 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:45.378 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2971720 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: finishing write tid 1 to smithi04268048-47 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: finishing write tid 2 to smithi04268048-47 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: finishing write tid 3 to smithi04268048-47 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: finishing write tid 4 to smithi04268048-47 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: oid 47 updating version 0 to 678 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: oid 47 updating version 678 to 679 2024-09-20T19:26:45.392 INFO:tasks.rados.rados.0.smithi042.stdout:1367: oid 47 updating version 679 to 680 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 680 (ObjNum 483 snap 158 seq_num 483) dirty exists 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1367: left oid 47 (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1362: done (6 left) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1365: done (5 left) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1366: done (4 left) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1367: done (3 left) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1368: done (2 left) 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1372: write oid 14 current snap is 158 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: finishing write tid 1 to smithi04268048-40 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: finishing write tid 2 to smithi04268048-40 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: finishing write tid 3 to smithi04268048-40 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: finishing write tid 4 to smithi04268048-40 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: oid 40 updating version 0 to 573 2024-09-20T19:26:45.393 INFO:tasks.rados.rados.0.smithi042.stdout:1370: oid 40 version 573 is already newer than 572 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:1370: oid 40 updating version 573 to 574 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 574 (ObjNum 484 snap 158 seq_num 484) dirty exists 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:1370: left oid 40 (ObjNum 484 snap 158 seq_num 484) 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:1371: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:1371: finishing copy_from to smithi04268048-3 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 568 (ObjNum 471 snap 152 seq_num 471) dirty exists 2024-09-20T19:26:45.394 INFO:tasks.rados.rados.0.smithi042.stdout:1372: seq_num 485 ranges {551395=626004,1848002=303393} 2024-09-20T19:26:45.398 INFO:tasks.rados.rados.0.smithi042.stdout:1372: writing smithi04268048-14 from 551395 to 1177399 tid 1 2024-09-20T19:26:45.404 INFO:tasks.rados.rados.0.smithi042.stdout:1372: writing smithi04268048-14 from 1848002 to 2151395 tid 2 2024-09-20T19:26:45.404 INFO:tasks.rados.rados.0.smithi042.stdout:1370: done (2 left) 2024-09-20T19:26:45.404 INFO:tasks.rados.rados.0.smithi042.stdout:1371: done (1 left) 2024-09-20T19:26:45.404 INFO:tasks.rados.rados.0.smithi042.stdout:1373: delete oid 20 current snap is 158 2024-09-20T19:26:45.404 INFO:tasks.rados.rados.0.smithi042.stdout:1372: finishing write tid 1 to smithi04268048-14 2024-09-20T19:26:45.406 INFO:tasks.rados.rados.0.smithi042.stdout:1373: done (1 left) 2024-09-20T19:26:45.406 INFO:tasks.rados.rados.0.smithi042.stdout:1374: write oid 21 current snap is 158 2024-09-20T19:26:45.407 INFO:tasks.rados.rados.0.smithi042.stdout:1374: seq_num 487 ranges {585009=492855,1564966=620043} 2024-09-20T19:26:45.412 INFO:tasks.rados.rados.0.smithi042.stdout:1374: writing smithi04268048-21 from 585009 to 1077864 tid 1 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1374: writing smithi04268048-21 from 1564966 to 2185009 tid 2 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1375: read oid 43 snap 156 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1375: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1376: read oid 20 snap -1 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1376: expect deleted 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1377: snap_create 2024-09-20T19:26:45.420 INFO:tasks.rados.rados.0.smithi042.stdout:1372: finishing write tid 2 to smithi04268048-14 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: finishing write tid 3 to smithi04268048-14 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: finishing write tid 4 to smithi04268048-14 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: oid 14 updating version 0 to 406 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: oid 14 version 406 is already newer than 405 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: oid 14 version 406 is already newer than 404 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 406 (ObjNum 485 snap 158 seq_num 485) dirty exists 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1372: left oid 14 (ObjNum 485 snap 158 seq_num 485) 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1374: finishing write tid 1 to smithi04268048-21 2024-09-20T19:26:45.421 INFO:tasks.rados.rados.0.smithi042.stdout:1375: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:26:45.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:45 smithi162 bash[24557]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.448 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:45.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:45.480 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:45.508 INFO:tasks.rados.rados.0.smithi042.stdout:1374: finishing write tid 2 to smithi04268048-21 2024-09-20T19:26:45.508 INFO:tasks.rados.rados.0.smithi042.stdout:1374: finishing write tid 3 to smithi04268048-21 2024-09-20T19:26:45.508 INFO:tasks.rados.rados.0.smithi042.stdout:1374: finishing write tid 4 to smithi04268048-21 2024-09-20T19:26:45.508 INFO:tasks.rados.rados.0.smithi042.stdout:1374: oid 21 updating version 0 to 711 2024-09-20T19:26:45.509 INFO:tasks.rados.rados.0.smithi042.stdout:1374: oid 21 updating version 711 to 713 2024-09-20T19:26:45.509 INFO:tasks.rados.rados.0.smithi042.stdout:1374: oid 21 version 713 is already newer than 712 2024-09-20T19:26:45.509 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 713 (ObjNum 487 snap 158 seq_num 487) dirty exists 2024-09-20T19:26:45.509 INFO:tasks.rados.rados.0.smithi042.stdout:1374: left oid 21 (ObjNum 487 snap 158 seq_num 487) 2024-09-20T19:26:45.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[32075]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: cluster 2024-09-20T19:26:44.078720+0000 mgr.y (mgr.24491) 336 : cluster [DBG] pgmap v560: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: cluster 2024-09-20T19:26:44.266670+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e415: 8 total, 7 up, 8 in 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: audit 2024-09-20T19:26:45.231761+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.42:0/1248346642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:45 smithi042 bash[22706]: audit 2024-09-20T19:26:45.232055+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25184 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:26:45.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:45.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:45.839 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:45.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:45.970 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:45.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:46.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:46.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:46.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:46.234 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:46.282 INFO:tasks.rados.rados.0.smithi042.stdout:1372: done (4 left) 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1374: done (3 left) 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1375: done (2 left) 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1376: done (1 left) 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1377: done (0 left) 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1378: rollback oid 48 current snap is 159 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 158 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1379: read oid 24 snap -1 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1379: expect deleted 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1380: copy_from oid 2 from oid 20 current snap is 159 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1381: write (excl) oid 44 current snap is 159 2024-09-20T19:26:46.283 INFO:tasks.rados.rados.0.smithi042.stdout:1381: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-20T19:26:46.292 INFO:tasks.rados.rados.0.smithi042.stdout:1381: writing smithi04268048-44 from 601816 to 1169920 tid 1 2024-09-20T19:26:46.300 INFO:tasks.rados.rados.0.smithi042.stdout:1381: writing smithi04268048-44 from 1623448 to 2075271 tid 2 2024-09-20T19:26:46.300 INFO:tasks.rados.rados.0.smithi042.stdout:1382: read oid 40 snap 136 2024-09-20T19:26:46.300 INFO:tasks.rados.rados.0.smithi042.stdout:1382: expect deleted 2024-09-20T19:26:46.300 INFO:tasks.rados.rados.0.smithi042.stdout:1383: write oid 45 current snap is 159 2024-09-20T19:26:46.300 INFO:tasks.rados.rados.0.smithi042.stdout:1383: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-20T19:26:46.310 INFO:tasks.rados.rados.0.smithi042.stdout:1383: writing smithi04268048-45 from 618623 to 1261976 tid 1 2024-09-20T19:26:46.318 INFO:tasks.rados.rados.0.smithi042.stdout:1383: writing smithi04268048-45 from 1681930 to 2218623 tid 2 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1384: delete oid 15 current snap is 159 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1380: finishing copy_from to smithi04268048-2 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1380: got expected ENOENT (src dne) 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1380: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: finishing write tid 1 to smithi04268048-44 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1378: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1378: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1378: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 717 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: finishing write tid 2 to smithi04268048-44 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: finishing write tid 3 to smithi04268048-44 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: finishing write tid 4 to smithi04268048-44 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: oid 44 updating version 0 to 762 2024-09-20T19:26:46.319 INFO:tasks.rados.rados.0.smithi042.stdout:1381: oid 44 updating version 762 to 763 2024-09-20T19:26:46.320 INFO:tasks.rados.rados.0.smithi042.stdout:1381: oid 44 version 763 is already newer than 761 2024-09-20T19:26:46.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 763 (ObjNum 488 snap 159 seq_num 488) dirty exists 2024-09-20T19:26:46.320 INFO:tasks.rados.rados.0.smithi042.stdout:1381: left oid 44 (ObjNum 488 snap 159 seq_num 488) 2024-09-20T19:26:46.320 INFO:tasks.rados.rados.0.smithi042.stdout:1383: finishing write tid 1 to smithi04268048-45 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1378: done (6 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1379: done (5 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1380: done (4 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1381: done (3 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1382: done (2 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1384: done (1 left) 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1385: write (excl) oid 31 current snap is 159 2024-09-20T19:26:46.324 INFO:tasks.rados.rados.0.smithi042.stdout:1385: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-20T19:26:46.335 INFO:tasks.rados.rados.0.smithi042.stdout:1385: writing smithi04268048-31 from 652237 to 1446088 tid 1 2024-09-20T19:26:46.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:46.342 INFO:tasks.rados.rados.0.smithi042.stdout:1385: writing smithi04268048-31 from 2198894 to 2252237 tid 2 2024-09-20T19:26:46.342 INFO:tasks.rados.rados.0.smithi042.stdout:1386: copy_from oid 35 from oid 6 current snap is 159 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: finishing write tid 2 to smithi04268048-45 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: finishing write tid 3 to smithi04268048-45 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: finishing write tid 4 to smithi04268048-45 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: oid 45 updating version 0 to 766 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: oid 45 updating version 766 to 767 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: oid 45 version 767 is already newer than 765 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 767 (ObjNum 489 snap 159 seq_num 489) dirty exists 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: left oid 45 (ObjNum 489 snap 159 seq_num 489) 2024-09-20T19:26:46.343 INFO:tasks.rados.rados.0.smithi042.stdout:1383: done (2 left) 2024-09-20T19:26:46.344 INFO:tasks.rados.rados.0.smithi042.stdout:1387: write oid 32 current snap is 159 2024-09-20T19:26:46.344 INFO:tasks.rados.rados.0.smithi042.stdout:1387: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:46 smithi162 bash[24557]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.354 INFO:tasks.rados.rados.0.smithi042.stdout:1387: writing smithi04268048-32 from 669044 to 1138144 tid 1 2024-09-20T19:26:46.363 INFO:tasks.rados.rados.0.smithi042.stdout:1387: writing smithi04268048-32 from 1741023 to 2200184 tid 2 2024-09-20T19:26:46.363 INFO:tasks.rados.rados.0.smithi042.stdout:1388: delete oid 17 current snap is 159 2024-09-20T19:26:46.363 INFO:tasks.rados.rados.0.smithi042.stdout:1385: finishing write tid 1 to smithi04268048-31 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: finishing write tid 2 to smithi04268048-31 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: finishing write tid 3 to smithi04268048-31 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: finishing write tid 4 to smithi04268048-31 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: oid 31 updating version 0 to 509 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: oid 31 version 509 is already newer than 508 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: oid 31 updating version 509 to 510 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 510 (ObjNum 491 snap 159 seq_num 491) dirty exists 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1385: left oid 31 (ObjNum 491 snap 159 seq_num 491) 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1386: finishing copy_from to smithi04268048-35 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1386: got expected ENOENT (src dne) 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1386: finishing copy_from racing read to smithi04268048-35 2024-09-20T19:26:46.364 INFO:tasks.rados.rados.0.smithi042.stdout:1387: finishing write tid 1 to smithi04268048-32 2024-09-20T19:26:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1385: done (3 left) 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1386: done (2 left) 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1388: done (1 left) 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1389: copy_from oid 48 from oid 31 current snap is 159 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1390: read oid 18 snap -1 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1390: expect deleted 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1391: rollback oid 27 current snap is 159 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 133 2024-09-20T19:26:46.366 INFO:tasks.rados.rados.0.smithi042.stdout:1392: snap_create 2024-09-20T19:26:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:46.368 INFO:tasks.rados.rados.0.smithi042.stdout:1387: finishing write tid 2 to smithi04268048-32 2024-09-20T19:26:46.368 INFO:tasks.rados.rados.0.smithi042.stdout:1387: finishing write tid 3 to smithi04268048-32 2024-09-20T19:26:46.368 INFO:tasks.rados.rados.0.smithi042.stdout:1391: finishing rollback tid 0 to smithi04268048-27 2024-09-20T19:26:46.369 INFO:tasks.rados.rados.0.smithi042.stdout:1391: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:26:46.369 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 683 (ObjNum 129 snap 0 seq_num 1415403056) dirty dne 2024-09-20T19:26:46.369 INFO:tasks.rados.rados.0.smithi042.stdout:1387: finishing write tid 4 to smithi04268048-32 2024-09-20T19:26:46.370 INFO:tasks.rados.rados.0.smithi042.stdout:1387: oid 32 updating version 0 to 770 2024-09-20T19:26:46.370 INFO:tasks.rados.rados.0.smithi042.stdout:1387: oid 32 version 770 is already newer than 769 2024-09-20T19:26:46.370 INFO:tasks.rados.rados.0.smithi042.stdout:1387: oid 32 updating version 770 to 771 2024-09-20T19:26:46.370 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 771 (ObjNum 492 snap 159 seq_num 492) dirty exists 2024-09-20T19:26:46.370 INFO:tasks.rados.rados.0.smithi042.stdout:1387: left oid 32 (ObjNum 492 snap 159 seq_num 492) 2024-09-20T19:26:46.375 INFO:tasks.rados.rados.0.smithi042.stdout:1389: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:26:46.378 INFO:tasks.rados.rados.0.smithi042.stdout:1389: finishing copy_from to smithi04268048-48 2024-09-20T19:26:46.378 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 510 (ObjNum 491 snap 159 seq_num 491) dirty exists 2024-09-20T19:26:46.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:46.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:46.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:46.678 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[22706]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: cluster 2024-09-20T19:26:45.259850+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: audit 2024-09-20T19:26:45.268355+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25184 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:46 smithi042 bash[32075]: cluster 2024-09-20T19:26:45.271881+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e416: 8 total, 7 up, 8 in 2024-09-20T19:26:46.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:46.758 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:46.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:46.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:46.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:47.021 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:47.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:47.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:47.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:47.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1387: done (4 left) 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1389: done (3 left) 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1390: done (2 left) 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1391: done (1 left) 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1392: done (0 left) 2024-09-20T19:26:47.296 INFO:tasks.rados.rados.0.smithi042.stdout:1393: snap_remove snap 132 2024-09-20T19:26:47.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:47.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:47 smithi162 bash[24557]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:47 smithi162 bash[24557]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:47 smithi162 bash[24557]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:47 smithi162 bash[24557]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:47.518 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:47.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[22706]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[22706]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[22706]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[22706]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[32075]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[32075]: cluster 2024-09-20T19:26:46.079790+0000 mgr.y (mgr.24491) 337 : cluster [DBG] pgmap v563: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 446 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 187/1233 objects degraded (15.166%) 2024-09-20T19:26:47.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[32075]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:47 smithi042 bash[32075]: cluster 2024-09-20T19:26:46.280919+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e417: 8 total, 7 up, 8 in 2024-09-20T19:26:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:47.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:47.678 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:47.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:47.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:47.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:47.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:47.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:48.041 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:48.061 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:48.141 INFO:tasks.rados.rados.0.smithi042.stdout:1393: done (0 left) 2024-09-20T19:26:48.141 INFO:tasks.rados.rados.0.smithi042.stdout:1394: read oid 20 snap -1 2024-09-20T19:26:48.141 INFO:tasks.rados.rados.0.smithi042.stdout:1394: expect deleted 2024-09-20T19:26:48.141 INFO:tasks.rados.rados.0.smithi042.stdout:1395: snap_remove snap 151 2024-09-20T19:26:48.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:48.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:48.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[32075]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[32075]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[22706]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[22706]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.301 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:48 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:48.327 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:26:48.346 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:26:48.347 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:26:48.364 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:26:48.365 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:26:48.391 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:26:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:26:48.412 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:26:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:26:48.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:48 smithi162 bash[24557]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:48 smithi162 bash[24557]: cluster 2024-09-20T19:26:47.295945+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e418: 8 total, 7 up, 8 in 2024-09-20T19:26:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:48 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:48 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.140989+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e419: 8 total, 7 up, 8 in 2024-09-20T19:26:48.433 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:26:48.433 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:26:48.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:48.452 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:26:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:26:48.460 INFO:teuthology.orchestra.run.smithi042.stderr:Exporting 7.d info 7.d( v 393'279 (0'0,393'279] local-lis/les=112/113 n=15 ec=112/112 lis/c=112/112 les/c/f=113/113/0 sis=112) 2024-09-20T19:26:48.461 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:42# 2024-09-20T19:26:48.464 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:83# 2024-09-20T19:26:48.468 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:87# 2024-09-20T19:26:48.471 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:26:48.471 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:26:48.471 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:89# 2024-09-20T19:26:48.474 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:96# 2024-09-20T19:26:48.478 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b3515764:::smithi04268048-3:head# 2024-09-20T19:26:48.483 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b90bdf9e:::smithi04268048-23:43# 2024-09-20T19:26:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b90bdf9e:::smithi04268048-23:7c# 2024-09-20T19:26:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b90bdf9e:::smithi04268048-23:91# 2024-09-20T19:26:48.491 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:b90bdf9e:::smithi04268048-23:head# 2024-09-20T19:26:48.496 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:baa4e37b:::smithi04268048-17:53# 2024-09-20T19:26:48.498 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:baa4e37b:::smithi04268048-17:7c# 2024-09-20T19:26:48.503 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:baa4e37b:::smithi04268048-17:91# 2024-09-20T19:26:48.506 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:baa4e37b:::smithi04268048-17:96# 2024-09-20T19:26:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:Read #7:baa4e37b:::smithi04268048-17:head# 2024-09-20T19:26:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:Export successful 2024-09-20T19:26:48.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:48.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:48.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:48.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:48.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:48.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:48.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:48.959 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:48.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:49.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:49.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:49.144 INFO:tasks.rados.rados.0.smithi042.stdout:1395: done (1 left) 2024-09-20T19:26:49.144 INFO:tasks.rados.rados.0.smithi042.stdout:1396: write (excl) oid 37 current snap is 160 2024-09-20T19:26:49.144 INFO:tasks.rados.rados.0.smithi042.stdout:1396: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-20T19:26:49.164 INFO:tasks.rados.rados.0.smithi042.stdout:1396: writing smithi04268048-37 from 702658 to 1322256 tid 1 2024-09-20T19:26:49.179 INFO:tasks.rados.rados.0.smithi042.stdout:1396: writing smithi04268048-37 from 1857987 to 2302658 tid 2 2024-09-20T19:26:49.179 INFO:tasks.rados.rados.0.smithi042.stdout:1397: write oid 8 current snap is 160 2024-09-20T19:26:49.179 INFO:tasks.rados.rados.0.smithi042.stdout:1397: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-20T19:26:49.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:49.195 INFO:tasks.rados.rados.0.smithi042.stdout:1397: writing smithi04268048-8 from 719465 to 1130665 tid 1 2024-09-20T19:26:49.209 INFO:tasks.rados.rados.0.smithi042.stdout:1397: writing smithi04268048-8 from 1632822 to 2319465 tid 2 2024-09-20T19:26:49.209 INFO:tasks.rados.rados.0.smithi042.stdout:1398: read oid 24 snap -1 2024-09-20T19:26:49.209 INFO:tasks.rados.rados.0.smithi042.stdout:1398: expect deleted 2024-09-20T19:26:49.209 INFO:tasks.rados.rados.0.smithi042.stdout:1399: read oid 10 snap -1 2024-09-20T19:26:49.210 INFO:tasks.rados.rados.0.smithi042.stdout:1399: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:49.210 INFO:tasks.rados.rados.0.smithi042.stdout:1400: snap_remove snap 133 2024-09-20T19:26:49.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:49.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:49.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:49.357 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:26:49.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.477 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:49.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:49 smithi162 bash[24557]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:49.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[22706]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.080469+0000 mgr.y (mgr.24491) 338 : cluster [DBG] pgmap v566: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:48.287498+0000 mon.a (mon.0) 1483 : cluster [WRN] Health check update: Degraded data redundancy: 195/1269 objects degraded (15.366%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: audit 2024-09-20T19:26:48.327185+0000 mon.a (mon.0) 1484 : audit [DBG] from='client.? v1:172.21.15.42:0/1785932720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:49 smithi042 bash[32075]: cluster 2024-09-20T19:26:49.144279+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e420: 8 total, 7 up, 8 in 2024-09-20T19:26:49.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:49.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:49.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:49.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:49.998 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:50.101 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:50.126 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:50.143 INFO:tasks.rados.rados.0.smithi042.stdout:1400: done (5 left) 2024-09-20T19:26:50.144 INFO:tasks.rados.rados.0.smithi042.stdout:1401: read oid 26 snap 134 2024-09-20T19:26:50.144 INFO:tasks.rados.rados.0.smithi042.stdout:1401: expect (ObjNum 412 snap 129 seq_num 412) 2024-09-20T19:26:50.144 INFO:tasks.rados.rados.0.smithi042.stdout:1402: copy_from oid 50 from oid 39 current snap is 160 2024-09-20T19:26:50.144 INFO:tasks.rados.rados.0.smithi042.stdout:1399: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:26:50.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:50.279 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1598169 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1397: finishing write tid 1 to smithi04268048-8 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1397: finishing write tid 2 to smithi04268048-8 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1397: finishing write tid 3 to smithi04268048-8 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1394: done (6 left) 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1398: done (5 left) 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1399: done (4 left) 2024-09-20T19:26:50.305 INFO:tasks.rados.rados.0.smithi042.stdout:1403: snap_remove snap 130 2024-09-20T19:26:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:26:50.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:26:50.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:50.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:50.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:50.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.625 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:50 smithi162 bash[24557]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.626 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:50 smithi162 bash[24557]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.626 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:50 smithi162 bash[24557]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.626 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:50 smithi162 bash[24557]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:50.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[32075]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[32075]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[32075]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[32075]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[22706]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[22706]: audit 2024-09-20T19:26:49.280120+0000 mgr.y (mgr.24491) 339 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[22706]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:50 smithi042 bash[22706]: cluster 2024-09-20T19:26:50.141745+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e421: 8 total, 7 up, 8 in 2024-09-20T19:26:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:50.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 7.d 2024-09-20T19:26:50.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:50.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:51.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:51.030 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:51.062 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:26:51.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:51.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:51.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:51.288 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:51.305 INFO:tasks.rados.rados.0.smithi042.stdout:1403: done (4 left) 2024-09-20T19:26:51.305 INFO:tasks.rados.rados.0.smithi042.stdout:1404: read oid 9 snap -1 2024-09-20T19:26:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1404: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:26:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1405: read oid 46 snap -1 2024-09-20T19:26:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1405: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1406: copy_from oid 2 from oid 38 current snap is 160 2024-09-20T19:26:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1407: snap_remove snap 146 2024-09-20T19:26:51.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:51.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[32075]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[32075]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[32075]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.404 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[32075]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[22706]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.405 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[22706]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.405 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[22706]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.405 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:51 smithi042 bash[22706]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.417 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:51.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:51.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:51 smithi162 bash[24557]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:51 smithi162 bash[24557]: cluster 2024-09-20T19:26:50.081419+0000 mgr.y (mgr.24491) 340 : cluster [DBG] pgmap v569: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 464 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.1 MiB/s wr, 24 op/s; 195/1269 objects degraded (15.366%) 2024-09-20T19:26:51.534 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:51 smithi162 bash[24557]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.534 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:51 smithi162 bash[24557]: cluster 2024-09-20T19:26:51.305948+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e422: 8 total, 7 up, 8 in 2024-09-20T19:26:51.549 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:51.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:51.680 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:51.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:51.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:51.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:51.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:51.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:52.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:52.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:52.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1407: done (7 left) 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1408: rollback oid 21 current snap is 160 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 153 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1396: finishing write tid 1 to smithi04268048-37 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1397: finishing write tid 4 to smithi04268048-8 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1397: oid 8 updating version 0 to 513 2024-09-20T19:26:52.319 INFO:tasks.rados.rados.0.smithi042.stdout:1397: oid 8 version 513 is already newer than 512 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1397: oid 8 version 513 is already newer than 511 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 513 (ObjNum 495 snap 160 seq_num 495) dirty exists 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1397: left oid 8 (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: finishing write tid 2 to smithi04268048-37 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: finishing write tid 3 to smithi04268048-37 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: finishing write tid 4 to smithi04268048-37 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: oid 37 updating version 0 to 409 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: oid 37 version 409 is already newer than 407 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: oid 37 version 409 is already newer than 408 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 409 (ObjNum 494 snap 160 seq_num 494) dirty exists 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1396: left oid 37 (ObjNum 494 snap 160 seq_num 494) 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:1401: expect (ObjNum 412 snap 129 seq_num 412) 2024-09-20T19:26:52.320 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:52.368 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1864249 2024-09-20T19:26:52.413 INFO:tasks.rados.rados.0.smithi042.stdout:1402: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:26:52.414 INFO:tasks.rados.rados.0.smithi042.stdout:1402: finishing copy_from to smithi04268048-50 2024-09-20T19:26:52.414 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 433 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:26:52.414 INFO:tasks.rados.rados.0.smithi042.stdout:1405: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:26:52.417 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 640607 2024-09-20T19:26:52.436 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:52.518 INFO:tasks.rados.rados.0.smithi042.stdout:1406: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:26:52.518 INFO:tasks.rados.rados.0.smithi042.stdout:1404: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:26:52.518 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:52.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:52.590 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:52.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:52.624 INFO:tasks.rados.rados.0.smithi042.stdout:1406: finishing copy_from to smithi04268048-2 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 571 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1408: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1408: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1408: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 722 (ObjNum 442 snap 145 seq_num 442) dirty exists 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1396: done (7 left) 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1397: done (6 left) 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1401: done (5 left) 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1402: done (4 left) 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1404: done (3 left) 2024-09-20T19:26:52.625 INFO:tasks.rados.rados.0.smithi042.stdout:1405: done (2 left) 2024-09-20T19:26:52.626 INFO:tasks.rados.rados.0.smithi042.stdout:1406: done (1 left) 2024-09-20T19:26:52.626 INFO:tasks.rados.rados.0.smithi042.stdout:1408: done (0 left) 2024-09-20T19:26:52.626 INFO:tasks.rados.rados.0.smithi042.stdout:1409: snap_remove snap 156 2024-09-20T19:26:52.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.714 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:52.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:52.847 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:52.950 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:52.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:53.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:53.110 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:53.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:53.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:53.337 INFO:tasks.rados.rados.0.smithi042.stdout:1409: done (0 left) 2024-09-20T19:26:53.338 INFO:tasks.rados.rados.0.smithi042.stdout:1410: copy_from oid 12 from oid 41 current snap is 160 2024-09-20T19:26:53.338 INFO:tasks.rados.rados.0.smithi042.stdout:1411: copy_from oid 2 from oid 18 current snap is 160 2024-09-20T19:26:53.338 INFO:tasks.rados.rados.0.smithi042.stdout:1412: delete oid 25 current snap is 160 2024-09-20T19:26:53.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.344 INFO:tasks.rados.rados.0.smithi042.stdout:1411: finishing copy_from to smithi04268048-2 2024-09-20T19:26:53.344 INFO:tasks.rados.rados.0.smithi042.stdout:1411: got expected ENOENT (src dne) 2024-09-20T19:26:53.344 INFO:tasks.rados.rados.0.smithi042.stdout:1411: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:26:53.346 INFO:tasks.rados.rados.0.smithi042.stdout:1411: done (2 left) 2024-09-20T19:26:53.346 INFO:tasks.rados.rados.0.smithi042.stdout:1412: done (1 left) 2024-09-20T19:26:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:1413: write oid 37 current snap is 160 2024-09-20T19:26:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:1413: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-20T19:26:53.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.368 INFO:tasks.rados.rados.0.smithi042.stdout:1413: writing smithi04268048-37 from 753079 to 1314777 tid 1 2024-09-20T19:26:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1413: writing smithi04268048-37 from 2033433 to 2353079 tid 2 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1414: snap_create 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1410: finishing copy_from to smithi04268048-12 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1410: got expected ENOENT (src dne) 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1410: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:26:53.384 INFO:tasks.rados.rados.0.smithi042.stdout:1413: finishing write tid 1 to smithi04268048-37 2024-09-20T19:26:53.388 INFO:tasks.rados.rados.0.smithi042.stdout:1413: finishing write tid 2 to smithi04268048-37 2024-09-20T19:26:53.389 INFO:tasks.rados.rados.0.smithi042.stdout:1413: finishing write tid 3 to smithi04268048-37 2024-09-20T19:26:53.389 INFO:tasks.rados.rados.0.smithi042.stdout:1413: finishing write tid 4 to smithi04268048-37 2024-09-20T19:26:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:1413: oid 37 updating version 0 to 410 2024-09-20T19:26:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:1413: oid 37 updating version 410 to 412 2024-09-20T19:26:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:1413: oid 37 version 412 is already newer than 411 2024-09-20T19:26:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 412 (ObjNum 497 snap 160 seq_num 497) dirty exists 2024-09-20T19:26:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:1413: left oid 37 (ObjNum 497 snap 160 seq_num 497) 2024-09-20T19:26:53.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[22706]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[22706]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[22706]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[22706]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[32075]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[32075]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[32075]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:53 smithi042 bash[32075]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:53 smithi162 bash[24557]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:53 smithi162 bash[24557]: cluster 2024-09-20T19:26:52.083337+0000 mgr.y (mgr.24491) 341 : cluster [DBG] pgmap v572: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:53.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:53 smithi162 bash[24557]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:53 smithi162 bash[24557]: cluster 2024-09-20T19:26:52.312610+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e423: 8 total, 7 up, 8 in 2024-09-20T19:26:53.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:26:53.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:53.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:53.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:53.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:53.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:53.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:53.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:54.010 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:54.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:54.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:54.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:54.346 INFO:tasks.rados.rados.0.smithi042.stdout:1410: done (2 left) 2024-09-20T19:26:54.346 INFO:tasks.rados.rados.0.smithi042.stdout:1413: done (1 left) 2024-09-20T19:26:54.346 INFO:tasks.rados.rados.0.smithi042.stdout:1414: done (0 left) 2024-09-20T19:26:54.346 INFO:tasks.rados.rados.0.smithi042.stdout:1415: snap_create 2024-09-20T19:26:54.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:54.400 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:54.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:54.514 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:54 smithi042 bash[32075]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.514 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:54 smithi042 bash[32075]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:54 smithi042 bash[22706]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:54 smithi042 bash[22706]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:54.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:54 smithi162 bash[24557]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:54 smithi162 bash[24557]: cluster 2024-09-20T19:26:53.333777+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e424: 8 total, 7 up, 8 in 2024-09-20T19:26:54.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:54.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:54.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:54.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:54.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:54.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:55.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:55.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:55.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:55.176 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:55.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:55.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:55.349 INFO:tasks.rados.rados.0.smithi042.stdout:1415: done (0 left) 2024-09-20T19:26:55.349 INFO:tasks.rados.rados.0.smithi042.stdout:1416: snap_create 2024-09-20T19:26:55.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:55.438 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:55.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:55.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:55 smithi162 bash[24557]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:55 smithi162 bash[24557]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:55 smithi162 bash[24557]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:55 smithi162 bash[24557]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.568 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:55.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:55.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[32075]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[32075]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[32075]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[32075]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[22706]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[22706]: cluster 2024-09-20T19:26:54.084016+0000 mgr.y (mgr.24491) 342 : cluster [DBG] pgmap v575: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[22706]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:55 smithi042 bash[22706]: cluster 2024-09-20T19:26:54.341474+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e425: 8 total, 7 up, 8 in 2024-09-20T19:26:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:55.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:55.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:55.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:55.956 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:55.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:56.059 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:56.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:56.189 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:26:56.189 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:26:56.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:56.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:56.322 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:26:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:26:56.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:56.342 INFO:tasks.rados.rados.0.smithi042.stdout:1416: done (0 left) 2024-09-20T19:26:56.342 INFO:tasks.rados.rados.0.smithi042.stdout:1417: write (excl) oid 4 current snap is 163 2024-09-20T19:26:56.342 INFO:tasks.rados.rados.0.smithi042.stdout:1417: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-20T19:26:56.350 INFO:tasks.rados.rados.0.smithi042.stdout:1417: writing smithi04268048-4 from 769886 to 1406833 tid 1 2024-09-20T19:26:56.356 INFO:tasks.rados.rados.0.smithi042.stdout:1417: writing smithi04268048-4 from 2091915 to 2369886 tid 2 2024-09-20T19:26:56.356 INFO:tasks.rados.rados.0.smithi042.stdout:1418: delete oid 25 current snap is 163 2024-09-20T19:26:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1418: done (1 left) 2024-09-20T19:26:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1419: read oid 3 snap -1 2024-09-20T19:26:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1419: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:26:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1420: snap_remove snap 139 2024-09-20T19:26:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:26:56.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:56 smithi162 bash[24557]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:56.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[32075]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: cluster 2024-09-20T19:26:55.345687+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e426: 8 total, 7 up, 8 in 2024-09-20T19:26:56.605 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: audit 2024-09-20T19:26:56.158738+0000 mon.a (mon.0) 1492 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:26:56.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:56 smithi042 bash[22706]: cluster 2024-09-20T19:26:56.343367+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e427: 8 total, 7 up, 8 in 2024-09-20T19:26:56.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:56.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:56.810 INFO:teuthology.orchestra.run.smithi042.stdout: marking collection for removal 2024-09-20T19:26:56.810 INFO:teuthology.orchestra.run.smithi042.stdout:setting '_remove' omap key 2024-09-20T19:26:56.811 INFO:teuthology.orchestra.run.smithi042.stdout:finish_remove_pgs 7.d_head removing 7.d 2024-09-20T19:26:56.813 INFO:teuthology.orchestra.run.smithi042.stdout:Remove successful 2024-09-20T19:26:56.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:56.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:56.963 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:56.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:57.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:57.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:57.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:57.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:57.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:57.346 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:57.364 INFO:tasks.rados.rados.0.smithi042.stdout:1420: done (2 left) 2024-09-20T19:26:57.364 INFO:tasks.rados.rados.0.smithi042.stdout:1421: write (excl) oid 28 current snap is 163 2024-09-20T19:26:57.364 INFO:tasks.rados.rados.0.smithi042.stdout:1417: finishing write tid 1 to smithi04268048-4 2024-09-20T19:26:57.364 INFO:tasks.rados.rados.0.smithi042.stdout:1417: finishing write tid 2 to smithi04268048-4 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: finishing write tid 3 to smithi04268048-4 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: finishing write tid 4 to smithi04268048-4 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: oid 4 updating version 0 to 686 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: oid 4 updating version 686 to 687 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: oid 4 updating version 687 to 688 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 688 (ObjNum 498 snap 163 seq_num 498) dirty exists 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1417: left oid 4 (ObjNum 498 snap 163 seq_num 498) 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:1419: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:26:57.365 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:57.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:57.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:57.517 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:26:57.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:57.589 INFO:tasks.rados.rados.0.smithi042.stdout:1421: seq_num 500 ranges {403500=787445,1692526=473837} 2024-09-20T19:26:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:57.609 INFO:tasks.rados.rados.0.smithi042.stdout:1421: writing smithi04268048-28 from 403500 to 1190945 tid 1 2024-09-20T19:26:57.625 INFO:tasks.rados.rados.0.smithi042.stdout:1421: writing smithi04268048-28 from 1692526 to 2166363 tid 2 2024-09-20T19:26:57.625 INFO:tasks.rados.rados.0.smithi042.stdout:1417: done (2 left) 2024-09-20T19:26:57.625 INFO:tasks.rados.rados.0.smithi042.stdout:1419: done (1 left) 2024-09-20T19:26:57.625 INFO:tasks.rados.rados.0.smithi042.stdout:1422: delete oid 27 current snap is 163 2024-09-20T19:26:57.625 INFO:tasks.rados.rados.0.smithi042.stdout:1421: finishing write tid 1 to smithi04268048-28 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1422: done (1 left) 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1423: rollback oid 43 current snap is 163 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 150 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1424: read oid 27 snap -1 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1424: expect deleted 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1425: read oid 9 snap -1 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1425: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:1426: rollback oid 24 current snap is 163 2024-09-20T19:26:57.629 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 153 2024-09-20T19:26:57.630 INFO:tasks.rados.rados.0.smithi042.stdout:1427: write oid 45 current snap is 163 2024-09-20T19:26:57.630 INFO:tasks.rados.rados.0.smithi042.stdout:1427: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-20T19:26:57.647 INFO:tasks.rados.rados.0.smithi042.stdout:1427: writing smithi04268048-45 from 437114 to 1091410 tid 1 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1427: writing smithi04268048-45 from 1525843 to 2003349 tid 2 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1428: snap_create 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1426: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: finishing write tid 2 to smithi04268048-28 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: finishing write tid 3 to smithi04268048-28 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: finishing write tid 4 to smithi04268048-28 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: oid 28 updating version 0 to 398 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: oid 28 version 398 is already newer than 397 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: oid 28 updating version 398 to 399 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 399 (ObjNum 500 snap 163 seq_num 500) dirty exists 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1421: left oid 28 (ObjNum 500 snap 163 seq_num 500) 2024-09-20T19:26:57.662 INFO:tasks.rados.rados.0.smithi042.stdout:1423: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:26:57.663 INFO:tasks.rados.rados.0.smithi042.stdout:1423: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:26:57.663 INFO:tasks.rados.rados.0.smithi042.stdout:1425: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[32075]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:57 smithi042 bash[22706]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:57.729 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1886233 2024-09-20T19:26:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:57.749 INFO:tasks.rados.rados.0.smithi042.stdout:1423: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:26:57.749 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 439 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: finishing write tid 1 to smithi04268048-45 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: finishing write tid 2 to smithi04268048-45 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: finishing write tid 3 to smithi04268048-45 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: finishing write tid 4 to smithi04268048-45 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: oid 45 updating version 0 to 836 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: oid 45 updating version 836 to 837 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: oid 45 version 837 is already newer than 835 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 837 (ObjNum 502 snap 163 seq_num 502) dirty exists 2024-09-20T19:26:57.750 INFO:tasks.rados.rados.0.smithi042.stdout:1427: left oid 45 (ObjNum 502 snap 163 seq_num 502) 2024-09-20T19:26:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: cluster 2024-09-20T19:26:56.084549+0000 mgr.y (mgr.24491) 343 : cluster [DBG] pgmap v578: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 469 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 197/1275 objects degraded (15.451%) 2024-09-20T19:26:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.864545+0000 mon.a (mon.0) 1494 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:26:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.865619+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:26:57.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:56.873740+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:26:57.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:57 smithi162 bash[24557]: audit 2024-09-20T19:26:57.301223+0000 mon.a (mon.0) 1497 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:26:57.835 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:57.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:57.988 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:26:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:57.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:26:58.091 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:58.117 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:26:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:26:58.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1421: done (6 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1423: done (5 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1424: done (4 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1425: done (3 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1426: done (2 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1427: done (1 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1428: done (0 left) 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1429: read oid 17 snap 163 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1429: expect deleted 2024-09-20T19:26:58.141 INFO:tasks.rados.rados.0.smithi042.stdout:1430: delete oid 44 current snap is 164 2024-09-20T19:26:58.155 INFO:tasks.rados.rados.0.smithi042.stdout:1429: done (1 left) 2024-09-20T19:26:58.155 INFO:tasks.rados.rados.0.smithi042.stdout:1430: done (0 left) 2024-09-20T19:26:58.155 INFO:tasks.rados.rados.0.smithi042.stdout:1431: delete oid 1 current snap is 164 2024-09-20T19:26:58.158 INFO:tasks.rados.rados.0.smithi042.stdout:1431: done (0 left) 2024-09-20T19:26:58.159 INFO:tasks.rados.rados.0.smithi042.stdout:1432: rollback oid 8 current snap is 164 2024-09-20T19:26:58.159 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 161 2024-09-20T19:26:58.159 INFO:tasks.rados.rados.0.smithi042.stdout:1433: snap_remove snap 125 2024-09-20T19:26:58.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:58.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:58.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:58.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:58.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:58.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:58 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:58.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:58.623 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:57.364743+0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e428: 8 total, 7 up, 8 in 2024-09-20T19:26:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.134063+0000 mon.a (mon.0) 1499 : cluster [WRN] Health check update: Degraded data redundancy: 197/1275 objects degraded (15.451%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:26:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:58 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.140257+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e429: 8 total, 7 up, 8 in 2024-09-20T19:26:58.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:58.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:58.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:58.883 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:58.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:26:58.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:59.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:59.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:59.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1433: done (1 left) 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1434: write oid 32 current snap is 164 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1432: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1432: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1432: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 517 (ObjNum 495 snap 160 seq_num 495) dirty exists 2024-09-20T19:26:59.158 INFO:tasks.rados.rados.0.smithi042.stdout:1434: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-20T19:26:59.174 INFO:tasks.rados.rados.0.smithi042.stdout:1434: writing smithi04268048-32 from 487535 to 967578 tid 1 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1434: writing smithi04268048-32 from 1584936 to 2209769 tid 2 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1432: done (1 left) 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1435: rollback oid 13 current snap is 164 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 136 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1434: finishing write tid 1 to smithi04268048-32 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1436: write (excl) oid 9 current snap is 164 2024-09-20T19:26:59.195 INFO:tasks.rados.rados.0.smithi042.stdout:1436: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-20T19:26:59.208 INFO:tasks.rados.rados.0.smithi042.stdout:1436: writing smithi04268048-9 from 504342 to 1059634 tid 1 2024-09-20T19:26:59.221 INFO:tasks.rados.rados.0.smithi042.stdout:1436: writing smithi04268048-9 from 1643418 to 2128262 tid 2 2024-09-20T19:26:59.221 INFO:tasks.rados.rados.0.smithi042.stdout:1437: rollback oid 11 current snap is 164 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 159 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1435: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1438: read oid 44 snap -1 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1438: expect deleted 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1439: copy_from oid 33 from oid 25 current snap is 164 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1435: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 561 (ObjNum 129 snap 0 seq_num 1415901224) dirty dne 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1435: done (5 left) 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1440: read oid 43 snap 137 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1440: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1441: read oid 16 snap -1 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1441: expect (ObjNum 481 snap 158 seq_num 481) 2024-09-20T19:26:59.222 INFO:tasks.rados.rados.0.smithi042.stdout:1442: read oid 24 snap 134 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1442: expect deleted 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1443: rollback oid 19 current snap is 164 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 134 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: finishing write tid 2 to smithi04268048-32 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: finishing write tid 3 to smithi04268048-32 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: finishing write tid 4 to smithi04268048-32 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: oid 32 updating version 0 to 851 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: oid 32 updating version 851 to 852 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: oid 32 updating version 852 to 853 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 853 (ObjNum 505 snap 164 seq_num 505) dirty exists 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: left oid 32 (ObjNum 505 snap 164 seq_num 505) 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1436: finishing write tid 1 to smithi04268048-9 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1434: done (8 left) 2024-09-20T19:26:59.223 INFO:tasks.rados.rados.0.smithi042.stdout:1444: read oid 12 snap -1 2024-09-20T19:26:59.224 INFO:tasks.rados.rados.0.smithi042.stdout:1444: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:26:59.224 INFO:tasks.rados.rados.0.smithi042.stdout:1445: snap_remove snap 153 2024-09-20T19:26:59.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:59.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:59.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:59.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:26:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:26:59.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:59.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:26:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:26:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:59 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:59 smithi162 bash[24557]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:59 smithi162 bash[24557]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:26:59 smithi162 bash[24557]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:26:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:26:59.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:59.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:26:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:26:59.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[32075]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[32075]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[32075]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[22706]: cluster 2024-09-20T19:26:58.085277+0000 mgr.y (mgr.24491) 344 : cluster [DBG] pgmap v581: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[22706]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:26:59 smithi042 bash[22706]: cluster 2024-09-20T19:26:59.155376+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e430: 8 total, 7 up, 8 in 2024-09-20T19:26:59.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:59.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:26:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:26:59.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:59.900 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.7.d.0 2024-09-20T19:26:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:26:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:26:59.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:00.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:00.046 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:00.092 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:27:00.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1445: done (9 left) 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1446: snap_create 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1439: finishing copy_from to smithi04268048-33 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1439: got expected ENOENT (src dne) 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1439: finishing copy_from racing read to smithi04268048-33 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1436: finishing write tid 2 to smithi04268048-9 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:1440: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:27:00.160 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:00.214 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 952062 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1436: finishing write tid 3 to smithi04268048-9 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1443: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1436: finishing write tid 4 to smithi04268048-9 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1436: oid 9 updating version 0 to 691 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1436: oid 9 updating version 691 to 692 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:1436: oid 9 updating version 692 to 693 2024-09-20T19:27:00.267 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 693 (ObjNum 506 snap 164 seq_num 506) dirty exists 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1436: left oid 9 (ObjNum 506 snap 164 seq_num 506) 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1437: finishing rollback tid 0 to smithi04268048-11 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1443: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 401 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1437: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1437: finishing rollback tid 2 to smithi04268048-11 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 697 (ObjNum 430 snap 134 seq_num 430) dirty exists 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:1441: expect (ObjNum 481 snap 158 seq_num 481) 2024-09-20T19:27:00.268 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:00.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:00.325 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:00.354 INFO:tasks.rados.rados.0.smithi042.stdout:1444: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:27:00.356 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1486762 2024-09-20T19:27:00.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:00.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[22706]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[22706]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.476 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[32075]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[32075]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.477 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:00.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:00 smithi162 bash[24557]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:00 smithi162 bash[24557]: audit 2024-09-20T19:26:59.288043+0000 mgr.y (mgr.24491) 345 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:00.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:00.160251+0000 mon.a (mon.0) 1502 : cluster [DBG] osdmap e431: 8 total, 7 up, 8 in 2024-09-20T19:27:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:00.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:00.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:00.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:00.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:00.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:00.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:01.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1436: done (9 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1437: done (8 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1438: done (7 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1439: done (6 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1440: done (5 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1441: done (4 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1442: done (3 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1443: done (2 left) 2024-09-20T19:27:01.159 INFO:tasks.rados.rados.0.smithi042.stdout:1444: done (1 left) 2024-09-20T19:27:01.160 INFO:tasks.rados.rados.0.smithi042.stdout:1446: done (0 left) 2024-09-20T19:27:01.160 INFO:tasks.rados.rados.0.smithi042.stdout:1447: write oid 7 current snap is 165 2024-09-20T19:27:01.160 INFO:tasks.rados.rados.0.smithi042.stdout:1447: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-20T19:27:01.174 INFO:tasks.rados.rados.0.smithi042.stdout:1447: writing smithi04268048-7 from 521149 to 1151690 tid 1 2024-09-20T19:27:01.177 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:01.190 INFO:tasks.rados.rados.0.smithi042.stdout:1447: writing smithi04268048-7 from 1701900 to 2330402 tid 2 2024-09-20T19:27:01.190 INFO:tasks.rados.rados.0.smithi042.stdout:1448: copy_from oid 9 from oid 43 current snap is 165 2024-09-20T19:27:01.190 INFO:tasks.rados.rados.0.smithi042.stdout:1449: write oid 10 current snap is 165 2024-09-20T19:27:01.191 INFO:tasks.rados.rados.0.smithi042.stdout:1449: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-20T19:27:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:01.205 INFO:tasks.rados.rados.0.smithi042.stdout:1449: writing smithi04268048-10 from 537956 to 1243746 tid 1 2024-09-20T19:27:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1449: writing smithi04268048-10 from 1644029 to 2416189 tid 2 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1450: delete oid 49 current snap is 165 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: finishing write tid 1 to smithi04268048-7 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: finishing write tid 2 to smithi04268048-7 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: finishing write tid 3 to smithi04268048-7 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: finishing write tid 4 to smithi04268048-7 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: oid 7 updating version 0 to 619 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: oid 7 version 619 is already newer than 618 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: oid 7 updating version 619 to 620 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 620 (ObjNum 507 snap 165 seq_num 507) dirty exists 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1447: left oid 7 (ObjNum 507 snap 165 seq_num 507) 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1448: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:1448: finishing copy_from to smithi04268048-9 2024-09-20T19:27:01.213 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 439 (ObjNum 240 snap 70 seq_num 240) dirty exists 2024-09-20T19:27:01.219 INFO:tasks.rados.rados.0.smithi042.stdout:1447: done (3 left) 2024-09-20T19:27:01.219 INFO:tasks.rados.rados.0.smithi042.stdout:1448: done (2 left) 2024-09-20T19:27:01.219 INFO:tasks.rados.rados.0.smithi042.stdout:1450: done (1 left) 2024-09-20T19:27:01.219 INFO:tasks.rados.rados.0.smithi042.stdout:1451: delete oid 1 current snap is 165 2024-09-20T19:27:01.221 INFO:tasks.rados.rados.0.smithi042.stdout:1449: finishing write tid 1 to smithi04268048-10 2024-09-20T19:27:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:1449: finishing write tid 2 to smithi04268048-10 2024-09-20T19:27:01.225 INFO:tasks.rados.rados.0.smithi042.stdout:1449: finishing write tid 3 to smithi04268048-10 2024-09-20T19:27:01.226 INFO:tasks.rados.rados.0.smithi042.stdout:1449: finishing write tid 4 to smithi04268048-10 2024-09-20T19:27:01.227 INFO:tasks.rados.rados.0.smithi042.stdout:1449: oid 10 updating version 0 to 576 2024-09-20T19:27:01.227 INFO:tasks.rados.rados.0.smithi042.stdout:1449: oid 10 version 576 is already newer than 575 2024-09-20T19:27:01.227 INFO:tasks.rados.rados.0.smithi042.stdout:1449: oid 10 updating version 576 to 577 2024-09-20T19:27:01.227 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 577 (ObjNum 508 snap 165 seq_num 508) dirty exists 2024-09-20T19:27:01.227 INFO:tasks.rados.rados.0.smithi042.stdout:1449: left oid 10 (ObjNum 508 snap 165 seq_num 508) 2024-09-20T19:27:01.229 INFO:tasks.rados.rados.0.smithi042.stdout:1449: done (1 left) 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:1451: done (0 left) 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:1452: rollback oid 38 current snap is 165 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 164 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:1453: read oid 43 snap -1 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:1453: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:27:01.230 INFO:tasks.rados.rados.0.smithi042.stdout:1454: snap_create 2024-09-20T19:27:01.233 INFO:tasks.rados.rados.0.smithi042.stdout:1453: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:27:01.293 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:01.308 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:01.323 INFO:tasks.rados.rados.0.smithi042.stdout:1452: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:27:01.323 INFO:tasks.rados.rados.0.smithi042.stdout:1452: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:27:01.323 INFO:tasks.rados.rados.0.smithi042.stdout:1452: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:27:01.324 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 580 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:27:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:01.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:01.445 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[32075]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.445 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[32075]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[32075]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[32075]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[22706]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[22706]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[22706]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.446 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:01 smithi042 bash[22706]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:01.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:01.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:01.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:01 smithi162 bash[24557]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:01 smithi162 bash[24557]: cluster 2024-09-20T19:27:00.086277+0000 mgr.y (mgr.24491) 346 : cluster [DBG] pgmap v584: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 471 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 439 KiB/s wr, 7 op/s; 200/1281 objects degraded (15.613%) 2024-09-20T19:27:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:01 smithi162 bash[24557]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:01 smithi162 bash[24557]: cluster 2024-09-20T19:27:01.159318+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e432: 8 total, 7 up, 8 in 2024-09-20T19:27:01.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:01.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:01.885 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:27:01.903 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:27:01.903 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:27:01.924 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:27:01.925 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:27:01.942 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:27:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:27:01.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:01.977 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:27:01.977 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:27:01.985 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:01.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:02.011 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:27:02.012 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:27:02.037 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:27:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:27:02.055 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:27:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:27:02.088 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:02.109 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1452: done (2 left) 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1453: done (1 left) 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1454: done (0 left) 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1455: read oid 32 snap -1 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1455: expect (ObjNum 505 snap 164 seq_num 505) 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1456: copy_from oid 45 from oid 10 current snap is 166 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1457: read oid 8 snap -1 2024-09-20T19:27:02.159 INFO:tasks.rados.rados.0.smithi042.stdout:1457: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:27:02.160 INFO:tasks.rados.rados.0.smithi042.stdout:1458: write (excl) oid 33 current snap is 166 2024-09-20T19:27:02.160 INFO:tasks.rados.rados.0.smithi042.stdout:1458: seq_num 511 ranges {588377=647890,1935828=635840} 2024-09-20T19:27:02.166 INFO:tasks.rados.rados.0.smithi042.stdout:1458: writing smithi04268048-33 from 588377 to 1236267 tid 1 2024-09-20T19:27:02.174 INFO:tasks.rados.rados.0.smithi042.stdout:1458: writing smithi04268048-33 from 1935828 to 2571668 tid 2 2024-09-20T19:27:02.174 INFO:tasks.rados.rados.0.smithi042.stdout:1459: delete oid 9 current snap is 166 2024-09-20T19:27:02.174 INFO:tasks.rados.rados.0.smithi042.stdout:1455: expect (ObjNum 505 snap 164 seq_num 505) 2024-09-20T19:27:02.211 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2294261 2024-09-20T19:27:02.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.218 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:02.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:02.312 INFO:tasks.rados.rados.0.smithi042.stdout:1458: finishing write tid 1 to smithi04268048-33 2024-09-20T19:27:02.312 INFO:tasks.rados.rados.0.smithi042.stdout:1457: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:27:02.312 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:02.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.361 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2285546 2024-09-20T19:27:02.361 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 263413 2024-09-20T19:27:02.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1456: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1458: finishing write tid 2 to smithi04268048-33 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1458: finishing write tid 3 to smithi04268048-33 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1458: finishing write tid 4 to smithi04268048-33 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1458: oid 33 updating version 0 to 574 2024-09-20T19:27:02.391 INFO:tasks.rados.rados.0.smithi042.stdout:1458: oid 33 version 574 is already newer than 572 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1458: oid 33 version 574 is already newer than 573 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 574 (ObjNum 511 snap 166 seq_num 511) dirty exists 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1458: left oid 33 (ObjNum 511 snap 166 seq_num 511) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1456: finishing copy_from to smithi04268048-45 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 577 (ObjNum 508 snap 165 seq_num 508) dirty exists 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1455: done (4 left) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1456: done (3 left) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1457: done (2 left) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1458: done (1 left) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1459: done (0 left) 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1460: write oid 49 current snap is 166 2024-09-20T19:27:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:1460: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-20T19:27:02.415 INFO:tasks.rados.rados.0.smithi042.stdout:1460: writing smithi04268048-49 from 621991 to 1420379 tid 1 2024-09-20T19:27:02.435 INFO:tasks.rados.rados.0.smithi042.stdout:1460: writing smithi04268048-49 from 1936439 to 2575948 tid 2 2024-09-20T19:27:02.435 INFO:tasks.rados.rados.0.smithi042.stdout:1461: write (excl) oid 17 current snap is 166 2024-09-20T19:27:02.435 INFO:tasks.rados.rados.0.smithi042.stdout:1460: finishing write tid 1 to smithi04268048-49 2024-09-20T19:27:02.435 INFO:tasks.rados.rados.0.smithi042.stdout:1461: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-20T19:27:02.453 INFO:tasks.rados.rados.0.smithi042.stdout:1461: writing smithi04268048-17 from 638798 to 1112435 tid 1 2024-09-20T19:27:02.469 INFO:tasks.rados.rados.0.smithi042.stdout:1461: writing smithi04268048-17 from 1594921 to 2378088 tid 2 2024-09-20T19:27:02.470 INFO:tasks.rados.rados.0.smithi042.stdout:1462: read oid 2 snap -1 2024-09-20T19:27:02.470 INFO:tasks.rados.rados.0.smithi042.stdout:1462: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:27:02.470 INFO:tasks.rados.rados.0.smithi042.stdout:1463: snap_remove snap 127 2024-09-20T19:27:02.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:02 smithi162 bash[24557]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:02.601 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:02.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[22706]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: audit 2024-09-20T19:27:01.884490+0000 mon.a (mon.0) 1504 : audit [DBG] from='client.? v1:172.21.15.42:0/2052358098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: cluster 2024-09-20T19:27:02.157970+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e433: 8 total, 7 up, 8 in 2024-09-20T19:27:02.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:02 smithi042 bash[32075]: audit 2024-09-20T19:27:02.187376+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:02.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:02.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.758 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:02.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:02.889 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:02.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:02.923 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:02.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.021 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:03.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:03.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:03.155 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:27:03.167 INFO:tasks.rados.rados.0.smithi042.stdout:1463: done (3 left) 2024-09-20T19:27:03.167 INFO:tasks.rados.rados.0.smithi042.stdout:1464: snap_remove snap 160 2024-09-20T19:27:03.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:03.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:03.298 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:27:03.310 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:27:03.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:03.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.578 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[32075]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:03 smithi042 bash[22706]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:03.649 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.649 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:02.087203+0000 mgr.y (mgr.24491) 347 : cluster [DBG] pgmap v587: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.136062+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: Degraded data redundancy: 205/1314 objects degraded (15.601%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.154749+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: audit 2024-09-20T19:27:03.156586+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? v1:172.21.15.42:0/3387540091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:03 smithi162 bash[24557]: cluster 2024-09-20T19:27:03.165787+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e434: 8 total, 7 up, 8 in 2024-09-20T19:27:03.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:03.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:03.899 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:03.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:04.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:04.055 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:04.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1464: done (3 left) 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1465: read oid 46 snap -1 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1465: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1466: copy_from oid 31 from oid 19 current snap is 166 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1467: write (excl) oid 6 current snap is 166 2024-09-20T19:27:04.173 INFO:tasks.rados.rados.0.smithi042.stdout:1467: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-20T19:27:04.187 INFO:tasks.rados.rados.0.smithi042.stdout:1467: writing smithi04268048-6 from 655605 to 1204491 tid 1 2024-09-20T19:27:04.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:04.194 INFO:tasks.rados.rados.0.smithi042.stdout:1467: writing smithi04268048-6 from 1653403 to 2296581 tid 2 2024-09-20T19:27:04.194 INFO:tasks.rados.rados.0.smithi042.stdout:1468: snap_create 2024-09-20T19:27:04.194 INFO:tasks.rados.rados.0.smithi042.stdout:1460: finishing write tid 2 to smithi04268048-49 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: finishing write tid 3 to smithi04268048-49 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: finishing write tid 4 to smithi04268048-49 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: oid 49 updating version 0 to 865 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: oid 49 version 865 is already newer than 864 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: oid 49 version 865 is already newer than 863 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 865 (ObjNum 513 snap 166 seq_num 513) dirty exists 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1460: left oid 49 (ObjNum 513 snap 166 seq_num 513) 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1461: finishing write tid 1 to smithi04268048-17 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1461: finishing write tid 2 to smithi04268048-17 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:1462: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:27:04.195 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:04.242 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1222405 2024-09-20T19:27:04.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:04.296 INFO:tasks.rados.rados.0.smithi042.stdout:1461: finishing write tid 3 to smithi04268048-17 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1461: finishing write tid 4 to smithi04268048-17 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1461: oid 17 updating version 0 to 673 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1461: oid 17 updating version 673 to 674 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1461: oid 17 updating version 674 to 675 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 675 (ObjNum 514 snap 166 seq_num 514) dirty exists 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1461: left oid 17 (ObjNum 514 snap 166 seq_num 514) 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:1465: expect (ObjNum 226 snap 66 seq_num 226) 2024-09-20T19:27:04.297 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:04.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:04.343 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1313096 2024-09-20T19:27:04.381 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3051920 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1466: finishing copy_from to smithi04268048-31 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1466: got expected ENOENT (src dne) 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1466: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: finishing write tid 1 to smithi04268048-6 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: finishing write tid 2 to smithi04268048-6 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: finishing write tid 3 to smithi04268048-6 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: finishing write tid 4 to smithi04268048-6 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: oid 6 updating version 0 to 533 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: oid 6 version 533 is already newer than 531 2024-09-20T19:27:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:1467: oid 6 version 533 is already newer than 532 2024-09-20T19:27:04.395 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 533 (ObjNum 515 snap 166 seq_num 515) dirty exists 2024-09-20T19:27:04.395 INFO:tasks.rados.rados.0.smithi042.stdout:1467: left oid 6 (ObjNum 515 snap 166 seq_num 515) 2024-09-20T19:27:04.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:04.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:04.563 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:27:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:27:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:27:04.563 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:27:04.574 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:04.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:04.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:04.933 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:04.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:05.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1460: done (6 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1461: done (5 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1462: done (4 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1465: done (3 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1466: done (2 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1467: done (1 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1468: done (0 left) 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1469: write oid 18 current snap is 167 2024-09-20T19:27:05.173 INFO:tasks.rados.rados.0.smithi042.stdout:1469: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-20T19:27:05.188 INFO:tasks.rados.rados.0.smithi042.stdout:1469: writing smithi04268048-18 from 672412 to 1296547 tid 1 2024-09-20T19:27:05.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.204 INFO:tasks.rados.rados.0.smithi042.stdout:1469: writing smithi04268048-18 from 1995532 to 2672412 tid 2 2024-09-20T19:27:05.204 INFO:tasks.rados.rados.0.smithi042.stdout:1470: delete oid 10 current snap is 167 2024-09-20T19:27:05.204 INFO:tasks.rados.rados.0.smithi042.stdout:1469: finishing write tid 1 to smithi04268048-18 2024-09-20T19:27:05.213 INFO:tasks.rados.rados.0.smithi042.stdout:1469: finishing write tid 2 to smithi04268048-18 2024-09-20T19:27:05.213 INFO:tasks.rados.rados.0.smithi042.stdout:1469: finishing write tid 3 to smithi04268048-18 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1470: done (1 left) 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1471: delete oid 7 current snap is 167 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1469: finishing write tid 4 to smithi04268048-18 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1469: oid 18 updating version 0 to 727 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1469: oid 18 updating version 727 to 728 2024-09-20T19:27:05.215 INFO:tasks.rados.rados.0.smithi042.stdout:1469: oid 18 version 728 is already newer than 726 2024-09-20T19:27:05.216 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 728 (ObjNum 516 snap 167 seq_num 516) dirty exists 2024-09-20T19:27:05.216 INFO:tasks.rados.rados.0.smithi042.stdout:1469: left oid 18 (ObjNum 516 snap 167 seq_num 516) 2024-09-20T19:27:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1469: done (1 left) 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1471: done (0 left) 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1472: read oid 33 snap 165 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1472: expect deleted 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1473: read oid 27 snap -1 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1473: expect deleted 2024-09-20T19:27:05.225 INFO:tasks.rados.rados.0.smithi042.stdout:1474: read oid 23 snap -1 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1474: expect (ObjNum 436 snap 144 seq_num 436) 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1475: read oid 22 snap 157 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1475: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1476: copy_from oid 24 from oid 42 current snap is 167 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1477: write oid 16 current snap is 167 2024-09-20T19:27:05.226 INFO:tasks.rados.rados.0.smithi042.stdout:1477: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-20T19:27:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:05.232 INFO:tasks.rados.rados.0.smithi042.stdout:1477: writing smithi04268048-16 from 722833 to 1289068 tid 1 2024-09-20T19:27:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:1477: writing smithi04268048-16 from 1887331 to 2537847 tid 2 2024-09-20T19:27:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:1478: delete oid 9 current snap is 167 2024-09-20T19:27:05.239 INFO:tasks.rados.rados.0.smithi042.stdout:1475: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:27:05.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:27:05.328 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.329 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[22706]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.339 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:05 smithi042 bash[32075]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T19:27:05.355 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:05.425 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:05.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:05.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.087967+0000 mgr.y (mgr.24491) 348 : cluster [DBG] pgmap v590: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 23 op/s; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.165071+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e435: 8 total, 7 up, 8 in 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.481534+0000 osd.5 (osd.5) 153 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:05 smithi162 bash[24557]: cluster 2024-09-20T19:27:04.483785+0000 osd.5 (osd.5) 154 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:27:05.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:05.566 INFO:tasks.rados.rados.0.smithi042.stdout:1472: done (6 left) 2024-09-20T19:27:05.566 INFO:tasks.rados.rados.0.smithi042.stdout:1473: done (5 left) 2024-09-20T19:27:05.566 INFO:tasks.rados.rados.0.smithi042.stdout:1475: done (4 left) 2024-09-20T19:27:05.566 INFO:tasks.rados.rados.0.smithi042.stdout:1478: done (3 left) 2024-09-20T19:27:05.566 INFO:tasks.rados.rados.0.smithi042.stdout:1479: read oid 13 snap -1 2024-09-20T19:27:05.567 INFO:tasks.rados.rados.0.smithi042.stdout:1479: expect deleted 2024-09-20T19:27:05.567 INFO:tasks.rados.rados.0.smithi042.stdout:1480: copy_from oid 39 from oid 34 current snap is 167 2024-09-20T19:27:05.567 INFO:tasks.rados.rados.0.smithi042.stdout:1474: expect (ObjNum 436 snap 144 seq_num 436) 2024-09-20T19:27:05.567 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:05.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:05.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1476: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1477: finishing write tid 1 to smithi04268048-16 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1476: finishing copy_from to smithi04268048-24 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 541 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1477: finishing write tid 2 to smithi04268048-16 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1474: done (4 left) 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1476: done (3 left) 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1481: read oid 8 snap -1 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1481: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1482: snap_create 2024-09-20T19:27:05.692 INFO:tasks.rados.rados.0.smithi042.stdout:1477: finishing write tid 3 to smithi04268048-16 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1477: finishing write tid 4 to smithi04268048-16 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1477: oid 16 updating version 0 to 679 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1477: oid 16 version 679 is already newer than 677 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1477: oid 16 version 679 is already newer than 678 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 679 (ObjNum 519 snap 167 seq_num 519) dirty exists 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1477: left oid 16 (ObjNum 519 snap 167 seq_num 519) 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1480: finishing copy_from racing read to smithi04268048-39 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:1480: finishing copy_from to smithi04268048-39 2024-09-20T19:27:05.693 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 313 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:27:05.700 INFO:tasks.rados.rados.0.smithi042.stdout:1481: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:27:05.700 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:05.770 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1567967 2024-09-20T19:27:05.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:05.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:05.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:05.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:05.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:06.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:06.054 INFO:teuthology.orchestra.run.smithi042.stdout:get_pg_num_history pg_num_history pg_num_history(e393 pg_nums {1={20=1,259=11},2={69=32},3={70=3},4={71=32},5={73=32},6={75=32},7={112=16}} deleted_pools ) 2024-09-20T19:27:06.054 INFO:teuthology.orchestra.run.smithi042.stdout:Importing pgid 7.d 2024-09-20T19:27:06.055 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:42# 2024-09-20T19:27:06.059 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:83# 2024-09-20T19:27:06.065 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:87# 2024-09-20T19:27:06.069 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:89# 2024-09-20T19:27:06.072 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:96# 2024-09-20T19:27:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:06.077 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b3515764:::smithi04268048-3:head# 2024-09-20T19:27:06.077 INFO:teuthology.orchestra.run.smithi042.stdout:snapset 96=[]:{42=[41],83=[83,82,81,7f,7c,79,72],87=[87,86,85,84],89=[89,88],96=[96,95,94,92,91,8f,8e,8d,8c,8b,8a]} 2024-09-20T19:27:06.081 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b90bdf9e:::smithi04268048-23:43# 2024-09-20T19:27:06.084 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b90bdf9e:::smithi04268048-23:7c# 2024-09-20T19:27:06.087 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b90bdf9e:::smithi04268048-23:91# 2024-09-20T19:27:06.089 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:b90bdf9e:::smithi04268048-23:head# 2024-09-20T19:27:06.090 INFO:teuthology.orchestra.run.smithi042.stdout:snapset 91=[]:{43=[41],7c=[7c,79],91=[91,8f,8e,8d,8b,8a,89,88,87,86,85,84,83,82,81,7f]} 2024-09-20T19:27:06.094 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:baa4e37b:::smithi04268048-17:53# 2024-09-20T19:27:06.097 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:baa4e37b:::smithi04268048-17:7c# 2024-09-20T19:27:06.100 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:baa4e37b:::smithi04268048-17:91# 2024-09-20T19:27:06.103 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:baa4e37b:::smithi04268048-17:96# 2024-09-20T19:27:06.107 INFO:teuthology.orchestra.run.smithi042.stdout:Write #7:baa4e37b:::smithi04268048-17:head# 2024-09-20T19:27:06.107 INFO:teuthology.orchestra.run.smithi042.stdout:snapset 96=[]:{53=[41],7c=[7c,79,72],91=[91,8f,8e,8d,8b,8a],96=[96,95,94,92]} 2024-09-20T19:27:06.107 INFO:teuthology.orchestra.run.smithi042.stdout:write_pg epoch 392 info 7.d( v 393'279 (0'0,393'279] local-lis/les=112/113 n=15 ec=112/112 lis/c=112/112 les/c/f=113/113/0 sis=112) 2024-09-20T19:27:06.108 INFO:teuthology.orchestra.run.smithi042.stdout:Import successful 2024-09-20T19:27:06.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:06.197 INFO:tasks.rados.rados.0.smithi042.stdout:1477: done (4 left) 2024-09-20T19:27:06.197 INFO:tasks.rados.rados.0.smithi042.stdout:1479: done (3 left) 2024-09-20T19:27:06.197 INFO:tasks.rados.rados.0.smithi042.stdout:1480: done (2 left) 2024-09-20T19:27:06.197 INFO:tasks.rados.rados.0.smithi042.stdout:1481: done (1 left) 2024-09-20T19:27:06.198 INFO:tasks.rados.rados.0.smithi042.stdout:1482: done (0 left) 2024-09-20T19:27:06.198 INFO:tasks.rados.rados.0.smithi042.stdout:1483: read oid 14 snap -1 2024-09-20T19:27:06.198 INFO:tasks.rados.rados.0.smithi042.stdout:1483: expect (ObjNum 485 snap 158 seq_num 485) 2024-09-20T19:27:06.198 INFO:tasks.rados.rados.0.smithi042.stdout:1484: snap_create 2024-09-20T19:27:06.205 INFO:tasks.rados.rados.0.smithi042.stdout:1483: expect (ObjNum 485 snap 158 seq_num 485) 2024-09-20T19:27:06.206 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:06.213 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 948826 2024-09-20T19:27:06.239 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 647033 2024-09-20T19:27:06.271 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1603325 2024-09-20T19:27:06.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:06.334 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:06.428 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[32075]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:06 smithi042 bash[22706]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:06.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.171826+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e436: 8 total, 7 up, 8 in 2024-09-20T19:27:06.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.203723+0000 osd.1 (osd.1) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:06.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:06 smithi162 bash[24557]: cluster 2024-09-20T19:27:05.204728+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:06.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:06.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:06.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:06.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:06.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:27:06.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:06.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:06.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:06.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:07.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1483: done (1 left) 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1484: done (0 left) 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1485: read oid 43 snap -1 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1485: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1486: read oid 17 snap 159 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1486: expect deleted 2024-09-20T19:27:07.194 INFO:tasks.rados.rados.0.smithi042.stdout:1487: write (excl) oid 18 current snap is 169 2024-09-20T19:27:07.195 INFO:tasks.rados.rados.0.smithi042.stdout:1487: seq_num 521 ranges {756447=716733,1887942=537832} 2024-09-20T19:27:07.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:07.217 INFO:tasks.rados.rados.0.smithi042.stdout:1487: writing smithi04268048-18 from 756447 to 1473180 tid 1 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1487: writing smithi04268048-18 from 1887942 to 2425774 tid 2 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1488: read oid 9 snap 134 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1488: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1489: read oid 6 snap -1 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1489: expect (ObjNum 515 snap 166 seq_num 515) 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:1490: rollback oid 1 current snap is 169 2024-09-20T19:27:07.232 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 167 2024-09-20T19:27:07.233 INFO:tasks.rados.rados.0.smithi042.stdout:1485: expect (ObjNum 240 snap 70 seq_num 240) 2024-09-20T19:27:07.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:07.269 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:07.312 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:07.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:07.342 INFO:tasks.rados.rados.0.smithi042.stdout:1487: finishing write tid 1 to smithi04268048-18 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: finishing write tid 2 to smithi04268048-18 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: finishing write tid 3 to smithi04268048-18 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: finishing write tid 4 to smithi04268048-18 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: oid 18 updating version 0 to 731 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: oid 18 updating version 731 to 732 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: oid 18 version 732 is already newer than 730 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 732 (ObjNum 521 snap 169 seq_num 521) dirty exists 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1487: left oid 18 (ObjNum 521 snap 169 seq_num 521) 2024-09-20T19:27:07.343 INFO:tasks.rados.rados.0.smithi042.stdout:1489: expect (ObjNum 515 snap 166 seq_num 515) 2024-09-20T19:27:07.347 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 775881 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[22706]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.348 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.349 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:07 smithi042 bash[32075]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.360 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:07.413 INFO:tasks.rados.rados.0.smithi042.stdout:1490: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:27:07.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 514 (ObjNum 33 snap 0 seq_num 1390889016) dirty dne 2024-09-20T19:27:07.413 INFO:tasks.rados.rados.0.smithi042.stdout:1488: expect (ObjNum 426 snap 133 seq_num 426) 2024-09-20T19:27:07.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:07.475 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.088599+0000 mgr.y (mgr.24491) 349 : cluster [DBG] pgmap v593: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 477 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 205/1314 objects degraded (15.601%) 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.194757+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e437: 8 total, 7 up, 8 in 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.689763+0000 osd.4 (osd.4) 173 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:07 smithi162 bash[24557]: cluster 2024-09-20T19:27:06.697800+0000 osd.4 (osd.4) 174 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:27:07.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1485: done (5 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1486: done (4 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1487: done (3 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1488: done (2 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1489: done (1 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1490: done (0 left) 2024-09-20T19:27:07.516 INFO:tasks.rados.rados.0.smithi042.stdout:1491: write (excl) oid 5 current snap is 169 2024-09-20T19:27:07.517 INFO:tasks.rados.rados.0.smithi042.stdout:1491: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-20T19:27:07.525 INFO:tasks.rados.rados.0.smithi042.stdout:1491: writing smithi04268048-5 from 773254 to 1565236 tid 1 2024-09-20T19:27:07.532 INFO:tasks.rados.rados.0.smithi042.stdout:1491: writing smithi04268048-5 from 2346424 to 2773254 tid 2 2024-09-20T19:27:07.532 INFO:tasks.rados.rados.0.smithi042.stdout:1492: write oid 6 current snap is 169 2024-09-20T19:27:07.532 INFO:tasks.rados.rados.0.smithi042.stdout:1492: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-20T19:27:07.539 INFO:tasks.rados.rados.0.smithi042.stdout:1492: writing smithi04268048-6 from 790061 to 1257292 tid 1 2024-09-20T19:27:07.547 INFO:tasks.rados.rados.0.smithi042.stdout:1492: writing smithi04268048-6 from 2004906 to 2546407 tid 2 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1493: read oid 34 snap -1 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1493: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1494: copy_from oid 48 from oid 25 current snap is 169 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: finishing write tid 1 to smithi04268048-5 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: finishing write tid 2 to smithi04268048-5 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: finishing write tid 3 to smithi04268048-5 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: finishing write tid 4 to smithi04268048-5 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: oid 5 updating version 0 to 413 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: oid 5 updating version 413 to 414 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: oid 5 updating version 414 to 415 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 415 (ObjNum 522 snap 169 seq_num 522) dirty exists 2024-09-20T19:27:07.548 INFO:tasks.rados.rados.0.smithi042.stdout:1491: left oid 5 (ObjNum 522 snap 169 seq_num 522) 2024-09-20T19:27:07.549 INFO:tasks.rados.rados.0.smithi042.stdout:1492: finishing write tid 1 to smithi04268048-6 2024-09-20T19:27:07.549 INFO:tasks.rados.rados.0.smithi042.stdout:1491: done (3 left) 2024-09-20T19:27:07.549 INFO:tasks.rados.rados.0.smithi042.stdout:1495: write oid 33 current snap is 169 2024-09-20T19:27:07.549 INFO:tasks.rados.rados.0.smithi042.stdout:1495: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-20T19:27:07.556 INFO:tasks.rados.rados.0.smithi042.stdout:1495: writing smithi04268048-33 from 406868 to 949348 tid 1 2024-09-20T19:27:07.564 INFO:tasks.rados.rados.0.smithi042.stdout:1495: writing smithi04268048-33 from 1547035 to 1948547 tid 2 2024-09-20T19:27:07.570 INFO:tasks.rados.rados.0.smithi042.stdout:1495: writing smithi04268048-33 from 2690354 to 2806868 tid 3 2024-09-20T19:27:07.571 INFO:tasks.rados.rados.0.smithi042.stdout:1496: snap_remove snap 136 2024-09-20T19:27:07.594 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:07.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/exp.7.d.0 2024-09-20T19:27:07.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:07.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:07.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:07.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:07.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:07.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:07.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:08.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:08.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:08.145 INFO:tasks.rados.rados.0.smithi042.stdout:1496: done (4 left) 2024-09-20T19:27:08.145 INFO:tasks.rados.rados.0.smithi042.stdout:1497: delete oid 24 current snap is 169 2024-09-20T19:27:08.152 INFO:tasks.rados.rados.0.smithi042.stdout:1494: finishing copy_from to smithi04268048-48 2024-09-20T19:27:08.152 INFO:tasks.rados.rados.0.smithi042.stdout:1494: got expected ENOENT (src dne) 2024-09-20T19:27:08.152 INFO:tasks.rados.rados.0.smithi042.stdout:1494: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:27:08.152 INFO:tasks.rados.rados.0.smithi042.stdout:1493: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:27:08.153 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1603667 2024-09-20T19:27:08.168 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:08.204 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1464225 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: finishing write tid 2 to smithi04268048-6 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: finishing write tid 3 to smithi04268048-6 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: finishing write tid 4 to smithi04268048-6 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: oid 6 updating version 0 to 542 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: oid 6 updating version 542 to 543 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:1492: oid 6 updating version 543 to 544 2024-09-20T19:27:08.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 544 (ObjNum 523 snap 169 seq_num 523) dirty exists 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1492: left oid 6 (ObjNum 523 snap 169 seq_num 523) 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: finishing write tid 1 to smithi04268048-33 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: finishing write tid 2 to smithi04268048-33 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: finishing write tid 3 to smithi04268048-33 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: finishing write tid 4 to smithi04268048-33 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: finishing write tid 5 to smithi04268048-33 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: oid 33 updating version 0 to 576 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: oid 33 updating version 576 to 577 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: oid 33 version 577 is already newer than 575 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: oid 33 updating version 577 to 578 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 578 (ObjNum 524 snap 169 seq_num 524) dirty exists 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1495: left oid 33 (ObjNum 524 snap 169 seq_num 524) 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1492: done (4 left) 2024-09-20T19:27:08.224 INFO:tasks.rados.rados.0.smithi042.stdout:1493: done (3 left) 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1494: done (2 left) 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1495: done (1 left) 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1497: done (0 left) 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1498: read oid 20 snap -1 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1498: expect deleted 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1499: rollback oid 45 current snap is 169 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 112 2024-09-20T19:27:08.225 INFO:tasks.rados.rados.0.smithi042.stdout:1500: snap_create 2024-09-20T19:27:08.229 INFO:tasks.rados.rados.0.smithi042.stdout:1499: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:27:08.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:08.231 INFO:tasks.rados.rados.0.smithi042.stdout:1499: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:27:08.231 INFO:tasks.rados.rados.0.smithi042.stdout:1499: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:27:08.231 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 895 (ObjNum 332 snap 108 seq_num 332) dirty exists 2024-09-20T19:27:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:08.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:08 smithi162 bash[24557]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:08 smithi162 bash[24557]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:08 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:08 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.359 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:08.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:08.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:08.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:08.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:08.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[32075]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[32075]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[22706]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[22706]: cluster 2024-09-20T19:27:07.193373+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e438: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:08 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.145662+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e439: 8 total, 7 up, 8 in 2024-09-20T19:27:08.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:08.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:08.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:09.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:09.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:09.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:09.160 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:09.213 INFO:tasks.rados.rados.0.smithi042.stdout:1498: done (2 left) 2024-09-20T19:27:09.213 INFO:tasks.rados.rados.0.smithi042.stdout:1499: done (1 left) 2024-09-20T19:27:09.213 INFO:tasks.rados.rados.0.smithi042.stdout:1500: done (0 left) 2024-09-20T19:27:09.213 INFO:tasks.rados.rados.0.smithi042.stdout:1501: delete oid 25 current snap is 170 2024-09-20T19:27:09.218 INFO:tasks.rados.rados.0.smithi042.stdout:1501: done (0 left) 2024-09-20T19:27:09.219 INFO:tasks.rados.rados.0.smithi042.stdout:1502: snap_remove snap 166 2024-09-20T19:27:09.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:09.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:09.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:27:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:27:09.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[22706]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:09 smithi042 bash[32075]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.535 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:09.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:09 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:09 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.089151+0000 mgr.y (mgr.24491) 350 : cluster [DBG] pgmap v596: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:09.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:09 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:09 smithi162 bash[24557]: cluster 2024-09-20T19:27:08.198657+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: Degraded data redundancy: 208/1308 objects degraded (15.902%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:09.549 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:09.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:09.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:09.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:09.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:10.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:10.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:10.126 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:10.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:10.231 INFO:tasks.rados.rados.0.smithi042.stdout:1502: done (0 left) 2024-09-20T19:27:10.231 INFO:tasks.rados.rados.0.smithi042.stdout:1503: rollback oid 14 current snap is 170 2024-09-20T19:27:10.231 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 157 2024-09-20T19:27:10.231 INFO:tasks.rados.rados.0.smithi042.stdout:1504: snap_remove snap 137 2024-09-20T19:27:10.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:10 smithi162 bash[24557]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:10 smithi162 bash[24557]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:10 smithi162 bash[24557]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:10 smithi162 bash[24557]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.326 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:10.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:10.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:10.557 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:10.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[22706]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[22706]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[22706]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[22706]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[32075]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[32075]: cluster 2024-09-20T19:27:09.213209+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e440: 8 total, 7 up, 8 in 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[32075]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:10 smithi042 bash[32075]: audit 2024-09-20T19:27:09.296027+0000 mgr.y (mgr.24491) 351 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:10.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:10.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:10.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:10.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:10.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:10.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:11.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:11.121 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:11.208 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1504: done (1 left) 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1505: read oid 44 snap 165 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1505: expect deleted 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1506: rollback oid 10 current snap is 170 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 134 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1503: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1503: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1503: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 418 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1503: done (2 left) 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1507: read oid 40 snap -1 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1507: expect (ObjNum 484 snap 158 seq_num 484) 2024-09-20T19:27:11.235 INFO:tasks.rados.rados.0.smithi042.stdout:1508: snap_remove snap 162 2024-09-20T19:27:11.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:11.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:11.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:11.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:11.497 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.497 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:11 smithi162 bash[24557]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:11.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[22706]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.090137+0000 mgr.y (mgr.24491) 352 : cluster [DBG] pgmap v599: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 4 active+clean+snaptrim, 25 active+undersized+degraded, 97 active+clean; 484 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 31 op/s; 208/1308 objects degraded (15.902%) 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.213697+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.215538+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.231429+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e441: 8 total, 7 up, 8 in 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.669175+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.e scrub starts 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.616 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:11 smithi042 bash[32075]: cluster 2024-09-20T19:27:10.673533+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.e scrub ok 2024-09-20T19:27:11.618 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:27:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:27:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-20T19:27:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-20T19:27:11.618 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:27:11.630 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:11.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:11.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:11.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:11.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:11.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:12.118 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:12.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:12.249 INFO:tasks.rados.rados.0.smithi042.stdout:1508: done (3 left) 2024-09-20T19:27:12.249 INFO:tasks.rados.rados.0.smithi042.stdout:1509: write oid 23 current snap is 170 2024-09-20T19:27:12.249 INFO:tasks.rados.rados.0.smithi042.stdout:1509: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-20T19:27:12.251 INFO:tasks.rados.rados.0.smithi042.stdout:1509: writing smithi04268048-23 from 457289 to 941869 tid 1 2024-09-20T19:27:12.261 INFO:tasks.rados.rados.0.smithi042.stdout:1509: writing smithi04268048-23 from 1438834 to 1987673 tid 2 2024-09-20T19:27:12.269 INFO:tasks.rados.rados.0.smithi042.stdout:1509: writing smithi04268048-23 from 2772623 to 2857289 tid 3 2024-09-20T19:27:12.269 INFO:tasks.rados.rados.0.smithi042.stdout:1510: read oid 4 snap -1 2024-09-20T19:27:12.269 INFO:tasks.rados.rados.0.smithi042.stdout:1510: expect (ObjNum 498 snap 163 seq_num 498) 2024-09-20T19:27:12.269 INFO:tasks.rados.rados.0.smithi042.stdout:1511: snap_create 2024-09-20T19:27:12.270 INFO:tasks.rados.rados.0.smithi042.stdout:1506: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:27:12.270 INFO:tasks.rados.rados.0.smithi042.stdout:1507: expect (ObjNum 484 snap 158 seq_num 484) 2024-09-20T19:27:12.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:12.326 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 515700 2024-09-20T19:27:12.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T19:27:12.363 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:12.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1506: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 583 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: finishing write tid 1 to smithi04268048-23 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: finishing write tid 2 to smithi04268048-23 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: finishing write tid 3 to smithi04268048-23 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: finishing write tid 4 to smithi04268048-23 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: finishing write tid 5 to smithi04268048-23 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: oid 23 updating version 0 to 679 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: oid 23 version 679 is already newer than 676 2024-09-20T19:27:12.392 INFO:tasks.rados.rados.0.smithi042.stdout:1509: oid 23 version 679 is already newer than 678 2024-09-20T19:27:12.393 INFO:tasks.rados.rados.0.smithi042.stdout:1509: oid 23 version 679 is already newer than 677 2024-09-20T19:27:12.393 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 679 (ObjNum 527 snap 170 seq_num 527) dirty exists 2024-09-20T19:27:12.393 INFO:tasks.rados.rados.0.smithi042.stdout:1509: left oid 23 (ObjNum 527 snap 170 seq_num 527) 2024-09-20T19:27:12.393 INFO:tasks.rados.rados.0.smithi042.stdout:1510: expect (ObjNum 498 snap 163 seq_num 498) 2024-09-20T19:27:12.393 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 423909 2024-09-20T19:27:12.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:12.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[22706]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[22706]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[32075]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[32075]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:12 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.470 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1929462 2024-09-20T19:27:12.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:12.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[24557]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[24557]: cluster 2024-09-20T19:27:11.233436+0000 mon.a (mon.0) 1519 : cluster [DBG] osdmap e442: 8 total, 7 up, 8 in 2024-09-20T19:27:12.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.515 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.239126+0000 mon.a (mon.0) 1520 : cluster [DBG] osdmap e443: 8 total, 7 up, 8 in 2024-09-20T19:27:12.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:12.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:27:12.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:27:12.630 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:27:12.630 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T19:27:12.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4 2024-09-20T19:27:12.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:12.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:12.776 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:12 smithi162 systemd[1]: Stopping Ceph osd.4 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:27:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:12.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:27:12.854 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:27:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:27:12.872 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:27:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:27:12.884 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.889 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:27:12.890 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:27:12.905 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:12.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:12.912 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:27:12.912 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:27:12.933 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:27:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:27:12.955 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:27:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:27:12.976 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:27:12.977 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:27:13.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.018 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[28094]: debug 2024-09-20T19:27:12.779+0000 7f43cacca640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:27:13.018 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[28094]: debug 2024-09-20T19:27:12.779+0000 7f43cacca640 -1 osd.4 443 *** Got signal Terminated *** 2024-09-20T19:27:13.018 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:12 smithi162 bash[28094]: debug 2024-09-20T19:27:12.779+0000 7f43cacca640 -1 osd.4 443 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:27:13.034 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:13.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:13.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:13.276 INFO:tasks.rados.rados.0.smithi042.stdout:1505: done (5 left) 2024-09-20T19:27:13.276 INFO:tasks.rados.rados.0.smithi042.stdout:1506: done (4 left) 2024-09-20T19:27:13.276 INFO:tasks.rados.rados.0.smithi042.stdout:1507: done (3 left) 2024-09-20T19:27:13.276 INFO:tasks.rados.rados.0.smithi042.stdout:1509: done (2 left) 2024-09-20T19:27:13.276 INFO:tasks.rados.rados.0.smithi042.stdout:1510: done (1 left) 2024-09-20T19:27:13.277 INFO:tasks.rados.rados.0.smithi042.stdout:1511: done (0 left) 2024-09-20T19:27:13.277 INFO:tasks.rados.rados.0.smithi042.stdout:1512: read oid 45 snap 170 2024-09-20T19:27:13.277 INFO:tasks.rados.rados.0.smithi042.stdout:1512: expect (ObjNum 332 snap 108 seq_num 332) 2024-09-20T19:27:13.277 INFO:tasks.rados.rados.0.smithi042.stdout:1513: write (excl) oid 19 current snap is 171 2024-09-20T19:27:13.277 INFO:tasks.rados.rados.0.smithi042.stdout:1513: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-20T19:27:13.285 INFO:tasks.rados.rados.0.smithi042.stdout:1513: writing smithi04268048-19 from 474096 to 1033925 tid 1 2024-09-20T19:27:13.294 INFO:tasks.rados.rados.0.smithi042.stdout:1513: writing smithi04268048-19 from 1780963 to 2189813 tid 2 2024-09-20T19:27:13.299 INFO:tasks.rados.rados.0.smithi042.stdout:1513: writing smithi04268048-19 from 2800046 to 2874096 tid 3 2024-09-20T19:27:13.299 INFO:tasks.rados.rados.0.smithi042.stdout:1514: write oid 29 current snap is 171 2024-09-20T19:27:13.299 INFO:tasks.rados.rados.0.smithi042.stdout:1514: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-20T19:27:13.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:13 smithi162 bash[24557]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.309 INFO:tasks.rados.rados.0.smithi042.stdout:1514: writing smithi04268048-29 from 490903 to 1125981 tid 1 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1514: writing smithi04268048-29 from 1839445 to 2391953 tid 2 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1515: snap_create 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: finishing write tid 1 to smithi04268048-19 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: finishing write tid 2 to smithi04268048-19 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: finishing write tid 3 to smithi04268048-19 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: finishing write tid 4 to smithi04268048-19 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: finishing write tid 5 to smithi04268048-19 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: oid 19 updating version 0 to 402 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: oid 19 updating version 402 to 404 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: oid 19 version 404 is already newer than 403 2024-09-20T19:27:13.318 INFO:tasks.rados.rados.0.smithi042.stdout:1513: oid 19 updating version 404 to 405 2024-09-20T19:27:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 405 (ObjNum 528 snap 171 seq_num 528) dirty exists 2024-09-20T19:27:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:1513: left oid 19 (ObjNum 528 snap 171 seq_num 528) 2024-09-20T19:27:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:1514: finishing write tid 1 to smithi04268048-29 2024-09-20T19:27:13.319 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:13.322 INFO:tasks.rados.rados.0.smithi042.stdout:1514: finishing write tid 2 to smithi04268048-29 2024-09-20T19:27:13.322 INFO:tasks.rados.rados.0.smithi042.stdout:1514: finishing write tid 3 to smithi04268048-29 2024-09-20T19:27:13.322 INFO:tasks.rados.rados.0.smithi042.stdout:1514: finishing write tid 4 to smithi04268048-29 2024-09-20T19:27:13.323 INFO:tasks.rados.rados.0.smithi042.stdout:1514: oid 29 updating version 0 to 586 2024-09-20T19:27:13.323 INFO:tasks.rados.rados.0.smithi042.stdout:1514: oid 29 version 586 is already newer than 584 2024-09-20T19:27:13.323 INFO:tasks.rados.rados.0.smithi042.stdout:1514: oid 29 version 586 is already newer than 585 2024-09-20T19:27:13.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 586 (ObjNum 529 snap 171 seq_num 529) dirty exists 2024-09-20T19:27:13.323 INFO:tasks.rados.rados.0.smithi042.stdout:1514: left oid 29 (ObjNum 529 snap 171 seq_num 529) 2024-09-20T19:27:13.422 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:13.449 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:13.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[22706]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.091359+0000 mgr.y (mgr.24491) 353 : cluster [DBG] pgmap v602: 158 pgs: 1 active+clean+snaptrim_wait, 31 active+undersized, 5 active+clean+snaptrim, 25 active+undersized+degraded, 96 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: audit 2024-09-20T19:27:12.317712+0000 mon.a (mon.0) 1521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.517102+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.519149+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: cluster 2024-09-20T19:27:12.778312+0000 mon.a (mon.0) 1522 : cluster [INF] osd.4 marked itself down and dead 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:13 smithi042 bash[32075]: audit 2024-09-20T19:27:12.836140+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? v1:172.21.15.42:0/3127910489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:13.679 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:13.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:13.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:13.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:14.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:14.094 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:14.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:14.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:14.281 INFO:tasks.rados.rados.0.smithi042.stdout:1513: done (3 left) 2024-09-20T19:27:14.282 INFO:tasks.rados.rados.0.smithi042.stdout:1514: done (2 left) 2024-09-20T19:27:14.282 INFO:tasks.rados.rados.0.smithi042.stdout:1515: done (1 left) 2024-09-20T19:27:14.282 INFO:tasks.rados.rados.0.smithi042.stdout:1516: snap_create 2024-09-20T19:27:14.311 INFO:tasks.rados.rados.0.smithi042.stdout:1512: expect (ObjNum 332 snap 108 seq_num 332) 2024-09-20T19:27:14.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:27:14.325 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:27:14.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[24557]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[24557]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[24557]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[24557]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:14.380 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 385919 2024-09-20T19:27:14.448 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:14.456 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:14.458 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[32075]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.458 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[32075]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.458 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[32075]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.458 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[32075]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[22706]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[22706]: cluster 2024-09-20T19:27:13.256329+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:27:14.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[22706]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.459 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:14 smithi042 bash[22706]: cluster 2024-09-20T19:27:13.276616+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e444: 8 total, 6 up, 8 in 2024-09-20T19:27:14.471 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:14.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:14.597 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:14.700 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:14.726 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:14.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:14.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:14.914 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[68200]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-4 2024-09-20T19:27:14.915 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:27:14 smithi162 bash[68317]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-4 2024-09-20T19:27:14.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:14.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:14.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:15.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:15.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:15.242 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:15.276 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:27:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1512: done (1 left) 2024-09-20T19:27:15.283 INFO:tasks.rados.rados.0.smithi042.stdout:1516: done (0 left) 2024-09-20T19:27:15.284 INFO:tasks.rados.rados.0.smithi042.stdout:1517: delete oid 42 current snap is 173 2024-09-20T19:27:15.296 INFO:tasks.rados.rados.0.smithi042.stdout:1517: done (0 left) 2024-09-20T19:27:15.296 INFO:tasks.rados.rados.0.smithi042.stdout:1518: write (excl) oid 46 current snap is 173 2024-09-20T19:27:15.296 INFO:tasks.rados.rados.0.smithi042.stdout:1518: seq_num 531 ranges {524517=785576,1956409=556177} 2024-09-20T19:27:15.307 INFO:tasks.rados.rados.0.smithi042.stdout:1518: writing smithi04268048-46 from 524517 to 1310093 tid 1 2024-09-20T19:27:15.316 INFO:tasks.rados.rados.0.smithi042.stdout:1518: writing smithi04268048-46 from 1956409 to 2512586 tid 2 2024-09-20T19:27:15.316 INFO:tasks.rados.rados.0.smithi042.stdout:1519: read oid 21 snap 169 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1519: expect (ObjNum 442 snap 145 seq_num 442) 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1520: read oid 3 snap -1 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1520: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1521: write (excl) oid 49 current snap is 173 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1518: finishing write tid 1 to smithi04268048-46 2024-09-20T19:27:15.317 INFO:tasks.rados.rados.0.smithi042.stdout:1521: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-09-20T19:27:15.323 INFO:tasks.rados.rados.0.smithi042.stdout:1521: writing smithi04268048-49 from 541324 to 1002149 tid 1 2024-09-20T19:27:15.330 INFO:tasks.rados.rados.0.smithi042.stdout:1521: writing smithi04268048-49 from 1498538 to 1914726 tid 2 2024-09-20T19:27:15.337 INFO:tasks.rados.rados.0.smithi042.stdout:1521: writing smithi04268048-49 from 2393385 to 2941324 tid 3 2024-09-20T19:27:15.337 INFO:tasks.rados.rados.0.smithi042.stdout:1522: snap_remove snap 158 2024-09-20T19:27:15.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:15.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:15.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:15.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[22706]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.507 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:15 smithi042 bash[32075]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.018948+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.022113+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.092356+0000 mgr.y (mgr.24491) 354 : cluster [DBG] pgmap v605: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 19 op/s; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: cluster 2024-09-20T19:27:14.279794+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e445: 8 total, 6 up, 8 in 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:15 smithi162 bash[24557]: audit 2024-09-20T19:27:14.584021+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:15.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:15.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:15.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:15.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:15.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:15.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:15.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:16.027 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:16.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:16.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.282 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:16.288 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:16.310 INFO:tasks.rados.rados.0.smithi042.stdout:1522: done (4 left) 2024-09-20T19:27:16.310 INFO:tasks.rados.rados.0.smithi042.stdout:1523: write oid 11 current snap is 173 2024-09-20T19:27:16.310 INFO:tasks.rados.rados.0.smithi042.stdout:1518: finishing write tid 2 to smithi04268048-46 2024-09-20T19:27:16.310 INFO:tasks.rados.rados.0.smithi042.stdout:1523: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-09-20T19:27:16.319 INFO:tasks.rados.rados.0.smithi042.stdout:1523: writing smithi04268048-11 from 558131 to 1210558 tid 1 2024-09-20T19:27:16.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:27:16.327 INFO:tasks.rados.rados.0.smithi042.stdout:1523: writing smithi04268048-11 from 1673373 to 2233219 tid 2 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1523: writing smithi04268048-11 from 2820808 to 2958131 tid 3 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1524: write (excl) oid 6 current snap is 173 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1518: finishing write tid 3 to smithi04268048-46 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1518: finishing write tid 4 to smithi04268048-46 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1518: oid 46 updating version 0 to 916 2024-09-20T19:27:16.334 INFO:tasks.rados.rados.0.smithi042.stdout:1518: oid 46 version 916 is already newer than 915 2024-09-20T19:27:16.335 INFO:tasks.rados.rados.0.smithi042.stdout:1518: oid 46 updating version 916 to 917 2024-09-20T19:27:16.335 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 917 (ObjNum 531 snap 173 seq_num 531) dirty exists 2024-09-20T19:27:16.335 INFO:tasks.rados.rados.0.smithi042.stdout:1518: left oid 46 (ObjNum 531 snap 173 seq_num 531) 2024-09-20T19:27:16.335 INFO:tasks.rados.rados.0.smithi042.stdout:1521: finishing write tid 1 to smithi04268048-49 2024-09-20T19:27:16.335 INFO:tasks.rados.rados.0.smithi042.stdout:1524: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-20T19:27:16.343 INFO:tasks.rados.rados.0.smithi042.stdout:1524: writing smithi04268048-6 from 574938 to 1302614 tid 1 2024-09-20T19:27:16.356 INFO:tasks.rados.rados.0.smithi042.stdout:1524: writing smithi04268048-6 from 1731855 to 2435359 tid 2 2024-09-20T19:27:16.358 INFO:tasks.rados.rados.0.smithi042.stdout:1524: writing smithi04268048-6 from 2848231 to 2974938 tid 3 2024-09-20T19:27:16.358 INFO:tasks.rados.rados.0.smithi042.stdout:1518: done (5 left) 2024-09-20T19:27:16.358 INFO:tasks.rados.rados.0.smithi042.stdout:1525: write (excl) oid 48 current snap is 173 2024-09-20T19:27:16.358 INFO:tasks.rados.rados.0.smithi042.stdout:1525: seq_num 535 ranges {591745=402925,1790337=563515} 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:16 smithi162 bash[24557]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.366 INFO:tasks.rados.rados.0.smithi042.stdout:1525: writing smithi04268048-48 from 591745 to 994670 tid 1 2024-09-20T19:27:16.375 INFO:tasks.rados.rados.0.smithi042.stdout:1525: writing smithi04268048-48 from 1790337 to 2353852 tid 2 2024-09-20T19:27:16.376 INFO:tasks.rados.rados.0.smithi042.stdout:1526: write oid 1 current snap is 173 2024-09-20T19:27:16.376 INFO:tasks.rados.rados.0.smithi042.stdout:1526: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-20T19:27:16.383 INFO:tasks.rados.rados.0.smithi042.stdout:1526: writing smithi04268048-1 from 608552 to 1086726 tid 1 2024-09-20T19:27:16.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:16.392 INFO:tasks.rados.rados.0.smithi042.stdout:1526: writing smithi04268048-1 from 1848819 to 2555992 tid 2 2024-09-20T19:27:16.392 INFO:tasks.rados.rados.0.smithi042.stdout:1527: write (excl) oid 31 current snap is 173 2024-09-20T19:27:16.392 INFO:tasks.rados.rados.0.smithi042.stdout:1521: finishing write tid 2 to smithi04268048-49 2024-09-20T19:27:16.393 INFO:tasks.rados.rados.0.smithi042.stdout:1520: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:27:16.393 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:16.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:16.470 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:16.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:16.521 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[22706]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: cluster 2024-09-20T19:27:15.275784+0000 mon.a (mon.0) 1528 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: audit 2024-09-20T19:27:15.277523+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.42:0/564269031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:16 smithi042 bash[32075]: cluster 2024-09-20T19:27:15.283364+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e446: 8 total, 6 up, 8 in 2024-09-20T19:27:16.533 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:16.617 INFO:tasks.rados.rados.0.smithi042.stdout:1521: finishing write tid 3 to smithi04268048-49 2024-09-20T19:27:16.618 INFO:tasks.rados.rados.0.smithi042.stdout:1521: finishing write tid 4 to smithi04268048-49 2024-09-20T19:27:16.618 INFO:tasks.rados.rados.0.smithi042.stdout:1527: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-20T19:27:16.624 INFO:tasks.rados.rados.0.smithi042.stdout:1527: writing smithi04268048-31 from 625359 to 1178782 tid 1 2024-09-20T19:27:16.631 INFO:tasks.rados.rados.0.smithi042.stdout:1527: writing smithi04268048-31 from 1790948 to 2358132 tid 2 2024-09-20T19:27:16.635 INFO:tasks.rados.rados.0.smithi042.stdout:1527: writing smithi04268048-31 from 2930500 to 3025359 tid 3 2024-09-20T19:27:16.635 INFO:tasks.rados.rados.0.smithi042.stdout:1520: done (7 left) 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1528: copy_from oid 45 from oid 17 current snap is 173 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: finishing write tid 5 to smithi04268048-49 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: oid 49 updating version 0 to 919 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: oid 49 updating version 919 to 920 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: oid 49 updating version 920 to 921 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: oid 49 updating version 921 to 922 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 922 (ObjNum 532 snap 173 seq_num 532) dirty exists 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: left oid 49 (ObjNum 532 snap 173 seq_num 532) 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1521: done (7 left) 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1529: write (excl) oid 2 current snap is 173 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1523: finishing write tid 1 to smithi04268048-11 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1523: finishing write tid 2 to smithi04268048-11 2024-09-20T19:27:16.636 INFO:tasks.rados.rados.0.smithi042.stdout:1529: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-20T19:27:16.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:16.645 INFO:tasks.rados.rados.0.smithi042.stdout:1529: writing smithi04268048-2 from 642166 to 1270838 tid 1 2024-09-20T19:27:16.657 INFO:tasks.rados.rados.0.smithi042.stdout:1529: writing smithi04268048-2 from 1849430 to 2560272 tid 2 2024-09-20T19:27:16.657 INFO:tasks.rados.rados.0.smithi042.stdout:1530: delete oid 26 current snap is 173 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: finishing write tid 3 to smithi04268048-11 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: finishing write tid 4 to smithi04268048-11 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: finishing write tid 5 to smithi04268048-11 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: oid 11 updating version 0 to 709 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: oid 11 version 709 is already newer than 708 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: oid 11 version 709 is already newer than 706 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: oid 11 version 709 is already newer than 707 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 709 (ObjNum 533 snap 173 seq_num 533) dirty exists 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1523: left oid 11 (ObjNum 533 snap 173 seq_num 533) 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1524: finishing write tid 1 to smithi04268048-6 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1524: finishing write tid 2 to smithi04268048-6 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1524: finishing write tid 3 to smithi04268048-6 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1524: finishing write tid 4 to smithi04268048-6 2024-09-20T19:27:16.658 INFO:tasks.rados.rados.0.smithi042.stdout:1524: finishing write tid 5 to smithi04268048-6 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1524: oid 6 updating version 0 to 560 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1524: oid 6 updating version 560 to 563 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1524: oid 6 version 563 is already newer than 562 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1524: oid 6 version 563 is already newer than 561 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 563 (ObjNum 534 snap 173 seq_num 534) dirty exists 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1524: left oid 6 (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: finishing write tid 1 to smithi04268048-1 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: finishing write tid 2 to smithi04268048-1 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: finishing write tid 3 to smithi04268048-1 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: finishing write tid 4 to smithi04268048-1 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: oid 1 updating version 0 to 564 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: oid 1 updating version 564 to 566 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:1526: oid 1 version 566 is already newer than 565 2024-09-20T19:27:16.659 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 566 (ObjNum 536 snap 173 seq_num 536) dirty exists 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1526: left oid 1 (ObjNum 536 snap 173 seq_num 536) 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: finishing write tid 1 to smithi04268048-31 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: finishing write tid 2 to smithi04268048-31 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: finishing write tid 3 to smithi04268048-31 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: finishing write tid 4 to smithi04268048-31 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: finishing write tid 5 to smithi04268048-31 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: oid 31 updating version 0 to 569 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: oid 31 version 569 is already newer than 568 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: oid 31 updating version 569 to 571 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: oid 31 version 571 is already newer than 570 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 571 (ObjNum 537 snap 173 seq_num 537) dirty exists 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1527: left oid 31 (ObjNum 537 snap 173 seq_num 537) 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1528: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:27:16.660 INFO:tasks.rados.rados.0.smithi042.stdout:1529: finishing write tid 1 to smithi04268048-2 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1528: finishing copy_from to smithi04268048-45 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 675 (ObjNum 514 snap 166 seq_num 514) dirty exists 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1523: done (8 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1524: done (7 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1526: done (6 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1527: done (5 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1528: done (4 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1530: done (3 left) 2024-09-20T19:27:16.661 INFO:tasks.rados.rados.0.smithi042.stdout:1531: snap_remove snap 163 2024-09-20T19:27:16.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:16.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:16.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:16.927 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:16.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:17.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:17.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4.service' 2024-09-20T19:27:17.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:27:17.151 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-20T19:27:17.151 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-20T19:27:17.151 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi162.front.sepia.ceph.com 2024-09-20T19:27:17.151 INFO:tasks.thrashosds.thrasher:remote for exp osd 0 is ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T19:27:17.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-20T19:27:17.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:17.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1531: done (3 left) 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1532: snap_create 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1529: finishing write tid 2 to smithi04268048-2 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1529: finishing write tid 3 to smithi04268048-2 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1529: finishing write tid 4 to smithi04268048-2 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1529: oid 2 updating version 0 to 581 2024-09-20T19:27:17.319 INFO:tasks.rados.rados.0.smithi042.stdout:1529: oid 2 version 581 is already newer than 580 2024-09-20T19:27:17.320 INFO:tasks.rados.rados.0.smithi042.stdout:1529: oid 2 version 581 is already newer than 579 2024-09-20T19:27:17.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 581 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:27:17.320 INFO:tasks.rados.rados.0.smithi042.stdout:1529: left oid 2 (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:27:17.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:17.347 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:27:17.348 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[22706]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[22706]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[22706]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[22706]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[32075]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[32075]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[32075]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:17 smithi042 bash[32075]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.449 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:17 smithi162 bash[24557]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:17 smithi162 bash[24557]: cluster 2024-09-20T19:27:16.093413+0000 mgr.y (mgr.24491) 355 : cluster [DBG] pgmap v608: 158 pgs: 1 stale+active+undersized, 5 stale+active+undersized+degraded, 18 stale+active+clean, 1 active+clean+snaptrim_wait, 30 active+undersized, 5 active+clean+snaptrim, 20 active+undersized+degraded, 78 active+clean; 495 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 211/1320 objects degraded (15.985%) 2024-09-20T19:27:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:17 smithi162 bash[24557]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:17 smithi162 bash[24557]: cluster 2024-09-20T19:27:16.307522+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e447: 8 total, 6 up, 8 in 2024-09-20T19:27:17.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:17.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:17.615 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:17.704 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.733 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:17.836 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:17.863 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:17.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:17.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:18.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.125 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:18.148 INFO:tasks.rados.rados.0.smithi042.stdout:1529: done (3 left) 2024-09-20T19:27:18.149 INFO:tasks.rados.rados.0.smithi042.stdout:1532: done (2 left) 2024-09-20T19:27:18.149 INFO:tasks.rados.rados.0.smithi042.stdout:1533: delete oid 49 current snap is 174 2024-09-20T19:27:18.156 INFO:tasks.rados.rados.0.smithi042.stdout:1533: done (2 left) 2024-09-20T19:27:18.156 INFO:tasks.rados.rados.0.smithi042.stdout:1534: read oid 6 snap -1 2024-09-20T19:27:18.156 INFO:tasks.rados.rados.0.smithi042.stdout:1534: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:27:18.156 INFO:tasks.rados.rados.0.smithi042.stdout:1535: snap_remove snap 112 2024-09-20T19:27:18.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:18.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:18.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:27:18.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:18.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:18.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:18 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.517 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:18.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:18.640 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:18.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.769 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:17.316430+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e448: 8 total, 6 up, 8 in 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.142870+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: Degraded data redundancy: 211/1320 objects degraded (15.985%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:18 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.149710+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e449: 8 total, 6 up, 8 in 2024-09-20T19:27:18.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:18.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:18.947 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:27:18.947 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:27:19.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:19.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:19.153 INFO:tasks.rados.rados.0.smithi042.stdout:1535: done (3 left) 2024-09-20T19:27:19.153 INFO:tasks.rados.rados.0.smithi042.stdout:1536: rollback oid 16 current snap is 174 2024-09-20T19:27:19.153 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 134 2024-09-20T19:27:19.153 INFO:tasks.rados.rados.0.smithi042.stdout:1534: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:27:19.158 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 858655 2024-09-20T19:27:19.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:19.229 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1815238 2024-09-20T19:27:19.262 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:19.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:19.321 INFO:tasks.rados.rados.0.smithi042.stdout:1536: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:1536: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 682 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:1534: done (3 left) 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:1536: done (2 left) 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:1537: write (excl) oid 1 current snap is 174 2024-09-20T19:27:19.322 INFO:tasks.rados.rados.0.smithi042.stdout:1537: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-20T19:27:19.343 INFO:tasks.rados.rados.0.smithi042.stdout:1537: writing smithi04268048-1 from 692587 to 1263359 tid 1 2024-09-20T19:27:19.365 INFO:tasks.rados.rados.0.smithi042.stdout:1537: writing smithi04268048-1 from 2024876 to 2599398 tid 2 2024-09-20T19:27:19.372 INFO:tasks.rados.rados.0.smithi042.stdout:1537: writing smithi04268048-1 from 3040192 to 3092587 tid 3 2024-09-20T19:27:19.373 INFO:tasks.rados.rados.0.smithi042.stdout:1538: snap_create 2024-09-20T19:27:19.373 INFO:tasks.rados.rados.0.smithi042.stdout:1537: finishing write tid 1 to smithi04268048-1 2024-09-20T19:27:19.373 INFO:tasks.rados.rados.0.smithi042.stdout:1537: finishing write tid 2 to smithi04268048-1 2024-09-20T19:27:19.376 INFO:tasks.rados.rados.0.smithi042.stdout:1537: finishing write tid 3 to smithi04268048-1 2024-09-20T19:27:19.377 INFO:tasks.rados.rados.0.smithi042.stdout:1537: finishing write tid 4 to smithi04268048-1 2024-09-20T19:27:19.377 INFO:tasks.rados.rados.0.smithi042.stdout:1537: finishing write tid 5 to smithi04268048-1 2024-09-20T19:27:19.377 INFO:tasks.rados.rados.0.smithi042.stdout:1537: oid 1 updating version 0 to 578 2024-09-20T19:27:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1537: oid 1 version 578 is already newer than 577 2024-09-20T19:27:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1537: oid 1 version 578 is already newer than 575 2024-09-20T19:27:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1537: oid 1 version 578 is already newer than 576 2024-09-20T19:27:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 578 (ObjNum 541 snap 174 seq_num 541) dirty exists 2024-09-20T19:27:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1537: left oid 1 (ObjNum 541 snap 174 seq_num 541) 2024-09-20T19:27:19.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:19.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:19.530 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[32075]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:19 smithi042 bash[22706]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.545 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:19.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:19.676 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:19.677 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.094734+0000 mgr.y (mgr.24491) 356 : cluster [DBG] pgmap v611: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:19.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:18.308808+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T19:27:19.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:19 smithi162 bash[24557]: cluster 2024-09-20T19:27:19.152200+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e450: 8 total, 6 up, 8 in 2024-09-20T19:27:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:19.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:19.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:27:19.809 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:19.837 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:19.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:19.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:20.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:20.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:20.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:20.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:20.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:20.328 INFO:tasks.rados.rados.0.smithi042.stdout:1537: done (3 left) 2024-09-20T19:27:20.328 INFO:tasks.rados.rados.0.smithi042.stdout:1538: done (2 left) 2024-09-20T19:27:20.328 INFO:tasks.rados.rados.0.smithi042.stdout:1539: read oid 3 snap 165 2024-09-20T19:27:20.329 INFO:tasks.rados.rados.0.smithi042.stdout:1539: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:27:20.329 INFO:tasks.rados.rados.0.smithi042.stdout:1540: copy_from oid 8 from oid 42 current snap is 175 2024-09-20T19:27:20.329 INFO:tasks.rados.rados.0.smithi042.stdout:1541: read oid 49 snap -1 2024-09-20T19:27:20.329 INFO:tasks.rados.rados.0.smithi042.stdout:1541: expect deleted 2024-09-20T19:27:20.329 INFO:tasks.rados.rados.0.smithi042.stdout:1542: snap_create 2024-09-20T19:27:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:20.348 INFO:tasks.rados.rados.0.smithi042.stdout:1539: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:27:20.348 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:27:20.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:20.454 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:20.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:20 smithi162 bash[24557]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:20 smithi162 bash[24557]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.551 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3716837 2024-09-20T19:27:20.553 INFO:tasks.rados.rados.0.smithi042.stdout:1540: finishing copy_from to smithi04268048-8 2024-09-20T19:27:20.553 INFO:tasks.rados.rados.0.smithi042.stdout:1540: got expected ENOENT (src dne) 2024-09-20T19:27:20.553 INFO:tasks.rados.rados.0.smithi042.stdout:1540: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:27:20.557 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:20.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:20.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:20 smithi042 bash[32075]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:20 smithi042 bash[32075]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.584 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:20 smithi042 bash[22706]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.584 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:20 smithi042 bash[22706]: audit 2024-09-20T19:27:19.304031+0000 mgr.y (mgr.24491) 357 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:20.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:20.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:20.841 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:20.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:20.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:20.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:21.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:21.096 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:21.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:21.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:21.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:21.335 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1539: done (5 left) 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1540: done (4 left) 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1541: done (3 left) 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1542: done (2 left) 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1543: read oid 32 snap -1 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1543: expect (ObjNum 505 snap 164 seq_num 505) 2024-09-20T19:27:21.339 INFO:tasks.rados.rados.0.smithi042.stdout:1544: snap_create 2024-09-20T19:27:21.348 INFO:tasks.rados.rados.0.smithi042.stdout:1543: expect (ObjNum 505 snap 164 seq_num 505) 2024-09-20T19:27:21.365 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:21.374 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1636604 2024-09-20T19:27:21.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:21.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:21.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:21.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[32075]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[32075]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[32075]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.605 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[32075]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[22706]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[22706]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[22706]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.606 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:21 smithi042 bash[22706]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:21.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:21.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:21 smithi162 bash[24557]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:21 smithi162 bash[24557]: cluster 2024-09-20T19:27:20.095614+0000 mgr.y (mgr.24491) 358 : cluster [DBG] pgmap v614: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 512 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 23 op/s; 382/1347 objects degraded (28.359%); 13 KiB/s, 0 objects/s recovering 2024-09-20T19:27:21.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:21 smithi162 bash[24557]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:21 smithi162 bash[24557]: cluster 2024-09-20T19:27:20.329274+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e451: 8 total, 6 up, 8 in 2024-09-20T19:27:21.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:21.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:21.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:21.978 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:7.d 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:7.5 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:7.4 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:6.d 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:6.9 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:6.16 2024-09-20T19:27:21.990 INFO:teuthology.orchestra.run.smithi042.stdout:6.11 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:6.10 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:6.0 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.e 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.c 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:7.e 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.8 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.5 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.2 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:7.7 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.1e 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:5.1d 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:2.1f 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:2.1e 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:7.9 2024-09-20T19:27:21.991 INFO:teuthology.orchestra.run.smithi042.stdout:2.1 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:5.1b 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:2.19 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:4.1e 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:2.13 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:1.6 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:7.6 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:1.2 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:1.0 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:6.a 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:2.4 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:2.1d 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:1.1 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:2.15 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:5.3 2024-09-20T19:27:21.992 INFO:teuthology.orchestra.run.smithi042.stdout:4.b 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:2.5 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:2.c 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:2.10 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.19 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:3.1 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.0 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:7.f 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.16 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:6.13 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.3 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.1a 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.6 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:4.7 2024-09-20T19:27:21.993 INFO:teuthology.orchestra.run.smithi042.stdout:5.b 2024-09-20T19:27:21.994 INFO:teuthology.orchestra.run.smithi042.stdout:2.0 2024-09-20T19:27:21.994 INFO:teuthology.orchestra.run.smithi042.stdout:2.f 2024-09-20T19:27:21.994 INFO:teuthology.orchestra.run.smithi042.stdout:5.13 2024-09-20T19:27:21.994 INFO:teuthology.orchestra.run.smithi042.stdout:5.7 2024-09-20T19:27:21.994 INFO:teuthology.orchestra.run.smithi042.stdout:5.15 2024-09-20T19:27:22.000 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:22.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:22.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:22.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:22.333 INFO:tasks.rados.rados.0.smithi042.stdout:1543: done (3 left) 2024-09-20T19:27:22.334 INFO:tasks.rados.rados.0.smithi042.stdout:1544: done (2 left) 2024-09-20T19:27:22.334 INFO:tasks.rados.rados.0.smithi042.stdout:1545: write oid 28 current snap is 177 2024-09-20T19:27:22.334 INFO:tasks.rados.rados.0.smithi042.stdout:1545: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-20T19:27:22.345 INFO:tasks.rados.rados.0.smithi042.stdout:1545: writing smithi04268048-28 from 709394 to 1355415 tid 1 2024-09-20T19:27:22.358 INFO:tasks.rados.rados.0.smithi042.stdout:1545: writing smithi04268048-28 from 2083358 to 2801538 tid 2 2024-09-20T19:27:22.358 INFO:tasks.rados.rados.0.smithi042.stdout:1546: write oid 42 current snap is 177 2024-09-20T19:27:22.358 INFO:tasks.rados.rados.0.smithi042.stdout:1545: finishing write tid 1 to smithi04268048-28 2024-09-20T19:27:22.358 INFO:tasks.rados.rados.0.smithi042.stdout:1546: seq_num 543 ranges {726201=721270,2141840=578191} 2024-09-20T19:27:22.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:22.370 INFO:tasks.rados.rados.0.smithi042.stdout:1546: writing smithi04268048-42 from 726201 to 1447471 tid 1 2024-09-20T19:27:22.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:22 smithi162 bash[24557]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:22 smithi162 bash[24557]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:22 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:22 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.380 INFO:tasks.rados.rados.0.smithi042.stdout:1546: writing smithi04268048-42 from 2141840 to 2720031 tid 2 2024-09-20T19:27:22.381 INFO:tasks.rados.rados.0.smithi042.stdout:1547: write (excl) oid 11 current snap is 177 2024-09-20T19:27:22.381 INFO:tasks.rados.rados.0.smithi042.stdout:1547: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-20T19:27:22.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:22.393 INFO:tasks.rados.rados.0.smithi042.stdout:1547: writing smithi04268048-11 from 743008 to 1539527 tid 1 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1547: writing smithi04268048-11 from 2200322 to 2922171 tid 2 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1548: copy_from oid 20 from oid 26 current snap is 177 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: finishing write tid 2 to smithi04268048-28 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: finishing write tid 3 to smithi04268048-28 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: finishing write tid 4 to smithi04268048-28 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: oid 28 updating version 0 to 406 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: oid 28 updating version 406 to 408 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:1545: oid 28 version 408 is already newer than 407 2024-09-20T19:27:22.405 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 408 (ObjNum 542 snap 177 seq_num 542) dirty exists 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1545: left oid 28 (ObjNum 542 snap 177 seq_num 542) 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: finishing write tid 1 to smithi04268048-42 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: finishing write tid 2 to smithi04268048-42 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1545: done (5 left) 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1549: rollback oid 27 current snap is 177 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 170 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: finishing write tid 3 to smithi04268048-42 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: finishing write tid 4 to smithi04268048-42 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: oid 42 updating version 0 to 711 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: oid 42 updating version 711 to 712 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: oid 42 version 712 is already newer than 710 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 712 (ObjNum 543 snap 177 seq_num 543) dirty exists 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: left oid 42 (ObjNum 543 snap 177 seq_num 543) 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1547: finishing write tid 1 to smithi04268048-11 2024-09-20T19:27:22.406 INFO:tasks.rados.rados.0.smithi042.stdout:1546: done (5 left) 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1550: copy_from oid 34 from oid 2 current snap is 177 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1551: rollback oid 28 current snap is 177 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 154 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1552: read oid 16 snap -1 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1552: expect deleted 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1553: write oid 32 current snap is 177 2024-09-20T19:27:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:1553: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-20T19:27:22.413 INFO:tasks.rados.rados.0.smithi042.stdout:1553: writing smithi04268048-32 from 759815 to 1231583 tid 1 2024-09-20T19:27:22.421 INFO:tasks.rados.rados.0.smithi042.stdout:1553: writing smithi04268048-32 from 1742451 to 2207958 tid 2 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1553: writing smithi04268048-32 from 2917178 to 3159815 tid 3 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1554: delete oid 13 current snap is 177 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1548: finishing copy_from to smithi04268048-20 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1548: got expected ENOENT (src dne) 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1548: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1547: finishing write tid 2 to smithi04268048-11 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1547: finishing write tid 3 to smithi04268048-11 2024-09-20T19:27:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:1547: finishing write tid 4 to smithi04268048-11 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1547: oid 11 updating version 0 to 714 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1547: oid 11 updating version 714 to 716 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1547: oid 11 version 716 is already newer than 715 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 716 (ObjNum 544 snap 177 seq_num 544) dirty exists 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1547: left oid 11 (ObjNum 544 snap 177 seq_num 544) 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1549: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 683 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1551: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1551: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1551: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 411 (ObjNum 463 snap 151 seq_num 463) dirty exists 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1553: finishing write tid 1 to smithi04268048-32 2024-09-20T19:27:22.429 INFO:tasks.rados.rados.0.smithi042.stdout:1550: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:27:22.430 INFO:tasks.rados.rados.0.smithi042.stdout:1550: finishing copy_from to smithi04268048-34 2024-09-20T19:27:22.430 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 581 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:27:22.430 INFO:tasks.rados.rados.0.smithi042.stdout:1553: finishing write tid 2 to smithi04268048-32 2024-09-20T19:27:22.430 INFO:tasks.rados.rados.0.smithi042.stdout:1553: finishing write tid 3 to smithi04268048-32 2024-09-20T19:27:22.437 INFO:tasks.rados.rados.0.smithi042.stdout:1553: finishing write tid 4 to smithi04268048-32 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: finishing write tid 5 to smithi04268048-32 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: oid 32 updating version 0 to 928 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: oid 32 updating version 928 to 931 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: oid 32 version 931 is already newer than 929 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: oid 32 version 931 is already newer than 930 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 931 (ObjNum 545 snap 177 seq_num 545) dirty exists 2024-09-20T19:27:22.438 INFO:tasks.rados.rados.0.smithi042.stdout:1553: left oid 32 (ObjNum 545 snap 177 seq_num 545) 2024-09-20T19:27:22.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:22.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:22.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:22.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[22706]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[22706]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[32075]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[32075]: cluster 2024-09-20T19:27:21.335463+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e452: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:22 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.334259+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e453: 8 total, 6 up, 8 in 2024-09-20T19:27:22.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:22.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:22.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:22.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:23.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:23.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:23.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:23.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.413 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.414 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.415 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:23 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:23.533 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.097750+0000 mgr.y (mgr.24491) 359 : cluster [DBG] pgmap v617: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.502828+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.506334+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.507004+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:22.511421+0000 osd.3 (osd.3) 170 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:27:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.144559+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded (PG_DEGRADED) 2024-09-20T19:27:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:23 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.153116+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e454: 8 total, 6 up, 8 in 2024-09-20T19:27:23.561 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:23.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:23.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:23.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:23.822 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:23.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:23.954 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:23.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:24.056 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:24.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:24.212 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:24.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:24.343 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:24.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:24.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:24.564 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:24.577 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:24.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[32075]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:24 smithi042 bash[22706]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:24 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:24 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.491496+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub starts 2024-09-20T19:27:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:24 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:24 smithi162 bash[24557]: cluster 2024-09-20T19:27:23.499184+0000 osd.3 (osd.3) 172 : cluster [DBG] 7.a scrub ok 2024-09-20T19:27:24.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:24.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.1f --file /var/log/ceph/exp.2.1f.0 2024-09-20T19:27:24.736 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:24.797 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:27:24.815 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:27:24.815 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:27:24.835 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:27:24.835 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:27:24.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:24.855 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:27:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:27:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:27:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:24.876 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:27:24.877 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:27:24.898 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:27:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:27:24.920 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:27:24.920 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:27:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:27:24.960 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:24.978 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:24.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:25.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:25.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:25.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:25.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:25.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:27:25.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:25.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:25.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:25.617 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[22706]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:25 smithi042 bash[32075]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.098656+0000 mgr.y (mgr.24491) 360 : cluster [DBG] pgmap v620: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 293 KiB/s wr, 10 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: audit 2024-09-20T19:27:24.797030+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? v1:172.21.15.42:0/1515217582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.841624+0000 osd.2 (osd.2) 153 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: cluster 2024-09-20T19:27:24.842348+0000 osd.2 (osd.2) 154 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.731 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:25 smithi162 bash[24557]: audit 2024-09-20T19:27:25.251024+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:27:25.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:25.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:25.878 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:25.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:25.981 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:26.007 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:26.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.116 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:27:26.116 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:27:26.116 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:27:26.116 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:27:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:26.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:26.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.385 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:26.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[32075]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.515 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:26 smithi042 bash[22706]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: cluster 2024-09-20T19:27:25.375983+0000 mon.a (mon.0) 1543 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: audit 2024-09-20T19:27:25.377534+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? v1:172.21.15.42:0/3036995328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:26 smithi162 bash[24557]: cluster 2024-09-20T19:27:25.384418+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e455: 8 total, 6 up, 8 in 2024-09-20T19:27:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:27:26.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:27:26.634 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:26.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:26.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:26.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:27:26.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:26.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:26.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:27.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:27.069 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:27.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:27.197 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:27.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:27.326 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:27.429 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:27.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:27.573 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.573 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.573 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.573 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:27 smithi162 bash[24557]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:27.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[22706]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: cluster 2024-09-20T19:27:26.099731+0000 mgr.y (mgr.24491) 361 : cluster [DBG] pgmap v622: 158 pgs: 7 undersized+peered, 51 active+undersized, 3 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 53 active+clean; 486 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 246 KiB/s wr, 9 op/s; 384/1323 objects degraded (29.025%) 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: audit 2024-09-20T19:27:27.347939+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:27 smithi042 bash[32075]: audit 2024-09-20T19:27:27.350276+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:27.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:27.823 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:27.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:27.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:27.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:27.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:28.077 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:28.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:28.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:28.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:28.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:28.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:28.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:27:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:28.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:28 smithi042 bash[32075]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:28 smithi042 bash[32075]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:28 smithi042 bash[22706]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:28 smithi042 bash[22706]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:28.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:28 smithi162 bash[24557]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:28 smithi162 bash[24557]: cluster 2024-09-20T19:27:28.146911+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Degraded data redundancy: 384/1323 objects degraded (29.025%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:28.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:28.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:28.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:28.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:29.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.138 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:29.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:29.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:29.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:29.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:29 smithi162 bash[24557]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:29 smithi162 bash[24557]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:27:29.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:29.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:29 smithi042 bash[32075]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:29 smithi042 bash[32075]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:29 smithi042 bash[22706]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:29 smithi042 bash[22706]: cluster 2024-09-20T19:27:28.101062+0000 mgr.y (mgr.24491) 362 : cluster [DBG] pgmap v623: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:29.654 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:29.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:29.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:29.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:29.913 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:29.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:30.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:30.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:30.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.174 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:30.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:30.300 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:30.403 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:30.529 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.541 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:30 smithi042 bash[32075]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.541 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:30 smithi042 bash[32075]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:30 smithi042 bash[22706]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:30 smithi042 bash[22706]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.558 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:30.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:Exporting 2.1f info 2.1f( empty local-lis/les=69/70 n=0 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-09-20T19:27:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:Export successful 2024-09-20T19:27:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:30.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:30.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:30.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:30 smithi162 bash[24557]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:30 smithi162 bash[24557]: audit 2024-09-20T19:27:29.311929+0000 mgr.y (mgr.24491) 363 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:30.814 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:30.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:30.946 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:30.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:31.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.077 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:31.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:31.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:31.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:31.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:31.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:31.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:31 smithi162 bash[24557]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:31 smithi162 bash[24557]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:31.565 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.577 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:31 smithi042 bash[32075]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.577 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:31 smithi042 bash[32075]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:31 smithi042 bash[22706]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:31 smithi042 bash[22706]: cluster 2024-09-20T19:27:30.102507+0000 mgr.y (mgr.24491) 364 : cluster [DBG] pgmap v624: 158 pgs: 7 undersized+peered, 51 active+undersized, 2 active+clean+snaptrim, 7 undersized+degraded+peered, 37 active+undersized+degraded, 54 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:31.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:31.681 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:31.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:31.725 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:31.829 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.857 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:31.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:31.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:31.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.1f 2024-09-20T19:27:32.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:27:32.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:32.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:32.770 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:32.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:32.901 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:32.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:33.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:33.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.148 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:27:33.148 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:27:33.148 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-20T19:27:33.149 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-20T19:27:33.149 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:27:33.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:33.263 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:33.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:33.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:33.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:33 smithi162 bash[24557]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:33 smithi162 bash[24557]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:33 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:33 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[22706]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[22706]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[32075]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[32075]: cluster 2024-09-20T19:27:32.104936+0000 mgr.y (mgr.24491) 365 : cluster [DBG] pgmap v625: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 917 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:33.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:33 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.148631+0000 mon.a (mon.0) 1549 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-20T19:27:33.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:33.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:33.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:33.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:27:33.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:33.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:34.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:34.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:34.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:34.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:34.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:34.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:34.442 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.442 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[32075]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:34 smithi042 bash[22706]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:34.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:34.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:34.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:34 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:34 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.778463+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:34.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:34 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:34 smithi162 bash[24557]: cluster 2024-09-20T19:27:33.780677+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:34.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:34.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:34.842 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:34.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:34.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:35.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:35.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:35.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:35.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:35.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:35 smithi162 bash[24557]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:35 smithi162 bash[24557]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:35 smithi162 bash[24557]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:35 smithi162 bash[24557]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:35.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.748 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:35.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:35.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[22706]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[22706]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[22706]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[22706]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[32075]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[32075]: cluster 2024-09-20T19:27:34.106468+0000 mgr.y (mgr.24491) 366 : cluster [DBG] pgmap v626: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:35.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[32075]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:35 smithi042 bash[32075]: cluster 2024-09-20T19:27:34.441155+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e456: 8 total, 6 up, 8 in 2024-09-20T19:27:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:35.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:35.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:36.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:36.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:36.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:36.381 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:36.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:36.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:36.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.660 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:36.763 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:36.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:36.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:37.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:37.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:37.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:37.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:37.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:37.412 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:37.441 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:37.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:37.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:37.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:37 smithi042 bash[32075]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:37 smithi042 bash[32075]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:37 smithi042 bash[22706]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:37 smithi042 bash[22706]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:37.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:37 smithi162 bash[24557]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:37 smithi162 bash[24557]: cluster 2024-09-20T19:27:36.107919+0000 mgr.y (mgr.24491) 367 : cluster [DBG] pgmap v628: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 820 KiB/s wr, 3 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:37.804 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.824 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:37.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:37.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:37.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:38.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:38.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:38.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:38.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:38.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:27:38.360 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:27:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:27:38.380 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:27:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:27:38.397 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:27:38.397 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:27:38.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:27:38.417 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:27:38.418 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:27:38.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.441 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:27:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:27:38.460 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:27:38.460 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:27:38.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:38.475 INFO:teuthology.orchestra.run.smithi042.stdout: marking collection for removal 2024-09-20T19:27:38.475 INFO:teuthology.orchestra.run.smithi042.stdout:setting '_remove' omap key 2024-09-20T19:27:38.475 INFO:teuthology.orchestra.run.smithi042.stdout:finish_remove_pgs 2.1f_head removing 2.1f 2024-09-20T19:27:38.475 INFO:teuthology.orchestra.run.smithi042.stdout:Remove successful 2024-09-20T19:27:38.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:38.668 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:27:38.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:38.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:38.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:38.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:38.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:39.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:39.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:39.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:39.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:39.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:39.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:39.354 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.354 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.354 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.355 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:39 smithi162 bash[24557]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:39.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[22706]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: cluster 2024-09-20T19:27:38.109561+0000 mgr.y (mgr.24491) 368 : cluster [DBG] pgmap v629: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: cluster 2024-09-20T19:27:38.156994+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e457: 8 total, 6 up, 8 in 2024-09-20T19:27:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: audit 2024-09-20T19:27:38.342059+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? v1:172.21.15.42:0/3227318165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:39 smithi042 bash[32075]: audit 2024-09-20T19:27:38.523077+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:27:39.439 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:27:39.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:27:39.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:27:39.596 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:39.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:39.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:39.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:39.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:39.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:39.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:40.009 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:40.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:40.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.256 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:40.271 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:27:40.271 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[22706]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.345 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.346 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:40 smithi042 bash[32075]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: audit 2024-09-20T19:27:39.319914+0000 mgr.y (mgr.24491) 369 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: cluster 2024-09-20T19:27:39.437441+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: audit 2024-09-20T19:27:39.439899+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? v1:172.21.15.42:0/1508589360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:40 smithi162 bash[24557]: cluster 2024-09-20T19:27:39.448057+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e458: 8 total, 6 up, 8 in 2024-09-20T19:27:40.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:40.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:40.511 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:40.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:40.643 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:40.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:40.774 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:40.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:40.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:40.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:41.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:41.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:27:41.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:41.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.4 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-20T19:27:41.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:41.258 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.4/config 2024-09-20T19:27:41.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:41.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:41.419 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:41.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:41 smithi162 bash[24557]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:41 smithi162 bash[24557]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:27:41.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:41.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:41 smithi042 bash[22706]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:41 smithi042 bash[22706]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:41 smithi042 bash[32075]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:41 smithi042 bash[32075]: cluster 2024-09-20T19:27:40.111023+0000 mgr.y (mgr.24491) 370 : cluster [DBG] pgmap v632: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:41.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:41.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:41.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:41.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:41.932 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:41.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:42.034 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:42.063 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:42.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:42.194 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:42.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:42.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:42.451 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:42.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:42.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.704 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:42 smithi162 bash[24557]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:42.808 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:42.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.819 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[32075]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.429676+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.431272+0000 osd.3 (osd.3) 174 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:42.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: cluster 2024-09-20T19:27:41.453773+0000 mon.a (mon.0) 1557 : cluster [DBG] osdmap e459: 8 total, 6 up, 8 in 2024-09-20T19:27:42.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:42 smithi042 bash[22706]: audit 2024-09-20T19:27:42.379961+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:42.835 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:42.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:42.966 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:42.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:43.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:43.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:43.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:43.316 INFO:teuthology.orchestra.run.smithi162.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:43.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:43.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:43.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:27:43.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:43.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:43 smithi162 bash[24557]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:43 smithi162 bash[24557]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:43.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:43.600 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:43.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:43 smithi042 bash[22706]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:43 smithi042 bash[22706]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:43 smithi042 bash[32075]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:43 smithi042 bash[32075]: cluster 2024-09-20T19:27:42.113408+0000 mgr.y (mgr.24491) 371 : cluster [DBG] pgmap v634: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:43.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:43.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:43.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:43.965 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:43.993 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:43.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:44.096 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:44.124 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:44.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:44.256 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:44.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:44.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:44.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:44.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:44.641 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:44.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:44.771 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:44.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:44.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:45.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.008 INFO:teuthology.orchestra.run.smithi162.stdout:7.9 2024-09-20T19:27:45.008 INFO:teuthology.orchestra.run.smithi162.stdout:4.5 2024-09-20T19:27:45.008 INFO:teuthology.orchestra.run.smithi162.stdout:4.1 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:2.1f 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.e 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.2 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.9 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:3.1 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:5.a 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:1.3 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:1.2 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.18 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.1b 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.c 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:4.b 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:2.5 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:2.11 2024-09-20T19:27:45.009 INFO:teuthology.orchestra.run.smithi162.stdout:6.12 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:5.1d 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:4.13 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:1.5 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:5.0 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:2.6 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:6.15 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:4.f 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:4.d 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:4.1d 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:7.3 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:2.f 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:5.10 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:2.d 2024-09-20T19:27:45.010 INFO:teuthology.orchestra.run.smithi162.stdout:5.11 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.18 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.1a 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:6.1d 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:2.18 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.1c 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.1f 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.5 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.9 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.c 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:6.17 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:2.13 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.e 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:5.f 2024-09-20T19:27:45.011 INFO:teuthology.orchestra.run.smithi162.stdout:6.1e 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.14 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.6 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:2.b 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.1a 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.2 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:4.1a 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:7.8 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.7 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.e 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:6.f 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:2.1a 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:7.1 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:5.1 2024-09-20T19:27:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:7.4 2024-09-20T19:27:45.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:45.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:45.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:45.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:45.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:45.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.518 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:45 smithi162 bash[24557]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:45.518 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:45 smithi162 bash[24557]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:45.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:45.634 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:45.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.789 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:45.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:45.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:45.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:45 smithi042 bash[32075]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:45.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:45 smithi042 bash[32075]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:45.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:45 smithi042 bash[22706]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:45.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:45 smithi042 bash[22706]: cluster 2024-09-20T19:27:44.114910+0000 mgr.y (mgr.24491) 372 : cluster [DBG] pgmap v635: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:46.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:46.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.181 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:46.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.1f.0 2024-09-20T19:27:46.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:46.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:27:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:46.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:46.565 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:46.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:46.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:46.798 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.809 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:46.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:46.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:47.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:47.087 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:47.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:47.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:47.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:47.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:47.480 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:47.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:47.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:47.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:47 smithi162 bash[24557]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:47 smithi162 bash[24557]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:47.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.866 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:47.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:47.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:47.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:47 smithi042 bash[32075]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:47 smithi042 bash[32075]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:47 smithi042 bash[22706]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:47 smithi042 bash[22706]: cluster 2024-09-20T19:27:46.116347+0000 mgr.y (mgr.24491) 373 : cluster [DBG] pgmap v636: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:47.974 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:48.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:48.134 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:48.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:48.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:48.371 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:48.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:48.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[22706]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[22706]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[22706]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[22706]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:48.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:48 smithi162 bash[24557]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:48 smithi162 bash[24557]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:48 smithi162 bash[24557]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:48 smithi162 bash[24557]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:48.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:48.896 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[32075]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.896 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[32075]: cluster 2024-09-20T19:27:47.541291+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:27:48.896 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[32075]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.897 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:48 smithi042 bash[32075]: cluster 2024-09-20T19:27:47.541947+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:27:48.914 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:48.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:49.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:49.041 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:49.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:49.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:49.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:49.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:49.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:49 smithi162 bash[24557]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:49 smithi162 bash[24557]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:49 smithi162 bash[24557]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:49 smithi162 bash[24557]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:49.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.683 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:49.786 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:49.797 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[32075]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.797 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[32075]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.797 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[32075]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[32075]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[22706]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[22706]: cluster 2024-09-20T19:27:48.118714+0000 mgr.y (mgr.24491) 374 : cluster [DBG] pgmap v637: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:49.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[22706]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:49 smithi042 bash[22706]: cluster 2024-09-20T19:27:48.480085+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e460: 8 total, 6 up, 8 in 2024-09-20T19:27:49.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:49.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:49.945 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:49.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:50.049 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:50.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:50.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:50.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.334 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:50.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:50.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:50.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.578 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:50 smithi042 bash[22706]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.578 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:50 smithi042 bash[22706]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:50 smithi042 bash[32075]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:50 smithi042 bash[32075]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.592 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:50.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:50 smithi162 bash[24557]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:50 smithi162 bash[24557]: audit 2024-09-20T19:27:49.327891+0000 mgr.y (mgr.24491) 375 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:27:50.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:50.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:50.880 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:27:50.881 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:27:50.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:50.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:50.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:51.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.103 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:51.118 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:27:51.136 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:27:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:27:51.154 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:27:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:27:51.171 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:27:51.172 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:27:51.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:27:51.193 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:27:51.194 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:27:51.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.216 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:27:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:27:51.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:51.239 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:27:51.239 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:27:51.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:51.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:51.458 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:51.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:51.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:51.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:51.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:51.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:51 smithi162 bash[24557]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:51 smithi162 bash[24557]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:51 smithi162 bash[24557]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:51 smithi162 bash[24557]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:51.845 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[22706]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[22706]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[22706]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[22706]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[32075]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[32075]: cluster 2024-09-20T19:27:50.119995+0000 mgr.y (mgr.24491) 376 : cluster [DBG] pgmap v639: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:51.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[32075]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.858 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:51 smithi042 bash[32075]: audit 2024-09-20T19:27:51.117679+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? v1:172.21.15.42:0/292122576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:27:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:51.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:51.977 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:52.005 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:52.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:52.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:52.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:52.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.396 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:52.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:52.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:52.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:52.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.787 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:52.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:52.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:52.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:53.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.045 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:53.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:53.157 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:27:53.177 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:53.283 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:53.301 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:53.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:53.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:53.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:53.594 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:27:53.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:53 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:53.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.808 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.808 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:52.122258+0000 mgr.y (mgr.24491) 377 : cluster [DBG] pgmap v640: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: audit 2024-09-20T19:27:52.845887+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.155940+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: audit 2024-09-20T19:27:53.158393+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? v1:172.21.15.42:0/765119759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:53 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.165273+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e461: 8 total, 6 up, 8 in 2024-09-20T19:27:53.822 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:53.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:53.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:54.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:54.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:54.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:54.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:27:54.218 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:27:54.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:54.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:54.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:27:54.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:54 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:54.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:54.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.897986+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:53.899736+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.252076+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:54 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.253844+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:27:54.837 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:54.968 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:54.995 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:54.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:55.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:55.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:55.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:55.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:55.386 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:55.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[32075]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:55 smithi042 bash[22706]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:55.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:55.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:55.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.123686+0000 mgr.y (mgr.24491) 378 : cluster [DBG] pgmap v642: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.464507+0000 osd.5 (osd.5) 161 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.474386+0000 osd.5 (osd.5) 162 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.665264+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.666880+0000 osd.6 (osd.6) 124 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.932252+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:55 smithi162 bash[24557]: cluster 2024-09-20T19:27:54.933844+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-09-20T19:27:55.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:55.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:55.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:55.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:56.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:56.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:56.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:56.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:27:56.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:56.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:56.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:56.512 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:27:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.516 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:56.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:56.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:56 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:56.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:56.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:55.359210+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:55.361501+0000 osd.7 (osd.7) 134 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.227331+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.c scrub starts 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:56 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.229496+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.c scrub ok 2024-09-20T19:27:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:56.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:56.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:56.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:57.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:57.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:57.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:57.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:27:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:27:57.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:57.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:57.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.468 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:57.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:57.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[32075]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:57 smithi042 bash[22706]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:57.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: cluster 2024-09-20T19:27:56.125131+0000 mgr.y (mgr.24491) 379 : cluster [DBG] pgmap v643: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: audit 2024-09-20T19:27:57.336401+0000 mon.a (mon.0) 1565 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:27:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:57 smithi162 bash[24557]: audit 2024-09-20T19:27:57.408867+0000 mon.a (mon.0) 1566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:27:57.806 INFO:teuthology.orchestra.run.smithi042.stdout:get_pg_num_history pg_num_history pg_num_history(e393 pg_nums {1={20=1,259=11},2={69=32},3={70=3},4={71=32},5={73=32},6={75=32},7={112=16}} deleted_pools ) 2024-09-20T19:27:57.806 INFO:teuthology.orchestra.run.smithi042.stdout:Importing pgid 2.1f 2024-09-20T19:27:57.807 INFO:teuthology.orchestra.run.smithi042.stdout:write_pg epoch 384 info 2.1f( empty local-lis/les=69/70 n=0 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-09-20T19:27:57.807 INFO:teuthology.orchestra.run.smithi042.stdout:Import successful 2024-09-20T19:27:57.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:27:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:27:57.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:27:57.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:57.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:58.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:58.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:58.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.213 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:58.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:58.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:58.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:58.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:58.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:58 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:58.703 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.066934+0000 mon.a (mon.0) 1567 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.067716+0000 mon.a (mon.0) 1568 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: audit 2024-09-20T19:27:58.072974+0000 mon.a (mon.0) 1569 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.279323+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:58 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.281407+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:27:58.833 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:58.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:58.990 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:58.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:59.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:59.120 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:27:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:27:59.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:59.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:59.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/exp.2.1f.0 2024-09-20T19:27:59.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.380 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:59.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:59.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:27:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:27:59.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:27:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:27:59.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:59.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:59 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:27:59 smithi162 bash[24557]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:27:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:27:59.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:59 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:27:59 smithi042 bash[22706]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:59 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:27:59 smithi042 bash[32075]: cluster 2024-09-20T19:27:58.126190+0000 mgr.y (mgr.24491) 380 : cluster [DBG] pgmap v644: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:27:59.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:27:59.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:27:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:27:59.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:00.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:00.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:00.163 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:00.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:00.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:00.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:00.424 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:00.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:00.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[22706]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:00 smithi042 bash[32075]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:00.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:00.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: audit 2024-09-20T19:27:59.338874+0000 mgr.y (mgr.24491) 381 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:00.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:59.343144+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:28:00.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:00 smithi162 bash[24557]: cluster 2024-09-20T19:27:59.345388+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:28:00.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:00.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:00.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:01.045 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:01.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:01.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.202 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:28:01.203 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:28:01.203 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:28:01.203 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:28:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:28:01.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:01.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:01.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:01.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:01.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:01.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[32075]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:01 smithi042 bash[22706]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.127542+0000 mgr.y (mgr.24491) 382 : cluster [DBG] pgmap v645: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.415085+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:00.417252+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:01.259459+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:01 smithi162 bash[24557]: cluster 2024-09-20T19:28:01.260482+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:28:01.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:01.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:01.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:01.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:02.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:28:02.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:02.222 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:02.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:02.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:02.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:02.608 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:02.738 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:02.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:02.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:03.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:03.131 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:03.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:03.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:03.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.419 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:03.512 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:03.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:03.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:03.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[32075]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:03 smithi042 bash[22706]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.129867+0000 mgr.y (mgr.24491) 383 : cluster [DBG] pgmap v646: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.446938+0000 osd.2 (osd.2) 157 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.451844+0000 osd.2 (osd.2) 158 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.695913+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:03 smithi162 bash[24557]: cluster 2024-09-20T19:28:02.698038+0000 osd.6 (osd.6) 126 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:28:03.813 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:03.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:03.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:03.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:04.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:04.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:04.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.330 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 4] live_osds: [1, 2, 3, 5, 6, 7] 2024-09-20T19:28:04.331 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:28:04.331 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-20T19:28:04.331 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T19:28:04.331 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T19:28:04.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-20T19:28:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:04.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:04.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:04.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:04.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[22706]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:04 smithi042 bash[32075]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:04.708 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:04.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.364391+0000 osd.5 (osd.5) 165 : cluster [DBG] 1.a scrub starts 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.366351+0000 osd.5 (osd.5) 166 : cluster [DBG] 1.a scrub ok 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.708433+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:04 smithi162 bash[24557]: cluster 2024-09-20T19:28:03.711361+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-09-20T19:28:04.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.838 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:04.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:04.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:04.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:28:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:28:04.999 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:28:05.016 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:28:05.017 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:28:05.035 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:28:05.035 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:28:05.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:28:05.056 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:28:05.056 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:28:05.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:05.078 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:28:05.079 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:28:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:05.101 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:28:05.101 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:28:05.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:05.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:05.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:05.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:05.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:05.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:05.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:05 smithi162 bash[24557]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:05 smithi162 bash[24557]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:05 smithi162 bash[24557]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:05 smithi162 bash[24557]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:05.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[22706]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[22706]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[22706]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[22706]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[32075]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[32075]: cluster 2024-09-20T19:28:04.130662+0000 mgr.y (mgr.24491) 384 : cluster [DBG] pgmap v647: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[32075]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:05 smithi042 bash[32075]: audit 2024-09-20T19:28:04.980597+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.42:0/1012358867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:05.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:05.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:05.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:05.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:05.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:05.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:06.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:06.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:06.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:06.246 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:06.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:06.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.493 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:06.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:28:06.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:28:06.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.630 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:06.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:06.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:06.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:06.888 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:06.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:06.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:07.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:07.121 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:07.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:07.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:07.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:07.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:07.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:07.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:07.558 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:28:07.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:07 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.656 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:07.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.132042+0000 mgr.y (mgr.24491) 385 : cluster [DBG] pgmap v648: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.697081+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.699537+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.894544+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:06.895898+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:28:07.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: audit 2024-09-20T19:28:06.899968+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.42:0/1382836698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: audit 2024-09-20T19:28:06.900291+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25268 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.244942+0000 osd.3 (osd.3) 183 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:07 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.247973+0000 osd.3 (osd.3) 184 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:28:07.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:07.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:07.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:07.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:08.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:08.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:08.149 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.182 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:08.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:08.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:08.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:08.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:08.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:08.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:28:08.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:08.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:08.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:08 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.545541+0000 mon.a (mon.0) 1571 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: audit 2024-09-20T19:28:07.547629+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25268 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:07.557211+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e462: 8 total, 6 up, 8 in 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.174710+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e463: 8 total, 6 up, 8 in 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.346335+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.d scrub starts 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:08 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.349094+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.d scrub ok 2024-09-20T19:28:08.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:08.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:09.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:09.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.219 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:09.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:09.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:09.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:09.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:09.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:09 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:09 smithi162 bash[24557]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:09.613 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:09.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:09 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:09 smithi042 bash[32075]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.725 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:09 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.725 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:09 smithi042 bash[22706]: cluster 2024-09-20T19:28:08.133663+0000 mgr.y (mgr.24491) 386 : cluster [DBG] pgmap v650: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:09.737 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:09.784 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:28:09.784 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:28:09.840 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:09.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:09.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.987 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:09.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:10.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:10.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:10.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:10.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.512 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:10.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:10.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:10 smithi042 bash[32075]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:10.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:10 smithi042 bash[32075]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:10.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:10 smithi042 bash[22706]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:10.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:10 smithi042 bash[22706]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:10.747 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.773 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:28:10.775 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:10.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:10.903 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:10.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:11.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:11.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:10 smithi162 bash[24557]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:11.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:10 smithi162 bash[24557]: audit 2024-09-20T19:28:09.347677+0000 mgr.y (mgr.24491) 387 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:11.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:11.240 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:11.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:11.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:11.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:28:11.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:11.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:11.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:11 smithi042 bash[22706]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:11 smithi042 bash[22706]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.677 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:11.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:11 smithi162 bash[24557]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:11 smithi162 bash[24557]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.806 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:11.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:11.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:11 smithi042 bash[32075]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:11 smithi042 bash[32075]: cluster 2024-09-20T19:28:10.135188+0000 mgr.y (mgr.24491) 388 : cluster [DBG] pgmap v652: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:11.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:11.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:12.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:12.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:12.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:12.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:12.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:12.323 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:12.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:12.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:12.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:12.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:12.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[22706]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:12 smithi042 bash[32075]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:12.843 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:12.924 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:28:12.924 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T19:28:12.924 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:28:12.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.168523+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.171719+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: audit 2024-09-20T19:28:12.437595+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:12 smithi162 bash[24557]: audit 2024-09-20T19:28:12.440350+0000 mon.a (mon.0) 1576 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:12.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:12.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:13.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:13.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:13.228 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:13.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:13.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:13.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:13.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:13.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:13.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:13.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:13 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:13.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.137484+0000 mgr.y (mgr.24491) 389 : cluster [DBG] pgmap v653: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:12.597040+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e464: 8 total, 6 up, 8 in 2024-09-20T19:28:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:13 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.175522+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e465: 8 total, 6 up, 8 in 2024-09-20T19:28:13.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:14.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:14.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:14.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:14.245 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:14.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:14.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:14.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[32075]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:14 smithi042 bash[22706]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:14.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.799 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:14.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:14 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:14 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.306147+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:28:14.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:14 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:14 smithi162 bash[24557]: cluster 2024-09-20T19:28:13.311019+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:28:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:14.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:14.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:15.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:15.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:15.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:15.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:15.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:15.302 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:15.302 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:15.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:15.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:15.554 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:15.581 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:15.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:15.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:15 smithi162 bash[24557]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:15.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[32075]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.138997+0000 mgr.y (mgr.24491) 390 : cluster [DBG] pgmap v656: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.593551+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.940602+0000 osd.1 (osd.1) 181 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:15 smithi042 bash[22706]: cluster 2024-09-20T19:28:14.942052+0000 osd.1 (osd.1) 182 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:28:15.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:15.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:16.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:16.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:16.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:16.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:16.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:16.496 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:16.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:16.623 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:16.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:16.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:16.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:16.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[22706]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[22706]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[22706]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[22706]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[32075]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[32075]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[32075]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:16.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:16 smithi042 bash[32075]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:16.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:16.998 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:16 smithi162 bash[24557]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.999 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:16 smithi162 bash[24557]: cluster 2024-09-20T19:28:16.140754+0000 mgr.y (mgr.24491) 391 : cluster [DBG] pgmap v657: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 642 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:16.999 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:16 smithi162 bash[24557]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:16.999 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:16 smithi162 bash[24557]: cluster 2024-09-20T19:28:16.607337+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check update: Degraded data redundancy: 386/1329 objects degraded (29.044%), 44 pgs degraded, 102 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:17.016 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:17.119 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:28:17.167 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:28:17.167 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:28:17.185 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:28:17.185 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:28:17.202 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:28:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:28:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:28:17.224 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:28:17.224 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:28:17.244 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.245 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:28:17.245 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:28:17.266 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:28:17.266 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:28:17.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:17.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:17.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:17.525 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:17.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:17.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:17.783 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:17.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:17.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:17.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:17 smithi162 bash[24557]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:17.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:17 smithi162 bash[24557]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:17.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:17.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:17 smithi042 bash[22706]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:17.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:17 smithi042 bash[22706]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:17.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:17 smithi042 bash[32075]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:17.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:17 smithi042 bash[32075]: audit 2024-09-20T19:28:17.149824+0000 mon.a (mon.0) 1581 : audit [DBG] from='client.? v1:172.21.15.42:0/3276247277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:18.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:18.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:18.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:18.273 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:18.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:18.434 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:18.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:18.563 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:18.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:18.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:18.757 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 4] live_osds: [1, 2, 3, 5, 6, 7] 2024-09-20T19:28:18.757 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:28:18.757 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-20T19:28:18.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:28:18.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.828 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:18.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.954 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:18.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:19.057 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:19.188 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:28:19.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:28:19.197 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.198 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[32075]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:19 smithi042 bash[22706]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.212 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:19.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:19.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:19.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:19.457 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:19.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.143548+0000 mgr.y (mgr.24491) 392 : cluster [DBG] pgmap v658: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:19.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.240990+0000 osd.3 (osd.3) 185 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:28:19.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:19 smithi162 bash[24557]: cluster 2024-09-20T19:28:18.243960+0000 osd.3 (osd.3) 186 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:28:19.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:19.570 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:28:19.570 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:28:19.570 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:28:19.570 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:28:19.577 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.598 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:19.701 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:19.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:19.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:19.849 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:19.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:19.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:19.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:20.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:20.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:28:20.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:20.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:20 smithi162 bash[24557]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:20.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:20.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:20.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[32075]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.355470+0000 mgr.y (mgr.24491) 393 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.804324+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.42:0/4112346843' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:20 smithi042 bash[22706]: audit 2024-09-20T19:28:19.804674+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25280 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:28:20.624 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:20.727 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:20.763 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:28:20.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:20.881 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:20.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:20.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:21.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:21.133 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:21.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:28:21.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:21.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:21.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:21.393 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:21.420 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[32075]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.421 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.422 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:21 smithi042 bash[22706]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:21.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.145310+0000 mgr.y (mgr.24491) 394 : cluster [DBG] pgmap v659: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.169846+0000 mon.a (mon.0) 1583 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: audit 2024-09-20T19:28:20.172015+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:21 smithi162 bash[24557]: cluster 2024-09-20T19:28:20.183161+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e466: 8 total, 6 up, 8 in 2024-09-20T19:28:21.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:21.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:21.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:21.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:21.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:22.021 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:22.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:22.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:22.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:22.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:22.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:22.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:22.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:22.723 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:22.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:22.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:22.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:22.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:22.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:23.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:23.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:23.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:23.240 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:23.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:23 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:23.369 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:23.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:28:23.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:21.473055+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:22.148015+0000 mgr.y (mgr.24491) 395 : cluster [DBG] pgmap v661: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:23 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.178255+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e467: 8 total, 6 up, 8 in 2024-09-20T19:28:23.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:23.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:23.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:23.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:23.885 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:23.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:23.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:24.016 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:24.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:24.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:24.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.407 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:24.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:24.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:24.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:24.665 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:24.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:24.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:24.897 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:24.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:24.959 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:25.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:25.052 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:25.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:25.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:25.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:25.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:25.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:25.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[22706]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:25 smithi042 bash[32075]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.445 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:25.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:25.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.572571+0000 osd.7 (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:23.574433+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:24.149158+0000 mgr.y (mgr.24491) 396 : cluster [DBG] pgmap v663: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:25 smithi162 bash[24557]: cluster 2024-09-20T19:28:24.197270+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e468: 8 total, 6 up, 8 in 2024-09-20T19:28:25.575 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:25.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:25.706 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:25.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:25.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:25.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:25.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:25.971 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:25.972 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:25.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:26.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:26.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:26.230 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:26.333 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:26.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:26.490 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:26.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:26.604 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:26.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:26.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:26.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:26.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:26.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:26.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:26.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:27.013 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:27.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:27.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:27.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:27.379 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:27.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:27 smithi042 bash[22706]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:27 smithi042 bash[22706]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:27 smithi042 bash[32075]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.391 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:27 smithi042 bash[32075]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:27.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:27.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:27 smithi162 bash[24557]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:27 smithi162 bash[24557]: cluster 2024-09-20T19:28:26.150935+0000 mgr.y (mgr.24491) 397 : cluster [DBG] pgmap v665: 158 pgs: 1 active+clean+scrubbing, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 55 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:27.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:27.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.668 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:27.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:27.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:27.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:27.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:28.007 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:28.032 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:28.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:28.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:28.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:28.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:28.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:28.422 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:28:28.425 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:28:28.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:28.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:28.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:28.691 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:28.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[32075]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.693 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.694 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:28 smithi042 bash[22706]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:28.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:28.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: audit 2024-09-20T19:28:27.468157+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:27.468258+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: audit 2024-09-20T19:28:27.469894+0000 mon.a (mon.0) 1589 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:27.470747+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:28 smithi162 bash[24557]: cluster 2024-09-20T19:28:28.179115+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e469: 8 total, 6 up, 8 in 2024-09-20T19:28:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:28.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:28.954 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:28.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:29.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:29.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:29.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:29.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:29.445 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:29.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:29.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:29.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:29.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:29.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:29 smithi162 bash[24557]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:29 smithi162 bash[24557]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:29 smithi162 bash[24557]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:29 smithi162 bash[24557]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[22706]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[22706]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[22706]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[22706]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[32075]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[32075]: cluster 2024-09-20T19:28:28.152173+0000 mgr.y (mgr.24491) 398 : cluster [DBG] pgmap v666: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[32075]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:29 smithi042 bash[32075]: audit 2024-09-20T19:28:28.397410+0000 mgr.y (mgr.24491) 399 : audit [DBG] from='client.15432 v1:172.21.15.42:0/3222170934' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:28:29.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:29.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:29.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:30.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:30.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:30.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:30.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.262 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:30.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:30.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:30 smithi042 bash[22706]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:30 smithi042 bash[22706]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:30 smithi042 bash[32075]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:30 smithi042 bash[32075]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:30.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:30.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:30.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:30.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:30 smithi162 bash[24557]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:30 smithi162 bash[24557]: audit 2024-09-20T19:28:29.363781+0000 mgr.y (mgr.24491) 400 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:30.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:30.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:30.914 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:30.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:31.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:31.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:31.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:31.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.306 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:31.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:31.435 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:31.456 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-09-20T19:28:28.172793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":514950552,"num_objects":443,"num_object_clones":194,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":14,"num_read":4604,"num_read_kb":2436245,"num_write":2764,"num_write_kb":867783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":8743337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8200,"ondisk_log_size":8200,"up":358,"acting":358,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1365128,"kb_used_data":1094808,"kb_used_omap":12,"kb_used_meta":270131,"kb_avail":748596088,"statfs":{"total":767960285184,"available":766562394112,"internally_reserved":0,"allocated":1121083392,"data_stored":1508663652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":276614704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.027168"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":829,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.877784+0000","last_change":"2024-09-20T19:24:03.086154+0000","last_active":"2024-09-20T19:28:24.877784+0000","last_peered":"2024-09-20T19:28:24.877784+0000","last_clean":"2024-09-20T19:28:24.877784+0000","last_became_active":"2024-09-20T19:18:13.492719+0000","last_became_peered":"2024-09-20T19:18:13.492719+0000","last_unstale":"2024-09-20T19:28:24.877784+0000","last_undegraded":"2024-09-20T19:28:24.877784+0000","last_fullsized":"2024-09-20T19:28:24.877784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:03.085896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:03.085896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:38.397327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":159,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.205516+0000","last_change":"2024-09-20T19:27:14.296221+0000","last_active":"2024-09-20T19:27:13.293706+0000","last_peered":"2024-09-20T19:28:24.205516+0000","last_clean":"2024-09-20T19:18:06.213049+0000","last_became_active":"2024-09-20T19:26:24.157343+0000","last_became_peered":"2024-09-20T19:27:14.296221+0000","last_unstale":"2024-09-20T19:28:24.205516+0000","last_undegraded":"2024-09-20T19:28:24.205516+0000","last_fullsized":"2024-09-20T19:27:14.293485+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.967081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.967081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"465'17","reported_seq":869,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.878610+0000","last_change":"2024-09-20T19:26:24.175483+0000","last_active":"2024-09-20T19:28:24.878610+0000","last_peered":"2024-09-20T19:28:24.878610+0000","last_clean":"2024-09-20T19:26:22.379191+0000","last_became_active":"2024-09-20T19:26:24.175483+0000","last_became_peered":"2024-09-20T19:26:24.175483+0000","last_unstale":"2024-09-20T19:28:24.878610+0000","last_undegraded":"2024-09-20T19:26:24.149911+0000","last_fullsized":"2024-09-20T19:26:24.149790+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:25:18.620430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.620430+0000","objects_scrubbed":7,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:13.301452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.484315+0000","last_change":"2024-09-20T19:27:14.316944+0000","last_active":"2024-09-20T19:28:24.484315+0000","last_peered":"2024-09-20T19:28:24.484315+0000","last_clean":"2024-09-20T19:18:10.268062+0000","last_became_active":"2024-09-20T19:27:14.316944+0000","last_became_peered":"2024-09-20T19:27:14.316944+0000","last_unstale":"2024-09-20T19:28:24.484315+0000","last_undegraded":"2024-09-20T19:28:24.484315+0000","last_fullsized":"2024-09-20T19:27:14.295935+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:50.236477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.236477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:43.648971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":846,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.881067+0000","last_change":"2024-09-20T19:26:24.166470+0000","last_active":"2024-09-20T19:28:24.881067+0000","last_peered":"2024-09-20T19:28:24.881067+0000","last_clean":"2024-09-20T19:26:22.380061+0000","last_became_active":"2024-09-20T19:26:24.166470+0000","last_became_peered":"2024-09-20T19:26:24.166470+0000","last_unstale":"2024-09-20T19:28:24.881067+0000","last_undegraded":"2024-09-20T19:28:24.881067+0000","last_fullsized":"2024-09-20T19:26:24.151721+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.631202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.631202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'11","reported_seq":95,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.484065+0000","last_change":"2024-09-20T19:27:14.320521+0000","last_active":"2024-09-20T19:28:24.484065+0000","last_peered":"2024-09-20T19:28:24.484065+0000","last_clean":"2024-09-20T19:18:14.569805+0000","last_became_active":"2024-09-20T19:27:14.320521+0000","last_became_peered":"2024-09-20T19:27:14.320521+0000","last_unstale":"2024-09-20T19:28:24.484065+0000","last_undegraded":"2024-09-20T19:27:14.297343+0000","last_fullsized":"2024-09-20T19:27:14.297029+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:23:47.200185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.200185+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.540652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.198649+0000","last_change":"2024-09-20T19:25:23.446713+0000","last_active":"2024-09-20T19:28:24.198649+0000","last_peered":"2024-09-20T19:28:24.198649+0000","last_clean":"2024-09-20T19:28:24.198649+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:28:24.198649+0000","last_undegraded":"2024-09-20T19:28:24.198649+0000","last_fullsized":"2024-09-20T19:28:24.198649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:23.446585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:23.446585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:36.919000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882710+0000","last_change":"2024-09-20T19:27:14.287827+0000","last_active":"2024-09-20T19:28:24.882710+0000","last_peered":"2024-09-20T19:28:24.882710+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:27:14.287827+0000","last_became_peered":"2024-09-20T19:27:14.287827+0000","last_unstale":"2024-09-20T19:28:24.882710+0000","last_undegraded":"2024-09-20T19:28:24.882710+0000","last_fullsized":"2024-09-20T19:27:14.284380+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:18.903283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:18.903283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:25.461281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":843,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.206487+0000","last_change":"2024-09-20T19:26:24.150169+0000","last_active":"2024-09-20T19:28:24.206487+0000","last_peered":"2024-09-20T19:28:24.206487+0000","last_clean":"2024-09-20T19:26:22.373968+0000","last_became_active":"2024-09-20T19:26:24.150169+0000","last_became_peered":"2024-09-20T19:26:24.150169+0000","last_unstale":"2024-09-20T19:28:24.206487+0000","last_undegraded":"2024-09-20T19:28:24.206487+0000","last_fullsized":"2024-09-20T19:26:24.139098+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:46.297121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.297121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:39.160630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"77'5","reported_seq":85,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.878838+0000","last_change":"2024-09-20T19:27:14.301989+0000","last_active":"2024-09-20T19:28:24.878838+0000","last_peered":"2024-09-20T19:28:24.878838+0000","last_clean":"2024-09-20T19:18:14.690393+0000","last_became_active":"2024-09-20T19:27:14.301989+0000","last_became_peered":"2024-09-20T19:27:14.301989+0000","last_unstale":"2024-09-20T19:28:24.878838+0000","last_undegraded":"2024-09-20T19:27:14.294111+0000","last_fullsized":"2024-09-20T19:27:14.292163+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-20T19:26:11.110039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.110039+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.203109+0000","last_change":"2024-09-20T19:27:14.293596+0000","last_active":"2024-09-20T19:28:24.203109+0000","last_peered":"2024-09-20T19:28:24.203109+0000","last_clean":"2024-09-20T19:27:12.278605+0000","last_became_active":"2024-09-20T19:27:14.293596+0000","last_became_peered":"2024-09-20T19:27:14.293596+0000","last_unstale":"2024-09-20T19:28:24.203109+0000","last_undegraded":"2024-09-20T19:28:24.203109+0000","last_fullsized":"2024-09-20T19:27:14.288773+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:11.324315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.324315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.388181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.883698+0000","last_change":"2024-09-20T19:26:27.544490+0000","last_active":"2024-09-20T19:28:24.883698+0000","last_peered":"2024-09-20T19:28:24.883698+0000","last_clean":"2024-09-20T19:28:24.883698+0000","last_became_active":"2024-09-20T19:18:13.360854+0000","last_became_peered":"2024-09-20T19:18:13.360854+0000","last_unstale":"2024-09-20T19:28:24.883698+0000","last_undegraded":"2024-09-20T19:28:24.883698+0000","last_fullsized":"2024-09-20T19:28:24.883698+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:27.544371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:27.544371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":839,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.205711+0000","last_change":"2024-09-20T19:24:27.396341+0000","last_active":"2024-09-20T19:28:24.205711+0000","last_peered":"2024-09-20T19:28:24.205711+0000","last_clean":"2024-09-20T19:28:24.205711+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:28:24.205711+0000","last_undegraded":"2024-09-20T19:28:24.205711+0000","last_fullsized":"2024-09-20T19:28:24.205711+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:27.396275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:27.396275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:31.075665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"77'9","reported_seq":90,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.878647+0000","last_change":"2024-09-20T19:27:14.299565+0000","last_active":"2024-09-20T19:27:13.299811+0000","last_peered":"2024-09-20T19:28:24.878647+0000","last_clean":"2024-09-20T19:18:14.721316+0000","last_became_active":"2024-09-20T19:18:09.299047+0000","last_became_peered":"2024-09-20T19:27:14.299565+0000","last_unstale":"2024-09-20T19:28:24.878647+0000","last_undegraded":"2024-09-20T19:27:14.298899+0000","last_fullsized":"2024-09-20T19:27:14.298821+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:24:37.337852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:37.337852+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":827,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882936+0000","last_change":"2024-09-20T19:26:24.153532+0000","last_active":"2024-09-20T19:28:24.882936+0000","last_peered":"2024-09-20T19:28:24.882936+0000","last_clean":"2024-09-20T19:26:22.381776+0000","last_became_active":"2024-09-20T19:26:24.153532+0000","last_became_peered":"2024-09-20T19:26:24.153532+0000","last_unstale":"2024-09-20T19:28:24.882936+0000","last_undegraded":"2024-09-20T19:28:24.882936+0000","last_fullsized":"2024-09-20T19:26:24.135584+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:44.924865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.924865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:56.373905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":296,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.195521+0000","last_change":"2024-09-20T19:26:14.412183+0000","last_active":"2024-09-20T19:28:24.195521+0000","last_peered":"2024-09-20T19:28:24.195521+0000","last_clean":"2024-09-20T19:28:24.195521+0000","last_became_active":"2024-09-20T19:25:20.157858+0000","last_became_peered":"2024-09-20T19:25:20.157858+0000","last_unstale":"2024-09-20T19:28:24.195521+0000","last_undegraded":"2024-09-20T19:28:24.195521+0000","last_fullsized":"2024-09-20T19:28:24.195521+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:14.412131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:14.412131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:20.798023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":446,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.882299+0000","last_change":"2024-09-20T19:27:04.483849+0000","last_active":"2024-09-20T19:28:24.882299+0000","last_peered":"2024-09-20T19:28:24.882299+0000","last_clean":"2024-09-20T19:28:24.882299+0000","last_became_active":"2024-09-20T19:24:11.093457+0000","last_became_peered":"2024-09-20T19:24:11.093457+0000","last_unstale":"2024-09-20T19:28:24.882299+0000","last_undegraded":"2024-09-20T19:28:24.882299+0000","last_fullsized":"2024-09-20T19:28:24.882299+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:04.483789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:04.483789+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"416'117","reported_seq":1705,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.199238+0000","last_change":"2024-09-20T19:26:24.163133+0000","last_active":"2024-09-20T19:28:24.199238+0000","last_peered":"2024-09-20T19:28:24.199238+0000","last_clean":"2024-09-20T19:26:22.376098+0000","last_became_active":"2024-09-20T19:26:24.163133+0000","last_became_peered":"2024-09-20T19:26:24.163133+0000","last_unstale":"2024-09-20T19:28:24.199238+0000","last_undegraded":"2024-09-20T19:26:24.140894+0000","last_fullsized":"2024-09-20T19:26:24.140451+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'81","last_scrub_stamp":"2024-09-20T19:24:35.790673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.790673+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:28.263015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044367390000000003,"stat_sum":{"num_bytes":10021917,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":36,"num_read_kb":11863,"num_write":56,"num_write_kb":14224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8c","length":"1"}]},{"pgid":"4.d","version":"465'19","reported_seq":107,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.199490+0000","last_change":"2024-09-20T19:27:14.296681+0000","last_active":"2024-09-20T19:28:24.199490+0000","last_peered":"2024-09-20T19:28:24.199490+0000","last_clean":"2024-09-20T19:18:14.850498+0000","last_became_active":"2024-09-20T19:27:14.296681+0000","last_became_peered":"2024-09-20T19:27:14.296681+0000","last_unstale":"2024-09-20T19:28:24.199490+0000","last_undegraded":"2024-09-20T19:27:14.285690+0000","last_fullsized":"2024-09-20T19:27:14.285590+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-20T19:26:08.098575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:08.098575+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:14.340048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.204805+0000","last_change":"2024-09-20T19:27:14.305024+0000","last_active":"2024-09-20T19:28:24.204805+0000","last_peered":"2024-09-20T19:28:24.204805+0000","last_clean":"2024-09-20T19:27:12.279461+0000","last_became_active":"2024-09-20T19:27:14.305024+0000","last_became_peered":"2024-09-20T19:27:14.305024+0000","last_unstale":"2024-09-20T19:28:24.204805+0000","last_undegraded":"2024-09-20T19:28:24.204805+0000","last_fullsized":"2024-09-20T19:27:14.288888+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:10.363587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:10.363587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:19.354433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":842,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.196503+0000","last_change":"2024-09-20T19:27:14.285768+0000","last_active":"2024-09-20T19:27:13.275694+0000","last_peered":"2024-09-20T19:28:24.196503+0000","last_clean":"2024-09-20T19:26:22.365285+0000","last_became_active":"2024-09-20T19:26:24.150776+0000","last_became_peered":"2024-09-20T19:27:14.285768+0000","last_unstale":"2024-09-20T19:28:24.196503+0000","last_undegraded":"2024-09-20T19:28:24.196503+0000","last_fullsized":"2024-09-20T19:27:14.284600+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:34.311610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:34.311610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":834,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.199850+0000","last_change":"2024-09-20T19:27:14.289416+0000","last_active":"2024-09-20T19:28:24.199850+0000","last_peered":"2024-09-20T19:28:24.199850+0000","last_clean":"2024-09-20T19:27:12.507710+0000","last_became_active":"2024-09-20T19:27:14.289416+0000","last_became_peered":"2024-09-20T19:27:14.289416+0000","last_unstale":"2024-09-20T19:28:24.199850+0000","last_undegraded":"2024-09-20T19:28:24.199850+0000","last_fullsized":"2024-09-20T19:27:14.287088+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:18.833856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.833856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:41.208335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":902,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.203649+0000","last_change":"2024-09-20T19:27:55.361643+0000","last_active":"2024-09-20T19:28:24.203649+0000","last_peered":"2024-09-20T19:28:24.203649+0000","last_clean":"2024-09-20T19:28:24.203649+0000","last_became_active":"2024-09-20T19:24:11.093089+0000","last_became_peered":"2024-09-20T19:24:11.093089+0000","last_unstale":"2024-09-20T19:28:24.203649+0000","last_undegraded":"2024-09-20T19:28:24.203649+0000","last_fullsized":"2024-09-20T19:28:24.203649+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:55.361511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:55.361511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:19.196766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"441'367","reported_seq":1897,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.879812+0000","last_change":"2024-09-20T19:26:24.166111+0000","last_active":"2024-09-20T19:28:24.879812+0000","last_peered":"2024-09-20T19:28:24.879812+0000","last_clean":"2024-09-20T19:26:08.168738+0000","last_became_active":"2024-09-20T19:26:24.166111+0000","last_became_peered":"2024-09-20T19:26:24.166111+0000","last_unstale":"2024-09-20T19:28:24.879812+0000","last_undegraded":"2024-09-20T19:26:24.151935+0000","last_fullsized":"2024-09-20T19:26:24.151738+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'282","last_scrub_stamp":"2024-09-20T19:25:28.132134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.132134+0000","objects_scrubbed":11,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.014469515000000001,"stat_sum":{"num_bytes":30738780,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":163,"num_read_kb":148607,"num_write":113,"num_write_kb":48790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"}]},{"pgid":"4.c","version":"465'14","reported_seq":96,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.880415+0000","last_change":"2024-09-20T19:27:14.303968+0000","last_active":"2024-09-20T19:28:24.880415+0000","last_peered":"2024-09-20T19:28:24.880415+0000","last_clean":"2024-09-20T19:18:14.781171+0000","last_became_active":"2024-09-20T19:27:14.303968+0000","last_became_peered":"2024-09-20T19:27:14.303968+0000","last_unstale":"2024-09-20T19:28:24.880415+0000","last_undegraded":"2024-09-20T19:27:14.294861+0000","last_fullsized":"2024-09-20T19:27:14.294736+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:42.253363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.253363+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:15.298283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":843,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196932+0000","last_change":"2024-09-20T19:27:54.933931+0000","last_active":"2024-09-20T19:28:24.196932+0000","last_peered":"2024-09-20T19:28:24.196932+0000","last_clean":"2024-09-20T19:28:24.196932+0000","last_became_active":"2024-09-20T19:18:07.274138+0000","last_became_peered":"2024-09-20T19:18:07.274138+0000","last_unstale":"2024-09-20T19:28:24.196932+0000","last_undegraded":"2024-09-20T19:28:24.196932+0000","last_fullsized":"2024-09-20T19:28:24.196932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:54.933848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.933848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.175308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":979,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.199132+0000","last_change":"2024-09-20T19:28:08.349238+0000","last_active":"2024-09-20T19:28:24.199132+0000","last_peered":"2024-09-20T19:28:24.199132+0000","last_clean":"2024-09-20T19:28:24.199132+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:28:24.199132+0000","last_undegraded":"2024-09-20T19:28:24.199132+0000","last_fullsized":"2024-09-20T19:28:24.199132+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:28:08.349103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:08.349103+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:29.855283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":56,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.196536+0000","last_change":"2024-09-20T19:27:14.282251+0000","last_active":"2024-09-20T19:28:24.196536+0000","last_peered":"2024-09-20T19:28:24.196536+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:27:14.282251+0000","last_became_peered":"2024-09-20T19:27:14.282251+0000","last_unstale":"2024-09-20T19:28:24.196536+0000","last_undegraded":"2024-09-20T19:28:24.196536+0000","last_fullsized":"2024-09-20T19:27:14.280553+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:20.948186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:20.948186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:19.838889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":439,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.882616+0000","last_change":"2024-09-20T19:28:03.366431+0000","last_active":"2024-09-20T19:28:24.882616+0000","last_peered":"2024-09-20T19:28:24.882616+0000","last_clean":"2024-09-20T19:28:24.882616+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:28:24.882616+0000","last_undegraded":"2024-09-20T19:28:24.882616+0000","last_fullsized":"2024-09-20T19:28:24.882616+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:03.366356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.366356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:20.162121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"455'596","reported_seq":2328,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.485419+0000","last_change":"2024-09-20T19:27:31.584780+0000","last_active":"2024-09-20T19:28:24.485419+0000","last_peered":"2024-09-20T19:28:24.485419+0000","last_clean":"2024-09-20T19:28:24.485419+0000","last_became_active":"2024-09-20T19:21:30.255122+0000","last_became_peered":"2024-09-20T19:21:30.255122+0000","last_unstale":"2024-09-20T19:28:24.485419+0000","last_undegraded":"2024-09-20T19:28:24.485419+0000","last_fullsized":"2024-09-20T19:28:24.485419+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'237","last_scrub_stamp":"2024-09-20T19:24:35.542766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.542766+0000","objects_scrubbed":12,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:44.605783+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":6.011071662,"stat_sum":{"num_bytes":31174902,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":299,"num_read_kb":222888,"num_write":196,"num_write_kb":63655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"41","length":"1"},{"start":"72","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.f","version":"465'17","reported_seq":886,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.198344+0000","last_change":"2024-09-20T19:27:14.285334+0000","last_active":"2024-09-20T19:28:24.198344+0000","last_peered":"2024-09-20T19:28:24.198344+0000","last_clean":"2024-09-20T19:27:12.239383+0000","last_became_active":"2024-09-20T19:27:14.285334+0000","last_became_peered":"2024-09-20T19:27:14.285334+0000","last_unstale":"2024-09-20T19:28:24.198344+0000","last_undegraded":"2024-09-20T19:27:14.278665+0000","last_fullsized":"2024-09-20T19:27:14.278497+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:17.882060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":851,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.198336+0000","last_change":"2024-09-20T19:28:06.895952+0000","last_active":"2024-09-20T19:28:24.198336+0000","last_peered":"2024-09-20T19:28:24.198336+0000","last_clean":"2024-09-20T19:28:24.198336+0000","last_became_active":"2024-09-20T19:18:07.271488+0000","last_became_peered":"2024-09-20T19:18:07.271488+0000","last_unstale":"2024-09-20T19:28:24.198336+0000","last_undegraded":"2024-09-20T19:28:24.198336+0000","last_fullsized":"2024-09-20T19:28:24.198336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.895903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.895903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:35.042380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":82,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.883614+0000","last_change":"2024-09-20T19:27:14.306189+0000","last_active":"2024-09-20T19:27:13.284465+0000","last_peered":"2024-09-20T19:28:24.883614+0000","last_clean":"2024-09-20T19:18:14.954795+0000","last_became_active":"2024-09-20T19:18:11.309941+0000","last_became_peered":"2024-09-20T19:27:14.306189+0000","last_unstale":"2024-09-20T19:28:24.883614+0000","last_undegraded":"2024-09-20T19:27:14.300346+0000","last_fullsized":"2024-09-20T19:27:14.300079+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:03.226276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:03.226276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":831,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.883593+0000","last_change":"2024-09-20T19:26:24.147284+0000","last_active":"2024-09-20T19:28:24.883593+0000","last_peered":"2024-09-20T19:28:24.883593+0000","last_clean":"2024-09-20T19:26:22.382310+0000","last_became_active":"2024-09-20T19:26:24.147284+0000","last_became_peered":"2024-09-20T19:26:24.147284+0000","last_unstale":"2024-09-20T19:28:24.883593+0000","last_undegraded":"2024-09-20T19:28:24.883593+0000","last_fullsized":"2024-09-20T19:26:24.135536+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:26.791248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:26.791248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:17.249051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"443'304","reported_seq":1915,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.485598+0000","last_change":"2024-09-20T19:26:24.145707+0000","last_active":"2024-09-20T19:28:24.485598+0000","last_peered":"2024-09-20T19:28:24.485598+0000","last_clean":"2024-09-20T19:26:22.377764+0000","last_became_active":"2024-09-20T19:26:24.145707+0000","last_became_peered":"2024-09-20T19:26:24.145707+0000","last_unstale":"2024-09-20T19:28:24.485598+0000","last_undegraded":"2024-09-20T19:26:24.134489+0000","last_fullsized":"2024-09-20T19:26:24.134335+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'149","last_scrub_stamp":"2024-09-20T19:24:44.564761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.564761+0000","objects_scrubbed":9,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:50.533277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.87959363899999998,"stat_sum":{"num_bytes":48288159,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":210,"num_read_kb":194730,"num_write":121,"num_write_kb":52858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":20}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"465'13","reported_seq":98,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.486106+0000","last_change":"2024-09-20T19:27:14.317198+0000","last_active":"2024-09-20T19:28:24.486106+0000","last_peered":"2024-09-20T19:28:24.486106+0000","last_clean":"2024-09-20T19:18:14.855139+0000","last_became_active":"2024-09-20T19:27:14.317198+0000","last_became_peered":"2024-09-20T19:27:14.317198+0000","last_unstale":"2024-09-20T19:28:24.486106+0000","last_undegraded":"2024-09-20T19:27:14.297308+0000","last_fullsized":"2024-09-20T19:27:14.296980+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:27:10.673552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:10.673552+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:29.083737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":15,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":719,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.195149+0000","last_change":"2024-09-20T19:25:16.874919+0000","last_active":"2024-09-20T19:28:24.195149+0000","last_peered":"2024-09-20T19:28:24.195149+0000","last_clean":"2024-09-20T19:28:24.195149+0000","last_became_active":"2024-09-20T19:21:53.042537+0000","last_became_peered":"2024-09-20T19:21:53.042537+0000","last_unstale":"2024-09-20T19:28:24.195149+0000","last_undegraded":"2024-09-20T19:28:24.195149+0000","last_fullsized":"2024-09-20T19:28:24.195149+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:16.874763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.874763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:31.266919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":827,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.884424+0000","last_change":"2024-09-20T19:27:14.313879+0000","last_active":"2024-09-20T19:28:24.884424+0000","last_peered":"2024-09-20T19:28:24.884424+0000","last_clean":"2024-09-20T19:27:12.245082+0000","last_became_active":"2024-09-20T19:27:14.313879+0000","last_became_peered":"2024-09-20T19:27:14.313879+0000","last_unstale":"2024-09-20T19:28:24.884424+0000","last_undegraded":"2024-09-20T19:28:24.884424+0000","last_fullsized":"2024-09-20T19:27:14.301993+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.906652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.906652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:19.340693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":821,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.879229+0000","last_change":"2024-09-20T19:27:56.229631+0000","last_active":"2024-09-20T19:28:24.879229+0000","last_peered":"2024-09-20T19:28:24.879229+0000","last_clean":"2024-09-20T19:28:24.879229+0000","last_became_active":"2024-09-20T19:18:13.489303+0000","last_became_peered":"2024-09-20T19:18:13.489303+0000","last_unstale":"2024-09-20T19:28:24.879229+0000","last_undegraded":"2024-09-20T19:28:24.879229+0000","last_fullsized":"2024-09-20T19:28:24.879229+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:00.282248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":446,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.879030+0000","last_change":"2024-09-20T19:28:07.248122+0000","last_active":"2024-09-20T19:28:24.879030+0000","last_peered":"2024-09-20T19:28:24.879030+0000","last_clean":"2024-09-20T19:28:24.879030+0000","last_became_active":"2024-09-20T19:24:11.088414+0000","last_became_peered":"2024-09-20T19:24:11.088414+0000","last_unstale":"2024-09-20T19:28:24.879030+0000","last_undegraded":"2024-09-20T19:28:24.879030+0000","last_fullsized":"2024-09-20T19:28:24.879030+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:07.247985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:07.247985+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:09.459737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"450'149","reported_seq":1863,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.877520+0000","last_change":"2024-09-20T19:27:22.511607+0000","last_active":"2024-09-20T19:28:24.877520+0000","last_peered":"2024-09-20T19:28:24.877520+0000","last_clean":"2024-09-20T19:28:24.877520+0000","last_became_active":"2024-09-20T19:21:30.265959+0000","last_became_peered":"2024-09-20T19:21:30.265959+0000","last_unstale":"2024-09-20T19:28:24.877520+0000","last_undegraded":"2024-09-20T19:28:24.877520+0000","last_fullsized":"2024-09-20T19:28:24.877520+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:27:22.511438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:22.511438+0000","objects_scrubbed":2,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:47.436030+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0023136799999999998,"stat_sum":{"num_bytes":5586572,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":35507,"num_write":37,"num_write_kb":13841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"41","length":"1"},{"start":"72","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.1","version":"465'18","reported_seq":102,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.883838+0000","last_change":"2024-09-20T19:27:14.310617+0000","last_active":"2024-09-20T19:28:24.883838+0000","last_peered":"2024-09-20T19:28:24.883838+0000","last_clean":"2024-09-20T19:18:14.864493+0000","last_became_active":"2024-09-20T19:27:14.310617+0000","last_became_peered":"2024-09-20T19:27:14.310617+0000","last_unstale":"2024-09-20T19:28:24.883838+0000","last_undegraded":"2024-09-20T19:27:14.285932+0000","last_fullsized":"2024-09-20T19:27:14.285599+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-20T19:25:34.359269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:34.359269+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:23.279338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":848,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.487506+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:28:24.487506+0000","last_peered":"2024-09-20T19:28:24.487506+0000","last_clean":"2024-09-20T19:28:24.487506+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:28:24.487506+0000","last_undegraded":"2024-09-20T19:28:24.487506+0000","last_fullsized":"2024-09-20T19:28:24.487506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:09.149093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"445'12","reported_seq":965,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.877398+0000","last_change":"2024-09-20T19:27:14.300429+0000","last_active":"2024-09-20T19:28:24.877398+0000","last_peered":"2024-09-20T19:28:24.877398+0000","last_clean":"2024-09-20T19:27:12.758653+0000","last_became_active":"2024-09-20T19:27:14.300429+0000","last_became_peered":"2024-09-20T19:27:14.300429+0000","last_unstale":"2024-09-20T19:28:24.877398+0000","last_undegraded":"2024-09-20T19:27:14.277912+0000","last_fullsized":"2024-09-20T19:27:14.277785+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:32.089810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.089810+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:43.312913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":819,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.203989+0000","last_change":"2024-09-20T19:25:22.371471+0000","last_active":"2024-09-20T19:28:24.203989+0000","last_peered":"2024-09-20T19:28:24.203989+0000","last_clean":"2024-09-20T19:28:24.203989+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:28:24.203989+0000","last_undegraded":"2024-09-20T19:28:24.203989+0000","last_fullsized":"2024-09-20T19:28:24.203989+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:22.371404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:22.371404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:49.302044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":57,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.197207+0000","last_change":"2024-09-20T19:27:14.307519+0000","last_active":"2024-09-20T19:28:24.197207+0000","last_peered":"2024-09-20T19:28:24.197207+0000","last_clean":"2024-09-20T19:18:58.414056+0000","last_became_active":"2024-09-20T19:27:14.307519+0000","last_became_peered":"2024-09-20T19:27:14.307519+0000","last_unstale":"2024-09-20T19:28:24.197207+0000","last_undegraded":"2024-09-20T19:28:24.197207+0000","last_fullsized":"2024-09-20T19:27:14.284829+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:20.611001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.611001+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:29:15.290991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3","version":"443'337","reported_seq":2022,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.879591+0000","last_change":"2024-09-20T19:27:14.289341+0000","last_active":"2024-09-20T19:28:24.879591+0000","last_peered":"2024-09-20T19:28:24.879591+0000","last_clean":"2024-09-20T19:27:12.509667+0000","last_became_active":"2024-09-20T19:27:14.289341+0000","last_became_peered":"2024-09-20T19:27:14.289341+0000","last_unstale":"2024-09-20T19:28:24.879591+0000","last_undegraded":"2024-09-20T19:27:14.283764+0000","last_fullsized":"2024-09-20T19:27:14.283605+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'275","last_scrub_stamp":"2024-09-20T19:26:09.838497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:26:09.838497+0000","objects_scrubbed":8,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.23861025699999999,"stat_sum":{"num_bytes":14522704,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":176,"num_read_kb":150748,"num_write":89,"num_write_kb":32294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"41","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"2"},{"start":"8c","length":"4"},{"start":"91","length":"2"},{"start":"94","length":"3"},{"start":"9a","length":"1"}]},{"pgid":"4.0","version":"465'20","reported_seq":877,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.878287+0000","last_change":"2024-09-20T19:26:24.161827+0000","last_active":"2024-09-20T19:28:24.878287+0000","last_peered":"2024-09-20T19:28:24.878287+0000","last_clean":"2024-09-20T19:26:22.378770+0000","last_became_active":"2024-09-20T19:26:24.161827+0000","last_became_peered":"2024-09-20T19:26:24.161827+0000","last_unstale":"2024-09-20T19:28:24.878287+0000","last_undegraded":"2024-09-20T19:26:24.150549+0000","last_fullsized":"2024-09-20T19:26:24.150243+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'18","last_scrub_stamp":"2024-09-20T19:26:07.849948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:07.849948+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":25,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":840,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.198389+0000","last_change":"2024-09-20T19:27:14.308128+0000","last_active":"2024-09-20T19:28:24.198389+0000","last_peered":"2024-09-20T19:28:24.198389+0000","last_clean":"2024-09-20T19:27:12.241205+0000","last_became_active":"2024-09-20T19:27:14.308128+0000","last_became_peered":"2024-09-20T19:27:14.308128+0000","last_unstale":"2024-09-20T19:28:24.198389+0000","last_undegraded":"2024-09-20T19:27:14.281239+0000","last_fullsized":"2024-09-20T19:27:14.281101+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:23:51.507216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:51.507216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:29.012398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.877398+0000","last_change":"2024-09-20T19:27:14.305564+0000","last_active":"2024-09-20T19:28:24.877398+0000","last_peered":"2024-09-20T19:28:24.877398+0000","last_clean":"2024-09-20T19:18:10.268062+0000","last_became_active":"2024-09-20T19:27:14.305564+0000","last_became_peered":"2024-09-20T19:27:14.305564+0000","last_unstale":"2024-09-20T19:28:24.877398+0000","last_undegraded":"2024-09-20T19:28:24.877398+0000","last_fullsized":"2024-09-20T19:27:14.290586+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.482618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.482618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:24.219009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.878190+0000","last_change":"2024-09-20T19:27:14.299532+0000","last_active":"2024-09-20T19:28:24.878190+0000","last_peered":"2024-09-20T19:28:24.878190+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:27:14.299532+0000","last_became_peered":"2024-09-20T19:27:14.299532+0000","last_unstale":"2024-09-20T19:28:24.878190+0000","last_undegraded":"2024-09-20T19:28:24.878190+0000","last_fullsized":"2024-09-20T19:27:14.283514+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:17.258939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":539,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.486488+0000","last_change":"2024-09-20T19:26:24.175873+0000","last_active":"2024-09-20T19:28:24.486488+0000","last_peered":"2024-09-20T19:28:24.486488+0000","last_clean":"2024-09-20T19:26:22.364552+0000","last_became_active":"2024-09-20T19:26:24.175873+0000","last_became_peered":"2024-09-20T19:26:24.175873+0000","last_unstale":"2024-09-20T19:28:24.486488+0000","last_undegraded":"2024-09-20T19:28:24.486488+0000","last_fullsized":"2024-09-20T19:26:24.134569+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.382463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.382463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:58.662602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"455'561","reported_seq":2294,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.882299+0000","last_change":"2024-09-20T19:27:54.474587+0000","last_active":"2024-09-20T19:28:24.882299+0000","last_peered":"2024-09-20T19:28:24.882299+0000","last_clean":"2024-09-20T19:28:24.882299+0000","last_became_active":"2024-09-20T19:21:30.260949+0000","last_became_peered":"2024-09-20T19:21:30.260949+0000","last_unstale":"2024-09-20T19:28:24.882299+0000","last_undegraded":"2024-09-20T19:28:24.882299+0000","last_fullsized":"2024-09-20T19:28:24.882299+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'561","last_scrub_stamp":"2024-09-20T19:27:54.474400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.474400+0000","objects_scrubbed":11,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:02.807338+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":4.0231595230000003,"stat_sum":{"num_bytes":18856192,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":276,"num_read_kb":217988,"num_write":174,"num_write_kb":66893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"41","length":"1"},{"start":"72","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.3","version":"465'23","reported_seq":216,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.882969+0000","last_change":"2024-09-20T19:26:24.152888+0000","last_active":"2024-09-20T19:28:24.882969+0000","last_peered":"2024-09-20T19:28:24.882969+0000","last_clean":"2024-09-20T19:18:14.822642+0000","last_became_active":"2024-09-20T19:26:24.152888+0000","last_became_peered":"2024-09-20T19:26:24.152888+0000","last_unstale":"2024-09-20T19:28:24.882969+0000","last_undegraded":"2024-09-20T19:26:24.137982+0000","last_fullsized":"2024-09-20T19:26:24.137660+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:26:09.938801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:09.938801+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:24.286628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":44,"num_read_kb":30,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":795,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.199112+0000","last_change":"2024-09-20T19:27:14.287813+0000","last_active":"2024-09-20T19:27:13.288013+0000","last_peered":"2024-09-20T19:28:24.199112+0000","last_clean":"2024-09-20T19:26:22.371653+0000","last_became_active":"2024-09-20T19:26:24.144914+0000","last_became_peered":"2024-09-20T19:27:14.287813+0000","last_unstale":"2024-09-20T19:28:24.199112+0000","last_undegraded":"2024-09-20T19:28:24.199112+0000","last_fullsized":"2024-09-20T19:27:14.285255+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:22.169579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:22.169579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":836,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.484384+0000","last_change":"2024-09-20T19:26:24.145685+0000","last_active":"2024-09-20T19:28:24.484384+0000","last_peered":"2024-09-20T19:28:24.484384+0000","last_clean":"2024-09-20T19:26:22.363044+0000","last_became_active":"2024-09-20T19:26:24.145685+0000","last_became_peered":"2024-09-20T19:26:24.145685+0000","last_unstale":"2024-09-20T19:28:24.484384+0000","last_undegraded":"2024-09-20T19:28:24.484384+0000","last_fullsized":"2024-09-20T19:26:24.133505+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:59.649128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:59.649128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:42.691768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196046+0000","last_change":"2024-09-20T19:27:53.899785+0000","last_active":"2024-09-20T19:28:24.196046+0000","last_peered":"2024-09-20T19:28:24.196046+0000","last_clean":"2024-09-20T19:28:24.196046+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:28:24.196046+0000","last_undegraded":"2024-09-20T19:28:24.196046+0000","last_fullsized":"2024-09-20T19:28:24.196046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:53.899739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:53.899739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:07.824086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":543,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.485999+0000","last_change":"2024-09-20T19:27:54.666948+0000","last_active":"2024-09-20T19:28:24.485999+0000","last_peered":"2024-09-20T19:28:24.485999+0000","last_clean":"2024-09-20T19:28:24.485999+0000","last_became_active":"2024-09-20T19:24:11.094059+0000","last_became_peered":"2024-09-20T19:24:11.094059+0000","last_unstale":"2024-09-20T19:28:24.485999+0000","last_undegraded":"2024-09-20T19:28:24.485999+0000","last_fullsized":"2024-09-20T19:28:24.485999+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:22.700029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"450'437","reported_seq":2197,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.484894+0000","last_change":"2024-09-20T19:27:14.317520+0000","last_active":"2024-09-20T19:28:24.484894+0000","last_peered":"2024-09-20T19:28:24.484894+0000","last_clean":"2024-09-20T19:27:12.509954+0000","last_became_active":"2024-09-20T19:27:14.317520+0000","last_became_peered":"2024-09-20T19:27:14.317520+0000","last_unstale":"2024-09-20T19:28:24.484894+0000","last_undegraded":"2024-09-20T19:27:14.303083+0000","last_fullsized":"2024-09-20T19:27:14.302619+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'424","last_scrub_stamp":"2024-09-20T19:27:06.697816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:06.697816+0000","objects_scrubbed":10,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:24.481658+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.014584132,"stat_sum":{"num_bytes":22087644,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":149,"num_read_kb":97036,"num_write":132,"num_write_kb":50917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8689117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"41","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a8","length":"1"}]},{"pgid":"4.2","version":"465'18","reported_seq":870,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.196557+0000","last_change":"2024-09-20T19:27:14.285826+0000","last_active":"2024-09-20T19:28:24.196557+0000","last_peered":"2024-09-20T19:28:24.196557+0000","last_clean":"2024-09-20T19:27:12.241229+0000","last_became_active":"2024-09-20T19:27:14.285826+0000","last_became_peered":"2024-09-20T19:27:14.285826+0000","last_unstale":"2024-09-20T19:28:24.196557+0000","last_undegraded":"2024-09-20T19:27:14.279912+0000","last_fullsized":"2024-09-20T19:27:14.279781+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:54.536975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:54.536975+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":839,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.199411+0000","last_change":"2024-09-20T19:26:24.151794+0000","last_active":"2024-09-20T19:28:24.199411+0000","last_peered":"2024-09-20T19:28:24.199411+0000","last_clean":"2024-09-20T19:26:22.365819+0000","last_became_active":"2024-09-20T19:26:24.151794+0000","last_became_peered":"2024-09-20T19:26:24.151794+0000","last_unstale":"2024-09-20T19:28:24.199411+0000","last_undegraded":"2024-09-20T19:28:24.199411+0000","last_fullsized":"2024-09-20T19:26:24.141414+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:25.123201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"395'12","reported_seq":209,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.485504+0000","last_change":"2024-09-20T19:26:24.143115+0000","last_active":"2024-09-20T19:28:24.485504+0000","last_peered":"2024-09-20T19:28:24.485504+0000","last_clean":"2024-09-20T19:18:14.953653+0000","last_became_active":"2024-09-20T19:26:24.143115+0000","last_became_peered":"2024-09-20T19:26:24.143115+0000","last_unstale":"2024-09-20T19:28:24.485504+0000","last_undegraded":"2024-09-20T19:26:24.134038+0000","last_fullsized":"2024-09-20T19:26:24.133908+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.930350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.930350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:09.663564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":156,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.877356+0000","last_change":"2024-09-20T19:26:24.175498+0000","last_active":"2024-09-20T19:28:24.877356+0000","last_peered":"2024-09-20T19:28:24.877356+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:26:24.175498+0000","last_became_peered":"2024-09-20T19:26:24.175498+0000","last_unstale":"2024-09-20T19:28:24.877356+0000","last_undegraded":"2024-09-20T19:28:24.877356+0000","last_fullsized":"2024-09-20T19:26:24.149058+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:56.797049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:22:56.797049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:29.432351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":436,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882539+0000","last_change":"2024-09-20T19:26:24.153885+0000","last_active":"2024-09-20T19:28:24.882539+0000","last_peered":"2024-09-20T19:28:24.882539+0000","last_clean":"2024-09-20T19:26:22.379050+0000","last_became_active":"2024-09-20T19:26:24.153885+0000","last_became_peered":"2024-09-20T19:26:24.153885+0000","last_unstale":"2024-09-20T19:28:24.882539+0000","last_undegraded":"2024-09-20T19:28:24.882539+0000","last_fullsized":"2024-09-20T19:26:24.135530+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:19.799797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:19.799797+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:13.281398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"447'363","reported_seq":1973,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.486684+0000","last_change":"2024-09-20T19:26:24.167061+0000","last_active":"2024-09-20T19:28:24.486684+0000","last_peered":"2024-09-20T19:28:24.486684+0000","last_clean":"2024-09-20T19:26:22.365576+0000","last_became_active":"2024-09-20T19:26:24.167061+0000","last_became_peered":"2024-09-20T19:26:24.167061+0000","last_unstale":"2024-09-20T19:28:24.486684+0000","last_undegraded":"2024-09-20T19:26:24.134957+0000","last_fullsized":"2024-09-20T19:26:24.134804+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'119","last_scrub_stamp":"2024-09-20T19:23:46.636892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:23:46.636892+0000","objects_scrubbed":4,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:40.760310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.87886274600000003,"stat_sum":{"num_bytes":49329526,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":227,"num_read_kb":159968,"num_write":157,"num_write_kb":55239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":24}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"465'34","reported_seq":1027,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.199267+0000","last_change":"2024-09-20T19:27:14.022237+0000","last_active":"2024-09-20T19:28:24.199267+0000","last_peered":"2024-09-20T19:28:24.199267+0000","last_clean":"2024-09-20T19:28:24.199267+0000","last_became_active":"2024-09-20T19:18:09.298472+0000","last_became_peered":"2024-09-20T19:18:09.298472+0000","last_unstale":"2024-09-20T19:28:24.199267+0000","last_undegraded":"2024-09-20T19:28:24.199267+0000","last_fullsized":"2024-09-20T19:28:24.199267+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-20T19:27:14.022119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:14.022119+0000","objects_scrubbed":10,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":57,"num_read_kb":42,"num_write":30,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":847,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.884708+0000","last_change":"2024-09-20T19:27:12.519339+0000","last_active":"2024-09-20T19:28:24.884708+0000","last_peered":"2024-09-20T19:28:24.884708+0000","last_clean":"2024-09-20T19:28:24.884708+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:28:24.884708+0000","last_undegraded":"2024-09-20T19:28:24.884708+0000","last_fullsized":"2024-09-20T19:28:24.884708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:12.519164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:12.519164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:38.575993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":160,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.196706+0000","last_change":"2024-09-20T19:27:14.293695+0000","last_active":"2024-09-20T19:27:13.301662+0000","last_peered":"2024-09-20T19:28:24.196706+0000","last_clean":"2024-09-20T19:18:10.268062+0000","last_became_active":"2024-09-20T19:26:24.150755+0000","last_became_peered":"2024-09-20T19:27:14.293695+0000","last_unstale":"2024-09-20T19:28:24.196706+0000","last_undegraded":"2024-09-20T19:28:24.196706+0000","last_fullsized":"2024-09-20T19:27:14.291345+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.955124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.955124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":833,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.880323+0000","last_change":"2024-09-20T19:27:14.305123+0000","last_active":"2024-09-20T19:28:24.880323+0000","last_peered":"2024-09-20T19:28:24.880323+0000","last_clean":"2024-09-20T19:27:12.236233+0000","last_became_active":"2024-09-20T19:27:14.305123+0000","last_became_peered":"2024-09-20T19:27:14.305123+0000","last_unstale":"2024-09-20T19:28:24.880323+0000","last_undegraded":"2024-09-20T19:28:24.880323+0000","last_fullsized":"2024-09-20T19:27:14.289188+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.714321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.714321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":902,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.203763+0000","last_change":"2024-09-20T19:27:14.300880+0000","last_active":"2024-09-20T19:27:13.290896+0000","last_peered":"2024-09-20T19:28:24.203763+0000","last_clean":"2024-09-20T19:26:22.371510+0000","last_became_active":"2024-09-20T19:26:24.154908+0000","last_became_peered":"2024-09-20T19:27:14.300880+0000","last_unstale":"2024-09-20T19:28:24.203763+0000","last_undegraded":"2024-09-20T19:28:24.203763+0000","last_fullsized":"2024-09-20T19:27:14.291828+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:20.337772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.337772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"440'733","reported_seq":2281,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.196855+0000","last_change":"2024-09-20T19:27:14.289979+0000","last_active":"2024-09-20T19:27:13.277590+0000","last_peered":"2024-09-20T19:28:24.196855+0000","last_clean":"2024-09-20T19:26:22.364402+0000","last_became_active":"2024-09-20T19:26:24.150215+0000","last_became_peered":"2024-09-20T19:27:14.289979+0000","last_unstale":"2024-09-20T19:28:24.196855+0000","last_undegraded":"2024-09-20T19:27:14.286152+0000","last_fullsized":"2024-09-20T19:27:14.286025+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'652","last_scrub_stamp":"2024-09-20T19:26:03.006839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:26:03.006839+0000","objects_scrubbed":22,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.00456596,"stat_sum":{"num_bytes":63740501,"num_objects":30,"num_object_clones":25,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":240,"num_read_kb":136246,"num_write":249,"num_write_kb":95811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":30}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"}]},{"pgid":"4.7","version":"465'15","reported_seq":790,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.199436+0000","last_change":"2024-09-20T19:26:24.155740+0000","last_active":"2024-09-20T19:28:24.199436+0000","last_peered":"2024-09-20T19:28:24.199436+0000","last_clean":"2024-09-20T19:26:22.372454+0000","last_became_active":"2024-09-20T19:26:24.155740+0000","last_became_peered":"2024-09-20T19:26:24.155740+0000","last_unstale":"2024-09-20T19:28:24.199436+0000","last_undegraded":"2024-09-20T19:26:24.137088+0000","last_fullsized":"2024-09-20T19:26:24.136780+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-20T19:25:25.679635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:25.679635+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:46.247013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":1446,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:25.096593+0000","last_change":"2024-09-20T19:24:28.560164+0000","last_active":"2024-09-20T19:28:25.096593+0000","last_peered":"2024-09-20T19:28:25.096593+0000","last_clean":"2024-09-20T19:28:25.096593+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:28:25.096593+0000","last_undegraded":"2024-09-20T19:28:25.096593+0000","last_fullsized":"2024-09-20T19:28:25.096593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:24:28.560018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:28.560018+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:32.946269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":602,"num_read_kb":597,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":838,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.196614+0000","last_change":"2024-09-20T19:26:24.172120+0000","last_active":"2024-09-20T19:28:24.196614+0000","last_peered":"2024-09-20T19:28:24.196614+0000","last_clean":"2024-09-20T19:26:22.358020+0000","last_became_active":"2024-09-20T19:26:24.172120+0000","last_became_peered":"2024-09-20T19:26:24.172120+0000","last_unstale":"2024-09-20T19:28:24.196614+0000","last_undegraded":"2024-09-20T19:28:24.196614+0000","last_fullsized":"2024-09-20T19:26:24.128705+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:40.807648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:40.807648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:29.235977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":842,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.199587+0000","last_change":"2024-09-20T19:28:00.417353+0000","last_active":"2024-09-20T19:28:24.199587+0000","last_peered":"2024-09-20T19:28:24.199587+0000","last_clean":"2024-09-20T19:28:24.199587+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:28:24.199587+0000","last_undegraded":"2024-09-20T19:28:24.199587+0000","last_fullsized":"2024-09-20T19:28:24.199587+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:00.417262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:00.417262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:25.120008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":827,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.206074+0000","last_change":"2024-09-20T19:24:35.285541+0000","last_active":"2024-09-20T19:28:24.206074+0000","last_peered":"2024-09-20T19:28:24.206074+0000","last_clean":"2024-09-20T19:28:24.206074+0000","last_became_active":"2024-09-20T19:18:13.492180+0000","last_became_peered":"2024-09-20T19:18:13.492180+0000","last_unstale":"2024-09-20T19:28:24.206074+0000","last_undegraded":"2024-09-20T19:28:24.206074+0000","last_fullsized":"2024-09-20T19:28:24.206074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:35.285415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.285415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:30.116192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":438,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.198507+0000","last_change":"2024-09-20T19:27:14.308054+0000","last_active":"2024-09-20T19:28:24.198507+0000","last_peered":"2024-09-20T19:28:24.198507+0000","last_clean":"2024-09-20T19:27:12.242653+0000","last_became_active":"2024-09-20T19:27:14.308054+0000","last_became_peered":"2024-09-20T19:27:14.308054+0000","last_unstale":"2024-09-20T19:28:24.198507+0000","last_undegraded":"2024-09-20T19:28:24.198507+0000","last_fullsized":"2024-09-20T19:27:14.282887+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:26:13.400313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.400313+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"453'717","reported_seq":2348,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.204633+0000","last_change":"2024-09-20T19:26:24.156519+0000","last_active":"2024-09-20T19:28:24.204633+0000","last_peered":"2024-09-20T19:28:24.204633+0000","last_clean":"2024-09-20T19:26:22.380680+0000","last_became_active":"2024-09-20T19:26:24.156519+0000","last_became_peered":"2024-09-20T19:26:24.156519+0000","last_unstale":"2024-09-20T19:28:24.204633+0000","last_undegraded":"2024-09-20T19:26:24.142327+0000","last_fullsized":"2024-09-20T19:26:24.142080+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'286","last_scrub_stamp":"2024-09-20T19:24:01.346351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:01.346351+0000","objects_scrubbed":10,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":1.2270356179999999,"stat_sum":{"num_bytes":83866054,"num_objects":38,"num_object_clones":31,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":2,"num_read":453,"num_read_kb":323692,"num_write":300,"num_write_kb":122211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":38}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"465'16","reported_seq":200,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.198172+0000","last_change":"2024-09-20T19:26:24.175040+0000","last_active":"2024-09-20T19:28:24.198172+0000","last_peered":"2024-09-20T19:28:24.198172+0000","last_clean":"2024-09-20T19:18:14.758576+0000","last_became_active":"2024-09-20T19:26:24.175040+0000","last_became_peered":"2024-09-20T19:26:24.175040+0000","last_unstale":"2024-09-20T19:28:24.198172+0000","last_undegraded":"2024-09-20T19:26:24.140757+0000","last_fullsized":"2024-09-20T19:26:24.140413+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:21.127630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.127630+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:55.954255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":78,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.877973+0000","last_change":"2024-09-20T19:27:14.292054+0000","last_active":"2024-09-20T19:27:13.283039+0000","last_peered":"2024-09-20T19:28:24.877973+0000","last_clean":"2024-09-20T19:18:14.355184+0000","last_became_active":"2024-09-20T19:18:08.279045+0000","last_became_peered":"2024-09-20T19:27:14.292054+0000","last_unstale":"2024-09-20T19:28:24.877973+0000","last_undegraded":"2024-09-20T19:27:14.291014+0000","last_fullsized":"2024-09-20T19:27:14.290932+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:24:39.000765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.000765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":844,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.203178+0000","last_change":"2024-09-20T19:26:24.162911+0000","last_active":"2024-09-20T19:28:24.203178+0000","last_peered":"2024-09-20T19:28:24.203178+0000","last_clean":"2024-09-20T19:26:22.371455+0000","last_became_active":"2024-09-20T19:26:24.162911+0000","last_became_peered":"2024-09-20T19:26:24.162911+0000","last_unstale":"2024-09-20T19:28:24.203178+0000","last_undegraded":"2024-09-20T19:28:24.203178+0000","last_fullsized":"2024-09-20T19:26:24.139314+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:33.379111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:33.379111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:27.121055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.884417+0000","last_change":"2024-09-20T19:26:24.148238+0000","last_active":"2024-09-20T19:28:24.884417+0000","last_peered":"2024-09-20T19:28:24.884417+0000","last_clean":"2024-09-20T19:26:22.383595+0000","last_became_active":"2024-09-20T19:26:24.148238+0000","last_became_peered":"2024-09-20T19:26:24.148238+0000","last_unstale":"2024-09-20T19:28:24.884417+0000","last_undegraded":"2024-09-20T19:28:24.884417+0000","last_fullsized":"2024-09-20T19:26:24.139589+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.747335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.747335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:09.216680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":847,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196341+0000","last_change":"2024-09-20T19:28:14.942125+0000","last_active":"2024-09-20T19:28:24.196341+0000","last_peered":"2024-09-20T19:28:24.196341+0000","last_clean":"2024-09-20T19:28:24.196341+0000","last_became_active":"2024-09-20T19:18:13.367458+0000","last_became_peered":"2024-09-20T19:18:13.367458+0000","last_unstale":"2024-09-20T19:28:24.196341+0000","last_undegraded":"2024-09-20T19:28:24.196341+0000","last_fullsized":"2024-09-20T19:28:24.196341+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:14.942057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:14.942057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:26.446498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":1865,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.204413+0000","last_change":"2024-09-20T19:26:24.179042+0000","last_active":"2024-09-20T19:28:24.204413+0000","last_peered":"2024-09-20T19:28:24.204413+0000","last_clean":"2024-09-20T19:26:22.376820+0000","last_became_active":"2024-09-20T19:26:24.179042+0000","last_became_peered":"2024-09-20T19:26:24.179042+0000","last_unstale":"2024-09-20T19:28:24.204413+0000","last_undegraded":"2024-09-20T19:26:24.145475+0000","last_fullsized":"2024-09-20T19:26:24.145045+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'219","last_scrub_stamp":"2024-09-20T19:26:20.941464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:26:20.941464+0000","objects_scrubbed":10,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0060425940000000001,"stat_sum":{"num_bytes":33075790,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":149,"num_read_kb":133444,"num_write":94,"num_write_kb":36516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"}]},{"pgid":"4.5","version":"465'21","reported_seq":985,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.486417+0000","last_change":"2024-09-20T19:27:14.322535+0000","last_active":"2024-09-20T19:28:24.486417+0000","last_peered":"2024-09-20T19:28:24.486417+0000","last_clean":"2024-09-20T19:27:12.764421+0000","last_became_active":"2024-09-20T19:27:14.322535+0000","last_became_peered":"2024-09-20T19:27:14.322535+0000","last_unstale":"2024-09-20T19:28:24.486417+0000","last_undegraded":"2024-09-20T19:27:14.301818+0000","last_fullsized":"2024-09-20T19:27:14.301539+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:24:43.584143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.584143+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:51.568378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":21,"num_write":17,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":832,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.879294+0000","last_change":"2024-09-20T19:24:46.018528+0000","last_active":"2024-09-20T19:28:24.879294+0000","last_peered":"2024-09-20T19:28:24.879294+0000","last_clean":"2024-09-20T19:28:24.879294+0000","last_became_active":"2024-09-20T19:18:08.277707+0000","last_became_peered":"2024-09-20T19:18:08.277707+0000","last_unstale":"2024-09-20T19:28:24.879294+0000","last_undegraded":"2024-09-20T19:28:24.879294+0000","last_fullsized":"2024-09-20T19:28:24.879294+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:24:46.018399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.018399+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:48.289791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019666800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":831,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.883387+0000","last_change":"2024-09-20T19:24:37.868287+0000","last_active":"2024-09-20T19:28:24.883387+0000","last_peered":"2024-09-20T19:28:24.883387+0000","last_clean":"2024-09-20T19:28:24.883387+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:28:24.883387+0000","last_undegraded":"2024-09-20T19:28:24.883387+0000","last_fullsized":"2024-09-20T19:28:24.883387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:37.868230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:37.868230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:47.453775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":893,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.203576+0000","last_change":"2024-09-20T19:26:24.149721+0000","last_active":"2024-09-20T19:28:24.203576+0000","last_peered":"2024-09-20T19:28:24.203576+0000","last_clean":"2024-09-20T19:26:22.371984+0000","last_became_active":"2024-09-20T19:26:24.149721+0000","last_became_peered":"2024-09-20T19:26:24.149721+0000","last_unstale":"2024-09-20T19:28:24.203576+0000","last_undegraded":"2024-09-20T19:28:24.203576+0000","last_fullsized":"2024-09-20T19:26:24.139098+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:26:13.424049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.424049+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:21.292574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.204861+0000","last_change":"2024-09-20T19:24:30.320332+0000","last_active":"2024-09-20T19:28:24.204861+0000","last_peered":"2024-09-20T19:28:24.204861+0000","last_clean":"2024-09-20T19:28:24.204861+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:28:24.204861+0000","last_undegraded":"2024-09-20T19:28:24.204861+0000","last_fullsized":"2024-09-20T19:28:24.204861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:30.320193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.320193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:29.103749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.884601+0000","last_change":"2024-09-20T19:27:14.309581+0000","last_active":"2024-09-20T19:28:24.884601+0000","last_peered":"2024-09-20T19:28:24.884601+0000","last_clean":"2024-09-20T19:27:12.245335+0000","last_became_active":"2024-09-20T19:27:14.309581+0000","last_became_peered":"2024-09-20T19:27:14.309581+0000","last_unstale":"2024-09-20T19:28:24.884601+0000","last_undegraded":"2024-09-20T19:28:24.884601+0000","last_fullsized":"2024-09-20T19:27:14.304053+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:49.765546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:49.765546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:18.297223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"430'282","reported_seq":1880,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.195560+0000","last_change":"2024-09-20T19:27:14.288514+0000","last_active":"2024-09-20T19:27:13.295931+0000","last_peered":"2024-09-20T19:28:24.195560+0000","last_clean":"2024-09-20T19:26:22.365237+0000","last_became_active":"2024-09-20T19:26:24.138324+0000","last_became_peered":"2024-09-20T19:27:14.288514+0000","last_unstale":"2024-09-20T19:28:24.195560+0000","last_undegraded":"2024-09-20T19:27:14.285883+0000","last_fullsized":"2024-09-20T19:27:14.285758+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'256","last_scrub_stamp":"2024-09-20T19:26:07.943812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:26:07.943812+0000","objects_scrubbed":7,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.005231299,"stat_sum":{"num_bytes":16776658,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":153,"num_read_kb":130993,"num_write":90,"num_write_kb":34982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"}]},{"pgid":"4.a","version":"465'23","reported_seq":873,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.484990+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:28:24.484990+0000","last_peered":"2024-09-20T19:28:24.484990+0000","last_clean":"2024-09-20T19:28:24.484990+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:28:24.484990+0000","last_undegraded":"2024-09-20T19:28:24.484990+0000","last_fullsized":"2024-09-20T19:28:24.484990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:07.716259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":27,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":846,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.198798+0000","last_change":"2024-09-20T19:26:24.155499+0000","last_active":"2024-09-20T19:28:24.198798+0000","last_peered":"2024-09-20T19:28:24.198798+0000","last_clean":"2024-09-20T19:26:22.371653+0000","last_became_active":"2024-09-20T19:26:24.155499+0000","last_became_peered":"2024-09-20T19:26:24.155499+0000","last_unstale":"2024-09-20T19:28:24.198798+0000","last_undegraded":"2024-09-20T19:28:24.198798+0000","last_fullsized":"2024-09-20T19:26:24.134014+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:07.306212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:07.306212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:47.279325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":838,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.197504+0000","last_change":"2024-09-20T19:26:24.157294+0000","last_active":"2024-09-20T19:28:24.197504+0000","last_peered":"2024-09-20T19:28:24.197504+0000","last_clean":"2024-09-20T19:26:22.371098+0000","last_became_active":"2024-09-20T19:26:24.157294+0000","last_became_peered":"2024-09-20T19:26:24.157294+0000","last_unstale":"2024-09-20T19:28:24.197504+0000","last_undegraded":"2024-09-20T19:28:24.197504+0000","last_fullsized":"2024-09-20T19:26:24.132357+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:14.776619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:14.776619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:33.212385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":811,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.203301+0000","last_change":"2024-09-20T19:23:42.246466+0000","last_active":"2024-09-20T19:28:24.203301+0000","last_peered":"2024-09-20T19:28:24.203301+0000","last_clean":"2024-09-20T19:28:24.203301+0000","last_became_active":"2024-09-20T19:18:13.370158+0000","last_became_peered":"2024-09-20T19:18:13.370158+0000","last_unstale":"2024-09-20T19:28:24.203301+0000","last_undegraded":"2024-09-20T19:28:24.203301+0000","last_fullsized":"2024-09-20T19:28:24.203301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.246406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.246406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:50.345697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"453'931","reported_seq":2598,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.195135+0000","last_change":"2024-09-20T19:27:14.302624+0000","last_active":"2024-09-20T19:28:24.195135+0000","last_peered":"2024-09-20T19:28:24.195135+0000","last_clean":"2024-09-20T19:27:12.514955+0000","last_became_active":"2024-09-20T19:27:14.302624+0000","last_became_peered":"2024-09-20T19:27:14.302624+0000","last_unstale":"2024-09-20T19:28:24.195135+0000","last_undegraded":"2024-09-20T19:27:14.282249+0000","last_fullsized":"2024-09-20T19:27:14.282032+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'671","last_scrub_stamp":"2024-09-20T19:26:27.104871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:26:27.104871+0000","objects_scrubbed":21,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.246491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.010678783000000001,"stat_sum":{"num_bytes":57237050,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":2,"num_read":303,"num_read_kb":248459,"num_write":258,"num_write_kb":100266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"41","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a8","length":"1"}]},{"pgid":"4.b","version":"77'9","reported_seq":195,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.198561+0000","last_change":"2024-09-20T19:27:14.285317+0000","last_active":"2024-09-20T19:27:13.296810+0000","last_peered":"2024-09-20T19:28:24.198561+0000","last_clean":"2024-09-20T19:18:14.815617+0000","last_became_active":"2024-09-20T19:26:24.147678+0000","last_became_peered":"2024-09-20T19:27:14.285317+0000","last_unstale":"2024-09-20T19:28:24.198561+0000","last_undegraded":"2024-09-20T19:27:14.283364+0000","last_fullsized":"2024-09-20T19:27:14.283283+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:25:22.114696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:22.114696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":847,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.198883+0000","last_change":"2024-09-20T19:27:14.302119+0000","last_active":"2024-09-20T19:28:24.198883+0000","last_peered":"2024-09-20T19:28:24.198883+0000","last_clean":"2024-09-20T19:27:12.244801+0000","last_became_active":"2024-09-20T19:27:14.302119+0000","last_became_peered":"2024-09-20T19:27:14.302119+0000","last_unstale":"2024-09-20T19:28:24.198883+0000","last_undegraded":"2024-09-20T19:28:24.198883+0000","last_fullsized":"2024-09-20T19:27:14.292360+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:08.329699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:08.329699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:09.955193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":838,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.197164+0000","last_change":"2024-09-20T19:27:14.287481+0000","last_active":"2024-09-20T19:28:24.197164+0000","last_peered":"2024-09-20T19:28:24.197164+0000","last_clean":"2024-09-20T19:27:12.505078+0000","last_became_active":"2024-09-20T19:27:14.287481+0000","last_became_peered":"2024-09-20T19:27:14.287481+0000","last_unstale":"2024-09-20T19:28:24.197164+0000","last_undegraded":"2024-09-20T19:28:24.197164+0000","last_fullsized":"2024-09-20T19:27:14.283118+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:35.635707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:35.635707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:42.185175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":168,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.202856+0000","last_change":"2024-09-20T19:26:24.154825+0000","last_active":"2024-09-20T19:28:24.202856+0000","last_peered":"2024-09-20T19:28:24.202856+0000","last_clean":"2024-09-20T19:18:14.356390+0000","last_became_active":"2024-09-20T19:26:24.154825+0000","last_became_peered":"2024-09-20T19:26:24.154825+0000","last_unstale":"2024-09-20T19:28:24.202856+0000","last_undegraded":"2024-09-20T19:26:24.140991+0000","last_fullsized":"2024-09-20T19:26:24.140632+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:25:31.993519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.993519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:57.367621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"453'257","reported_seq":1993,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196350+0000","last_change":"2024-09-20T19:27:21.161044+0000","last_active":"2024-09-20T19:28:24.196350+0000","last_peered":"2024-09-20T19:28:24.196350+0000","last_clean":"2024-09-20T19:28:24.196350+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:28:24.196350+0000","last_undegraded":"2024-09-20T19:28:24.196350+0000","last_fullsized":"2024-09-20T19:28:24.196350+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'122","last_scrub_stamp":"2024-09-20T19:25:18.429559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.429559+0000","objects_scrubbed":7,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:37.916610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.83043360300000002,"stat_sum":{"num_bytes":11244147,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":99789,"num_write":67,"num_write_kb":33337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"41","length":"1"},{"start":"72","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.8","version":"465'17","reported_seq":863,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.882456+0000","last_change":"2024-09-20T19:26:06.678159+0000","last_active":"2024-09-20T19:28:24.882456+0000","last_peered":"2024-09-20T19:28:24.882456+0000","last_clean":"2024-09-20T19:28:24.882456+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:28:24.882456+0000","last_undegraded":"2024-09-20T19:28:24.882456+0000","last_fullsized":"2024-09-20T19:28:24.882456+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:26:06.678021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:06.678021+0000","objects_scrubbed":7,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:30.411281+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":858,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.199728+0000","last_change":"2024-09-20T19:25:31.622707+0000","last_active":"2024-09-20T19:28:24.199728+0000","last_peered":"2024-09-20T19:28:24.199728+0000","last_clean":"2024-09-20T19:28:24.199728+0000","last_became_active":"2024-09-20T19:18:07.263940+0000","last_became_peered":"2024-09-20T19:18:07.263940+0000","last_unstale":"2024-09-20T19:28:24.199728+0000","last_undegraded":"2024-09-20T19:28:24.199728+0000","last_fullsized":"2024-09-20T19:28:24.199728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:25:31.622605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.622605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:39.150589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"445'12","reported_seq":970,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.203072+0000","last_change":"2024-09-20T19:27:14.316391+0000","last_active":"2024-09-20T19:28:24.203072+0000","last_peered":"2024-09-20T19:28:24.203072+0000","last_clean":"2024-09-20T19:27:12.764859+0000","last_became_active":"2024-09-20T19:27:14.316391+0000","last_became_peered":"2024-09-20T19:27:14.316391+0000","last_unstale":"2024-09-20T19:28:24.203072+0000","last_undegraded":"2024-09-20T19:27:14.290516+0000","last_fullsized":"2024-09-20T19:27:14.290117+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:34.334743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:34.334743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:41.255110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882954+0000","last_change":"2024-09-20T19:26:24.152272+0000","last_active":"2024-09-20T19:28:24.882954+0000","last_peered":"2024-09-20T19:28:24.882954+0000","last_clean":"2024-09-20T19:26:22.381982+0000","last_became_active":"2024-09-20T19:26:24.152272+0000","last_became_peered":"2024-09-20T19:26:24.152272+0000","last_unstale":"2024-09-20T19:28:24.882954+0000","last_undegraded":"2024-09-20T19:28:24.882954+0000","last_fullsized":"2024-09-20T19:26:24.134833+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:32.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:48.478406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"453'344","reported_seq":2064,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.877491+0000","last_change":"2024-09-20T19:27:23.499394+0000","last_active":"2024-09-20T19:28:24.877491+0000","last_peered":"2024-09-20T19:28:24.877491+0000","last_clean":"2024-09-20T19:28:24.877491+0000","last_became_active":"2024-09-20T19:21:30.257404+0000","last_became_peered":"2024-09-20T19:21:30.257404+0000","last_unstale":"2024-09-20T19:28:24.877491+0000","last_undegraded":"2024-09-20T19:28:24.877491+0000","last_fullsized":"2024-09-20T19:28:24.877491+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'344","last_scrub_stamp":"2024-09-20T19:27:23.499201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:23.499201+0000","objects_scrubbed":7,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:45.263992+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":1.0124598380000001,"stat_sum":{"num_bytes":17753779,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":153,"num_read_kb":123089,"num_write":109,"num_write_kb":45756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"41","length":"1"},{"start":"72","length":"1"},{"start":"79","length":"1"},{"start":"7c","length":"1"},{"start":"7f","length":"1"},{"start":"81","length":"7"},{"start":"89","length":"7"},{"start":"91","length":"2"},{"start":"94","length":"4"},{"start":"99","length":"3"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.9","version":"465'14","reported_seq":158,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.882555+0000","last_change":"2024-09-20T19:27:14.308148+0000","last_active":"2024-09-20T19:28:24.882555+0000","last_peered":"2024-09-20T19:28:24.882555+0000","last_clean":"2024-09-20T19:19:53.983037+0000","last_became_active":"2024-09-20T19:27:14.308148+0000","last_became_peered":"2024-09-20T19:27:14.308148+0000","last_unstale":"2024-09-20T19:28:24.882555+0000","last_undegraded":"2024-09-20T19:27:14.284765+0000","last_fullsized":"2024-09-20T19:27:14.284436+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:26:06.088980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:06.088980+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:57.381397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":19,"num_write":16,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":65,"reported_epoch":468,"state":"undersized+degraded+peered","last_fresh":"2024-09-20T19:28:24.205756+0000","last_change":"2024-09-20T19:27:14.299033+0000","last_active":"2024-09-20T19:27:13.294658+0000","last_peered":"2024-09-20T19:28:24.205756+0000","last_clean":"2024-09-20T19:18:07.298165+0000","last_became_active":"2024-09-20T19:18:07.264732+0000","last_became_peered":"2024-09-20T19:27:14.299033+0000","last_unstale":"2024-09-20T19:28:24.205756+0000","last_undegraded":"2024-09-20T19:27:14.294121+0000","last_fullsized":"2024-09-20T19:27:14.293905+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:25:28.472194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:28.472194+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":838,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.195333+0000","last_change":"2024-09-20T19:26:24.149738+0000","last_active":"2024-09-20T19:28:24.195333+0000","last_peered":"2024-09-20T19:28:24.195333+0000","last_clean":"2024-09-20T19:26:22.356804+0000","last_became_active":"2024-09-20T19:26:24.149738+0000","last_became_peered":"2024-09-20T19:26:24.149738+0000","last_unstale":"2024-09-20T19:28:24.195333+0000","last_undegraded":"2024-09-20T19:28:24.195333+0000","last_fullsized":"2024-09-20T19:26:24.127827+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:19.834608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:19.834608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:38.165182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":821,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.878547+0000","last_change":"2024-09-20T19:25:35.724736+0000","last_active":"2024-09-20T19:28:24.878547+0000","last_peered":"2024-09-20T19:28:24.878547+0000","last_clean":"2024-09-20T19:28:24.878547+0000","last_became_active":"2024-09-20T19:18:13.370296+0000","last_became_peered":"2024-09-20T19:18:13.370296+0000","last_unstale":"2024-09-20T19:28:24.878547+0000","last_undegraded":"2024-09-20T19:28:24.878547+0000","last_fullsized":"2024-09-20T19:28:24.878547+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:35.724674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:35.724674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"465'13","reported_seq":197,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.880605+0000","last_change":"2024-09-20T19:26:24.167373+0000","last_active":"2024-09-20T19:28:24.880605+0000","last_peered":"2024-09-20T19:28:24.880605+0000","last_clean":"2024-09-20T19:18:14.614735+0000","last_became_active":"2024-09-20T19:26:24.167373+0000","last_became_peered":"2024-09-20T19:26:24.167373+0000","last_unstale":"2024-09-20T19:28:24.880605+0000","last_undegraded":"2024-09-20T19:26:24.162530+0000","last_fullsized":"2024-09-20T19:26:24.162219+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:25:23.161060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:23.161060+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":862,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.195619+0000","last_change":"2024-09-20T19:26:24.155683+0000","last_active":"2024-09-20T19:28:24.195619+0000","last_peered":"2024-09-20T19:28:24.195619+0000","last_clean":"2024-09-20T19:26:22.357789+0000","last_became_active":"2024-09-20T19:26:24.155683+0000","last_became_peered":"2024-09-20T19:26:24.155683+0000","last_unstale":"2024-09-20T19:28:24.195619+0000","last_undegraded":"2024-09-20T19:28:24.195619+0000","last_fullsized":"2024-09-20T19:26:24.128294+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:19.224546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:19.224546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:05.438859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":825,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.880894+0000","last_change":"2024-09-20T19:28:01.260540+0000","last_active":"2024-09-20T19:28:24.880894+0000","last_peered":"2024-09-20T19:28:24.880894+0000","last_clean":"2024-09-20T19:28:24.880894+0000","last_became_active":"2024-09-20T19:18:11.330881+0000","last_became_peered":"2024-09-20T19:18:11.330881+0000","last_unstale":"2024-09-20T19:28:24.880894+0000","last_undegraded":"2024-09-20T19:28:24.880894+0000","last_fullsized":"2024-09-20T19:28:24.880894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:01.260486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:01.260486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:20.243568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":834,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.197885+0000","last_change":"2024-09-20T19:27:14.292683+0000","last_active":"2024-09-20T19:28:24.197885+0000","last_peered":"2024-09-20T19:28:24.197885+0000","last_clean":"2024-09-20T19:27:12.505390+0000","last_became_active":"2024-09-20T19:27:14.292683+0000","last_became_peered":"2024-09-20T19:27:14.292683+0000","last_unstale":"2024-09-20T19:28:24.197885+0000","last_undegraded":"2024-09-20T19:28:24.197885+0000","last_fullsized":"2024-09-20T19:27:14.284175+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:31.849481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:31.849481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:27.289028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":844,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.484967+0000","last_change":"2024-09-20T19:27:14.320488+0000","last_active":"2024-09-20T19:28:24.484967+0000","last_peered":"2024-09-20T19:28:24.484967+0000","last_clean":"2024-09-20T19:27:12.263753+0000","last_became_active":"2024-09-20T19:27:14.320488+0000","last_became_peered":"2024-09-20T19:27:14.320488+0000","last_unstale":"2024-09-20T19:28:24.484967+0000","last_undegraded":"2024-09-20T19:28:24.484967+0000","last_fullsized":"2024-09-20T19:27:14.303742+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.575807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.575807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:46.559024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"465'8","reported_seq":841,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.881081+0000","last_change":"2024-09-20T19:24:29.131043+0000","last_active":"2024-09-20T19:28:24.881081+0000","last_peered":"2024-09-20T19:28:24.881081+0000","last_clean":"2024-09-20T19:28:24.881081+0000","last_became_active":"2024-09-20T19:18:10.110855+0000","last_became_peered":"2024-09-20T19:18:10.110855+0000","last_unstale":"2024-09-20T19:28:24.881081+0000","last_undegraded":"2024-09-20T19:28:24.881081+0000","last_fullsized":"2024-09-20T19:28:24.881081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-20T19:24:29.130909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:29.130909+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:45.322099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":839,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.883758+0000","last_change":"2024-09-20T19:27:59.345556+0000","last_active":"2024-09-20T19:28:24.883758+0000","last_peered":"2024-09-20T19:28:24.883758+0000","last_clean":"2024-09-20T19:28:24.883758+0000","last_became_active":"2024-09-20T19:18:11.316286+0000","last_became_peered":"2024-09-20T19:18:11.316286+0000","last_unstale":"2024-09-20T19:28:24.883758+0000","last_undegraded":"2024-09-20T19:28:24.883758+0000","last_fullsized":"2024-09-20T19:28:24.883758+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:59.345399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:27:59.345399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:07.598525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.205656+0000","last_change":"2024-09-20T19:27:14.320822+0000","last_active":"2024-09-20T19:28:24.205656+0000","last_peered":"2024-09-20T19:28:24.205656+0000","last_clean":"2024-09-20T19:27:12.279232+0000","last_became_active":"2024-09-20T19:27:14.320822+0000","last_became_peered":"2024-09-20T19:27:14.320822+0000","last_unstale":"2024-09-20T19:28:24.205656+0000","last_undegraded":"2024-09-20T19:28:24.205656+0000","last_fullsized":"2024-09-20T19:27:14.298664+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.296694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.296694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:48.319192+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"465'12","reported_seq":847,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.881990+0000","last_change":"2024-09-20T19:23:50.024939+0000","last_active":"2024-09-20T19:28:24.881990+0000","last_peered":"2024-09-20T19:28:24.881990+0000","last_clean":"2024-09-20T19:28:24.881990+0000","last_became_active":"2024-09-20T19:18:10.109849+0000","last_became_peered":"2024-09-20T19:18:10.109849+0000","last_unstale":"2024-09-20T19:28:24.881990+0000","last_undegraded":"2024-09-20T19:28:24.881990+0000","last_fullsized":"2024-09-20T19:28:24.881990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:23:50.024860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:50.024860+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:39.380740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":839,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.884375+0000","last_change":"2024-09-20T19:23:56.858288+0000","last_active":"2024-09-20T19:28:24.884375+0000","last_peered":"2024-09-20T19:28:24.884375+0000","last_clean":"2024-09-20T19:28:24.884375+0000","last_became_active":"2024-09-20T19:18:07.264758+0000","last_became_peered":"2024-09-20T19:18:07.264758+0000","last_unstale":"2024-09-20T19:28:24.884375+0000","last_undegraded":"2024-09-20T19:28:24.884375+0000","last_fullsized":"2024-09-20T19:28:24.884375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:56.858227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:56.858227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:43.416208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"395'12","reported_seq":970,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.883867+0000","last_change":"2024-09-20T19:26:24.148043+0000","last_active":"2024-09-20T19:28:24.883867+0000","last_peered":"2024-09-20T19:28:24.883867+0000","last_clean":"2024-09-20T19:26:22.724761+0000","last_became_active":"2024-09-20T19:26:24.148043+0000","last_became_peered":"2024-09-20T19:26:24.148043+0000","last_unstale":"2024-09-20T19:28:24.883867+0000","last_undegraded":"2024-09-20T19:26:24.135854+0000","last_fullsized":"2024-09-20T19:26:24.135723+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:23:53.824647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:53.824647+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:38.408447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":156,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.205261+0000","last_change":"2024-09-20T19:26:24.176974+0000","last_active":"2024-09-20T19:28:24.205261+0000","last_peered":"2024-09-20T19:28:24.205261+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:26:24.176974+0000","last_became_peered":"2024-09-20T19:26:24.176974+0000","last_unstale":"2024-09-20T19:28:24.205261+0000","last_undegraded":"2024-09-20T19:28:24.205261+0000","last_fullsized":"2024-09-20T19:26:24.145450+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:30.079831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:30.079831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:00.354898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"465'11","reported_seq":862,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.883969+0000","last_change":"2024-09-20T19:27:22.506477+0000","last_active":"2024-09-20T19:28:24.883969+0000","last_peered":"2024-09-20T19:28:24.883969+0000","last_clean":"2024-09-20T19:28:24.883969+0000","last_became_active":"2024-09-20T19:18:10.114604+0000","last_became_peered":"2024-09-20T19:18:10.114604+0000","last_unstale":"2024-09-20T19:28:24.883969+0000","last_undegraded":"2024-09-20T19:28:24.883969+0000","last_fullsized":"2024-09-20T19:28:24.883969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:27:22.506346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:22.506346+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:51.371605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":56,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.879630+0000","last_change":"2024-09-20T19:27:14.295331+0000","last_active":"2024-09-20T19:18:06.213049+0000","last_peered":"2024-09-20T19:28:24.879630+0000","last_clean":"2024-09-20T19:18:06.213049+0000","last_became_active":"0.000000","last_became_peered":"2024-09-20T19:27:14.295331+0000","last_unstale":"2024-09-20T19:28:24.879630+0000","last_undegraded":"2024-09-20T19:28:24.879630+0000","last_fullsized":"2024-09-20T19:27:14.294522+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:06.971815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:06.971815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"77'11","reported_seq":967,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.879805+0000","last_change":"2024-09-20T19:28:18.244111+0000","last_active":"2024-09-20T19:28:24.879805+0000","last_peered":"2024-09-20T19:28:24.879805+0000","last_clean":"2024-09-20T19:28:24.879805+0000","last_became_active":"2024-09-20T19:18:11.329711+0000","last_became_peered":"2024-09-20T19:18:11.329711+0000","last_unstale":"2024-09-20T19:28:24.879805+0000","last_undegraded":"2024-09-20T19:28:24.879805+0000","last_fullsized":"2024-09-20T19:28:24.879805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:28:18.243973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:18.243973+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:44.645661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":56,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.199716+0000","last_change":"2024-09-20T19:27:14.297199+0000","last_active":"2024-09-20T19:28:24.199716+0000","last_peered":"2024-09-20T19:28:24.199716+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:27:14.297199+0000","last_became_peered":"2024-09-20T19:27:14.297199+0000","last_unstale":"2024-09-20T19:28:24.199716+0000","last_undegraded":"2024-09-20T19:28:24.199716+0000","last_fullsized":"2024-09-20T19:27:14.287014+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:17.884618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:17.884618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:48.296668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"77'9","reported_seq":855,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196399+0000","last_change":"2024-09-20T19:26:16.401327+0000","last_active":"2024-09-20T19:28:24.196399+0000","last_peered":"2024-09-20T19:28:24.196399+0000","last_clean":"2024-09-20T19:28:24.196399+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:28:24.196399+0000","last_undegraded":"2024-09-20T19:28:24.196399+0000","last_fullsized":"2024-09-20T19:28:24.196399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:26:16.401264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:16.401264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:21.792404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":832,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.486002+0000","last_change":"2024-09-20T19:24:42.625537+0000","last_active":"2024-09-20T19:28:24.486002+0000","last_peered":"2024-09-20T19:28:24.486002+0000","last_clean":"2024-09-20T19:28:24.486002+0000","last_became_active":"2024-09-20T19:18:07.259301+0000","last_became_peered":"2024-09-20T19:18:07.259301+0000","last_unstale":"2024-09-20T19:28:24.486002+0000","last_undegraded":"2024-09-20T19:28:24.486002+0000","last_fullsized":"2024-09-20T19:28:24.486002+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.625450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.625450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:49.558348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":821,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.878602+0000","last_change":"2024-09-20T19:26:24.170420+0000","last_active":"2024-09-20T19:28:24.878602+0000","last_peered":"2024-09-20T19:28:24.878602+0000","last_clean":"2024-09-20T19:26:22.378978+0000","last_became_active":"2024-09-20T19:26:24.170420+0000","last_became_peered":"2024-09-20T19:26:24.170420+0000","last_unstale":"2024-09-20T19:28:24.878602+0000","last_undegraded":"2024-09-20T19:28:24.878602+0000","last_fullsized":"2024-09-20T19:26:24.150120+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:21:05.014540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:21:05.014540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:25.484349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882454+0000","last_change":"2024-09-20T19:26:24.148189+0000","last_active":"2024-09-20T19:28:24.882454+0000","last_peered":"2024-09-20T19:28:24.882454+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:26:24.148189+0000","last_became_peered":"2024-09-20T19:26:24.148189+0000","last_unstale":"2024-09-20T19:28:24.882454+0000","last_undegraded":"2024-09-20T19:28:24.882454+0000","last_fullsized":"2024-09-20T19:26:24.137174+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:31.041093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.041093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:15.205487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"465'15","reported_seq":101,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.485566+0000","last_change":"2024-09-20T19:27:14.318838+0000","last_active":"2024-09-20T19:28:24.485566+0000","last_peered":"2024-09-20T19:28:24.485566+0000","last_clean":"2024-09-20T19:18:14.816616+0000","last_became_active":"2024-09-20T19:27:14.318838+0000","last_became_peered":"2024-09-20T19:27:14.318838+0000","last_unstale":"2024-09-20T19:28:24.485566+0000","last_undegraded":"2024-09-20T19:27:14.298395+0000","last_fullsized":"2024-09-20T19:27:14.298078+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:10.092637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:10.092637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:12.591767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":855,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.198843+0000","last_change":"2024-09-20T19:26:24.149794+0000","last_active":"2024-09-20T19:28:24.198843+0000","last_peered":"2024-09-20T19:28:24.198843+0000","last_clean":"2024-09-20T19:26:22.359582+0000","last_became_active":"2024-09-20T19:26:24.149794+0000","last_became_peered":"2024-09-20T19:26:24.149794+0000","last_unstale":"2024-09-20T19:28:24.198843+0000","last_undegraded":"2024-09-20T19:26:24.136789+0000","last_fullsized":"2024-09-20T19:26:24.136379+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:25:16.437044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:25:16.437044+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:40.911765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":834,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.196548+0000","last_change":"2024-09-20T19:25:20.158995+0000","last_active":"2024-09-20T19:28:24.196548+0000","last_peered":"2024-09-20T19:28:24.196548+0000","last_clean":"2024-09-20T19:28:24.196548+0000","last_became_active":"2024-09-20T19:25:20.158758+0000","last_became_peered":"2024-09-20T19:25:20.158758+0000","last_unstale":"2024-09-20T19:28:24.196548+0000","last_undegraded":"2024-09-20T19:28:24.196548+0000","last_fullsized":"2024-09-20T19:28:24.196548+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:44.572407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:44.572407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:46.893113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":817,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.877200+0000","last_change":"2024-09-20T19:26:24.169002+0000","last_active":"2024-09-20T19:28:24.877200+0000","last_peered":"2024-09-20T19:28:24.877200+0000","last_clean":"2024-09-20T19:26:22.378732+0000","last_became_active":"2024-09-20T19:26:24.169002+0000","last_became_peered":"2024-09-20T19:26:24.169002+0000","last_unstale":"2024-09-20T19:28:24.877200+0000","last_undegraded":"2024-09-20T19:28:24.877200+0000","last_fullsized":"2024-09-20T19:26:24.151608+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:47.003602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:47.003602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:32.397663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"465'6","reported_seq":846,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.877869+0000","last_change":"2024-09-20T19:26:03.880292+0000","last_active":"2024-09-20T19:28:24.877869+0000","last_peered":"2024-09-20T19:28:24.877869+0000","last_clean":"2024-09-20T19:28:24.877869+0000","last_became_active":"2024-09-20T19:18:09.292536+0000","last_became_peered":"2024-09-20T19:18:09.292536+0000","last_unstale":"2024-09-20T19:28:24.877869+0000","last_undegraded":"2024-09-20T19:28:24.877869+0000","last_fullsized":"2024-09-20T19:28:24.877869+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-20T19:26:03.880121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:03.880121+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":831,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.884056+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:28:24.884056+0000","last_peered":"2024-09-20T19:28:24.884056+0000","last_clean":"2024-09-20T19:28:24.884056+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:28:24.884056+0000","last_undegraded":"2024-09-20T19:28:24.884056+0000","last_fullsized":"2024-09-20T19:28:24.884056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:35.475039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":836,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.484315+0000","last_change":"2024-09-20T19:27:14.322249+0000","last_active":"2024-09-20T19:28:24.484315+0000","last_peered":"2024-09-20T19:28:24.484315+0000","last_clean":"2024-09-20T19:27:12.262865+0000","last_became_active":"2024-09-20T19:27:14.322249+0000","last_became_peered":"2024-09-20T19:27:14.322249+0000","last_unstale":"2024-09-20T19:28:24.484315+0000","last_undegraded":"2024-09-20T19:28:24.484315+0000","last_fullsized":"2024-09-20T19:27:14.301319+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.544996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.544996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:04.730831+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":815,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.202794+0000","last_change":"2024-09-20T19:27:14.304486+0000","last_active":"2024-09-20T19:28:24.202794+0000","last_peered":"2024-09-20T19:28:24.202794+0000","last_clean":"2024-09-20T19:27:12.278439+0000","last_became_active":"2024-09-20T19:27:14.304486+0000","last_became_peered":"2024-09-20T19:27:14.304486+0000","last_unstale":"2024-09-20T19:28:24.202794+0000","last_undegraded":"2024-09-20T19:28:24.202794+0000","last_fullsized":"2024-09-20T19:27:14.290380+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:43.313654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.313654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:43.286716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"77'11","reported_seq":856,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.878309+0000","last_change":"2024-09-20T19:26:11.789544+0000","last_active":"2024-09-20T19:28:24.878309+0000","last_peered":"2024-09-20T19:28:24.878309+0000","last_clean":"2024-09-20T19:28:24.878309+0000","last_became_active":"2024-09-20T19:18:10.109694+0000","last_became_peered":"2024-09-20T19:18:10.109694+0000","last_unstale":"2024-09-20T19:28:24.878309+0000","last_undegraded":"2024-09-20T19:28:24.878309+0000","last_fullsized":"2024-09-20T19:28:24.878309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:11.789341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.789341+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":848,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.487490+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:28:24.487490+0000","last_peered":"2024-09-20T19:28:24.487490+0000","last_clean":"2024-09-20T19:28:24.487490+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:28:24.487490+0000","last_undegraded":"2024-09-20T19:28:24.487490+0000","last_fullsized":"2024-09-20T19:28:24.487490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:29.360397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.202440+0000","last_change":"2024-09-20T19:27:14.317037+0000","last_active":"2024-09-20T19:28:24.202440+0000","last_peered":"2024-09-20T19:28:24.202440+0000","last_clean":"2024-09-20T19:27:12.278118+0000","last_became_active":"2024-09-20T19:27:14.317037+0000","last_became_peered":"2024-09-20T19:27:14.317037+0000","last_unstale":"2024-09-20T19:28:24.202440+0000","last_undegraded":"2024-09-20T19:28:24.202440+0000","last_fullsized":"2024-09-20T19:27:14.290333+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:39.314605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.314605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:35.086854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":833,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.878018+0000","last_change":"2024-09-20T19:26:24.161749+0000","last_active":"2024-09-20T19:28:24.878018+0000","last_peered":"2024-09-20T19:28:24.878018+0000","last_clean":"2024-09-20T19:26:22.380251+0000","last_became_active":"2024-09-20T19:26:24.161749+0000","last_became_peered":"2024-09-20T19:26:24.161749+0000","last_unstale":"2024-09-20T19:28:24.878018+0000","last_undegraded":"2024-09-20T19:28:24.878018+0000","last_fullsized":"2024-09-20T19:26:24.150545+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.682353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.682353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:57.250404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":829,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.204590+0000","last_change":"2024-09-20T19:28:13.311223+0000","last_active":"2024-09-20T19:28:24.204590+0000","last_peered":"2024-09-20T19:28:24.204590+0000","last_clean":"2024-09-20T19:28:24.204590+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:28:24.204590+0000","last_undegraded":"2024-09-20T19:28:24.204590+0000","last_fullsized":"2024-09-20T19:28:24.204590+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:28:13.311036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:13.311036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:41.187320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":851,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.882582+0000","last_change":"2024-09-20T19:27:14.308612+0000","last_active":"2024-09-20T19:28:24.882582+0000","last_peered":"2024-09-20T19:28:24.882582+0000","last_clean":"2024-09-20T19:27:12.242967+0000","last_became_active":"2024-09-20T19:27:14.308612+0000","last_became_peered":"2024-09-20T19:27:14.308612+0000","last_unstale":"2024-09-20T19:28:24.882582+0000","last_undegraded":"2024-09-20T19:28:24.882582+0000","last_fullsized":"2024-09-20T19:27:14.284810+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:25.557584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:25.557584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:26.497151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"77'10","reported_seq":192,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.205921+0000","last_change":"2024-09-20T19:26:24.178367+0000","last_active":"2024-09-20T19:28:24.205921+0000","last_peered":"2024-09-20T19:28:24.205921+0000","last_clean":"2024-09-20T19:18:14.854961+0000","last_became_active":"2024-09-20T19:26:24.178367+0000","last_became_peered":"2024-09-20T19:26:24.178367+0000","last_unstale":"2024-09-20T19:28:24.205921+0000","last_undegraded":"2024-09-20T19:26:24.146690+0000","last_fullsized":"2024-09-20T19:26:24.146566+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-20T19:24:30.887749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.887749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"445'12","reported_seq":982,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.486684+0000","last_change":"2024-09-20T19:27:14.321965+0000","last_active":"2024-09-20T19:28:24.486684+0000","last_peered":"2024-09-20T19:28:24.486684+0000","last_clean":"2024-09-20T19:27:12.764296+0000","last_became_active":"2024-09-20T19:27:14.321965+0000","last_became_peered":"2024-09-20T19:27:14.321965+0000","last_unstale":"2024-09-20T19:28:24.486684+0000","last_undegraded":"2024-09-20T19:27:14.299847+0000","last_fullsized":"2024-09-20T19:27:14.299536+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:18.523807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":835,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.196411+0000","last_change":"2024-09-20T19:27:14.300270+0000","last_active":"2024-09-20T19:28:24.196411+0000","last_peered":"2024-09-20T19:28:24.196411+0000","last_clean":"2024-09-20T19:27:12.242048+0000","last_became_active":"2024-09-20T19:27:14.300270+0000","last_became_peered":"2024-09-20T19:27:14.300270+0000","last_unstale":"2024-09-20T19:28:24.196411+0000","last_undegraded":"2024-09-20T19:28:24.196411+0000","last_fullsized":"2024-09-20T19:27:14.291013+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.432349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.432349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:50.907861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":846,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.878925+0000","last_change":"2024-09-20T19:26:24.168540+0000","last_active":"2024-09-20T19:28:24.878925+0000","last_peered":"2024-09-20T19:28:24.878925+0000","last_clean":"2024-09-20T19:26:22.379843+0000","last_became_active":"2024-09-20T19:26:24.168540+0000","last_became_peered":"2024-09-20T19:26:24.168540+0000","last_unstale":"2024-09-20T19:28:24.878925+0000","last_undegraded":"2024-09-20T19:26:24.150958+0000","last_fullsized":"2024-09-20T19:26:24.149083+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:24:30.106245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.106245+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:52.206408+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"77'11","reported_seq":885,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.487248+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:28:24.487248+0000","last_peered":"2024-09-20T19:28:24.487248+0000","last_clean":"2024-09-20T19:28:24.487248+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:28:24.487248+0000","last_undegraded":"2024-09-20T19:28:24.487248+0000","last_fullsized":"2024-09-20T19:28:24.487248+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:51.474565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":158,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.205020+0000","last_change":"2024-09-20T19:26:24.181498+0000","last_active":"2024-09-20T19:28:24.205020+0000","last_peered":"2024-09-20T19:28:24.205020+0000","last_clean":"2024-09-20T19:18:10.268062+0000","last_became_active":"2024-09-20T19:26:24.181498+0000","last_became_peered":"2024-09-20T19:26:24.181498+0000","last_unstale":"2024-09-20T19:28:24.205020+0000","last_undegraded":"2024-09-20T19:28:24.205020+0000","last_fullsized":"2024-09-20T19:26:24.144978+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:36.003180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:36.003180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:14.354356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":56,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.485692+0000","last_change":"2024-09-20T19:27:14.310564+0000","last_active":"2024-09-20T19:28:24.485692+0000","last_peered":"2024-09-20T19:28:24.485692+0000","last_clean":"2024-09-20T19:18:12.302527+0000","last_became_active":"2024-09-20T19:27:14.310564+0000","last_became_peered":"2024-09-20T19:27:14.310564+0000","last_unstale":"2024-09-20T19:28:24.485692+0000","last_undegraded":"2024-09-20T19:28:24.485692+0000","last_fullsized":"2024-09-20T19:27:14.299319+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:19.940559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:19.940559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:19.504239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":844,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.485363+0000","last_change":"2024-09-20T19:27:14.320887+0000","last_active":"2024-09-20T19:28:24.485363+0000","last_peered":"2024-09-20T19:28:24.485363+0000","last_clean":"2024-09-20T19:27:12.263174+0000","last_became_active":"2024-09-20T19:27:14.320887+0000","last_became_peered":"2024-09-20T19:27:14.320887+0000","last_unstale":"2024-09-20T19:28:24.485363+0000","last_undegraded":"2024-09-20T19:28:24.485363+0000","last_fullsized":"2024-09-20T19:27:14.298004+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:52.648578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:52.648578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:30.654694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"465'21","reported_seq":880,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.198437+0000","last_change":"2024-09-20T19:28:02.452053+0000","last_active":"2024-09-20T19:28:24.198437+0000","last_peered":"2024-09-20T19:28:24.198437+0000","last_clean":"2024-09-20T19:28:24.198437+0000","last_became_active":"2024-09-20T19:18:09.288491+0000","last_became_peered":"2024-09-20T19:18:09.288491+0000","last_unstale":"2024-09-20T19:28:24.198437+0000","last_undegraded":"2024-09-20T19:28:24.198437+0000","last_fullsized":"2024-09-20T19:28:24.198437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:28:02.451854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.451854+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:30.578790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":35,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":842,"reported_epoch":468,"state":"undersized+peered","last_fresh":"2024-09-20T19:28:24.198040+0000","last_change":"2024-09-20T19:27:14.293701+0000","last_active":"2024-09-20T19:27:13.304689+0000","last_peered":"2024-09-20T19:28:24.198040+0000","last_clean":"2024-09-20T19:26:22.359139+0000","last_became_active":"2024-09-20T19:26:24.158183+0000","last_became_peered":"2024-09-20T19:27:14.293701+0000","last_unstale":"2024-09-20T19:28:24.198040+0000","last_undegraded":"2024-09-20T19:28:24.198040+0000","last_fullsized":"2024-09-20T19:27:14.291595+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:57.510542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:57.510542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":821,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.880630+0000","last_change":"2024-09-20T19:27:54.253979+0000","last_active":"2024-09-20T19:28:24.880630+0000","last_peered":"2024-09-20T19:28:24.880630+0000","last_clean":"2024-09-20T19:28:24.880630+0000","last_became_active":"2024-09-20T19:18:13.489462+0000","last_became_peered":"2024-09-20T19:18:13.489462+0000","last_unstale":"2024-09-20T19:28:24.880630+0000","last_undegraded":"2024-09-20T19:28:24.880630+0000","last_fullsized":"2024-09-20T19:28:24.880630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:54.253853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.253853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.057379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":833,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-20T19:28:24.879371+0000","last_change":"2024-09-20T19:27:58.281533+0000","last_active":"2024-09-20T19:28:24.879371+0000","last_peered":"2024-09-20T19:28:24.879371+0000","last_clean":"2024-09-20T19:28:24.879371+0000","last_became_active":"2024-09-20T19:18:07.264936+0000","last_became_peered":"2024-09-20T19:18:07.264936+0000","last_unstale":"2024-09-20T19:28:24.879371+0000","last_undegraded":"2024-09-20T19:28:24.879371+0000","last_fullsized":"2024-09-20T19:28:24.879371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:58.281416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:58.281416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:28:59.374455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"77'12","reported_seq":856,"reported_epoch":468,"state":"active+undersized+degraded","last_fresh":"2024-09-20T19:28:24.878220+0000","last_change":"2024-09-20T19:27:14.301067+0000","last_active":"2024-09-20T19:28:24.878220+0000","last_peered":"2024-09-20T19:28:24.878220+0000","last_clean":"2024-09-20T19:27:12.239627+0000","last_became_active":"2024-09-20T19:27:14.301067+0000","last_became_peered":"2024-09-20T19:27:14.301067+0000","last_unstale":"2024-09-20T19:28:24.878220+0000","last_undegraded":"2024-09-20T19:27:14.291415+0000","last_fullsized":"2024-09-20T19:27:14.291266+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:25.699697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:25.699697+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:29:16.283848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":468,"state":"active+undersized","last_fresh":"2024-09-20T19:28:24.879996+0000","last_change":"2024-09-20T19:27:14.295740+0000","last_active":"2024-09-20T19:28:24.879996+0000","last_peered":"2024-09-20T19:28:24.879996+0000","last_clean":"2024-09-20T19:18:10.268062+0000","last_became_active":"2024-09-20T19:27:14.295740+0000","last_became_peered":"2024-09-20T19:27:14.295740+0000","last_unstale":"2024-09-20T19:28:24.879996+0000","last_undegraded":"2024-09-20T19:28:24.879996+0000","last_fullsized":"2024-09-20T19:27:14.287661+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:15.968310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:15.968310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":514300375,"num_objects":244,"num_object_clones":194,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":14,"num_read":3168,"num_read_kb":2435047,"num_write":2242,"num_write_kb":867590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8689117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097981952,"data_stored":1494409422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6749,"ondisk_log_size":6749,"up":35,"acting":35,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":76,"acting":76,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":68,"acting":68,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":801,"num_read_kb":558,"num_write":493,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":73,"acting":73,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":602,"num_read_kb":597,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":73,"acting":73,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":26,"acting":26,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907077,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":221668,"kb_used_data":178000,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93523484,"statfs":{"total":95995035648,"available":95768047616,"internally_reserved":0,"allocated":182272000,"data_stored":248094978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234128,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167292,"kb_used_data":127780,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93577860,"statfs":{"total":95995035648,"available":95823728640,"internally_reserved":0,"allocated":130846720,"data_stored":158177598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561178,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80372,"kb_used_data":45024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93664780,"statfs":{"total":95995035648,"available":95912734720,"internally_reserved":0,"allocated":46104576,"data_stored":66460513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855511,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158336,"kb_used_data":127144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93586816,"statfs":{"total":95995035648,"available":95832899584,"internally_reserved":0,"allocated":130195456,"data_stored":188730262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084464,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175812,"kb_used_data":148784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93569340,"statfs":{"total":95995035648,"available":95815004160,"internally_reserved":0,"allocated":152354816,"data_stored":213585269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378814,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176912,"kb_used_data":141560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93568240,"statfs":{"total":95995035648,"available":95813877760,"internally_reserved":0,"allocated":144957440,"data_stored":204297978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168308,"kb_used_data":141276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93576844,"statfs":{"total":95995035648,"available":95822688256,"internally_reserved":0,"allocated":144666624,"data_stored":200490569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738557,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216428,"kb_used_data":185240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93528724,"statfs":{"total":95995035648,"available":95773413376,"internally_reserved":0,"allocated":189685760,"data_stored":228826485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185520128,"data_stored":226426727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142159872,"data_stored":199021652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142192640,"data_stored":202566890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149250048,"data_stored":211526801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":128020480,"data_stored":187456101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43024384,"data_stored":64401709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":128364544,"data_stored":156710253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":179449856,"data_stored":246299289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:28:31.463 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T19:28:31.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-20T19:28:31.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:31.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:31.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:31.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:31 smithi042 bash[22706]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:31 smithi042 bash[22706]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:31 smithi042 bash[32075]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:31 smithi042 bash[32075]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.698 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:31.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:31 smithi162 bash[24557]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:31 smithi162 bash[24557]: cluster 2024-09-20T19:28:30.153844+0000 mgr.y (mgr.24491) 401 : cluster [DBG] pgmap v668: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 733 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:31.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:31.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:31.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:31.966 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:28:31.984 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:28:31.984 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:28:31.984 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:28:31.984 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:28:31.984 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:28:31.985 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:28:31.985 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:28:32.004 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:28:32.005 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:28:32.021 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:28:32.022 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:28:32.022 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:28:32.043 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:28:32.043 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:28:32.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.065 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:28:32.065 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:28:32.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:32.085 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:28:32.085 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:28:32.182 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:32.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.325 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:32.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:32.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:32.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:32.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.712 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:32 smithi042 bash[22706]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.712 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:32 smithi042 bash[22706]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.712 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:32 smithi042 bash[32075]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.712 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:32 smithi042 bash[32075]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:32.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:32 smithi162 bash[24557]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:32 smithi162 bash[24557]: audit 2024-09-20T19:28:31.965997+0000 mon.a (mon.0) 1591 : audit [DBG] from='client.? v1:172.21.15.42:0/213488192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:32.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:32.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:32.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:32.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:33.077 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:33.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:33.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:33.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:33.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:33.508 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:28:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:28:33.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:28:33.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:28:33.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:33.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:33.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[32075]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:33 smithi042 bash[22706]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: cluster 2024-09-20T19:28:32.155208+0000 mgr.y (mgr.24491) 402 : cluster [DBG] pgmap v669: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: audit 2024-09-20T19:28:32.487924+0000 mon.c (mon.2) 69 : audit [INF] from='client.? v1:172.21.15.42:0/2040598580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:33 smithi162 bash[24557]: audit 2024-09-20T19:28:32.492086+0000 mon.a (mon.0) 1592 : audit [INF] from='client.25298 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:28:33.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:33.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:33.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:33.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:33.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:34.017 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:34.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:34.249 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:34.278 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:34.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:34.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:34.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:34.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:34.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:34.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[22706]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.651 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:34 smithi042 bash[32075]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:34.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: cluster 2024-09-20T19:28:33.490271+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: audit 2024-09-20T19:28:33.500466+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:34 smithi162 bash[24557]: cluster 2024-09-20T19:28:33.506189+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e470: 8 total, 6 up, 8 in 2024-09-20T19:28:34.798 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:34.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:34.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:34.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:35.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:35.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:35.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:35.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:35.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:35.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:35.447 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:35.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[22706]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:35 smithi042 bash[32075]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.573 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:28:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:35.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.704 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:35.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.156918+0000 mgr.y (mgr.24491) 403 : cluster [DBG] pgmap v671: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:35.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.556651+0000 osd.6 (osd.6) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:28:35.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:35 smithi162 bash[24557]: cluster 2024-09-20T19:28:34.557288+0000 osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:28:35.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:35.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:35.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:35.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:35.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:36.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.094 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:36.197 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:36.328 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:36.339 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:36.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:28:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:36.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:36.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:36 smithi042 bash[32075]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:36 smithi042 bash[32075]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:36 smithi042 bash[22706]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:36 smithi042 bash[22706]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.610 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:36.713 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:36.740 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:36.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:36 smithi162 bash[24557]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:36 smithi162 bash[24557]: cluster 2024-09-20T19:28:35.508099+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e471: 8 total, 6 up, 8 in 2024-09-20T19:28:36.843 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.870 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:36.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:36.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:37.002 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:37.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:37.134 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:37.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:37.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:37.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:37.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:37.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.637 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:37 smithi162 bash[24557]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.637 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:37 smithi162 bash[24557]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:37.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:37.785 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:37.786 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:37 smithi042 bash[22706]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:37 smithi042 bash[22706]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:37 smithi042 bash[32075]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:37 smithi042 bash[32075]: cluster 2024-09-20T19:28:36.157872+0000 mgr.y (mgr.24491) 404 : cluster [DBG] pgmap v673: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:37.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:37.918 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:37.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:38.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:38.118 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:38.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:38.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:38.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:38.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:38.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:38.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:38.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:38.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.688 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:38.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:38.816 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:38.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:38.946 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:38.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:39.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:39.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:39.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:39.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:39.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:39.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:39.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:39 smithi042 bash[22706]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:39 smithi042 bash[22706]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:39 smithi042 bash[32075]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:39 smithi042 bash[32075]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.597 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:39.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:39.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:39.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:39 smithi162 bash[24557]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:39 smithi162 bash[24557]: cluster 2024-09-20T19:28:38.159138+0000 mgr.y (mgr.24491) 405 : cluster [DBG] pgmap v674: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:39.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:39.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:39.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:39.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:39.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:40.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:40.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.221 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:28:40.221 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:28:40.228 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:40.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:40.358 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:40.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:40.539 INFO:teuthology.orchestra.run.smithi042.stderr:set pool 7 pg_num to 26 2024-09-20T19:28:40.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:40.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:40 smithi162 bash[24557]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:40 smithi162 bash[24557]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:40 smithi162 bash[24557]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:40 smithi162 bash[24557]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:40.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:40.750 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:40.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:40.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:40.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[32075]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[32075]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[32075]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[32075]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[22706]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[22706]: audit 2024-09-20T19:28:39.374866+0000 mgr.y (mgr.24491) 406 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[22706]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:40 smithi042 bash[22706]: audit 2024-09-20T19:28:40.140115+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-20T19:28:40.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:41.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:41.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:41.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:41.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:41.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:41.400 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:41.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:41.633 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.659 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:41.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:41.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:41 smithi162 bash[24557]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.847 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[32075]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: cluster 2024-09-20T19:28:40.159916+0000 mgr.y (mgr.24491) 407 : cluster [DBG] pgmap v675: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: audit 2024-09-20T19:28:40.540269+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.42:0/4009480979' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:41 smithi042 bash[22706]: cluster 2024-09-20T19:28:40.547560+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e472: 8 total, 6 up, 8 in 2024-09-20T19:28:41.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:41.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:41.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:42.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.051 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:42.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:42.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:42.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:42.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:42.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:42 smithi162 bash[24557]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:42 smithi162 bash[24557]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:42 smithi162 bash[24557]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:42 smithi162 bash[24557]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.568 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:42.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:42.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:42.798 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.814 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[32075]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[32075]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[32075]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[32075]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[22706]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[22706]: audit 2024-09-20T19:28:42.163345+0000 mon.a (mon.0) 1600 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[22706]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:42 smithi042 bash[22706]: audit 2024-09-20T19:28:42.503752+0000 mon.a (mon.0) 1601 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:42.832 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:42.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:28:42.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:42.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:43.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:43.091 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:43.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:43.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:28:43.294 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:28:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:28:43.312 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:28:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:28:43.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:43.329 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:28:43.330 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:28:43.330 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:28:43.349 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:28:43.349 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:28:43.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:43.368 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:28:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:28:43.389 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:28:43.389 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:28:43.406 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:43.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:43.482 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:43.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:43.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[32075]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:43 smithi042 bash[22706]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: cluster 2024-09-20T19:28:42.161679+0000 mgr.y (mgr.24491) 408 : cluster [DBG] pgmap v677: 158 pgs: 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: audit 2024-09-20T19:28:42.553962+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: cluster 2024-09-20T19:28:42.566481+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e473: 8 total, 6 up, 8 in 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:43 smithi162 bash[24557]: audit 2024-09-20T19:28:43.276636+0000 mon.c (mon.2) 70 : audit [DBG] from='client.? v1:172.21.15.42:0/453385537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:43.737 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:43.840 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:43.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:43.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:43.969 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:43.998 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:44.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:44.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:44.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:44.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:44.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:44.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:44.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:44.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:44.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:44 smithi162 bash[24557]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:44 smithi162 bash[24557]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:44 smithi162 bash[24557]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:44 smithi162 bash[24557]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:44.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:44.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[32075]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[32075]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[32075]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[32075]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[22706]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[22706]: cluster 2024-09-20T19:28:43.565469+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e474: 8 total, 6 up, 8 in 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[22706]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:44 smithi042 bash[22706]: audit 2024-09-20T19:28:44.165180+0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-20T19:28:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:44.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:44.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:44.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:45.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.038 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:28:45.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:28:45.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:45.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:45.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:45.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:45.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:45.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[32075]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:45 smithi042 bash[22706]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:45.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:45.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:45.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: cluster 2024-09-20T19:28:44.163529+0000 mgr.y (mgr.24491) 409 : cluster [DBG] pgmap v680: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: audit 2024-09-20T19:28:44.579990+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:45 smithi162 bash[24557]: cluster 2024-09-20T19:28:44.586791+0000 mon.a (mon.0) 1607 : cluster [DBG] osdmap e475: 8 total, 6 up, 8 in 2024-09-20T19:28:45.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:45.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:46.045 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:46.065 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:46.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:46.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:46.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:46.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:46.428 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:46.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:46.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:46.591 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:28:46.652 INFO:tasks.rados.rados.0.smithi042.stdout:1519: expect (ObjNum 442 snap 145 seq_num 442) 2024-09-20T19:28:46.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:46.745 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:28:46.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.838 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[22706]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[22706]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[22706]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[22706]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[32075]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[32075]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[32075]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:46 smithi042 bash[32075]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:46.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:46 smithi162 bash[24557]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:46 smithi162 bash[24557]: cluster 2024-09-20T19:28:45.593971+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e476: 8 total, 6 up, 8 in 2024-09-20T19:28:46.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:46 smithi162 bash[24557]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.952 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:46 smithi162 bash[24557]: audit 2024-09-20T19:28:46.215492+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:28:46.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:46.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:47.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.091 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:47.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:28:47.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:47.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:47.326 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:47.343 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:47.445 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.469 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:47.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:28:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:28:47.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:47.726 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:47.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:47.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[22706]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:47 smithi042 bash[32075]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.940 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 4] live_osds: [1, 2, 3, 5, 6, 7] 2024-09-20T19:28:47.940 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:28:47.940 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 2, 3, 5, 6, 7] 2024-09-20T19:28:47.940 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T19:28:47.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3 2024-09-20T19:28:47.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:47.968 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.165239+0000 mgr.y (mgr.24491) 410 : cluster [DBG] pgmap v683: 168 pgs: 10 unknown, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 386/1329 objects degraded (29.044%) 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.589712+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: audit 2024-09-20T19:28:46.591931+0000 mon.a (mon.0) 1611 : audit [INF] from='client.? v1:172.21.15.42:0/3361038273' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.969 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:47 smithi162 bash[24557]: cluster 2024-09-20T19:28:46.600881+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e477: 8 total, 6 up, 8 in 2024-09-20T19:28:47.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:47.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:48.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:48.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:48.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:48.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:48.428 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:47 smithi042 systemd[1]: Stopping Ceph osd.3 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:28:48.429 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[51953]: debug 2024-09-20T19:28:48.106+0000 7f3ae42a0640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:28:48.429 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[51953]: debug 2024-09-20T19:28:48.106+0000 7f3ae42a0640 -1 osd.3 478 *** Got signal Terminated *** 2024-09-20T19:28:48.429 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[51953]: debug 2024-09-20T19:28:48.106+0000 7f3ae42a0640 -1 osd.3 478 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:28:48.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:48.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:48.662 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:48.691 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:48.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:48 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:48.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.422461+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.424463+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.569476+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.571910+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:47.628101+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e478: 8 total, 6 up, 8 in 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.112503+0000 mon.a (mon.0) 1614 : cluster [INF] osd.3 marked itself down and dead 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.175799+0000 mon.a (mon.0) 1615 : cluster [WRN] Health check update: 3 osds down (OSD_DOWN) 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:48 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.182131+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e479: 8 total, 5 up, 8 in 2024-09-20T19:28:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:48.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:49.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:49.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:49.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:49.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:49.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:49.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:49.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:49.605 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:49.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:49.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:28:49.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:49.732 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[103246]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:28:49.733 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[103367]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[22706]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.734 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:49 smithi042 bash[32075]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.167066+0000 mgr.y (mgr.24491) 411 : cluster [DBG] pgmap v686: 168 pgs: 9 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 undersized+peered, 51 active+undersized, 7 undersized+degraded+peered, 37 active+undersized+degraded, 56 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.423139+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.424980+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.597524+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.600373+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:28:49.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:48.625279+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: Degraded data redundancy: 282/1329 objects degraded (21.219%), 45 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.163397+0000 osd.2 (osd.2) 163 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.166983+0000 osd.2 (osd.2) 164 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:49 smithi162 bash[24557]: cluster 2024-09-20T19:28:49.187601+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e480: 8 total, 5 up, 8 in 2024-09-20T19:28:49.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:49.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:49.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:50.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:50.120 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:50.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:50.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:50.353 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:50.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:50.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:50.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:50.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:50.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[32075]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[32075]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[32075]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[32075]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[22706]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[22706]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[22706]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:50 smithi042 bash[22706]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:50 smithi162 bash[24557]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:50 smithi162 bash[24557]: audit 2024-09-20T19:28:49.383584+0000 mgr.y (mgr.24491) 412 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:28:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:50 smithi162 bash[24557]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:50 smithi162 bash[24557]: cluster 2024-09-20T19:28:50.185232+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e481: 8 total, 5 up, 8 in 2024-09-20T19:28:50.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:50.898 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:50.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:51.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:51.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.162 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:51.265 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:51.294 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:51.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:51.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:51.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:51.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:51.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:51.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:51 smithi162 bash[24557]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:51.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[22706]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:50.168262+0000 mgr.y (mgr.24491) 413 : cluster [DBG] pgmap v689: 168 pgs: 1 stale+active+undersized+remapped, 9 stale+active+undersized, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 10 stale+active+undersized+degraded, 15 stale+active+clean, 8 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 undersized+peered, 42 active+undersized, 5 undersized+degraded+peered, 27 active+undersized+degraded, 41 active+clean; 491 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2 op/s; 282/1329 objects degraded (21.219%); 38/1329 objects misplaced (2.859%); 31 MiB/s, 45 objects/s recovering 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:51.160593+0000 osd.2 (osd.2) 165 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:28:51.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:51 smithi042 bash[32075]: cluster 2024-09-20T19:28:51.165596+0000 osd.2 (osd.2) 166 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:28:51.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:51.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:52.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:52.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:52.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:52.239 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:52.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:52.325 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:52.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:52.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:52.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:52.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:52.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:52.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[22706]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[22706]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[22706]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[22706]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[32075]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[32075]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[32075]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:52 smithi042 bash[32075]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:52.961 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:52 smithi162 bash[24557]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.961 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:52 smithi162 bash[24557]: cluster 2024-09-20T19:28:52.171095+0000 mgr.y (mgr.24491) 414 : cluster [DBG] pgmap v691: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:52.961 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:52 smithi162 bash[24557]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.961 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:52 smithi162 bash[24557]: cluster 2024-09-20T19:28:52.639986+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-09-20T19:28:52.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:52.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:53.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:53.104 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:53.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:53.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:53.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:53.363 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:53.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:53.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:53.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:53.622 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:53.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:53.745 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:53.794 INFO:teuthology.orchestra.run.smithi042.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:28:53.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:53.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:53.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:54.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:54.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:54.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:54.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:54.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:54.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:54.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T19:28:54.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:28:54.742 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T19:28:54.742 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-20T19:28:54.742 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T19:28:54.742 INFO:tasks.thrashosds.thrasher:remote for exp osd 0 is ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T19:28:54.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-20T19:28:54.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:54.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:54.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:54.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:54.973 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:28:55.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:55.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:55.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:55.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.284 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:55 smithi162 bash[24557]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.284 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:55 smithi162 bash[24557]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:55.401 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:55.533 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:55.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:55 smithi042 bash[32075]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:55 smithi042 bash[32075]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:55 smithi042 bash[22706]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:55 smithi042 bash[22706]: cluster 2024-09-20T19:28:54.172909+0000 mgr.y (mgr.24491) 415 : cluster [DBG] pgmap v692: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:55.561 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:55.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:55.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:55.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:55.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:55.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:56.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:56.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.209 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:56.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:56.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:56.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:56.494 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:28:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:28:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:28:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:28:56.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:28:56.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:28:56.673 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:28:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:28:56.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:28:56.692 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:28:56.692 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:28:56.692 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:28:56.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:56.713 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:28:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:28:56.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:56.734 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:28:56.734 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:28:56.755 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:28:56.756 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:28:56.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:56.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:56.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:28:57.087 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:57.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:57.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:57.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:57.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[22706]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[22706]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[22706]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[22706]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[32075]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[32075]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[32075]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:57 smithi042 bash[32075]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:57.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:57 smithi162 bash[24557]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:57 smithi162 bash[24557]: cluster 2024-09-20T19:28:56.174695+0000 mgr.y (mgr.24491) 416 : cluster [DBG] pgmap v693: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 2 active+clean+remapped+snaptrim, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 23 active+clean; 449 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 449/1266 objects degraded (35.466%); 28/1266 objects misplaced (2.212%) 2024-09-20T19:28:57.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:57 smithi162 bash[24557]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.478 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:57 smithi162 bash[24557]: audit 2024-09-20T19:28:56.655320+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.42:0/2228733649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:28:57.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:57.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:57.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.758 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:57.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:57.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:57.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:58.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:58.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:58.121 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:58.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:58 smithi162 bash[24557]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.263 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:58 smithi162 bash[24557]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.263 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:58 smithi162 bash[24557]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.263 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:58 smithi162 bash[24557]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:58.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:58.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[22706]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[22706]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[22706]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[22706]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[32075]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[32075]: audit 2024-09-20T19:28:57.533430+0000 mon.a (mon.0) 1621 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[32075]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:58 smithi042 bash[32075]: cluster 2024-09-20T19:28:58.178031+0000 mon.a (mon.0) 1622 : cluster [WRN] Health check update: Degraded data redundancy: 449/1266 objects degraded (35.466%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:28:58.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:58.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:58.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:58.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:28:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:28:58.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:58.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:58.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:59.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:59.045 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:28:59.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:59.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:59.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.378 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[32075]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:28:59 smithi042 bash[22706]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:28:59.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: cluster 2024-09-20T19:28:58.176126+0000 mgr.y (mgr.24491) 417 : cluster [DBG] pgmap v694: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:58.493032+0000 mon.a (mon.0) 1623 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.024785+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.030368+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.081846+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:28:59 smithi162 bash[24557]: audit 2024-09-20T19:28:59.087095+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:28:59.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:28:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:28:59.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:59.550 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:28:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:28:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:28:59.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:28:59.653 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:59.776 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:59.798 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:28:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:28:59.900 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:59.922 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:28:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:28:59.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:00.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:00.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:00.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:00.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:00 smithi162 bash[24557]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:00.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[32075]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.391267+0000 mgr.y (mgr.24491) 418 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.705782+0000 mon.a (mon.0) 1628 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.706701+0000 mon.a (mon.0) 1629 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:00 smithi042 bash[22706]: audit 2024-09-20T19:28:59.711705+0000 mon.a (mon.0) 1630 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:00.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:00.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:00.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:00.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:00.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:00.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:00.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:01.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:01.095 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:29:01.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.214 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:01.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:01.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:01 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:01 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:01.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:29:01.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:01 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.567 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:01 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:01 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:01 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.177158+0000 mgr.y (mgr.24491) 419 : cluster [DBG] pgmap v695: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 433 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1239 objects degraded (36.239%); 28/1239 objects misplaced (2.260%) 2024-09-20T19:29:01.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:01.602 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:01.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.734 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:01.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.863 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:01.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:01.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:02.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.123 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:02.220 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:29:02.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.235 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[32075]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.237 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.237 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:02 smithi042 bash[22706]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:02.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:02.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.470472+0000 osd.6 (osd.6) 134 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.472900+0000 osd.6 (osd.6) 135 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.753888+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: cluster 2024-09-20T19:29:00.755357+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: audit 2024-09-20T19:29:01.214052+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.42:0/376745150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:02 smithi162 bash[24557]: audit 2024-09-20T19:29:01.214445+0000 mon.a (mon.0) 1631 : audit [INF] from='client.25328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:02.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:02.586 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:02.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.637 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:02.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:02.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:02.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:02.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:03.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:03.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.274 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:03 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.285 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:03.388 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:01.438038+0000 osd.6 (osd.6) 136 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:01.442754+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.178709+0000 mgr.y (mgr.24491) 420 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.201618+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: audit 2024-09-20T19:29:02.211609+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:02.215791+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e482: 8 total, 5 up, 8 in 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.179042+0000 mon.a (mon.0) 1635 : cluster [WRN] Health check update: Degraded data redundancy: 449/1233 objects degraded (36.415%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:03 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.186846+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e483: 8 total, 5 up, 8 in 2024-09-20T19:29:03.418 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:03.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:03.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:03.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:03.703 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:03.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:03.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:03.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:03.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:03.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:03.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:7.d 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:7.5 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:7.4 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:6.d 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:6.9 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:6.16 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:6.11 2024-09-20T19:29:04.003 INFO:teuthology.orchestra.run.smithi042.stdout:6.10 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:6.0 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.e 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.c 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:7.e 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.8 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.5 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.2 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:7.7 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.1e 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.1d 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:2.1f 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:2.1e 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:7.9 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:2.1 2024-09-20T19:29:04.004 INFO:teuthology.orchestra.run.smithi042.stdout:5.1b 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:2.19 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:4.1e 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:2.13 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:1.6 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:7.6 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:1.2 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:1.0 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:6.a 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:2.4 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:2.1d 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:1.1 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:2.15 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:5.3 2024-09-20T19:29:04.005 INFO:teuthology.orchestra.run.smithi042.stdout:4.b 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:2.5 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:2.c 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:2.10 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.19 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:3.1 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.0 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:7.f 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.16 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:6.13 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.3 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.1a 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.6 2024-09-20T19:29:04.006 INFO:teuthology.orchestra.run.smithi042.stdout:4.7 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:5.b 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:2.0 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:2.f 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:5.13 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:5.7 2024-09-20T19:29:04.007 INFO:teuthology.orchestra.run.smithi042.stdout:5.15 2024-09-20T19:29:04.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:04.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:04.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:04.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:04.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:04.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:04.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:04.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:04.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:04.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:04.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:04.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:04.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:04.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:04.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:05.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.130 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:05.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:05.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.244 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:05 smithi162 bash[24557]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:05.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[32075]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.374 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.425519+0000 osd.6 (osd.6) 138 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:29:05.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:03.434582+0000 osd.6 (osd.6) 139 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:29:05.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:05 smithi042 bash[22706]: cluster 2024-09-20T19:29:04.180598+0000 mgr.y (mgr.24491) 421 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:05.391 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:05.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:05.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:05.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:05.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:05.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:05.881 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:05.910 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:06.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:06.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:06.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 5.e --file /var/log/ceph/exp.5.e.0 2024-09-20T19:29:06.140 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:06.185 INFO:teuthology.orchestra.run.smithi042.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:29:06.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:29:06.261 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:29:06.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:06.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:06.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:06.428 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:06.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:06.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:06.651 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.677 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:06.780 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:06.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:06.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:06.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:07.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:07.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:07.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:07.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:07.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:29:07.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:29:07.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:07.470 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:07 smithi042 bash[32075]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.476 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:07 smithi042 bash[32075]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.476 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:07 smithi042 bash[22706]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.476 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:07 smithi042 bash[22706]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:07 smithi162 bash[24557]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:07 smithi162 bash[24557]: cluster 2024-09-20T19:29:06.181427+0000 mgr.y (mgr.24491) 422 : cluster [DBG] pgmap v700: 168 pgs: 2 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 24 active+clean; 427 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1233 objects degraded (36.415%); 27/1233 objects misplaced (2.190%) 2024-09-20T19:29:07.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:07.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:07.730 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:07.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:07.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:07.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:07.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:07.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:08.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.118 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:08.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:08.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:08.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.361 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:08 smithi162 bash[24557]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.361 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:08 smithi162 bash[24557]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:08.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:08.506 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:08.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:08 smithi042 bash[32075]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.508 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:08 smithi042 bash[32075]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:08 smithi042 bash[22706]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.508 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:08 smithi042 bash[22706]: cluster 2024-09-20T19:29:08.211511+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: Degraded data redundancy: 449/1215 objects degraded (36.955%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:08.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:08.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.759 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:08.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:08.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:08.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:09.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:09.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:09.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:09.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:09.383 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:09 smithi042 bash[32075]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.395 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:09 smithi042 bash[32075]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:09 smithi042 bash[22706]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:09 smithi042 bash[22706]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.409 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:09.512 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:09.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:09.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:09 smithi162 bash[24557]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:09 smithi162 bash[24557]: cluster 2024-09-20T19:29:08.184124+0000 mgr.y (mgr.24491) 423 : cluster [DBG] pgmap v701: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:09.634 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:29:09.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.664 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:09.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:09.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:09.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:09.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:09.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:10.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.054 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:10.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:10.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:10.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:10.298 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:10 smithi042 bash[32075]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.299 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:10 smithi042 bash[32075]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:10 smithi042 bash[22706]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.299 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:10 smithi042 bash[22706]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.314 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:10.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:10.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:10 smithi162 bash[24557]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:10 smithi162 bash[24557]: audit 2024-09-20T19:29:09.402172+0000 mgr.y (mgr.24491) 424 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:10.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:10.575 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:10.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:10.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:10.839 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:10.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.968 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:10.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:11.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:11.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:11.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:11.227 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:11.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[32075]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:11 smithi042 bash[22706]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:11.491 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:11.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.185334+0000 mgr.y (mgr.24491) 425 : cluster [DBG] pgmap v702: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:11.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.629244+0000 osd.5 (osd.5) 171 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:11 smithi162 bash[24557]: cluster 2024-09-20T19:29:10.629664+0000 osd.5 (osd.5) 172 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:11.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:11.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:11.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:11.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:11.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:11.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:11.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:11.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:12.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:12.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:12.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:12.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:12.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:12.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:12 smithi042 bash[22706]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:12 smithi042 bash[22706]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:12 smithi042 bash[32075]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.522 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:12 smithi042 bash[32075]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:12.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:12 smithi162 bash[24557]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:12 smithi162 bash[24557]: cluster 2024-09-20T19:29:11.288982+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e484: 8 total, 5 up, 8 in 2024-09-20T19:29:12.634 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:12.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:12.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:29:12.738 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:29:12.738 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:29:12.757 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:29:12.757 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:29:12.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:29:12.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:29:12.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.779 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:29:12.779 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:29:12.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:12.801 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:29:12.801 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:29:12.822 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:29:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:29:12.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:12.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:12.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:13.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.051 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:13.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:13.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:13.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:13.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:13.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.438 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:13.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:13.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:13.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:13.804 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.815 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:13.816 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:13.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:13.818 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:13.818 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:13.818 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:13 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:13.829 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:13.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:13.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.187330+0000 mgr.y (mgr.24491) 426 : cluster [DBG] pgmap v704: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.567079+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.568396+0000 mon.a (mon.0) 1640 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: audit 2024-09-20T19:29:12.719644+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.42:0/1061693431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.866360+0000 osd.1 (osd.1) 189 : cluster [DBG] 7.b scrub starts 2024-09-20T19:29:14.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:14.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:12.871556+0000 osd.1 (osd.1) 190 : cluster [DBG] 7.b scrub ok 2024-09-20T19:29:14.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:14.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:13 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.189541+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e485: 8 total, 5 up, 8 in 2024-09-20T19:29:14.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:14.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:14.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:Exporting 5.e info 5.e( v 77'11 (0'0,77'11] local-lis/les=73/74 n=1 ec=73/73 lis/c=73/73 les/c/f=74/74/0 sis=73) 2024-09-20T19:29:14.194 INFO:teuthology.orchestra.run.smithi042.stderr:Read #5:7759931f:::notify.3:head# 2024-09-20T19:29:14.194 INFO:teuthology.orchestra.run.smithi042.stderr:Export successful 2024-09-20T19:29:14.215 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:14.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:14.346 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:14.449 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:14.476 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:14.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:14.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[22706]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:14 smithi042 bash[32075]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:14.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:14 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:14 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.830914+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:14 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:14 smithi162 bash[24557]: cluster 2024-09-20T19:29:13.831986+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:14.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:14.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:14.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:15.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:15.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:15.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:15.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:15.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:15.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:15.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:15.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:15.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:15 smithi162 bash[24557]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:15 smithi162 bash[24557]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:15 smithi162 bash[24557]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:15 smithi162 bash[24557]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:15.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:15.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[22706]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[22706]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[22706]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[22706]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[32075]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[32075]: cluster 2024-09-20T19:29:14.188237+0000 mgr.y (mgr.24491) 427 : cluster [DBG] pgmap v706: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[32075]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:15 smithi042 bash[32075]: cluster 2024-09-20T19:29:14.578387+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e486: 8 total, 5 up, 8 in 2024-09-20T19:29:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:29:15.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:29:15.879 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:15.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:16.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.029 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:16.132 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.156 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:16.259 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:16.285 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:16.388 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:16.399 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:16.416 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:16.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:16.545 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:16.584 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:29:16.590 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.590 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.590 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.590 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[32075]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:16 smithi042 bash[22706]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.675 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:16.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: cluster 2024-09-20T19:29:15.356952+0000 osd.7 (osd.7) 139 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:29:16.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: cluster 2024-09-20T19:29:15.367374+0000 osd.7 (osd.7) 140 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:29:16.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:16 smithi162 bash[24557]: audit 2024-09-20T19:29:16.251842+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:29:16.805 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:16.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:16.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:16.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:17.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:17.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.0 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 5.e 2024-09-20T19:29:17.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:17.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:17.340 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.0/config 2024-09-20T19:29:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:17.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:17.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:17.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:17.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:17.729 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[32075]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:17.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:17 smithi042 bash[22706]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:17.848 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:17.877 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:29:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:17.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:17.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.005 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:18.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:18.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.189356+0000 mgr.y (mgr.24491) 428 : cluster [DBG] pgmap v708: 168 pgs: 1 active+clean+snaptrim, 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 25 active+clean; 419 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1215 objects degraded (36.955%); 22/1215 objects misplaced (1.811%) 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.576279+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: audit 2024-09-20T19:29:16.584565+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.42:0/4022156906' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:18.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:17 smithi162 bash[24557]: cluster 2024-09-20T19:29:16.595910+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e487: 8 total, 5 up, 8 in 2024-09-20T19:29:18.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.130 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:18.233 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:18.361 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:18.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:18.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:29:18.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:18.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:18.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:18.882 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:18.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:19.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:19.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:19.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:19.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:19 smithi162 bash[24557]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.287 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:19 smithi162 bash[24557]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:19.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:19.434 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:19.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:19.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:19 smithi042 bash[32075]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:19 smithi042 bash[32075]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:19 smithi042 bash[22706]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:19 smithi042 bash[22706]: cluster 2024-09-20T19:29:18.190809+0000 mgr.y (mgr.24491) 429 : cluster [DBG] pgmap v710: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:19.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:19.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:19.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:19.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:19.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:19.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:19.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:19.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:20.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:20.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:20.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:20.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:20.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[22706]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[22706]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[22706]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[22706]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[32075]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[32075]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[32075]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:20 smithi042 bash[32075]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.451 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:29:20.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T19:29:20.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:20 smithi162 bash[24557]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:20 smithi162 bash[24557]: cluster 2024-09-20T19:29:19.187036+0000 mon.a (mon.0) 1647 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:20.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:20 smithi162 bash[24557]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:20 smithi162 bash[24557]: audit 2024-09-20T19:29:19.413364+0000 mgr.y (mgr.24491) 430 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:20.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:20.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:20.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:20.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:20.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:20.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:20.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:20.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:20.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:29:21.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:21.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:21.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:21.248 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:21 smithi162 bash[24557]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.248 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:21 smithi162 bash[24557]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:21.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:21.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.501 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:29:21.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:21.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:21 smithi042 bash[32075]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:21 smithi042 bash[32075]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:21 smithi042 bash[22706]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:21 smithi042 bash[22706]: cluster 2024-09-20T19:29:20.192543+0000 mgr.y (mgr.24491) 431 : cluster [DBG] pgmap v711: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:21.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:21.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:21.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:21.893 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:21.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:21.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:22.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:22.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:22.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:22.418 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:22.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:22.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:22.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:22.910 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:22.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:22.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:23.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:23.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:23.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:23.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:23.427 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:23 smithi042 bash[32075]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.428 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:23 smithi042 bash[32075]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:23 smithi042 bash[22706]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.428 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:23 smithi042 bash[22706]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.440 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:23.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:23.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:23 smithi162 bash[24557]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:23 smithi162 bash[24557]: cluster 2024-09-20T19:29:22.196275+0000 mgr.y (mgr.24491) 432 : cluster [DBG] pgmap v712: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:23.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:23.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:23.700 INFO:teuthology.orchestra.run.smithi042.stdout: marking collection for removal 2024-09-20T19:29:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:23.701 INFO:teuthology.orchestra.run.smithi042.stdout:setting '_remove' omap key 2024-09-20T19:29:23.701 INFO:teuthology.orchestra.run.smithi042.stdout:finish_remove_pgs 5.e_head removing 5.e 2024-09-20T19:29:23.701 INFO:teuthology.orchestra.run.smithi042.stdout:Remove successful 2024-09-20T19:29:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:23.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:23.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:23.933 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.961 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:23.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:24.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:24.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:24.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:24.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:24.564 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:29:24.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:24.582 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:29:24.582 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:29:24.601 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:29:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:29:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:29:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:29:24.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:24.620 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:29:24.620 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:29:24.639 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:29:24.639 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:29:24.658 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:29:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:29:24.708 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:24.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:24.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:24.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:25.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:25.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:25.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:25.269 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:29:25.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.3 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-20T19:29:25.351 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[22706]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[22706]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[22706]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[22706]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[32075]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[32075]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[32075]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.387 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:25 smithi042 bash[32075]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:25.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.516 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T19:29:25.523 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:25.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:25 smithi162 bash[24557]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:25 smithi162 bash[24557]: cluster 2024-09-20T19:29:24.198115+0000 mgr.y (mgr.24491) 433 : cluster [DBG] pgmap v713: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:25.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:25 smithi162 bash[24557]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:25 smithi162 bash[24557]: audit 2024-09-20T19:29:24.563477+0000 mon.a (mon.0) 1648 : audit [DBG] from='client.? v1:172.21.15.42:0/635392725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:25.626 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.652 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:25.755 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:25.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:25.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:25.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:26.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:26.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:29:26.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:29:26.272 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:26.301 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:26.404 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:26.432 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:26.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:26.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:26.674 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:26.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:26.826 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:29:26.826 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:29:26.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:26.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:26.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:27.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:27.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:27.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:27 smithi162 bash[24557]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:27 smithi162 bash[24557]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:27.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:27 smithi042 bash[22706]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:27 smithi042 bash[22706]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:27 smithi042 bash[32075]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:27 smithi042 bash[32075]: cluster 2024-09-20T19:29:26.199892+0000 mgr.y (mgr.24491) 434 : cluster [DBG] pgmap v714: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:27.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:27.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:27.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T19:29:27.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:27.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:27.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:27.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:28.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:28.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:29:28.234 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:28.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:28.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:28.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:28.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:28.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:28 smithi162 bash[24557]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:28 smithi162 bash[24557]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:28 smithi162 bash[24557]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:28 smithi162 bash[24557]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.743 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:28.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:28.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:28.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[22706]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[22706]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[22706]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[22706]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[32075]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[32075]: audit 2024-09-20T19:29:27.589452+0000 mon.a (mon.0) 1649 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[32075]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:28 smithi042 bash[32075]: audit 2024-09-20T19:29:27.591638+0000 mon.a (mon.0) 1650 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:28.978 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:29.000 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:29.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:29.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:29.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:29.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:29:29.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:29.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:29.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:29.615 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:29:29.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:29.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:29 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:28.202650+0000 mgr.y (mgr.24491) 435 : cluster [DBG] pgmap v715: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: audit 2024-09-20T19:29:28.971398+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.42:0/2732659610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: audit 2024-09-20T19:29:28.971936+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.069825+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.825 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:29 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.071438+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:29.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:29.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:30.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:30.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:30.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:30.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:30.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:30.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:30.529 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.558 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:30.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:30.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:30 smithi162 bash[24557]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.688 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:30.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[22706]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: audit 2024-09-20T19:29:29.423154+0000 mgr.y (mgr.24491) 436 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.591418+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: audit 2024-09-20T19:29:29.604160+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:30.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.804 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:30 smithi042 bash[32075]: cluster 2024-09-20T19:29:29.612906+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e488: 8 total, 5 up, 8 in 2024-09-20T19:29:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:30.921 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:30.952 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:30.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:31.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:31.086 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:31.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:31.209 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:31.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:31.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:31.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:31.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:7.c 2024-09-20T19:29:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:7.7 2024-09-20T19:29:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:7.3 2024-09-20T19:29:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:7.16 2024-09-20T19:29:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:7.2 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.14 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.13 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.12 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.10 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:4.1b 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:3.2 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:4.14 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:4.10 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.a 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:6.19 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:7.0 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:4.0 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:4.19 2024-09-20T19:29:31.625 INFO:teuthology.orchestra.run.smithi042.stdout:2.e 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:2.1 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:2.a 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:1.4 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:5.13 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:4.17 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:5.0 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:1.6 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:3.1 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:5.a 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:4.9 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:4.15 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:7.6 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:1.7 2024-09-20T19:29:31.626 INFO:teuthology.orchestra.run.smithi042.stdout:5.16 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.12 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.19 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.1e 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:1.8 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:5.14 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.13 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:1.9 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:6.5 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.9 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.1b 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:6.1f 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:2.8 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:4.1c 2024-09-20T19:29:31.627 INFO:teuthology.orchestra.run.smithi042.stdout:6.13 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.11 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.1d 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:7.11 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.1e 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.1a 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.16 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:6.7 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:2.14 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.4 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:6.2 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:4.c 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:5.1 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:5.1c 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:7.f 2024-09-20T19:29:31.628 INFO:teuthology.orchestra.run.smithi042.stdout:6.b 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.0 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:7.17 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.6 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:5.17 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.11 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.16 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.1b 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:2.d 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.c 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.4 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.8 2024-09-20T19:29:31.629 INFO:teuthology.orchestra.run.smithi042.stdout:6.f 2024-09-20T19:29:31.704 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:31.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:31.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:31.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:31 smithi162 bash[24557]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:31 smithi162 bash[24557]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.863 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:31.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:31 smithi042 bash[22706]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:31 smithi042 bash[22706]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:31 smithi042 bash[32075]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:31 smithi042 bash[32075]: cluster 2024-09-20T19:29:30.204151+0000 mgr.y (mgr.24491) 437 : cluster [DBG] pgmap v717: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:31.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:31.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:32.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:32.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:32.259 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:32.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:32.390 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:32.439 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:29:32.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:32.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:32.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:32.753 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.784 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:32.844 INFO:tasks.thrashosds.thrasher:Moving pg 5.e from osd.0 to osd.3 2024-09-20T19:29:32.845 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell -n osd.3 --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.e.0 2024-09-20T19:29:32.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:32.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:32.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:33.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:29:33.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.022 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T19:29:33.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:33.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.171 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:33.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:33.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:33.423 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:33.526 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:33.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:33 smithi162 bash[24557]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:33 smithi162 bash[24557]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:33 smithi162 bash[24557]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:33 smithi162 bash[24557]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:33.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:33.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[32075]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[32075]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[32075]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[32075]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[22706]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[22706]: cluster 2024-09-20T19:29:32.207034+0000 mgr.y (mgr.24491) 438 : cluster [DBG] pgmap v718: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[22706]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:33 smithi042 bash[22706]: cluster 2024-09-20T19:29:33.193655+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e489: 8 total, 5 up, 8 in 2024-09-20T19:29:33.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:33.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:33.989 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:29:33.990 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:29:34.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:34.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:34.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:34.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:34.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:34.337 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:34.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.454 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:34.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:34.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:34.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:34.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T19:29:34.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:34.846 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:34.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:34.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:35.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:35.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:35.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:35.230 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:35.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:35.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:35.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:35.596 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:35.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:35.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:35.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:35.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:35 smithi162 bash[24557]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:35 smithi162 bash[24557]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:35.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:35.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:35 smithi042 bash[32075]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:35 smithi042 bash[32075]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:35 smithi042 bash[22706]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:35 smithi042 bash[22706]: cluster 2024-09-20T19:29:34.208261+0000 mgr.y (mgr.24491) 439 : cluster [DBG] pgmap v720: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:35.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:36.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:36.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:36.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:36.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:36.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:36.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:36.622 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:36.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.654 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:36 smithi162 bash[24557]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.654 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:36 smithi162 bash[24557]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.654 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:36 smithi162 bash[24557]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.654 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:36 smithi162 bash[24557]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.664 INFO:teuthology.orchestra.run.smithi042.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T19:29:36.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:36.773 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:36.794 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:29:36.893 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:29:36.893 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:29:36.897 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:36.905 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[32075]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.905 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[32075]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.905 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[32075]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.905 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[32075]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[22706]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.906 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[22706]: cluster 2024-09-20T19:29:35.655029+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:36.906 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[22706]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.906 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:36 smithi042 bash[22706]: cluster 2024-09-20T19:29:35.655973+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:36.911 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:29:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:29:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:29:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:29:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:36.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:36.934 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:29:36.934 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:29:36.955 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:29:36.955 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:29:36.976 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:29:36.976 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:29:37.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:37.147 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:37.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:37.387 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:37.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:37.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:37.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:37.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:37.798 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:37 smithi162 bash[24557]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.824 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:37.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[22706]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: cluster 2024-09-20T19:29:36.209968+0000 mgr.y (mgr.24491) 440 : cluster [DBG] pgmap v721: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: cluster 2024-09-20T19:29:36.627141+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e490: 8 total, 5 up, 8 in 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.852 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:37 smithi042 bash[32075]: audit 2024-09-20T19:29:36.871790+0000 mon.a (mon.0) 1657 : audit [DBG] from='client.? v1:172.21.15.42:0/1176961935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:37.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:37.954 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:37.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:38.057 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:38.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:38.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:38.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:38.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:38.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.475 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:38.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:38.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:38.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:38.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:38.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:38.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:38.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:39.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:29:39.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:29:39.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.253 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:39.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:39 smithi162 bash[24557]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:39 smithi162 bash[24557]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:39 smithi162 bash[24557]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:39 smithi162 bash[24557]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:39.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[22706]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[22706]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[22706]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[22706]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[32075]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[32075]: cluster 2024-09-20T19:29:38.206417+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e491: 8 total, 5 up, 8 in 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[32075]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:39 smithi042 bash[32075]: cluster 2024-09-20T19:29:38.211750+0000 mgr.y (mgr.24491) 441 : cluster [DBG] pgmap v724: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:39.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:39.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:39.556 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:29:39.618 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.646 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:39.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.776 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:39.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:39.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:40.008 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:40.029 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:40.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:40.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:40.390 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:40.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:40 smithi042 bash[32075]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:40 smithi042 bash[32075]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:40 smithi042 bash[22706]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.402 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:40 smithi042 bash[22706]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.416 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:40.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:40.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:40.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:40 smithi162 bash[24557]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:40 smithi162 bash[24557]: audit 2024-09-20T19:29:39.431193+0000 mgr.y (mgr.24491) 442 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:40.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:40.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:40.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:get_pg_num_history pg_num_history pg_num_history(e479 pg_nums {1={259=11},2={69=32},3={70=3},4={71=32},5={73=32},6={75=32},7={112=16,473=26}} deleted_pools ) 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:Importing pgid 5.e 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:Write #5:7759931f:::notify.3:head# 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:snapset 0=[]:{} 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:write_pg epoch 358 info 5.e( v 77'11 (0'0,77'11] local-lis/les=73/74 n=1 ec=73/73 lis/c=73/73 les/c/f=74/74/0 sis=73) 2024-09-20T19:29:40.868 INFO:teuthology.orchestra.run.smithi042.stdout:Import successful 2024-09-20T19:29:40.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:40.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:41.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:438 7.c 7:3b97d3fe:::smithi04268048-6:head [copy-from ver 31] snapc 11=[11,10,f,e,d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:12.130520+0000", 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.92434118099999, 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027192444999999999, 2024-09-20T19:29:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 438 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130520+0000", 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130520+0000", 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130527+0000", 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7360000000000004e-06 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130550+0000", 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3564e-05 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130590+0000", 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0005000000000001e-05 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130609+0000", 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8749e-05 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130762+0000", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015228 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130774+0000", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2293000000000001e-05 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.135796+0000", 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050217409999999997 2024-09-20T19:29:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.136029+0000", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000233384 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.146341+0000", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010312045000000001 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.156594+0000", 2024-09-20T19:29:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010253332 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.156810+0000", 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000215639 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.157563+0000", 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075317099999999996 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.157644+0000", 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0797999999999994e-05 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.157713+0000", 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8707999999999993e-05 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:439 7.c 7:3b97d3fe:::smithi04268048-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-20T19:29:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:12.130651+0000", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.92421015799999, 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025815695999999999, 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 439 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130651+0000", 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130651+0000", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130653+0000", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.227e-06 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130663+0000", 2024-09-20T19:29:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5599999999999999e-06 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130681+0000", 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7833e-05 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130697+0000", 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5926e-05 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130892+0000", 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019535900000000001 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.130904+0000", 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1396000000000001e-05 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.136559+0000", 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056551350000000004 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.136663+0000", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104652 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.156158+0000", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019494559000000002 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.156306+0000", 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014798599999999999 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:12.156467+0000", 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000161063 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:443 7.7 7:e75b8502:::smithi04268048-39:head [delete] snapc 11=[11,10,f,e,d,a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e144)", 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:13.010731+0000", 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.04413009899997, 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022600017999999999, 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 443 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010731+0000", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010731+0000", 2024-09-20T19:29:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010737+0000", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2029999999999997e-06 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010752+0000", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.471e-05 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010836+0000", 2024-09-20T19:29:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3807999999999994e-05 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.010863+0000", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6869e-05 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.011039+0000", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017660199999999999 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.011061+0000", 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1209999999999999e-05 2024-09-20T19:29:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.025402+0000", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014341536 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.025655+0000", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025268699999999997 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:13.033331+0000", 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076763930000000001 2024-09-20T19:29:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:736 7.d 7:b90bdf9e:::smithi04268048-23:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e172)", 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:41.347791+0000", 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 419.70707055299999, 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021811869000000001, 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 736 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347791+0000", 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347791+0000", 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347793+0000", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9039999999999999e-06 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347801+0000", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6859999999999995e-06 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347819+0000", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7790000000000001e-05 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.347840+0000", 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1183e-05 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.348134+0000", 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029329300000000003 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.348150+0000", 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6753e-05 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.361663+0000", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013512677000000001 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.361950+0000", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028666 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.369603+0000", 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076529229999999998 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1176 7.d 7:b90bdf9e:::smithi04268048-23:head [copy-from ver 162] snapc 39=[39,38,37,36,33,30,2c] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 368.813930153, 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028448029, 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1176 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240937+0000", 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4999999999999999e-06 2024-09-20T19:29:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240956+0000", 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9012999999999999e-05 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241002+0000", 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5838999999999999e-05 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241025+0000", 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3869000000000001e-05 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241233+0000", 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020778999999999999 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241494+0000", 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000260824 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.255282+0000", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013787785 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.266475+0000", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011193602 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.267752+0000", 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001276447 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269090+0000", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001338253 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269191+0000", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010124399999999999 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269379+0000", 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187863 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.080 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1177 7.d 7:b90bdf9e:::smithi04268048-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e223)", 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:32.241104+0000", 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 368.81375770699998, 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025277630999999998, 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1177 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.081 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241104+0000", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241104+0000", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241107+0000", 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.853e-06 2024-09-20T19:29:41.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241116+0000", 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000008e-06 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241137+0000", 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1376999999999999e-05 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241169+0000", 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241891+0000", 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00072182799999999999 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.242035+0000", 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000144027 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.265869+0000", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023833963999999999 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.266118+0000", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024860799999999999 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.266381+0000", 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026343800000000001 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.085 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1192 7.7 7:e8e5d7d7:::smithi04268048-2:head [copy-from ver 147] snapc 39=[39,38,37,33,30,2c] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:33.075835+0000", 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 367.97902683299998, 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027879094, 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.086 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1192 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075835+0000", 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075835+0000", 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075836+0000", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7999999999999993e-07 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075841+0000", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0440000000000003e-06 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075849+0000", 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0139999999999998e-06 2024-09-20T19:29:41.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.075855+0000", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6089999999999997e-06 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.077545+0000", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016895479999999999 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.077598+0000", 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3003000000000001e-05 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.087861+0000", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010263105999999999 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.091504+0000", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036434150000000001 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.094465+0000", 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029612319999999998 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.103545+0000", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0090792970000000001 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.103641+0000", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6284000000000002e-05 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:33.103714+0000", 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2562000000000001e-05 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2132 7.d 7:baa4e37b:::smithi04268048-17:head [copy-from ver 407] snapc 5f=[5f,5e,5d,5a,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.81334350499998, 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034554666999999997, 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2132 2024-09-20T19:29:41.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241522+0000", 2024-09-20T19:29:41.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4150000000000001e-06 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241538+0000", 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5338000000000001e-05 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241570+0000", 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2070999999999998e-05 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241585+0000", 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4982e-05 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241798+0000", 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021353400000000001 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241819+0000", 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0426000000000001e-05 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.246976+0000", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051576520000000004 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.247111+0000", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134358 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.257327+0000", 2024-09-20T19:29:41.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010216182000000001 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.266016+0000", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0086890890000000005 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.273265+0000", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072488439999999999 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.275923+0000", 2024-09-20T19:29:41.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026584500000000001 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.276003+0000", 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9417000000000002e-05 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.276073+0000", 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9908999999999998e-05 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2355 7.c 7:33675e62:::smithi04268048-1:head [copy-from ver 311] snapc 6c=[6c,6b,6a,69,68,67,66,65,63,62,5e,5d,5a,57,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.955104216, 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035881936000000003, 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2355 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099760+0000", 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.106e-06 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099770+0000", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3549999999999994e-06 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099788+0000", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8496999999999999e-05 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099798+0000", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0020000000000001e-05 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099977+0000", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017844000000000001 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099997+0000", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0628999999999998e-05 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105625+0000", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056274300000000001 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105761+0000", 2024-09-20T19:29:41.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136654 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.121616+0000", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015854995 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.132728+0000", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011111653000000001 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.132861+0000", 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133237 2024-09-20T19:29:41.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135569+0000", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002707845 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135607+0000", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8127999999999998e-05 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135639+0000", 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1946999999999998e-05 2024-09-20T19:29:41.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2356 7.c 7:33675e62:::smithi04268048-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e316)", 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:05.099820+0000", 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.95504170700002, 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027448381000000001, 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.105 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2356 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099820+0000", 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099820+0000", 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099820+0000", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5600000000000005e-07 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099823+0000", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1640000000000001e-06 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099829+0000", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4999999999999999e-06 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099833+0000", 2024-09-20T19:29:41.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.177e-06 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.100143+0000", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030972299999999998 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.100154+0000", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0986e-05 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105926+0000", 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057725809999999997 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105991+0000", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4813000000000004e-05 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.127129+0000", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021138049999999999 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.127201+0000", 2024-09-20T19:29:41.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1984999999999996e-05 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.127268+0000", 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6746000000000001e-05 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2586 7.d 7:b90bdf9e:::smithi04268048-23:head [rollback 69] snapc 74=[74,73,72,70,6d,6c,69,67,63,5e,5d,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-20T19:29:41.110 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 257.95368101600002, 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028973180000000001, 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2586 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:29:41.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101181+0000", 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1099999999999995e-07 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101184+0000", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9390000000000002e-06 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101191+0000", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.905e-06 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101194+0000", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5640000000000001e-06 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.113927+0000", 2024-09-20T19:29:41.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012732581 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.114158+0000", 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023086 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.130154+0000", 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015995720000000001 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2606 7.c 7:38c1396f:::smithi04268048-31:head [copy-from ver 311] snapc 74=[73,72,70,6d,6c,69,67,63,5e,5d,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:24.720521+0000", 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.33434052500002, 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027918247, 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.115 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2606 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720521+0000", 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720521+0000", 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720526+0000", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2399999999999998e-06 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720542+0000", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6049000000000002e-05 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720587+0000", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4372999999999997e-05 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720613+0000", 2024-09-20T19:29:41.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6662e-05 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720748+0000", 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013472000000000001 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720897+0000", 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148994 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.734788+0000", 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013890731 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.747455+0000", 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012667461 2024-09-20T19:29:41.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.747590+0000", 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013536099999999999 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.748258+0000", 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066737600000000002 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.748357+0000", 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9381000000000003e-05 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.748439+0000", 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1898999999999996e-05 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.121 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2607 7.c 7:38c1396f:::smithi04268048-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:24.720668+0000", 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.33419311500001, 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026657976, 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2607 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720668+0000", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720668+0000", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720670+0000", 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9099999999999999e-06 2024-09-20T19:29:41.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720677+0000", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.277e-06 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720695+0000", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7736000000000001e-05 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.720713+0000", 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7399999999999999e-05 2024-09-20T19:29:41.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.721290+0000", 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057769699999999995 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.721441+0000", 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015026699999999999 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.746711+0000", 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025270681999999999 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.747016+0000", 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030457500000000002 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:24.747326+0000", 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031043200000000003 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.126 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2611 7.d 7:b90bdf9e:::smithi04268048-23:head [write 416906~407733,stat] snapc 74=[73,72,70,6d,6c,69,67,5e,5d,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:25.270462+0000", 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.784399861, 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023328056999999999, 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.127 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2611 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.270462+0000", 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.270462+0000", 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.270464+0000", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4229999999999999e-06 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271056+0000", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059224099999999997 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271071+0000", 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5247e-05 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271078+0000", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8229999999999997e-06 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271142+0000", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4141999999999997e-05 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271234+0000", 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1279999999999993e-05 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.271366+0000", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013217599999999999 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.272400+0000", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001034325 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.274049+0000", 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016492340000000001 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.293661+0000", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019612081 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.293734+0000", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2312999999999997e-05 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:25.293790+0000", 2024-09-20T19:29:41.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5772000000000001e-05 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3145 7.c 7:3b97d3fe:::smithi04268048-6:head [copy-from ver 496] snapc 90=[90,8f,8e,8d,8c,8b,8a,89,88,87,86,85,84,83,82,81,7f,7c,79,72,41] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.77483068500001, 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030932989000000001, 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3145 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280037+0000", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1569999999999997e-06 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280055+0000", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8068999999999999e-05 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280092+0000", 2024-09-20T19:29:41.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7367000000000003e-05 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280111+0000", 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8732999999999998e-05 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280202+0000", 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1125000000000006e-05 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280219+0000", 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6418000000000001e-05 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283343+0000", 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031240460000000001 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283476+0000", 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013334299999999999 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.294947+0000", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011470721999999999 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.309429+0000", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014482116999999999 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310736+0000", 2024-09-20T19:29:41.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013068419999999999 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310763+0000", 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6913000000000001e-05 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310873+0000", 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010998200000000001 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310964+0000", 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1155000000000004e-05 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3146 7.c 7:3b97d3fe:::smithi04268048-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e383)", 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:12.280147+0000", 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.77471487, 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027763796, 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.140 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3146 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280147+0000", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280147+0000", 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280148+0000", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2640000000000001e-06 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280153+0000", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.49e-06 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280165+0000", 2024-09-20T19:29:41.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1446e-05 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280174+0000", 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1330000000000005e-06 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280341+0000", 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016689799999999999 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280363+0000", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1772000000000001e-05 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283634+0000", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00327159 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283690+0000", 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6249999999999998e-05 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.307531+0000", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023840276000000001 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.307724+0000", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019317299999999999 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.307910+0000", 2024-09-20T19:29:41.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018650399999999999 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3546 7.d 7:b3515764:::smithi04268048-3:head [copy-from ver 568] snapc 9f=[9f,9e,9d,9c,9b,99,98,94,8d,8b,8a,88,87,86,84,81,7f,72] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.75966951500001, 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033137336000000003, 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:29:41.146 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3546 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295197+0000", 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5999999999999997e-06 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295222+0000", 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4366e-05 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295259+0000", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6973e-05 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295278+0000", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9063e-05 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295461+0000", 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018272000000000001 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295477+0000", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6799999999999998e-05 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.303293+0000", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078156179999999999 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.303631+0000", 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033842200000000001 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.315863+0000", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012231367 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.327364+0000", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011501711 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328001+0000", 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063663100000000002 2024-09-20T19:29:41.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328115+0000", 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011380800000000001 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328329+0000", 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021425700000000001 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3547 7.d 7:b3515764:::smithi04268048-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e416)", 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.75954492700001, 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031928874000000003, 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3547 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.153 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295318+0000", 2024-09-20T19:29:41.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295324+0000", 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8499999999999999e-06 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295336+0000", 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1963e-05 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295345+0000", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6330000000000008e-06 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295580+0000", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023457999999999999 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295600+0000", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0346000000000001e-05 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.304072+0000", 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084717909999999993 2024-09-20T19:29:41.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.304244+0000", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017183500000000001 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.326603+0000", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022359176000000001 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.326840+0000", 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023657300000000001 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.327245+0000", 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040576700000000002 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(7.18 477/476 [PushOp(7:1b5f1bb3:::smithi04268048-46:7c, version: 408'725, data_included: [1927075~431074,2976554~514615], data_size: 945689, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:1b5f1bb3:::smithi04268048-46:7c@408'725, size: 3647119, copy_subset: [(0, 3647119)], clone_subset: {}, snapset: ae=[]:{7c: [72], 92: [88], ae: [ae, ad, ac, ab, aa, a9, a7, a6, a5, a3, a1, a0, 9f, 9d, 9c, 98]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3647119, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927075, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:28:46.680317+0000", 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.374543985000003, 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047509320000000002, 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.158 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.680317+0000", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.680317+0000", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.680319+0000", 2024-09-20T19:29:41.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.398e-06 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.681334+0000", 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010148539999999999 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.681361+0000", 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6937999999999999e-05 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.681380+0000", 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9279999999999998e-05 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.681498+0000", 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011784400000000001 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.681515+0000", 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7391999999999999e-05 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.685068+0000", 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003553226 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(7.18 477/476 [PushOp(7:1f0aed4a:::smithi04268048-44:7e, version: 408'729, data_included: [2457715~230195,3259733~152121], data_size: 382316, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:1f0aed4a:::smithi04268048-44:7e@408'729, size: 3411854, copy_subset: [(0, 3411854)], clone_subset: {}, snapset: a5=[]:{7e: [72], 91: [88], 9c: [9c, 98], a0: [a0, 9f, 9d], a5: [a5, a3, a1]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3411854, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2457715, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:28:46.694037+0000", 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.360824233999999, 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046587449999999997, 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:29:41.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.694037+0000", 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.694037+0000", 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.694039+0000", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9410000000000001e-06 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.695722+0000", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001683033 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.695736+0000", 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4083000000000001e-05 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.695748+0000", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2160999999999999e-05 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.696013+0000", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026464499999999998 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.696034+0000", 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0800000000000001e-05 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:29:41.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:46.698696+0000", 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026620820000000001 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:29:41.166 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:29:41.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:41.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:29:41.216 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:29:41.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:41.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.402 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[32075]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:41 smithi042 bash[22706]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.446 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:41.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:41.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: cluster 2024-09-20T19:29:40.212905+0000 mgr.y (mgr.24491) 443 : cluster [DBG] pgmap v725: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:41.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: audit 2024-09-20T19:29:40.331996+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.42:0/2178467977' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:41 smithi162 bash[24557]: audit 2024-09-20T19:29:40.332598+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25367 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:29:41.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:41.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.703 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:41.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:29:41.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:41.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:41.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:41.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:42.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:42.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:42.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:42.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:42.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:42.334 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:42.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:42.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[32075]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:42 smithi042 bash[22706]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:42.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: cluster 2024-09-20T19:29:41.203326+0000 mon.a (mon.0) 1660 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:29:42.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: audit 2024-09-20T19:29:41.211533+0000 mon.a (mon.0) 1661 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:29:42.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:42 smithi162 bash[24557]: cluster 2024-09-20T19:29:41.215525+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e492: 8 total, 5 up, 8 in 2024-09-20T19:29:42.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:42.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:42.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:42.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.852 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:42.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:42.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:43.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:43.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:43.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:43.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:43.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/exp.5.e.0 2024-09-20T19:29:43.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:43.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:43 smithi162 bash[24557]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:43 smithi162 bash[24557]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:43 smithi162 bash[24557]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.359 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:43 smithi162 bash[24557]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:43.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:43.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:43.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:43.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[32075]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[32075]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[32075]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[32075]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[22706]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[22706]: cluster 2024-09-20T19:29:42.215526+0000 mgr.y (mgr.24491) 444 : cluster [DBG] pgmap v727: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[22706]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.618 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:43 smithi042 bash[22706]: audit 2024-09-20T19:29:42.622385+0000 mon.a (mon.0) 1663 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:43.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:43.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:43.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:43.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:43.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:43.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:44.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:44.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:44.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:44.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:44 smithi162 bash[24557]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:44 smithi162 bash[24557]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.385 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:44.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:44 smithi042 bash[32075]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.397 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:44 smithi042 bash[32075]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:44 smithi042 bash[22706]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:44 smithi042 bash[22706]: cluster 2024-09-20T19:29:43.205968+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e493: 8 total, 5 up, 8 in 2024-09-20T19:29:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:44.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:44.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:44.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:29:44.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:44.802 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:44.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:44.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:45.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:45.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:45.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:45.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:45.289 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:45.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:45 smithi162 bash[24557]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:45 smithi162 bash[24557]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:45 smithi162 bash[24557]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:45 smithi162 bash[24557]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:45.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.448 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:45.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[32075]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[32075]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[32075]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[32075]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[22706]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[22706]: cluster 2024-09-20T19:29:44.217562+0000 mgr.y (mgr.24491) 445 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:45.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[22706]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:45 smithi042 bash[22706]: cluster 2024-09-20T19:29:44.220653+0000 mon.a (mon.0) 1665 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:45.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:45.706 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:45.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.835 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:45.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:45.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:45.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:46.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:46.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:46.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:46.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:46.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:46.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:46.487 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:46.546 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:29:46.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:46.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:46.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:46.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:46.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:46.840 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:46.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:46.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:46.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:46.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:47.005 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:47.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:47.136 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:47.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:47.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:47.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:47.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:47.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[22706]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:47 smithi042 bash[32075]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:47.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.572883+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.574103+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.804713+0000 osd.1 (osd.1) 193 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:45.808351+0000 osd.1 (osd.1) 194 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:46.219486+0000 mgr.y (mgr.24491) 446 : cluster [DBG] pgmap v730: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:47 smithi162 bash[24557]: cluster 2024-09-20T19:29:46.248327+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e494: 8 total, 5 up, 8 in 2024-09-20T19:29:47.633 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:47.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:47.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:47.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:47.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:47.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:47.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:48.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:48.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:48.179 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:48.192 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:29:48.192 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:29:48.192 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:29:48.193 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:29:48.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:48.351 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 4, 3] live_osds: [1, 2, 5, 6, 7] 2024-09-20T19:29:48.351 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-20T19:29:48.351 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-20T19:29:48.352 DEBUG:teuthology.orchestra.run.smithi042:osd.0> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T19:29:48.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl start ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0 2024-09-20T19:29:48.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:48.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:29:48.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:48.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:48.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:48.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.678 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:48 smithi042 systemd[1]: Started Ceph osd.0 for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:29:48.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:48.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:29:48.765 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:29:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:29:48.784 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:29:48.785 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:29:48.785 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:29:48.785 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:29:48.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.803 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:29:48.803 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:29:48.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:48.822 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:29:48.822 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:29:48.840 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:29:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:29:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:48.920 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:48.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:48.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:29:49.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:49.075 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:49.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:49.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[24557]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:49.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:49.457 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:49.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[22706]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: cluster 2024-09-20T19:29:48.202229+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e495: 8 total, 5 up, 8 in 2024-09-20T19:29:49.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: cluster 2024-09-20T19:29:48.220989+0000 mgr.y (mgr.24491) 447 : cluster [DBG] pgmap v733: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:49.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:49 smithi042 bash[32075]: audit 2024-09-20T19:29:48.747213+0000 mon.a (mon.0) 1668 : audit [DBG] from='client.? v1:172.21.15.42:0/2704068698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:29:49.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:49.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:49.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:49.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:49.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:49.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:50.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:50.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:50.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.225 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[32075]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.226 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:50 smithi042 bash[22706]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:50.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:50.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.205240+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: Reduced data availability: 44 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: audit 2024-09-20T19:29:49.442455+0000 mgr.y (mgr.24491) 448 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.726866+0000 osd.5 (osd.5) 173 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:50 smithi162 bash[24557]: cluster 2024-09-20T19:29:49.731364+0000 osd.5 (osd.5) 174 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:29:50.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:50.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:50.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:50.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:50.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:50.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:29:51.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:29:51.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:51.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:51.383 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:51.452 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:51 smithi042 bash[32075]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.452 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:51 smithi042 bash[32075]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.452 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:51 smithi042 bash[22706]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.452 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:51 smithi042 bash[22706]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:51 smithi162 bash[24557]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:51 smithi162 bash[24557]: cluster 2024-09-20T19:29:50.222969+0000 mgr.y (mgr.24491) 449 : cluster [DBG] pgmap v734: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:51.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:51.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:51.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:51.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.797 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:51.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:51.927 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:51.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:52.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:52.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:52.080 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[108995]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:29:52.080 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[108995]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:29:52.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:52.177 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:52.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:52.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:52 smithi162 bash[24557]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:52 smithi162 bash[24557]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:52.405 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:52.416 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[32075]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.416 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[32075]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.417 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[22706]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.417 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:52 smithi042 bash[22706]: cluster 2024-09-20T19:29:51.200586+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized (PG_DEGRADED) 2024-09-20T19:29:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:52.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:52.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:52.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:52.686 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:52.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:52.812 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:52.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:52.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:52.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:53.043 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:29:53.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:53.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:53.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:53.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:29:53.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[22706]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:53 smithi042 bash[32075]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.091841+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.092842+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: cluster 2024-09-20T19:29:52.224320+0000 mgr.y (mgr.24491) 450 : cluster [DBG] pgmap v735: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: audit 2024-09-20T19:29:52.427677+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.42:0/3739957505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:53 smithi162 bash[24557]: audit 2024-09-20T19:29:52.428002+0000 mon.a (mon.0) 1671 : audit [INF] from='client.25379 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:29:53.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:53.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:53.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:53.755 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:29:53.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:53.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:53.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:53.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:53.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:54.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:54.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:54.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:54.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:54.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:54.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.484 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:54 smithi162 bash[24557]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:54.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:54.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[22706]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: cluster 2024-09-20T19:29:53.211655+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: audit 2024-09-20T19:29:53.219566+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:54 smithi042 bash[32075]: cluster 2024-09-20T19:29:53.225075+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e496: 8 total, 5 up, 8 in 2024-09-20T19:29:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:54.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:54.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:54.854 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:54.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:54.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:55.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:55.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:55.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:55.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.253 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:55 smithi162 bash[24557]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:55.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:55.395 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:55.427 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:29:55.427 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:29:55.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:55.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:55.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[22706]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.082729+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.d scrub starts 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.085239+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.d scrub ok 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:55 smithi042 bash[32075]: cluster 2024-09-20T19:29:54.227427+0000 mgr.y (mgr.24491) 451 : cluster [DBG] pgmap v737: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:55.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:55.881 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:55.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:56.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:56.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:56.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:56.178 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-20T19:29:56.179 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:29:56.179 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:29:56.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:29:56.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:29:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:56.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:29:56.392 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:29:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:29:56.393 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:29:56.393 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-20T19:29:56.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:56.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:56.534 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:56.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.645 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:29:56.645 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:29:56.645 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:29:56.645 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:29:56.645 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:29:56 smithi042 bash[108995]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-20T19:29:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:56.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:56.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:56.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:56.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:57.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:57.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:57.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:57.282 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:57.300 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:57 smithi162 bash[24557]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:57 smithi162 bash[24557]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:57.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:57.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:57 smithi042 bash[22706]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:57 smithi042 bash[22706]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.534 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:57 smithi042 bash[32075]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.534 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:57 smithi042 bash[32075]: cluster 2024-09-20T19:29:56.228558+0000 mgr.y (mgr.24491) 452 : cluster [DBG] pgmap v738: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:57.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:57.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.701 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:57.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:57.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:57.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:29:58.060 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-20T19:29:58.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:58.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:58.191 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:58.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:58.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:58.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:58.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:58.588 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:58.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:58.655 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[51183]: logger=cleanup t=2024-09-20T19:29:58.489945312Z level=info msg="Completed cleanup jobs" duration=97.759647ms 2024-09-20T19:29:58.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:58.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.749 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.750 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:58 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:58.800 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[51183]: logger=plugins.update.checker t=2024-09-20T19:29:58.698417888Z level=info msg="Update check succeeded" duration=227.188706ms 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: audit 2024-09-20T19:29:57.655678+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: audit 2024-09-20T19:29:57.657729+0000 mon.a (mon.0) 1676 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.197538+0000 mon.a (mon.0) 1677 : cluster [WRN] Health check update: Reduced data availability: 45 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:58 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.214779+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e497: 8 total, 5 up, 8 in 2024-09-20T19:29:58.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:58.870 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:29:58.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:58.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:58.981 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:59.010 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:59.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:59.143 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:29:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:29:59.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:59.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:29:59.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:59.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:29:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:29:59.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:59.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:29:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:29:59.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:59.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:29:59.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:29:59.767 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:59.795 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:29:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:29:59.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:59 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:29:59 smithi162 bash[24557]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:59.927 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:29:59.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:59 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:29:59 smithi042 bash[32075]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:59 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:29:59 smithi042 bash[22706]: cluster 2024-09-20T19:29:58.230061+0000 mgr.y (mgr.24491) 453 : cluster [DBG] pgmap v740: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:29:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:59.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:00.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.060 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:00.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:00.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:00.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:00.420 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:00.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:00.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:00.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:00.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:00.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:00.840 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.841 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:00.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.843 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.845 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.845 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.845 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.845 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.850 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.851 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:00.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:00.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:00.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:00.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:00.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:00.874 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.875 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:00.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[22706]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:00.877 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:00.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:00.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:00.881 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:00.882 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:00.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:00.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:00 smithi042 bash[32075]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:00.928 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:00.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:01.011 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:30:01.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: audit 2024-09-20T19:29:59.451432+0000 mgr.y (mgr.24491) 454 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000281+0000 mon.a (mon.0) 1679 : cluster [WRN] Health detail: HEALTH_WARN 3 osds down; Reduced data availability: 45 pgs inactive, 3 pgs stale; Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:01.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000328+0000 mon.a (mon.0) 1680 : cluster [WRN] [WRN] OSD_DOWN: 3 osds down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000344+0000 mon.a (mon.0) 1681 : cluster [WRN] osd.0 (root=default,host=smithi042) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000361+0000 mon.a (mon.0) 1682 : cluster [WRN] osd.3 (root=default,host=smithi042) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000375+0000 mon.a (mon.0) 1683 : cluster [WRN] osd.4 (root=default,host=smithi162) is down 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000387+0000 mon.a (mon.0) 1684 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 45 pgs inactive, 3 pgs stale 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000408+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 1.2 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000427+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 1.6 is stuck inactive for 70s, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000444+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 2.1 is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000460+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 2.5 is stuck inactive for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000480+0000 mon.a (mon.0) 1689 : cluster [WRN] pg 2.d is stuck inactive for 70s, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000496+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 2.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:01.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000518+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 2.13 is stuck stale for 70s, current state stale+undersized+peered, last acting [3] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000535+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 2.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000552+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 2.1e is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000569+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 2.1f is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000588+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 3.1 is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000608+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000629+0000 mon.a (mon.0) 1697 : cluster [WRN] pg 4.9 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000646+0000 mon.a (mon.0) 1698 : cluster [WRN] pg 4.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000663+0000 mon.a (mon.0) 1699 : cluster [WRN] pg 4.c is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000682+0000 mon.a (mon.0) 1700 : cluster [WRN] pg 4.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000701+0000 mon.a (mon.0) 1701 : cluster [WRN] pg 4.19 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000722+0000 mon.a (mon.0) 1702 : cluster [WRN] pg 4.1a is stuck stale for 70s, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000743+0000 mon.a (mon.0) 1703 : cluster [WRN] pg 4.1b is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000758+0000 mon.a (mon.0) 1704 : cluster [WRN] pg 4.1d is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000776+0000 mon.a (mon.0) 1705 : cluster [WRN] pg 4.1e is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000794+0000 mon.a (mon.0) 1706 : cluster [WRN] pg 5.0 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000816+0000 mon.a (mon.0) 1707 : cluster [WRN] pg 5.1 is stuck inactive for 11m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000834+0000 mon.a (mon.0) 1708 : cluster [WRN] pg 5.5 is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000852+0000 mon.a (mon.0) 1709 : cluster [WRN] pg 5.a is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000875+0000 mon.a (mon.0) 1710 : cluster [WRN] pg 5.c is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000894+0000 mon.a (mon.0) 1711 : cluster [WRN] pg 5.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000915+0000 mon.a (mon.0) 1712 : cluster [WRN] pg 5.13 is stuck inactive for 3m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000935+0000 mon.a (mon.0) 1713 : cluster [WRN] pg 5.1c is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000957+0000 mon.a (mon.0) 1714 : cluster [WRN] pg 5.1d is stuck inactive for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000976+0000 mon.a (mon.0) 1715 : cluster [WRN] pg 6.0 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.000997+0000 mon.a (mon.0) 1716 : cluster [WRN] pg 6.2 is stuck inactive for 11m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001020+0000 mon.a (mon.0) 1717 : cluster [WRN] pg 6.6 is stuck inactive for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001039+0000 mon.a (mon.0) 1718 : cluster [WRN] pg 6.7 is stuck inactive for 70s, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001058+0000 mon.a (mon.0) 1719 : cluster [WRN] pg 6.f is stuck inactive for 70s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001074+0000 mon.a (mon.0) 1720 : cluster [WRN] pg 6.11 is stuck inactive for 3m, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001091+0000 mon.a (mon.0) 1721 : cluster [WRN] pg 6.13 is stuck inactive for 3m, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001113+0000 mon.a (mon.0) 1722 : cluster [WRN] pg 6.16 is stuck inactive for 70s, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001132+0000 mon.a (mon.0) 1723 : cluster [WRN] pg 7.3 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001151+0000 mon.a (mon.0) 1724 : cluster [WRN] pg 7.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001169+0000 mon.a (mon.0) 1725 : cluster [WRN] pg 7.6 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001185+0000 mon.a (mon.0) 1726 : cluster [WRN] pg 7.7 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001204+0000 mon.a (mon.0) 1727 : cluster [WRN] pg 7.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001234+0000 mon.a (mon.0) 1728 : cluster [WRN] pg 7.f is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001250+0000 mon.a (mon.0) 1729 : cluster [WRN] pg 7.16 is stuck inactive for 70s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001268+0000 mon.a (mon.0) 1730 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 449/1209 objects degraded (37.138%), 61 pgs degraded, 138 pgs undersized 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001285+0000 mon.a (mon.0) 1731 : cluster [WRN] pg 1.4 is stuck undersized for 69s, current state active+undersized, last acting [5,7] 2024-09-20T19:30:01.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001330+0000 mon.a (mon.0) 1732 : cluster [WRN] pg 1.5 is stuck undersized for 2m, current state active+undersized, last acting [2,6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001349+0000 mon.a (mon.0) 1733 : cluster [WRN] pg 1.6 is stuck undersized for 69s, current state undersized+peered, last acting [6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001372+0000 mon.a (mon.0) 1734 : cluster [WRN] pg 1.7 is stuck undersized for 69s, current state active+undersized, last acting [6,7] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001394+0000 mon.a (mon.0) 1735 : cluster [WRN] pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001413+0000 mon.a (mon.0) 1736 : cluster [WRN] pg 1.9 is stuck undersized for 69s, current state active+undersized, last acting [7,2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001432+0000 mon.a (mon.0) 1737 : cluster [WRN] pg 2.5 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001448+0000 mon.a (mon.0) 1738 : cluster [WRN] pg 2.6 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,6] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:01.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001465+0000 mon.a (mon.0) 1739 : cluster [WRN] pg 2.8 is stuck undersized for 69s, current state active+undersized, last acting [2,1] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001483+0000 mon.a (mon.0) 1740 : cluster [WRN] pg 2.9 is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001506+0000 mon.a (mon.0) 1741 : cluster [WRN] pg 2.a is stuck undersized for 69s, current state active+undersized, last acting [1,7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001523+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 2.b is stuck undersized for 2m, current state active+undersized, last acting [7,5] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001540+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001561+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 2.1e is stuck undersized for 69s, current state undersized+peered, last acting [5] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001579+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 2.1f is stuck undersized for 2m, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001598+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 4.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001621+0000 mon.a (mon.0) 1747 : cluster [WRN] pg 4.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,6] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001638+0000 mon.a (mon.0) 1748 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001661+0000 mon.a (mon.0) 1749 : cluster [WRN] pg 4.c is stuck undersized for 69s, current state undersized+degraded+peered, last acting [6] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001681+0000 mon.a (mon.0) 1750 : cluster [WRN] pg 4.d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001699+0000 mon.a (mon.0) 1751 : cluster [WRN] pg 4.e is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001716+0000 mon.a (mon.0) 1752 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:01.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001738+0000 mon.a (mon.0) 1753 : cluster [WRN] pg 4.18 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001757+0000 mon.a (mon.0) 1754 : cluster [WRN] pg 4.19 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001776+0000 mon.a (mon.0) 1755 : cluster [WRN] pg 4.1a is stuck undersized for 2m, current state stale+undersized+degraded+peered, last acting [3] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001795+0000 mon.a (mon.0) 1756 : cluster [WRN] pg 4.1b is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001814+0000 mon.a (mon.0) 1757 : cluster [WRN] pg 5.0 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001834+0000 mon.a (mon.0) 1758 : cluster [WRN] pg 5.1 is stuck undersized for 69s, current state undersized+peered, last acting [7] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001851+0000 mon.a (mon.0) 1759 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001869+0000 mon.a (mon.0) 1760 : cluster [WRN] pg 5.c is stuck undersized for 2m, current state undersized+peered, last acting [1] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001889+0000 mon.a (mon.0) 1761 : cluster [WRN] pg 5.e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001907+0000 mon.a (mon.0) 1762 : cluster [WRN] pg 5.f is stuck undersized for 2m, current state active+undersized, last acting [5,6] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001929+0000 mon.a (mon.0) 1763 : cluster [WRN] pg 5.18 is stuck undersized for 2m, current state active+undersized, last acting [6,1] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001948+0000 mon.a (mon.0) 1764 : cluster [WRN] pg 5.1a is stuck undersized for 2m, current state active+undersized, last acting [7,1] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001967+0000 mon.a (mon.0) 1765 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001982+0000 mon.a (mon.0) 1766 : cluster [WRN] pg 6.2 is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.001999+0000 mon.a (mon.0) 1767 : cluster [WRN] pg 6.c is stuck undersized for 69s, current state active+undersized, last acting [6,5] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002017+0000 mon.a (mon.0) 1768 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002037+0000 mon.a (mon.0) 1769 : cluster [WRN] pg 6.e is stuck undersized for 2m, current state active+undersized, last acting [1,2] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002053+0000 mon.a (mon.0) 1770 : cluster [WRN] pg 6.f is stuck undersized for 69s, current state undersized+peered, last acting [2] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002094+0000 mon.a (mon.0) 1771 : cluster [WRN] pg 6.19 is stuck undersized for 69s, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002112+0000 mon.a (mon.0) 1772 : cluster [WRN] pg 6.1a is stuck undersized for 2m, current state active+undersized, last acting [5,1] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:01.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002132+0000 mon.a (mon.0) 1773 : cluster [WRN] pg 6.1b is stuck undersized for 69s, current state active+undersized, last acting [7,6] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002151+0000 mon.a (mon.0) 1774 : cluster [WRN] pg 7.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [5,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002172+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 7.1 is active+undersized+degraded, acting [6,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002193+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 7.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,2] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002212+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 7.3 is stuck undersized for 69s, current state undersized+degraded+peered, last acting [2] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002231+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 7.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [6,7] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002253+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 7.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,1] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:01.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002270+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 7.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-09-20T19:30:01.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.002287+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 7.f is stuck undersized for 69s, current state undersized+degraded+peered, last acting [7] 2024-09-20T19:30:01.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:01.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:00 smithi162 bash[24557]: audit 2024-09-20T19:30:00.132976+0000 mon.a (mon.0) 1782 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:30:01.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:01.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-20T19:30:01.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:01.190 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:30:01.190 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:30:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:30:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:30:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:30:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:30:01.230 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:30:01.231 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:30:01.251 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:30:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:30:01.271 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:30:01.272 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:30:01.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:01.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:01.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:01.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:01.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:01.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:01.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:01.714 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:01.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.842 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:01.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[32075]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:01.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:01.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:01 smithi042 bash[22706]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:01.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:01.974 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:01.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:02.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cluster 2024-09-20T19:30:00.232357+0000 mgr.y (mgr.24491) 455 : cluster [DBG] pgmap v741: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.863802+0000 mon.a (mon.0) 1783 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.863967+0000 mon.a (mon.0) 1784 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.864725+0000 mon.a (mon.0) 1785 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.864901+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.865692+0000 mon.a (mon.0) 1787 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.865936+0000 mon.a (mon.0) 1788 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.866585+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.866784+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cephadm 2024-09-20T19:30:00.867202+0000 mgr.y (mgr.24491) 456 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3292M 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: cephadm 2024-09-20T19:30:00.867493+0000 mgr.y (mgr.24491) 457 : cephadm [INF] Adjusting osd_memory_target on smithi042 to 3293M 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.871141+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.878967+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.880957+0000 mon.a (mon.0) 1793 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.881619+0000 mon.a (mon.0) 1794 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:00.886701+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:02.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:01 smithi162 bash[24557]: audit 2024-09-20T19:30:01.171552+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.42:0/4225697168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:02.077 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:02.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:02.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:02.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:30:02.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:30:02.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:02.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:02.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:02.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:02.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:02.689 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:30:02.689 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:30:02.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:30:02.690 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:30:02.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.690 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:732 7.5 7:adc64224:::smithi04268048-47:head [copy-from ver 55] snapc 1d=[1d,10] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-09-20T19:30:02.690 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:41.254724+0000", 2024-09-20T19:30:02.690 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 441.43805224499999, 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021497031, 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 732 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254724+0000", 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254724+0000", 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254728+0000", 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.464e-06 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254744+0000", 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5043e-05 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254776+0000", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2276000000000001e-05 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254792+0000", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5733e-05 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254964+0000", 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000172139 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254985+0000", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0922999999999999e-05 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.259844+0000", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048596380000000003 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.260000+0000", 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155833 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.267697+0000", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076973040000000003 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.275690+0000", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0079926700000000003 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.275860+0000", 2024-09-20T19:30:02.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016997599999999999 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.275994+0000", 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013410399999999999 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.276097+0000", 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102594 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.276221+0000", 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012433400000000001 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:733 7.5 7:adc64224:::smithi04268048-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e172)", 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:22:41.254825+0000", 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 441.43795172300003, 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020443983999999998, 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.699 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 733 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254825+0000", 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254825+0000", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254826+0000", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6959999999999999e-06 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254834+0000", 2024-09-20T19:30:02.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3000000000000004e-06 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254851+0000", 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7765999999999999e-05 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.254866+0000", 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4803e-05 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.255130+0000", 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026360400000000002 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.255148+0000", 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8023e-05 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.260369+0000", 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052213169999999996 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.260466+0000", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6560999999999999e-05 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.274938+0000", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014472412 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.275087+0000", 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014900300000000001 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:22:41.275269+0000", 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000181499 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:941 7.5 7:a6db2e09:::smithi04268048-9:head [copy-from ver 86] snapc 2b=[2b,2a,29,26,24,22,20,10] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 419.64629696700001, 2024-09-20T19:30:02.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028695786000000001, 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 941 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:30:02.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046484+0000", 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.284e-06 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046498+0000", 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4596000000000001e-05 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046531+0000", 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3235999999999998e-05 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046549+0000", 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7540000000000001e-05 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046728+0000", 2024-09-20T19:30:02.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017878600000000001 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046750+0000", 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2566e-05 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053318+0000", 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065672559999999996 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053582+0000", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026449500000000001 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.064732+0000", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011149676000000001 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073497+0000", 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087651080000000006 2024-09-20T19:30:02.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073683+0000", 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018656599999999999 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.074975+0000", 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012920679999999999 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.075094+0000", 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118807 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.075175+0000", 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0802000000000003e-05 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:942 7.5 7:a6db2e09:::smithi04268048-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e194)", 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:30:02.713 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 419.64619628200001, 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026852612000000001, 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 942 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046581+0000", 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1370000000000001e-06 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046587+0000", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4060000000000004e-06 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046597+0000", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046606+0000", 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7620000000000001e-06 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046884+0000", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027780399999999998 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046895+0000", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1147e-05 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053916+0000", 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070209030000000002 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053994+0000", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8177999999999996e-05 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.072860+0000", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018865990999999999 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073105+0000", 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024436800000000002 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073433+0000", 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032792200000000002 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:968 7.5 7:adc64224:::smithi04268048-47:head [copy-from ver 27] snapc 2c=[2c,2a,29,26,24,20] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 415.63970930900001, 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037789484999999998, 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.719 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 968 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053071+0000", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1049999999999997e-06 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053084+0000", 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632e-05 2024-09-20T19:30:02.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053112+0000", 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8535999999999999e-05 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053124+0000", 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2041999999999999e-05 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053212+0000", 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7688999999999993e-05 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053223+0000", 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1336e-05 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.056699+0000", 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034760009999999998 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.056921+0000", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022168900000000001 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.070665+0000", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013744361 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.077380+0000", 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067147810000000004 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.079412+0000", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020316359999999999 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090764+0000", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011352704 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090804+0000", 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9332000000000002e-05 2024-09-20T19:30:02.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090856+0000", 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2641e-05 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:969 7.5 7:adc64224:::smithi04268048-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2024-09-20T19:30:02.726 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 415.63962578299999, 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024003268000000001, 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 969 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053151+0000", 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4099999999999997e-07 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053156+0000", 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5970000000000002e-06 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053166+0000", 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0023e-05 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053173+0000", 2024-09-20T19:30:02.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8739999999999997e-06 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053297+0000", 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123628 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053301+0000", 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.639e-06 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.057222+0000", 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039206919999999999 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.057310+0000", 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8491000000000004e-05 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.076556+0000", 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019245532999999999 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.076819+0000", 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026316700000000002 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.077154+0000", 2024-09-20T19:30:02.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033468300000000001 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:970 7.5 7:a6db2e09:::smithi04268048-9:head [write 588278~715406] snapc 2c=[2c,2a,29,26,24,20] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:07.064929+0000", 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 415.62784759700003, 2024-09-20T19:30:02.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020106315, 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 970 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.736 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.064929+0000", 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.064929+0000", 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.064931+0000", 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2000000000000001e-06 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.065865+0000", 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093423099999999999 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.065880+0000", 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4663e-05 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.065889+0000", 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1740000000000006e-06 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.066067+0000", 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017815100000000001 2024-09-20T19:30:02.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.066236+0000", 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000169407 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.066672+0000", 2024-09-20T19:30:02.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043503999999999998 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.076420+0000", 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.009748662 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.077277+0000", 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085701400000000002 2024-09-20T19:30:02.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.084933+0000", 2024-09-20T19:30:02.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076556660000000002 2024-09-20T19:30:02.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.084973+0000", 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9787999999999999e-05 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.085035+0000", 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2318999999999997e-05 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1899 7.5 7:a66d8c87:::smithi04268048-11:head [copy-from ver 152] snapc 51=[51,50,4f,4e,4c,4b,49,46,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-20T19:30:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 345.591168799, 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036523195000000001, 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1899 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101610+0000", 2024-09-20T19:30:02.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8270000000000002e-06 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101619+0000", 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5839999999999995e-06 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101633+0000", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3728e-05 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101643+0000", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.013e-05 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101762+0000", 2024-09-20T19:30:02.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011932 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101777+0000", 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107805+0000", 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006028299 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107879+0000", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4240999999999996e-05 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.122444+0000", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014564638 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137584+0000", 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015140278 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137772+0000", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018759700000000001 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137945+0000", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017279700000000001 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.138045+0000", 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010033499999999999 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.138131+0000", 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5514999999999997e-05 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1900 7.5 7:a66d8c87:::smithi04268048-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 345.59110843299999, 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031304029999999997, 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1900 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101669+0000", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0610000000000001e-06 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101673+0000", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.664e-06 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101680+0000", 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9979999999999997e-06 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101688+0000", 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1289999999999996e-06 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101896+0000", 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000208092 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101900+0000", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3889999999999998e-06 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107987+0000", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060870589999999997 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.108015+0000", 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7495e-05 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132810+0000", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024794817 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132889+0000", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9204999999999998e-05 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132972+0000", 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3121000000000006e-05 2024-09-20T19:30:02.759 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2130 7.5 7:a66d8c87:::smithi04268048-11:head [copy-from ver 340] snapc 5f=[5f,5e,5d,5a,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 316.45121907599997, 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023236039999999999, 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.760 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2130 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241561+0000", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7340000000000002e-06 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241580+0000", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8797999999999999e-05 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241600+0000", 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0602999999999999e-05 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241612+0000", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2043e-05 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241732+0000", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119838 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241849+0000", 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116729 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.254378+0000", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012528977 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.261386+0000", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070081910000000004 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.262140+0000", 2024-09-20T19:30:02.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075371699999999999 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264625+0000", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002485377 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264731+0000", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106048 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264793+0000", 2024-09-20T19:30:02.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1984999999999997e-05 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2131 7.5 7:a66d8c87:::smithi04268048-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:46.241650+0000", 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 316.45112611100001, 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018881251000000002, 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.766 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2131 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241650+0000", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241650+0000", 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241651+0000", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.099e-06 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241656+0000", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6929999999999998e-06 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241678+0000", 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1982999999999999e-05 2024-09-20T19:30:02.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241685+0000", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9430000000000003e-06 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.242043+0000", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035789200000000001 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.242104+0000", 2024-09-20T19:30:02.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0914e-05 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.260178+0000", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018074257999999999 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.260345+0000", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016725 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.260531+0000", 2024-09-20T19:30:02.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186219 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2314 7.5 7:adc64224:::smithi04268048-47:head [copy-from ver 414] snapc 69=[69,68,67,66,65,63,62,5e,5d,5a,57,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 300.49095794200002, 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034760207000000001, 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.771 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2314 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201824+0000", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6869999999999999e-06 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201842+0000", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8196999999999999e-05 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201880+0000", 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8219000000000001e-05 2024-09-20T19:30:02.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201900+0000", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9259e-05 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202083+0000", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000182907 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202105+0000", 2024-09-20T19:30:02.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2745e-05 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.205942+0000", 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038370309999999999 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206023+0000", 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0311999999999999e-05 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.218794+0000", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012771803999999999 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.233763+0000", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014968560000000001 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236138+0000", 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023753680000000001 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236390+0000", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025173900000000001 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236494+0000", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010427799999999999 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236579+0000", 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4101e-05 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.777 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2315 7.5 7:adc64224:::smithi04268048-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 300.49083823400002, 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029275061000000002, 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.778 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2315 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201940+0000", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6199999999999999e-06 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201954+0000", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4117000000000001e-05 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201999+0000", 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5701999999999997e-05 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202016+0000", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6693000000000001e-05 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202220+0000", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020356000000000001 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202243+0000", 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.304e-05 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206174+0000", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039308399999999997 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206306+0000", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013202099999999999 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.230853+0000", 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024547365000000002 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.231025+0000", 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017165 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.231213+0000", 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018845299999999999 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2703 7.5 7:acb1cf35:::smithi04268048-42:head [copy-from ver 393] snapc 77=[77,76,75,73,72,70,6d,6c,69,67,5e,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T19:30:02.783 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 273.29995088800001, 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025102850999999999, 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2703 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:30:02.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392852+0000", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6718999999999999e-05 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392870+0000", 2024-09-20T19:30:02.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8e-05 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392902+0000", 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1838999999999998e-05 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392918+0000", 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6456e-05 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393043+0000", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012482500000000001 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393277+0000", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000233983 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.404416+0000", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011138677 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417344+0000", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012927925999999999 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417615+0000", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027092099999999998 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417726+0000", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111159 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417838+0000", 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111965 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417928+0000", 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0381000000000001e-05 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2704 7.5 7:acb1cf35:::smithi04268048-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:30:02.789 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 273.299823328, 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023701936, 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2704 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:30:02.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392954+0000", 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.387e-06 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392960+0000", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392971+0000", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1413000000000001e-05 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392980+0000", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8799999999999997e-06 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393747+0000", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076712099999999995 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393847+0000", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100008 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416276+0000", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022428547 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416466+0000", 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190204 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416655+0000", 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018893299999999999 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3508 7.5 7:aa23a6e3:::smithi04268048-4:head [copy-from ver 556] snapc 9d=[9d,9c,9b,99,98,97,94,8f,8d,8b,8a,88,87,86,84,83,81,7f,72] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:30:02.794 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 202.45919186099999, 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042870617, 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3508 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:30:02.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233590+0000", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4070000000000001e-06 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233609+0000", 2024-09-20T19:30:02.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9473000000000001e-05 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233653+0000", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3251999999999999e-05 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233677+0000", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4773000000000002e-05 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233825+0000", 2024-09-20T19:30:02.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014807700000000001 2024-09-20T19:30:02.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233841+0000", 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.519e-05 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.246998+0000", 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013157515999999999 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.247358+0000", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036032499999999999 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.261838+0000", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014479984 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273713+0000", 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011874641999999999 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276154+0000", 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024411060000000002 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276267+0000", 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011283200000000001 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276455+0000", 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018804 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3509 7.5 7:aa23a6e3:::smithi04268048-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e411)", 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 202.45903766999999, 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039850053000000003, 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:30:02.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3509 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233741+0000", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233749+0000", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.089e-06 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233769+0000", 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0307000000000001e-05 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233785+0000", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.647e-05 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233929+0000", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014325000000000001 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233939+0000", 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0317e-05 2024-09-20T19:30:02.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.247951+0000", 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014011961999999999 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.248125+0000", 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000174272 2024-09-20T19:30:02.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273141+0000", 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025015565999999999 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273308+0000", 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000167088 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273589+0000", 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000280735 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.2.0:560 7.8 e419/112 7:1695abb3:::smithi04268048-45:99 v 419'773, mlcod=417'770)", 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:48.143462+0000", 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 194.549314563, 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019483037000000002, 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143462+0000", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143462+0000", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143463+0000", 2024-09-20T19:30:02.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143475+0000", 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2401999999999999e-05 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143528+0000", 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2602999999999998e-05 2024-09-20T19:30:02.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143550+0000", 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2266e-05 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143874+0000", 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032412899999999999 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.143952+0000", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8197999999999999e-05 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.162835+0000", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018882484000000001 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.162945+0000", 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109868 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.813 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.2.0:561 7.8 e419/112 7:1517470f:::smithi04268048-49:97 v 419'775, mlcod=417'770)", 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:48.144868+0000", 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 194.54790818500001, 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019456945, 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144868+0000", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144868+0000", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144871+0000", 2024-09-20T19:30:02.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4199999999999999e-06 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144888+0000", 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6246999999999999e-05 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144914+0000", 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6208999999999999e-05 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.144932+0000", 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.774e-05 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.145465+0000", 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053289199999999998 2024-09-20T19:30:02.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.145536+0000", 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1874999999999999e-05 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.164203+0000", 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018666471 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:48.164325+0000", 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012209100000000001 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.819 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2380 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94817798636544 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.921565114000003, 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45087614300000001, 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2380 2024-09-20T19:30:02.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:30:02.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771215+0000", 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2169999999999996e-06 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771228+0000", 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3073e-05 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771260+0000", 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1272999999999997e-05 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771276+0000", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6733000000000001e-05 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771453+0000", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017652400000000001 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771485+0000", 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2057000000000002e-05 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.219402+0000", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.447917073 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.219684+0000", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028212799999999998 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.222087+0000", 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024030649999999998 2024-09-20T19:30:02.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:30:02.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:30:02.825 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:30:02.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:02.826 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:02 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.826 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:02 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:02.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:02 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:02 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.930 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:30:02 smithi042 bash[109938]: debug 2024-09-20T19:30:02.725+0000 7fdcf1d20640 -1 Falling back to public interface 2024-09-20T19:30:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:02 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:02 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.234019+0000 mgr.y (mgr.24491) 458 : cluster [DBG] pgmap v742: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:02.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:02.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:02.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:03.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:30:03.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:03.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:03.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:03.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:03.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:30:03.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:03.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:03.723 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:03.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:03.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:03.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:03.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:03.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:03.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:03 smithi162 bash[24557]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:03.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:03.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:03.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:04.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:04.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:04.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[32075]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.528596+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:02.530850+0000 osd.5 (osd.5) 176 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:03.182875+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:03 smithi042 bash[22706]: cluster 2024-09-20T19:30:03.184547+0000 osd.7 (osd.7) 144 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:30:04.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:30:04.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:04.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:04.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:04.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.528 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:04.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.659 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:04.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:04 smithi162 bash[24557]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:04.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:04 smithi162 bash[24557]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:04.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:04.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:04.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:04.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:05.001 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:04 smithi042 bash[32075]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:05.001 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:04 smithi042 bash[32075]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:05.001 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:04 smithi042 bash[22706]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:05.002 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:04 smithi042 bash[22706]: cluster 2024-09-20T19:30:04.236237+0000 mgr.y (mgr.24491) 459 : cluster [DBG] pgmap v743: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:05.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:05.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:05.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:05.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:05.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:05.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:05.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:05.708 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:30:05.752 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:05.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:05.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.885 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:05 smithi162 bash[24557]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:05.885 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:05 smithi162 bash[24557]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:05.886 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:05 smithi162 bash[24557]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:05.886 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:05 smithi162 bash[24557]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:05.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:05.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:06.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:06.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:06.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:06.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[32075]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[32075]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[32075]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[32075]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[22706]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[22706]: audit 2024-09-20T19:30:05.285187+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.42:0/2542224647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[22706]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:05 smithi042 bash[22706]: audit 2024-09-20T19:30:05.285769+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25394 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:06.263 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:06.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:06.418 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:06.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.525 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:06.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:06.675 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:06.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.786 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[109938]: debug 2024-09-20T19:30:06.741+0000 7fdcf1d20640 -1 osd.0 393 log_to_monitors true 2024-09-20T19:30:06.786 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[32075]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:06.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:06.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:06.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:06.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:06.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:06.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:06 smithi042 bash[22706]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:06.900 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:06.925 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:06.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:07.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:05.690098+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: audit 2024-09-20T19:30:05.700305+0000 mon.a (mon.0) 1798 : audit [INF] from='client.25394 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:05.708231+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e498: 8 total, 5 up, 8 in 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:07.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:06 smithi162 bash[24557]: cluster 2024-09-20T19:30:06.238439+0000 mgr.y (mgr.24491) 460 : cluster [DBG] pgmap v745: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:07.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:07.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:07.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:07.410 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.439 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:07.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.567 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:07.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:07.698 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:07.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:07.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:07.943 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:07 smithi162 bash[24557]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:07.943 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:07 smithi162 bash[24557]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:07.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:07.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:07.989 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:07 smithi042 bash[22706]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:07.989 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:07 smithi042 bash[22706]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:07.990 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:07 smithi042 bash[32075]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:07.990 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:07 smithi042 bash[32075]: audit 2024-09-20T19:30:06.747645+0000 mon.a (mon.0) 1800 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:30:08.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:08.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.216 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:08.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:08.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:08.429 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:30:07 smithi042 bash[109938]: debug 2024-09-20T19:30:07.985+0000 7fdce9aca640 -1 osd.0 499 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:30:08.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:08.478 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:08.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:30:08.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:08.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:08.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:08.851 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:08.851 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:08 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:08.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:08.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:08.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:08.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:09.016 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:09.018 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:09.018 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:09.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.109 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:09.110 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: audit 2024-09-20T19:30:07.712815+0000 mon.a (mon.0) 1801 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.721497+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 8 total, 5 up, 8 in 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: audit 2024-09-20T19:30:07.722461+0000 mon.a (mon.0) 1803 : audit [INF] from='osd.0 v1:172.21.15.42:6801/701330554' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:09.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:08 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.239822+0000 mgr.y (mgr.24491) 461 : cluster [DBG] pgmap v747: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:09.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:09.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:09.488 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:09.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.647 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:09.747 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:09.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:09.799 INFO:tasks.rados.rados.0.smithi042.stdout:1525: finishing write tid 1 to smithi04268048-48 2024-09-20T19:30:09.800 INFO:tasks.rados.rados.0.smithi042.stdout:1525: finishing write tid 2 to smithi04268048-48 2024-09-20T19:30:09.800 INFO:tasks.rados.rados.0.smithi042.stdout:1525: finishing write tid 3 to smithi04268048-48 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1519: done (9 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1547: done (8 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1548: done (7 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1549: done (6 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1550: done (5 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1551: done (4 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1552: done (3 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1553: done (2 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1554: done (1 left) 2024-09-20T19:30:09.801 INFO:tasks.rados.rados.0.smithi042.stdout:1555: delete oid 3 current snap is 177 2024-09-20T19:30:09.802 INFO:tasks.rados.rados.0.smithi042.stdout:1525: finishing write tid 4 to smithi04268048-48 2024-09-20T19:30:09.803 INFO:tasks.rados.rados.0.smithi042.stdout:1525: oid 48 updating version 0 to 736 2024-09-20T19:30:09.803 INFO:tasks.rados.rados.0.smithi042.stdout:1525: oid 48 version 736 is already newer than 735 2024-09-20T19:30:09.803 INFO:tasks.rados.rados.0.smithi042.stdout:1525: oid 48 updating version 736 to 737 2024-09-20T19:30:09.803 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 737 (ObjNum 535 snap 173 seq_num 535) dirty exists 2024-09-20T19:30:09.803 INFO:tasks.rados.rados.0.smithi042.stdout:1525: left oid 48 (ObjNum 535 snap 173 seq_num 535) 2024-09-20T19:30:09.832 INFO:tasks.rados.rados.0.smithi042.stdout:1525: done (1 left) 2024-09-20T19:30:09.832 INFO:tasks.rados.rados.0.smithi042.stdout:1555: done (0 left) 2024-09-20T19:30:09.832 INFO:tasks.rados.rados.0.smithi042.stdout:1556: copy_from oid 21 from oid 45 current snap is 177 2024-09-20T19:30:09.832 INFO:tasks.rados.rados.0.smithi042.stdout:1557: delete oid 28 current snap is 177 2024-09-20T19:30:09.835 INFO:tasks.rados.rados.0.smithi042.stdout:1557: done (1 left) 2024-09-20T19:30:09.835 INFO:tasks.rados.rados.0.smithi042.stdout:1558: write (excl) oid 35 current snap is 177 2024-09-20T19:30:09.835 INFO:tasks.rados.rados.0.smithi042.stdout:1558: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-20T19:30:09.842 INFO:tasks.rados.rados.0.smithi042.stdout:1558: writing smithi04268048-35 from 427043 to 916160 tid 1 2024-09-20T19:30:09.849 INFO:tasks.rados.rados.0.smithi042.stdout:1558: writing smithi04268048-35 from 1576379 to 2049224 tid 2 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1558: writing smithi04268048-35 from 2626870 to 3227043 tid 3 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1559: read oid 26 snap -1 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1559: expect deleted 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1560: rollback oid 33 current snap is 177 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 164 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1556: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1556: finishing copy_from to smithi04268048-21 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 675 (ObjNum 514 snap 166 seq_num 514) dirty exists 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1558: finishing write tid 1 to smithi04268048-35 2024-09-20T19:30:09.857 INFO:tasks.rados.rados.0.smithi042.stdout:1556: done (3 left) 2024-09-20T19:30:09.858 INFO:tasks.rados.rados.0.smithi042.stdout:1561: read oid 23 snap -1 2024-09-20T19:30:09.858 INFO:tasks.rados.rados.0.smithi042.stdout:1561: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-20T19:30:09.858 INFO:tasks.rados.rados.0.smithi042.stdout:1562: write (excl) oid 46 current snap is 177 2024-09-20T19:30:09.858 INFO:tasks.rados.rados.0.smithi042.stdout:1562: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-09-20T19:30:09.863 INFO:tasks.rados.rados.0.smithi042.stdout:1562: writing smithi04268048-46 from 443850 to 1008216 tid 1 2024-09-20T19:30:09.873 INFO:tasks.rados.rados.0.smithi042.stdout:1562: writing smithi04268048-46 from 1634861 to 2367717 tid 2 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1562: writing smithi04268048-46 from 2770646 to 3243850 tid 3 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1563: copy_from oid 29 from oid 12 current snap is 177 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1558: finishing write tid 2 to smithi04268048-35 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1560: finishing rollback tid 0 to smithi04268048-33 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1560: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 583 (ObjNum 33 snap 0 seq_num 1415356176) dirty dne 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1559: done (5 left) 2024-09-20T19:30:09.879 INFO:tasks.rados.rados.0.smithi042.stdout:1560: done (4 left) 2024-09-20T19:30:09.880 INFO:tasks.rados.rados.0.smithi042.stdout:1564: write oid 17 current snap is 177 2024-09-20T19:30:09.880 INFO:tasks.rados.rados.0.smithi042.stdout:1564: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-20T19:30:09.886 INFO:tasks.rados.rados.0.smithi042.stdout:1564: writing smithi04268048-17 from 460657 to 1100272 tid 1 2024-09-20T19:30:09.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:09.893 INFO:tasks.rados.rados.0.smithi042.stdout:1564: writing smithi04268048-17 from 1693343 to 2169857 tid 2 2024-09-20T19:30:09.899 INFO:tasks.rados.rados.0.smithi042.stdout:1564: writing smithi04268048-17 from 2681716 to 3260657 tid 3 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1565: read oid 38 snap -1 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1565: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1566: rollback oid 41 current snap is 177 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 176 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: finishing write tid 3 to smithi04268048-35 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: finishing write tid 4 to smithi04268048-35 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: finishing write tid 5 to smithi04268048-35 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: oid 35 updating version 0 to 583 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: oid 35 version 583 is already newer than 582 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: oid 35 updating version 583 to 585 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: oid 35 version 585 is already newer than 584 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 585 (ObjNum 549 snap 177 seq_num 549) dirty exists 2024-09-20T19:30:09.900 INFO:tasks.rados.rados.0.smithi042.stdout:1558: left oid 35 (ObjNum 549 snap 177 seq_num 549) 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1558: done (6 left) 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1567: read oid 39 snap -1 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1567: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1568: rollback oid 30 current snap is 177 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 176 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1569: copy_from oid 16 from oid 42 current snap is 177 2024-09-20T19:30:09.901 INFO:tasks.rados.rados.0.smithi042.stdout:1561: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-20T19:30:09.904 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 851687 2024-09-20T19:30:09.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:09.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[22706]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:09.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:09 smithi042 bash[32075]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:10.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:10.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.713430+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.718908+0000 mon.a (mon.0) 1805 : cluster [INF] osd.0 v1:172.21.15.42:6801/701330554 boot 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: cluster 2024-09-20T19:30:08.722236+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 8 total, 6 up, 8 in 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: audit 2024-09-20T19:30:08.724413+0000 mon.a (mon.0) 1807 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:10.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:09 smithi162 bash[24557]: audit 2024-09-20T19:30:09.461681+0000 mgr.y (mgr.24491) 462 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:10.045 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:1564: finishing write tid 1 to smithi04268048-17 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:1564: finishing write tid 2 to smithi04268048-17 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:1568: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 686 (ObjNum 33 snap 0 seq_num 1414990736) dirty dne 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:1566: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 434 (ObjNum 33 snap 0 seq_num 1390889016) dirty dne 2024-09-20T19:30:10.104 INFO:tasks.rados.rados.0.smithi042.stdout:1565: expect (ObjNum 383 snap 123 seq_num 383) 2024-09-20T19:30:10.123 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1798440 2024-09-20T19:30:10.133 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:10.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.170 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:10.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:10.208 INFO:tasks.rados.rados.0.smithi042.stdout:1567: expect (ObjNum 283 snap 87 seq_num 283) 2024-09-20T19:30:10.239 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1808687 2024-09-20T19:30:10.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:10.286 INFO:tasks.rados.rados.0.smithi042.stdout:1564: finishing write tid 3 to smithi04268048-17 2024-09-20T19:30:10.286 INFO:tasks.rados.rados.0.smithi042.stdout:1564: finishing write tid 4 to smithi04268048-17 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: finishing write tid 5 to smithi04268048-17 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: oid 17 updating version 0 to 683 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: oid 17 version 683 is already newer than 682 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: oid 17 version 683 is already newer than 681 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: oid 17 updating version 683 to 684 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 684 (ObjNum 551 snap 177 seq_num 551) dirty exists 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: left oid 17 (ObjNum 551 snap 177 seq_num 551) 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1569: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1569: finishing copy_from to smithi04268048-16 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 712 (ObjNum 543 snap 177 seq_num 543) dirty exists 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1561: done (8 left) 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1564: done (7 left) 2024-09-20T19:30:10.287 INFO:tasks.rados.rados.0.smithi042.stdout:1565: done (6 left) 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1566: done (5 left) 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1567: done (4 left) 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1568: done (3 left) 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1569: done (2 left) 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1570: write (excl) oid 31 current snap is 177 2024-09-20T19:30:10.288 INFO:tasks.rados.rados.0.smithi042.stdout:1570: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-20T19:30:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:10.309 INFO:tasks.rados.rados.0.smithi042.stdout:1570: writing smithi04268048-31 from 477464 to 1192328 tid 1 2024-09-20T19:30:10.322 INFO:tasks.rados.rados.0.smithi042.stdout:1570: writing smithi04268048-31 from 1751825 to 2488350 tid 2 2024-09-20T19:30:10.331 INFO:tasks.rados.rados.0.smithi042.stdout:1570: writing smithi04268048-31 from 3225492 to 3277464 tid 3 2024-09-20T19:30:10.331 INFO:tasks.rados.rados.0.smithi042.stdout:1571: delete oid 35 current snap is 177 2024-09-20T19:30:10.331 INFO:tasks.rados.rados.0.smithi042.stdout:1570: finishing write tid 1 to smithi04268048-31 2024-09-20T19:30:10.331 INFO:tasks.rados.rados.0.smithi042.stdout:1570: finishing write tid 2 to smithi04268048-31 2024-09-20T19:30:10.333 INFO:tasks.rados.rados.0.smithi042.stdout:1571: done (3 left) 2024-09-20T19:30:10.333 INFO:tasks.rados.rados.0.smithi042.stdout:1572: read oid 48 snap -1 2024-09-20T19:30:10.333 INFO:tasks.rados.rados.0.smithi042.stdout:1572: expect (ObjNum 535 snap 173 seq_num 535) 2024-09-20T19:30:10.333 INFO:tasks.rados.rados.0.smithi042.stdout:1573: snap_create 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: finishing write tid 3 to smithi04268048-31 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: finishing write tid 4 to smithi04268048-31 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: finishing write tid 5 to smithi04268048-31 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: oid 31 updating version 0 to 599 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: oid 31 updating version 599 to 600 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: oid 31 updating version 600 to 601 2024-09-20T19:30:10.334 INFO:tasks.rados.rados.0.smithi042.stdout:1570: oid 31 version 601 is already newer than 598 2024-09-20T19:30:10.335 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 601 (ObjNum 552 snap 177 seq_num 552) dirty exists 2024-09-20T19:30:10.335 INFO:tasks.rados.rados.0.smithi042.stdout:1570: left oid 31 (ObjNum 552 snap 177 seq_num 552) 2024-09-20T19:30:10.339 INFO:tasks.rados.rados.0.smithi042.stdout:1572: expect (ObjNum 535 snap 173 seq_num 535) 2024-09-20T19:30:10.339 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:10.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.416 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:10.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:10.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:10.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1570: done (4 left) 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1572: done (3 left) 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1573: done (2 left) 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1574: read oid 8 snap 154 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1574: expect (ObjNum 376 snap 118 seq_num 376) 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1575: copy_from oid 20 from oid 3 current snap is 178 2024-09-20T19:30:10.768 INFO:tasks.rados.rados.0.smithi042.stdout:1576: write (excl) oid 31 current snap is 178 2024-09-20T19:30:10.769 INFO:tasks.rados.rados.0.smithi042.stdout:1576: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-20T19:30:10.778 INFO:tasks.rados.rados.0.smithi042.stdout:1576: writing smithi04268048-31 from 511078 to 976440 tid 1 2024-09-20T19:30:10.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.793 INFO:tasks.rados.rados.0.smithi042.stdout:1576: writing smithi04268048-31 from 1752436 to 2492630 tid 2 2024-09-20T19:30:10.800 INFO:tasks.rados.rados.0.smithi042.stdout:1576: writing smithi04268048-31 from 3163985 to 3311078 tid 3 2024-09-20T19:30:10.800 INFO:tasks.rados.rados.0.smithi042.stdout:1577: snap_remove snap 134 2024-09-20T19:30:10.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:10.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:10.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:10.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[22706]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:10.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:10.803 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:10 smithi042 bash[32075]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:10.817 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:10.920 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:10.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:11.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.761667+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:07.768746+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:09.740425+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e501: 8 total, 6 up, 8 in 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:11.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:10 smithi162 bash[24557]: cluster 2024-09-20T19:30:10.241084+0000 mgr.y (mgr.24491) 463 : cluster [DBG] pgmap v750: 168 pgs: 4 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 22 undersized+peered, 54 active+undersized, 20 undersized+degraded+peered, 39 active+undersized+degraded, 26 active+clean; 416 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 449/1209 objects degraded (37.138%); 22/1209 objects misplaced (1.820%) 2024-09-20T19:30:11.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:11.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:11.200 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:11.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:11.331 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:11.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.461 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:11.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:11.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:11.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:11.769 INFO:tasks.rados.rados.0.smithi042.stdout:1577: done (5 left) 2024-09-20T19:30:11.769 INFO:tasks.rados.rados.0.smithi042.stdout:1578: rollback oid 39 current snap is 178 2024-09-20T19:30:11.770 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 170 2024-09-20T19:30:11.770 INFO:tasks.rados.rados.0.smithi042.stdout:1579: read oid 19 snap -1 2024-09-20T19:30:11.770 INFO:tasks.rados.rados.0.smithi042.stdout:1579: expect (ObjNum 528 snap 171 seq_num 528) 2024-09-20T19:30:11.773 INFO:tasks.rados.rados.0.smithi042.stdout:1580: write oid 17 current snap is 178 2024-09-20T19:30:11.773 INFO:tasks.rados.rados.0.smithi042.stdout:1580: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-20T19:30:11.777 INFO:tasks.rados.rados.0.smithi042.stdout:1580: writing smithi04268048-17 from 527885 to 1184849 tid 1 2024-09-20T19:30:11.785 INFO:tasks.rados.rados.0.smithi042.stdout:1580: writing smithi04268048-17 from 1927271 to 2411123 tid 2 2024-09-20T19:30:11.790 INFO:tasks.rados.rados.0.smithi042.stdout:1580: writing smithi04268048-17 from 3191408 to 3327885 tid 3 2024-09-20T19:30:11.790 INFO:tasks.rados.rados.0.smithi042.stdout:1581: snap_remove snap 172 2024-09-20T19:30:11.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:11.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:11.919 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:30:11.919 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:30:11.944 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:30:11.944 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:30:11.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.960 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:11 smithi162 bash[24557]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:11.960 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:11 smithi162 bash[24557]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:11.966 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:30:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:30:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:30:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:30:11.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:11.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:11.984 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:30:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:30:12.004 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:30:12.005 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:30:12.023 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:30:12.024 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:30:12.027 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:11 smithi042 bash[22706]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:12.027 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:11 smithi042 bash[22706]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:12.028 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:11 smithi042 bash[32075]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:12.028 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:11 smithi042 bash[32075]: cluster 2024-09-20T19:30:10.765565+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e502: 8 total, 6 up, 8 in 2024-09-20T19:30:12.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:12.084 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:30:12.084 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:30:12.096 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:30:12.096 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:30:12.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:12.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:12.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:12.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:12.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:12.624 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:12.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:12.788 INFO:tasks.rados.rados.0.smithi042.stdout:1581: done (8 left) 2024-09-20T19:30:12.788 INFO:tasks.rados.rados.0.smithi042.stdout:1582: copy_from oid 32 from oid 23 current snap is 178 2024-09-20T19:30:12.788 INFO:tasks.rados.rados.0.smithi042.stdout:1583: snap_remove snap 165 2024-09-20T19:30:12.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:12.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:12.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[32075]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:12.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:12 smithi042 bash[22706]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:12.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: cluster 2024-09-20T19:30:11.768865+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e503: 8 total, 6 up, 8 in 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:11.900335+0000 mon.a (mon.0) 1811 : audit [DBG] from='client.? v1:172.21.15.42:0/3596538409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.243016+0000 mgr.y (mgr.24491) 464 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 2 active+clean+remapped, 1 stale+undersized+peered, 2 stale+undersized+degraded+peered, 1 active+clean+remapped+snaptrim, 16 undersized+peered, 46 active+undersized, 15 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 401 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 24 op/s; 371/1194 objects degraded (31.072%); 20/1194 objects misplaced (1.675%); 9.6 MiB/s, 7 objects/s recovering 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:12.695866+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:12.998 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:12.998 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:12 smithi162 bash[24557]: audit 2024-09-20T19:30:12.697872+0000 mon.a (mon.0) 1813 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:13.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:13.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:13.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:13.215 INFO:tasks.rados.rados.0.smithi042.stdout:1583: done (9 left) 2024-09-20T19:30:13.215 INFO:tasks.rados.rados.0.smithi042.stdout:1584: delete oid 45 current snap is 178 2024-09-20T19:30:13.216 INFO:tasks.rados.rados.0.smithi042.stdout:1575: finishing copy_from to smithi04268048-20 2024-09-20T19:30:13.216 INFO:tasks.rados.rados.0.smithi042.stdout:1575: got expected ENOENT (src dne) 2024-09-20T19:30:13.216 INFO:tasks.rados.rados.0.smithi042.stdout:1575: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:30:13.216 INFO:tasks.rados.rados.0.smithi042.stdout:1574: expect (ObjNum 376 snap 118 seq_num 376) 2024-09-20T19:30:13.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:13.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:1576: finishing write tid 1 to smithi04268048-31 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: finishing write tid 1 to smithi04268048-46 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: finishing write tid 2 to smithi04268048-46 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: finishing write tid 3 to smithi04268048-46 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: finishing write tid 4 to smithi04268048-46 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: finishing write tid 5 to smithi04268048-46 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: oid 46 updating version 0 to 943 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: oid 46 updating version 943 to 946 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: oid 46 version 946 is already newer than 944 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: oid 46 version 946 is already newer than 945 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 946 (ObjNum 550 snap 177 seq_num 550) dirty exists 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1562: left oid 46 (ObjNum 550 snap 177 seq_num 550) 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1576: finishing write tid 2 to smithi04268048-31 2024-09-20T19:30:13.333 INFO:tasks.rados.rados.0.smithi042.stdout:1563: finishing copy_from racing read to smithi04268048-29 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1563: finishing copy_from to smithi04268048-29 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 436 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: finishing write tid 3 to smithi04268048-31 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: finishing write tid 4 to smithi04268048-31 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: finishing write tid 5 to smithi04268048-31 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: oid 31 updating version 0 to 605 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: oid 31 version 605 is already newer than 603 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: oid 31 version 605 is already newer than 604 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: oid 31 updating version 605 to 606 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 606 (ObjNum 554 snap 178 seq_num 554) dirty exists 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1576: left oid 31 (ObjNum 554 snap 178 seq_num 554) 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1578: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1578: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:30:13.334 INFO:tasks.rados.rados.0.smithi042.stdout:1578: finishing rollback tid 2 to smithi04268048-39 2024-09-20T19:30:13.335 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 584 (ObjNum 283 snap 87 seq_num 283) dirty exists 2024-09-20T19:30:13.335 INFO:tasks.rados.rados.0.smithi042.stdout:1580: finishing write tid 1 to smithi04268048-17 2024-09-20T19:30:13.335 INFO:tasks.rados.rados.0.smithi042.stdout:1580: finishing write tid 2 to smithi04268048-17 2024-09-20T19:30:13.335 INFO:tasks.rados.rados.0.smithi042.stdout:1579: expect (ObjNum 528 snap 171 seq_num 528) 2024-09-20T19:30:13.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:13.394 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:13.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: finishing write tid 3 to smithi04268048-17 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: finishing write tid 4 to smithi04268048-17 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: finishing write tid 5 to smithi04268048-17 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: oid 17 updating version 0 to 687 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: oid 17 version 687 is already newer than 685 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: oid 17 updating version 687 to 688 2024-09-20T19:30:13.426 INFO:tasks.rados.rados.0.smithi042.stdout:1580: oid 17 version 688 is already newer than 686 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 688 (ObjNum 555 snap 178 seq_num 555) dirty exists 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1580: left oid 17 (ObjNum 555 snap 178 seq_num 555) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1582: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1582: finishing copy_from to smithi04268048-32 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 679 (ObjNum 527 snap 170 seq_num 527) dirty exists 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1562: done (9 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1563: done (8 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1574: done (7 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1575: done (6 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1576: done (5 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1578: done (4 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1579: done (3 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1580: done (2 left) 2024-09-20T19:30:13.427 INFO:tasks.rados.rados.0.smithi042.stdout:1582: done (1 left) 2024-09-20T19:30:13.428 INFO:tasks.rados.rados.0.smithi042.stdout:1584: done (0 left) 2024-09-20T19:30:13.428 INFO:tasks.rados.rados.0.smithi042.stdout:1585: snap_remove snap 173 2024-09-20T19:30:13.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:13.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:13.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:13.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:13.766 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:13.794 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:13.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:13.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:30:13.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:30:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:13.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:13.941 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-20T19:30:13.941 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-09-20T19:30:13.942 DEBUG:teuthology.orchestra.run.smithi162:osd.4> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T19:30:13.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:30:13.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl start ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4 2024-09-20T19:30:14.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:13 smithi162 bash[24557]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:30:14.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:14.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:14.160 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:14 smithi162 systemd[1]: Started Ceph osd.4 for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:30:14.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[22706]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.764605+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Reduced data availability: 34 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.764652+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check update: Degraded data redundancy: 371/1194 objects degraded (31.072%), 55 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:12.784924+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e504: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:13 smithi042 bash[32075]: cluster 2024-09-20T19:30:13.214041+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e505: 8 total, 6 up, 8 in 2024-09-20T19:30:14.241 INFO:tasks.rados.rados.0.smithi042.stdout:1585: done (0 left) 2024-09-20T19:30:14.241 INFO:tasks.rados.rados.0.smithi042.stdout:1586: delete oid 19 current snap is 178 2024-09-20T19:30:14.243 INFO:tasks.rados.rados.0.smithi042.stdout:1586: done (0 left) 2024-09-20T19:30:14.243 INFO:tasks.rados.rados.0.smithi042.stdout:1587: write oid 20 current snap is 178 2024-09-20T19:30:14.243 INFO:tasks.rados.rados.0.smithi042.stdout:1587: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-09-20T19:30:14.249 INFO:tasks.rados.rados.0.smithi042.stdout:1587: writing smithi04268048-20 from 578306 to 1061017 tid 1 2024-09-20T19:30:14.256 INFO:tasks.rados.rados.0.smithi042.stdout:1587: writing smithi04268048-20 from 1586364 to 2217543 tid 2 2024-09-20T19:30:14.262 INFO:tasks.rados.rados.0.smithi042.stdout:1587: writing smithi04268048-20 from 2757324 to 3242166 tid 3 2024-09-20T19:30:14.263 INFO:tasks.rados.rados.0.smithi042.stdout:1588: read oid 42 snap -1 2024-09-20T19:30:14.263 INFO:tasks.rados.rados.0.smithi042.stdout:1588: expect (ObjNum 543 snap 177 seq_num 543) 2024-09-20T19:30:14.263 INFO:tasks.rados.rados.0.smithi042.stdout:1589: snap_remove snap 170 2024-09-20T19:30:14.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:14.294 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:14.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:14.398 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:14.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:14.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:14.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:14.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:14.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:14.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:15.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:15.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.203 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:15.250 INFO:tasks.rados.rados.0.smithi042.stdout:1589: done (2 left) 2024-09-20T19:30:15.250 INFO:tasks.rados.rados.0.smithi042.stdout:1590: read oid 9 snap -1 2024-09-20T19:30:15.250 INFO:tasks.rados.rados.0.smithi042.stdout:1590: expect deleted 2024-09-20T19:30:15.250 INFO:tasks.rados.rados.0.smithi042.stdout:1591: delete oid 27 current snap is 178 2024-09-20T19:30:15.250 INFO:tasks.rados.rados.0.smithi042.stdout:1587: finishing write tid 1 to smithi04268048-20 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: finishing write tid 2 to smithi04268048-20 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: finishing write tid 3 to smithi04268048-20 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: finishing write tid 4 to smithi04268048-20 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: finishing write tid 5 to smithi04268048-20 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: oid 20 updating version 0 to 133 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: oid 20 updating version 133 to 134 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: oid 20 version 134 is already newer than 132 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: oid 20 updating version 134 to 135 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 135 (ObjNum 558 snap 178 seq_num 558) dirty exists 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1587: left oid 20 (ObjNum 558 snap 178 seq_num 558) 2024-09-20T19:30:15.251 INFO:tasks.rados.rados.0.smithi042.stdout:1588: expect (ObjNum 543 snap 177 seq_num 543) 2024-09-20T19:30:15.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.329 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:15.349 INFO:tasks.rados.rados.0.smithi042.stdout:1587: done (3 left) 2024-09-20T19:30:15.349 INFO:tasks.rados.rados.0.smithi042.stdout:1588: done (2 left) 2024-09-20T19:30:15.350 INFO:tasks.rados.rados.0.smithi042.stdout:1590: done (1 left) 2024-09-20T19:30:15.350 INFO:tasks.rados.rados.0.smithi042.stdout:1591: done (0 left) 2024-09-20T19:30:15.350 INFO:tasks.rados.rados.0.smithi042.stdout:1592: write oid 44 current snap is 178 2024-09-20T19:30:15.350 INFO:tasks.rados.rados.0.smithi042.stdout:1592: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-20T19:30:15.356 INFO:tasks.rados.rados.0.smithi042.stdout:1592: writing smithi04268048-44 from 611920 to 1245129 tid 1 2024-09-20T19:30:15.363 INFO:tasks.rados.rados.0.smithi042.stdout:1592: writing smithi04268048-44 from 1703328 to 2338176 tid 2 2024-09-20T19:30:15.369 INFO:tasks.rados.rados.0.smithi042.stdout:1592: writing smithi04268048-44 from 2812170 to 3319556 tid 3 2024-09-20T19:30:15.369 INFO:tasks.rados.rados.0.smithi042.stdout:1593: copy_from oid 20 from oid 10 current snap is 178 2024-09-20T19:30:15.369 INFO:tasks.rados.rados.0.smithi042.stdout:1592: finishing write tid 1 to smithi04268048-44 2024-09-20T19:30:15.369 INFO:tasks.rados.rados.0.smithi042.stdout:1594: write oid 24 current snap is 178 2024-09-20T19:30:15.369 INFO:tasks.rados.rados.0.smithi042.stdout:1594: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-20T19:30:15.376 INFO:tasks.rados.rados.0.smithi042.stdout:1594: writing smithi04268048-24 from 628727 to 1337185 tid 1 2024-09-20T19:30:15.383 INFO:tasks.rados.rados.0.smithi042.stdout:1594: writing smithi04268048-24 from 2045457 to 2540316 tid 2 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1594: writing smithi04268048-24 from 3123240 to 3428727 tid 3 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1595: snap_create 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: finishing write tid 2 to smithi04268048-44 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: finishing write tid 3 to smithi04268048-44 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: finishing write tid 4 to smithi04268048-44 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: finishing write tid 5 to smithi04268048-44 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: oid 44 updating version 0 to 953 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: oid 44 updating version 953 to 956 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: oid 44 version 956 is already newer than 955 2024-09-20T19:30:15.389 INFO:tasks.rados.rados.0.smithi042.stdout:1592: oid 44 version 956 is already newer than 954 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 956 (ObjNum 560 snap 178 seq_num 560) dirty exists 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:1592: left oid 44 (ObjNum 560 snap 178 seq_num 560) 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:1594: finishing write tid 1 to smithi04268048-24 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:1593: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:1594: finishing write tid 2 to smithi04268048-24 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:1593: finishing copy_from to smithi04268048-20 2024-09-20T19:30:15.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 583 (ObjNum 393 snap 124 seq_num 393) dirty exists 2024-09-20T19:30:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:1594: finishing write tid 3 to smithi04268048-24 2024-09-20T19:30:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:1594: finishing write tid 4 to smithi04268048-24 2024-09-20T19:30:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:1594: finishing write tid 5 to smithi04268048-24 2024-09-20T19:30:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:1594: oid 24 updating version 0 to 716 2024-09-20T19:30:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:1594: oid 24 version 716 is already newer than 715 2024-09-20T19:30:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:1594: oid 24 version 716 is already newer than 713 2024-09-20T19:30:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:1594: oid 24 version 716 is already newer than 714 2024-09-20T19:30:15.395 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 716 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:30:15.395 INFO:tasks.rados.rados.0.smithi042.stdout:1594: left oid 24 (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:30:15.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[32075]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[32075]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[32075]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[32075]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[22706]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.443 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[22706]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.444 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[22706]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.444 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:15 smithi042 bash[22706]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:15.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[24557]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[24557]: cluster 2024-09-20T19:30:14.241473+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e506: 8 total, 6 up, 8 in 2024-09-20T19:30:15.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[24557]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[24557]: cluster 2024-09-20T19:30:14.243807+0000 mgr.y (mgr.24491) 465 : cluster [DBG] pgmap v757: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.2 MiB/s wr, 36 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 26 MiB/s, 28 objects/s recovering 2024-09-20T19:30:15.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.580 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:15.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:15.801 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[77218]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:15.801 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:15 smithi162 bash[77218]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:15.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:15.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:15.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:30:15.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:15.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:16.095 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:16.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:16.245 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:16.257 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:30:16.259 INFO:tasks.rados.rados.0.smithi042.stdout:1592: done (3 left) 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:1593: done (2 left) 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:1594: done (1 left) 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:1595: done (0 left) 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:1596: rollback oid 17 current snap is 179 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 168 2024-09-20T19:30:16.260 INFO:tasks.rados.rados.0.smithi042.stdout:1597: rollback oid 32 current snap is 179 2024-09-20T19:30:16.261 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 177 2024-09-20T19:30:16.261 INFO:tasks.rados.rados.0.smithi042.stdout:1598: snap_remove snap 159 2024-09-20T19:30:16.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:16.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:16.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:16.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:16.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:16 smithi162 bash[24557]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:16.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:16.618 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[32075]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: cluster 2024-09-20T19:30:15.249182+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e507: 8 total, 6 up, 8 in 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: audit 2024-09-20T19:30:16.213058+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.42:0/3133709064' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:16 smithi042 bash[22706]: audit 2024-09-20T19:30:16.213851+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25409 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:16.635 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:16.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:16.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:16.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:16.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:16.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:16.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:17.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:17.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:17.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:17.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:17.272 INFO:tasks.rados.rados.0.smithi042.stdout:1598: done (2 left) 2024-09-20T19:30:17.272 INFO:tasks.rados.rados.0.smithi042.stdout:1599: delete oid 26 current snap is 179 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1597: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1597: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1597: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 939 (ObjNum 545 snap 177 seq_num 545) dirty exists 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1596: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1596: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:1596: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:30:17.273 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 691 (ObjNum 514 snap 166 seq_num 514) dirty exists 2024-09-20T19:30:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:17.277 INFO:tasks.rados.rados.0.smithi042.stdout:1596: done (2 left) 2024-09-20T19:30:17.277 INFO:tasks.rados.rados.0.smithi042.stdout:1597: done (1 left) 2024-09-20T19:30:17.277 INFO:tasks.rados.rados.0.smithi042.stdout:1599: done (0 left) 2024-09-20T19:30:17.277 INFO:tasks.rados.rados.0.smithi042.stdout:1600: snap_create 2024-09-20T19:30:17.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[22706]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:17 smithi042 bash[32075]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:17.534 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:17.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.040309+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.041262+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.243729+0000 mon.a (mon.0) 1821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.244679+0000 mgr.y (mgr.24491) 466 : cluster [DBG] pgmap v759: 168 pgs: 1 creating+peering, 3 down, 2 active+clean+snaptrim_wait, 2 peering, 4 active+clean+snaptrim, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 46 active+clean; 404 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 669 KiB/s wr, 6 op/s; 279/1200 objects degraded (23.250%); 12/1200 objects misplaced (1.000%); 13 MiB/s, 16 objects/s recovering 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: audit 2024-09-20T19:30:16.252719+0000 mon.a (mon.0) 1822 : audit [INF] from='client.25409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:17 smithi162 bash[24557]: cluster 2024-09-20T19:30:16.257648+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e508: 8 total, 6 up, 8 in 2024-09-20T19:30:17.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:17.767 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:17.799 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:17.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:17.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:17.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:18.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:18.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:18.281 INFO:tasks.rados.rados.0.smithi042.stdout:1600: done (0 left) 2024-09-20T19:30:18.281 INFO:tasks.rados.rados.0.smithi042.stdout:1601: read oid 41 snap 167 2024-09-20T19:30:18.282 INFO:tasks.rados.rados.0.smithi042.stdout:1601: expect deleted 2024-09-20T19:30:18.282 INFO:tasks.rados.rados.0.smithi042.stdout:1602: read oid 40 snap -1 2024-09-20T19:30:18.282 INFO:tasks.rados.rados.0.smithi042.stdout:1602: expect (ObjNum 484 snap 158 seq_num 484) 2024-09-20T19:30:18.282 INFO:tasks.rados.rados.0.smithi042.stdout:1603: snap_remove snap 155 2024-09-20T19:30:18.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:18.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:18 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.457 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:17.270865+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e509: 8 total, 6 up, 8 in 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.205897+0000 mon.a (mon.0) 1825 : cluster [WRN] Health check update: Reduced data availability: 14 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:18 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.205974+0000 mon.a (mon.0) 1826 : cluster [WRN] Health check update: Degraded data redundancy: 279/1200 objects degraded (23.250%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:18.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:18.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.714 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:18.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.847 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:18.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:18.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:19.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:19.105 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:19.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:30:19.208 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:19.234 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:19.290 INFO:tasks.rados.rados.0.smithi042.stdout:1603: done (2 left) 2024-09-20T19:30:19.290 INFO:tasks.rados.rados.0.smithi042.stdout:1604: write oid 23 current snap is 180 2024-09-20T19:30:19.290 INFO:tasks.rados.rados.0.smithi042.stdout:1604: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-20T19:30:19.299 INFO:tasks.rados.rados.0.smithi042.stdout:1604: writing smithi04268048-23 from 662341 to 1237650 tid 1 2024-09-20T19:30:19.307 INFO:tasks.rados.rados.0.smithi042.stdout:1604: writing smithi04268048-23 from 1878774 to 2377302 tid 2 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1604: writing smithi04268048-23 from 2894439 to 3351994 tid 3 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1605: read oid 26 snap -1 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1605: expect deleted 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1606: read oid 20 snap -1 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1606: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:30:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:1602: expect (ObjNum 484 snap 158 seq_num 484) 2024-09-20T19:30:19.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:19.340 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:19.346 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[22706]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.347 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:19 smithi042 bash[32075]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1604: finishing write tid 1 to smithi04268048-23 2024-09-20T19:30:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1604: finishing write tid 2 to smithi04268048-23 2024-09-20T19:30:19.378 INFO:tasks.rados.rados.0.smithi042.stdout:1606: expect (ObjNum 393 snap 124 seq_num 393) 2024-09-20T19:30:19.420 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2483040 2024-09-20T19:30:19.446 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1577393 2024-09-20T19:30:19.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: finishing write tid 3 to smithi04268048-23 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: finishing write tid 4 to smithi04268048-23 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: finishing write tid 5 to smithi04268048-23 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: oid 23 updating version 0 to 693 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: oid 23 updating version 693 to 695 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: oid 23 version 695 is already newer than 692 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: oid 23 version 695 is already newer than 694 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 695 (ObjNum 563 snap 180 seq_num 563) dirty exists 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1604: left oid 23 (ObjNum 563 snap 180 seq_num 563) 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1601: done (4 left) 2024-09-20T19:30:19.522 INFO:tasks.rados.rados.0.smithi042.stdout:1602: done (3 left) 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1604: done (2 left) 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1605: done (1 left) 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1606: done (0 left) 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1607: read oid 9 snap -1 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1607: expect deleted 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1608: read oid 48 snap -1 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1608: expect (ObjNum 535 snap 173 seq_num 535) 2024-09-20T19:30:19.523 INFO:tasks.rados.rados.0.smithi042.stdout:1609: delete oid 44 current snap is 180 2024-09-20T19:30:19.525 INFO:tasks.rados.rados.0.smithi042.stdout:1608: expect (ObjNum 535 snap 173 seq_num 535) 2024-09-20T19:30:19.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.245957+0000 mgr.y (mgr.24491) 467 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 769 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[24557]: cluster 2024-09-20T19:30:18.281690+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e510: 8 total, 6 up, 8 in 2024-09-20T19:30:19.589 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:19.614 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:19.649 INFO:tasks.rados.rados.0.smithi042.stdout:1607: done (2 left) 2024-09-20T19:30:19.649 INFO:tasks.rados.rados.0.smithi042.stdout:1608: done (1 left) 2024-09-20T19:30:19.649 INFO:tasks.rados.rados.0.smithi042.stdout:1609: done (0 left) 2024-09-20T19:30:19.649 INFO:tasks.rados.rados.0.smithi042.stdout:1610: write oid 31 current snap is 180 2024-09-20T19:30:19.649 INFO:tasks.rados.rados.0.smithi042.stdout:1610: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-20T19:30:19.657 INFO:tasks.rados.rados.0.smithi042.stdout:1610: writing smithi04268048-31 from 695955 to 1421762 tid 1 2024-09-20T19:30:19.662 INFO:tasks.rados.rados.0.smithi042.stdout:1610: writing smithi04268048-31 from 1879385 to 2381582 tid 2 2024-09-20T19:30:19.668 INFO:tasks.rados.rados.0.smithi042.stdout:1610: writing smithi04268048-31 from 2949285 to 3429384 tid 3 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1611: read oid 28 snap -1 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1611: expect deleted 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1612: rollback oid 14 current snap is 180 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 179 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1610: finishing write tid 1 to smithi04268048-31 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1610: finishing write tid 2 to smithi04268048-31 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1613: write oid 20 current snap is 180 2024-09-20T19:30:19.669 INFO:tasks.rados.rados.0.smithi042.stdout:1613: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-20T19:30:19.675 INFO:tasks.rados.rados.0.smithi042.stdout:1613: writing smithi04268048-20 from 712762 to 1113818 tid 1 2024-09-20T19:30:19.681 INFO:tasks.rados.rados.0.smithi042.stdout:1613: writing smithi04268048-20 from 1537867 to 2183722 tid 2 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1613: writing smithi04268048-20 from 2860355 to 3512762 tid 3 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1614: snap_create 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1612: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1612: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1610: finishing write tid 3 to smithi04268048-31 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1612: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 421 (ObjNum 217 snap 65 seq_num 217) dirty exists 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1610: finishing write tid 4 to smithi04268048-31 2024-09-20T19:30:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:1610: finishing write tid 5 to smithi04268048-31 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1610: oid 31 updating version 0 to 611 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1610: oid 31 version 611 is already newer than 610 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1610: oid 31 version 611 is already newer than 608 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1610: oid 31 version 611 is already newer than 609 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 611 (ObjNum 565 snap 180 seq_num 565) dirty exists 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1610: left oid 31 (ObjNum 565 snap 180 seq_num 565) 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1613: finishing write tid 1 to smithi04268048-20 2024-09-20T19:30:19.690 INFO:tasks.rados.rados.0.smithi042.stdout:1613: finishing write tid 2 to smithi04268048-20 2024-09-20T19:30:19.693 INFO:tasks.rados.rados.0.smithi042.stdout:1613: finishing write tid 3 to smithi04268048-20 2024-09-20T19:30:19.693 INFO:tasks.rados.rados.0.smithi042.stdout:1613: finishing write tid 4 to smithi04268048-20 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: finishing write tid 5 to smithi04268048-20 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: oid 20 updating version 0 to 584 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: oid 20 updating version 584 to 587 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: oid 20 version 587 is already newer than 586 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: oid 20 version 587 is already newer than 585 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 587 (ObjNum 566 snap 180 seq_num 566) dirty exists 2024-09-20T19:30:19.694 INFO:tasks.rados.rados.0.smithi042.stdout:1613: left oid 20 (ObjNum 566 snap 180 seq_num 566) 2024-09-20T19:30:19.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:19.727 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[77218]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-20T19:30:19.727 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[77218]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:19.727 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[77218]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:19.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:19.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:19.848 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:30:19.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:19.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:19.978 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[77218]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T19:30:19.978 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:19 smithi162 bash[77218]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2024-09-20T19:30:19.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:19.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:20.095 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:20.123 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:20.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:20.236 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[77218]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T19:30:20.236 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[77218]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2024-09-20T19:30:20.236 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[77218]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:30:20.236 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[77218]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T19:30:20.236 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[77218]: --> ceph-volume lvm activate successful for osd ID: 4 2024-09-20T19:30:20.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1610: done (4 left) 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1611: done (3 left) 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1612: done (2 left) 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1613: done (1 left) 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1614: done (0 left) 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1615: write oid 38 current snap is 181 2024-09-20T19:30:20.302 INFO:tasks.rados.rados.0.smithi042.stdout:1615: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-20T19:30:20.308 INFO:tasks.rados.rados.0.smithi042.stdout:1615: writing smithi04268048-38 from 729569 to 1205874 tid 1 2024-09-20T19:30:20.315 INFO:tasks.rados.rados.0.smithi042.stdout:1615: writing smithi04268048-38 from 1996349 to 2502215 tid 2 2024-09-20T19:30:20.321 INFO:tasks.rados.rados.0.smithi042.stdout:1615: writing smithi04268048-38 from 3004131 to 3529569 tid 3 2024-09-20T19:30:20.322 INFO:tasks.rados.rados.0.smithi042.stdout:1616: snap_remove snap 150 2024-09-20T19:30:20.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:20.362 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[24557]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.362 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[24557]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.362 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[24557]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.362 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:20 smithi162 bash[24557]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.371 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:30:20.371 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:30:20.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:20.379 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[32075]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[32075]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[32075]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[32075]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[22706]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[22706]: cluster 2024-09-20T19:30:19.289975+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e511: 8 total, 6 up, 8 in 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[22706]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:20 smithi042 bash[22706]: audit 2024-09-20T19:30:19.471355+0000 mgr.y (mgr.24491) 468 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:20.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:20.509 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:20.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:20.637 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:20.640 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:20.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:20.771 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:20.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:20.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:20.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:21.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:21.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:21.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:21.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1616: done (1 left) 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1617: delete oid 44 current snap is 181 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: finishing write tid 1 to smithi04268048-38 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: finishing write tid 2 to smithi04268048-38 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: finishing write tid 3 to smithi04268048-38 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: finishing write tid 4 to smithi04268048-38 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: finishing write tid 5 to smithi04268048-38 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: oid 38 updating version 0 to 587 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: oid 38 updating version 587 to 590 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: oid 38 version 590 is already newer than 588 2024-09-20T19:30:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1615: oid 38 version 590 is already newer than 589 2024-09-20T19:30:21.305 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 590 (ObjNum 567 snap 181 seq_num 567) dirty exists 2024-09-20T19:30:21.305 INFO:tasks.rados.rados.0.smithi042.stdout:1615: left oid 38 (ObjNum 567 snap 181 seq_num 567) 2024-09-20T19:30:21.330 INFO:tasks.rados.rados.0.smithi042.stdout:1615: done (1 left) 2024-09-20T19:30:21.330 INFO:tasks.rados.rados.0.smithi042.stdout:1617: done (0 left) 2024-09-20T19:30:21.330 INFO:tasks.rados.rados.0.smithi042.stdout:1618: delete oid 12 current snap is 181 2024-09-20T19:30:21.334 INFO:tasks.rados.rados.0.smithi042.stdout:1618: done (0 left) 2024-09-20T19:30:21.334 INFO:tasks.rados.rados.0.smithi042.stdout:1619: delete oid 38 current snap is 181 2024-09-20T19:30:21.337 INFO:tasks.rados.rados.0.smithi042.stdout:1619: done (0 left) 2024-09-20T19:30:21.337 INFO:tasks.rados.rados.0.smithi042.stdout:1620: snap_create 2024-09-20T19:30:21.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:21.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:21 smithi162 bash[24557]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:21 smithi162 bash[24557]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:21 smithi162 bash[24557]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:21 smithi162 bash[24557]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[32075]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[32075]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[32075]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[32075]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[22706]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[22706]: cluster 2024-09-20T19:30:20.246856+0000 mgr.y (mgr.24491) 469 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 48 active+clean; 397 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 770 KiB/s rd, 2.5 MiB/s wr, 10 op/s; 283/1182 objects degraded (23.942%); 12/1182 objects misplaced (1.015%) 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[22706]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:21 smithi042 bash[22706]: cluster 2024-09-20T19:30:20.295955+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e512: 8 total, 6 up, 8 in 2024-09-20T19:30:21.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:21.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:21.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.688 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:21.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:21.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:30:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:21.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:21.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:21.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:22.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:22.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:22.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.198 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:22.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:22.316 INFO:tasks.rados.rados.0.smithi042.stdout:1620: done (0 left) 2024-09-20T19:30:22.316 INFO:tasks.rados.rados.0.smithi042.stdout:1621: rollback oid 46 current snap is 182 2024-09-20T19:30:22.316 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 174 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1622: read oid 32 snap 154 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1622: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1623: rollback oid 12 current snap is 182 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 180 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1624: read oid 50 snap -1 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1624: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:30:22.317 INFO:tasks.rados.rados.0.smithi042.stdout:1625: delete oid 18 current snap is 182 2024-09-20T19:30:22.321 INFO:tasks.rados.rados.0.smithi042.stdout:1623: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:30:22.321 INFO:tasks.rados.rados.0.smithi042.stdout:1623: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:30:22.321 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 442 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:30:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:1622: expect (ObjNum 360 snap 116 seq_num 360) 2024-09-20T19:30:22.322 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:22.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:1621: finishing rollback tid 0 to smithi04268048-46 2024-09-20T19:30:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:1621: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:30:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:1621: finishing rollback tid 2 to smithi04268048-46 2024-09-20T19:30:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 973 (ObjNum 531 snap 173 seq_num 531) dirty exists 2024-09-20T19:30:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:1624: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:30:22.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:22.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:22 smithi042 bash[32075]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.448 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:22 smithi042 bash[32075]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:22 smithi042 bash[22706]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:22 smithi042 bash[22706]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:22.488 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:30:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:1621: done (4 left) 2024-09-20T19:30:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:1622: done (3 left) 2024-09-20T19:30:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:1623: done (2 left) 2024-09-20T19:30:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:1624: done (1 left) 2024-09-20T19:30:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:1625: done (0 left) 2024-09-20T19:30:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:1626: read oid 20 snap 174 2024-09-20T19:30:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:1626: expect deleted 2024-09-20T19:30:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:1627: snap_remove snap 178 2024-09-20T19:30:22.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:22.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:22.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:22 smithi162 bash[24557]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:22 smithi162 bash[24557]: cluster 2024-09-20T19:30:21.303974+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e513: 8 total, 6 up, 8 in 2024-09-20T19:30:22.815 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:30:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:30:22.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:22.833 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:30:22.833 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:30:22.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:22.849 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:30:22.850 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:30:22.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:30:22.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-20T19:30:22.868 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:30:22.868 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:30:22.886 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:30:22.886 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:30:22.904 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:30:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:30:22.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:22.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:22.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:23.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:23.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.222 INFO:tasks.rados.rados.0.smithi042.stdout:1627: done (1 left) 2024-09-20T19:30:23.222 INFO:tasks.rados.rados.0.smithi042.stdout:1628: write oid 12 current snap is 182 2024-09-20T19:30:23.223 INFO:tasks.rados.rados.0.smithi042.stdout:1628: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-20T19:30:23.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:23.240 INFO:tasks.rados.rados.0.smithi042.stdout:1628: writing smithi04268048-12 from 413604 to 982507 tid 1 2024-09-20T19:30:23.259 INFO:tasks.rados.rados.0.smithi042.stdout:1628: writing smithi04268048-12 from 1488759 to 2145621 tid 2 2024-09-20T19:30:23.275 INFO:tasks.rados.rados.0.smithi042.stdout:1628: writing smithi04268048-12 from 2624893 to 3216602 tid 3 2024-09-20T19:30:23.275 INFO:tasks.rados.rados.0.smithi042.stdout:1629: snap_remove snap 161 2024-09-20T19:30:23.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:23.366 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:23.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:23.496 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:23.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[22706]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:23 smithi042 bash[32075]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:23.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:22.248490+0000 mgr.y (mgr.24491) 470 : cluster [DBG] pgmap v768: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 379 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 285/1161 objects degraded (24.548%); 10/1161 objects misplaced (0.861%) 2024-09-20T19:30:23.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:22.313944+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e514: 8 total, 6 up, 8 in 2024-09-20T19:30:23.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: audit 2024-09-20T19:30:22.797397+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? v1:172.21.15.42:0/1337070952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:23.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:23.207948+0000 mon.a (mon.0) 1832 : cluster [WRN] Health check update: Degraded data redundancy: 285/1161 objects degraded (24.548%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:23.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:23 smithi162 bash[24557]: cluster 2024-09-20T19:30:23.219001+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e515: 8 total, 6 up, 8 in 2024-09-20T19:30:23.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:30:23.761 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:30:23.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:23.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:23.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:23.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:24.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:24.123 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:24.151 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:24.219 INFO:tasks.rados.rados.0.smithi042.stdout:1629: done (2 left) 2024-09-20T19:30:24.219 INFO:tasks.rados.rados.0.smithi042.stdout:1630: read oid 47 snap -1 2024-09-20T19:30:24.219 INFO:tasks.rados.rados.0.smithi042.stdout:1630: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:24.219 INFO:tasks.rados.rados.0.smithi042.stdout:1631: snap_remove snap 180 2024-09-20T19:30:24.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:24.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:24.312 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:24.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:30:24.495 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-20T19:30:24.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:24.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:24.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:24.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:24.796 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:24.899 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:24.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:25.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:25.059 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:25.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:25.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:25.232 INFO:tasks.rados.rados.0.smithi042.stdout:1631: done (3 left) 2024-09-20T19:30:25.232 INFO:tasks.rados.rados.0.smithi042.stdout:1632: delete oid 15 current snap is 182 2024-09-20T19:30:25.232 INFO:tasks.rados.rados.0.smithi042.stdout:1628: finishing write tid 1 to smithi04268048-12 2024-09-20T19:30:25.232 INFO:tasks.rados.rados.0.smithi042.stdout:1628: finishing write tid 2 to smithi04268048-12 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: finishing write tid 3 to smithi04268048-12 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: finishing write tid 4 to smithi04268048-12 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: finishing write tid 5 to smithi04268048-12 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: oid 12 updating version 0 to 443 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: oid 12 updating version 443 to 444 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: oid 12 updating version 444 to 445 2024-09-20T19:30:25.233 INFO:tasks.rados.rados.0.smithi042.stdout:1628: oid 12 updating version 445 to 446 2024-09-20T19:30:25.234 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 446 (ObjNum 572 snap 182 seq_num 572) dirty exists 2024-09-20T19:30:25.234 INFO:tasks.rados.rados.0.smithi042.stdout:1628: left oid 12 (ObjNum 572 snap 182 seq_num 572) 2024-09-20T19:30:25.234 INFO:tasks.rados.rados.0.smithi042.stdout:1630: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:25.264 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:25.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:25.298 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 773382 2024-09-20T19:30:25.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:25.318 INFO:tasks.rados.rados.0.smithi042.stdout:1626: done (3 left) 2024-09-20T19:30:25.319 INFO:tasks.rados.rados.0.smithi042.stdout:1628: done (2 left) 2024-09-20T19:30:25.319 INFO:tasks.rados.rados.0.smithi042.stdout:1630: done (1 left) 2024-09-20T19:30:25.319 INFO:tasks.rados.rados.0.smithi042.stdout:1632: done (0 left) 2024-09-20T19:30:25.319 INFO:tasks.rados.rados.0.smithi042.stdout:1633: snap_create 2024-09-20T19:30:25.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:25 smithi162 bash[24557]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:25 smithi162 bash[24557]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:25 smithi162 bash[24557]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:25 smithi162 bash[24557]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:25.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:25.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[32075]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[32075]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[32075]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[32075]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[22706]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[22706]: cluster 2024-09-20T19:30:24.217848+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e516: 8 total, 6 up, 8 in 2024-09-20T19:30:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[22706]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:25 smithi042 bash[22706]: cluster 2024-09-20T19:30:24.250370+0000 mgr.y (mgr.24491) 471 : cluster [DBG] pgmap v772: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:25.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:25.762 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:30:25.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:25.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:25.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:25.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:26.051 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:25 smithi162 bash[77847]: debug 2024-09-20T19:30:25.683+0000 7feaddca8640 -1 Falling back to public interface 2024-09-20T19:30:26.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:26.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:26.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:26.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:26.240 INFO:tasks.rados.rados.0.smithi042.stdout:1633: done (0 left) 2024-09-20T19:30:26.241 INFO:tasks.rados.rados.0.smithi042.stdout:1634: snap_remove snap 175 2024-09-20T19:30:26.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:26.332 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:26.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:26 smithi042 bash[32075]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.332 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:26 smithi042 bash[32075]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:26 smithi042 bash[22706]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:26 smithi042 bash[22706]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:26.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:26.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:26 smithi162 bash[24557]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:26 smithi162 bash[24557]: cluster 2024-09-20T19:30:25.226414+0000 mon.a (mon.0) 1835 : cluster [DBG] osdmap e517: 8 total, 6 up, 8 in 2024-09-20T19:30:26.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:26.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:26.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:26.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:26.734 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:26.837 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:26.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:26.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:26.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:26.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:27.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.126 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:27.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:30:27.243 INFO:tasks.rados.rados.0.smithi042.stdout:1634: done (0 left) 2024-09-20T19:30:27.243 INFO:tasks.rados.rados.0.smithi042.stdout:1635: snap_create 2024-09-20T19:30:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:27.360 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[22706]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.370 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.371 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.371 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.371 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.371 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:27 smithi042 bash[32075]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:27.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:27.506 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:26.238648+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e518: 8 total, 6 up, 8 in 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:26.251251+0000 mgr.y (mgr.24491) 472 : cluster [DBG] pgmap v775: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 down, 7 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 47 active+clean; 384 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 0 B/s wr, 3 op/s; 286/1170 objects degraded (24.444%); 10/1170 objects misplaced (0.855%) 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: audit 2024-09-20T19:30:26.462551+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:27.235294+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:27.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: audit 2024-09-20T19:30:27.237439+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.42:0/266038691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:30:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:27 smithi162 bash[24557]: cluster 2024-09-20T19:30:27.242797+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e519: 8 total, 6 up, 8 in 2024-09-20T19:30:27.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:27.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:27.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:27.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:27.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:27.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:27.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:27.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:28.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:28.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:28.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:28.267 INFO:tasks.rados.rados.0.smithi042.stdout:1635: done (0 left) 2024-09-20T19:30:28.267 INFO:tasks.rados.rados.0.smithi042.stdout:1636: write oid 7 current snap is 184 2024-09-20T19:30:28.267 INFO:tasks.rados.rados.0.smithi042.stdout:1636: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-20T19:30:28.274 INFO:tasks.rados.rados.0.smithi042.stdout:1636: writing smithi04268048-7 from 447218 to 1166619 tid 1 2024-09-20T19:30:28.280 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:28.283 INFO:tasks.rados.rados.0.smithi042.stdout:1636: writing smithi04268048-7 from 1889370 to 2549901 tid 2 2024-09-20T19:30:28.288 INFO:tasks.rados.rados.0.smithi042.stdout:1636: writing smithi04268048-7 from 2963386 to 3577639 tid 3 2024-09-20T19:30:28.288 INFO:tasks.rados.rados.0.smithi042.stdout:1637: read oid 26 snap -1 2024-09-20T19:30:28.288 INFO:tasks.rados.rados.0.smithi042.stdout:1637: expect deleted 2024-09-20T19:30:28.288 INFO:tasks.rados.rados.0.smithi042.stdout:1638: delete oid 10 current snap is 184 2024-09-20T19:30:28.288 INFO:tasks.rados.rados.0.smithi042.stdout:1636: finishing write tid 1 to smithi04268048-7 2024-09-20T19:30:28.289 INFO:tasks.rados.rados.0.smithi042.stdout:1636: finishing write tid 2 to smithi04268048-7 2024-09-20T19:30:28.292 INFO:tasks.rados.rados.0.smithi042.stdout:1636: finishing write tid 3 to smithi04268048-7 2024-09-20T19:30:28.292 INFO:tasks.rados.rados.0.smithi042.stdout:1636: finishing write tid 4 to smithi04268048-7 2024-09-20T19:30:28.292 INFO:tasks.rados.rados.0.smithi042.stdout:1636: finishing write tid 5 to smithi04268048-7 2024-09-20T19:30:28.292 INFO:tasks.rados.rados.0.smithi042.stdout:1636: oid 7 updating version 0 to 625 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1636: oid 7 updating version 625 to 626 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1636: oid 7 version 626 is already newer than 623 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1636: oid 7 version 626 is already newer than 624 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 626 (ObjNum 574 snap 184 seq_num 574) dirty exists 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1636: left oid 7 (ObjNum 574 snap 184 seq_num 574) 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1636: done (2 left) 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1637: done (1 left) 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1638: done (0 left) 2024-09-20T19:30:28.293 INFO:tasks.rados.rados.0.smithi042.stdout:1639: snap_remove snap 183 2024-09-20T19:30:28.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.410 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:28.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:28.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:28 smithi162 bash[24557]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:28 smithi162 bash[24557]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:28 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:28 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:28.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[32075]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[32075]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[22706]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[22706]: audit 2024-09-20T19:30:27.727406+0000 mon.a (mon.0) 1841 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:28.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:28 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.211032+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 286/1170 objects degraded (24.444%), 48 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:28.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:28.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:28.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:28.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:28.802 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:30:28.802 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:30:28.802 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:30:28.802 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:30:28.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:242 7.b 7:d579898a:::smithi04268048-35:head [copy-from ver 22] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 517.97968745399999, 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016001677999999998, 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 242 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.803 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821291+0000", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821293+0000", 2024-09-20T19:30:28.804 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0899999999999999e-06 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821299+0000", 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5559999999999998e-06 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821310+0000", 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1832e-05 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.805 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821318+0000", 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0879999999999995e-06 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821402+0000", 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3906000000000006e-05 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.821501+0000", 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8999999999999994e-05 2024-09-20T19:30:28.806 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.826058+0000", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045566590000000002 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.832133+0000", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060751090000000004 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.836775+0000", 2024-09-20T19:30:28.807 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0046420899999999998 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837179+0000", 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00040368899999999999 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837240+0000", 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0934999999999998e-05 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.808 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:21:50.837293+0000", 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.2723999999999999e-05 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2132 7.d e297/112 7:baa4e37b:::smithi04268048-17:head v 297'157, mlcod=296'153)", 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 342.54819025199998, 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017849924999999999, 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.809 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.252788+0000", 2024-09-20T19:30:28.810 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8000000000000002e-07 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259424+0000", 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0066358739999999999 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259460+0000", 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.5748999999999999e-05 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.811 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259482+0000", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1800000000000001e-05 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259597+0000", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000114894 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.259636+0000", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9044000000000001e-05 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.812 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.270493+0000", 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010857148 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:24:46.270638+0000", 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000145136 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.4.0:290 7.1 e334/333 7:892846d2:::smithi04268048-15:71 v 334'295, mlcod=329'291)", 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 305.69145967200001, 2024-09-20T19:30:28.813 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015268339000000001, 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109519+0000", 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109520+0000", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.824e-06 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109538+0000", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7889000000000001e-05 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109559+0000", 2024-09-20T19:30:28.815 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0863e-05 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109583+0000", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3574000000000002e-05 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109844+0000", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026147399999999998 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.109908+0000", 2024-09-20T19:30:28.816 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4127000000000005e-05 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124676+0000", 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014767656000000001 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124787+0000", 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000110932 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "pg_update_log_missing(7.d epoch 334/112 rep_tid 522 entries 334'201 (0'0) error 7:b90bdf9e:::smithi04268048-23:head by client.14904.0:2586 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 327'198)", 2024-09-20T19:30:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 305.69070408099998, 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014371523000000001, 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.818 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110274+0000", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110275+0000", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7999999999999996e-07 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110284+0000", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.816e-06 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.819 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110305+0000", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0985e-05 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.110320+0000", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5965e-05 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.120126+0000", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0098058480000000007 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.820 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124646+0000", 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0045194290000000002 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2589 7.1 e334/333 7:892846d2:::smithi04268048-15:head v 334'297, mlcod=329'291)", 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 305.67622465699998, 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013064749000000001, 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.821 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124754+0000", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.124757+0000", 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.157e-06 2024-09-20T19:30:28.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133124+0000", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0083669580000000007 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133149+0000", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4864000000000001e-05 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133161+0000", 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2856e-05 2024-09-20T19:30:28.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133235+0000", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3444999999999999e-05 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.133259+0000", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3730000000000001e-05 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.137670+0000", 2024-09-20T19:30:28.824 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044115889999999996 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:23.137818+0000", 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014815 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2604 7.0 e335/112 7:060c6a05:::smithi04268048-38:head v 335'313, mlcod=334'309)", 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 304.06347476399998, 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013697485000000001, 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.825 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.826 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.737504+0000", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0600000000000001e-07 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745768+0000", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0082640609999999996 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745786+0000", 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7507e-05 2024-09-20T19:30:28.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745793+0000", 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3640000000000001e-06 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745860+0000", 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7523999999999999e-05 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.828 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.745880+0000", 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9174999999999999e-05 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.751155+0000", 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0052757890000000003 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:24.751201+0000", 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5658999999999998e-05 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.829 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:2633 7.0 e337/112 7:03a4d37b:::smithi04268048-10:head v 337'317, mlcod=335'313)", 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 302.50293787499999, 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013586387, 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.830 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298040+0000", 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.298041+0000", 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8800000000000002e-07 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.831 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305314+0000", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0072731929999999998 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305347+0000", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3139000000000002e-05 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305362+0000", 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4559000000000001e-05 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305477+0000", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011479799999999999 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.305508+0000", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1517000000000003e-05 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.311557+0000", 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060486180000000004 2024-09-20T19:30:28.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:26.311627+0000", 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9875000000000004e-05 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2898 7.b 7:d579898a:::smithi04268048-35:head [append 0~0] snapc 81=[81,80,7f,7e,7d,7c,79,77,76,72,6d,6c,69,67,5e,5a,41] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 282.63907100400002, 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013430002, 2024-09-20T19:30:28.834 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2898 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161907+0000", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161909+0000", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.192e-06 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161913+0000", 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7770000000000002e-06 2024-09-20T19:30:28.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161929+0000", 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5282999999999999e-05 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.161940+0000", 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1268e-05 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162265+0000", 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000324691 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.162284+0000", 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9853000000000001e-05 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.170752+0000", 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0084678630000000008 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.171031+0000", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027857499999999998 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.171512+0000", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048082700000000002 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172465+0000", 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00095308600000000002 2024-09-20T19:30:28.839 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.172548+0000", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3375999999999996e-05 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175140+0000", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0025914649999999998 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175247+0000", 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107666 2024-09-20T19:30:28.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:46.175337+0000", 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0080000000000005e-05 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2992 7.b 7:d7080ac6:::smithi04268048-34:head [copy-from ver 487] snapc 83=[83,82,81,7f,7d,7c,79,72,6c,67,5e,41] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:55.113006+0000", 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 273.68797206199997, 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037505861000000001, 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.841 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2992 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113006+0000", 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.842 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113006+0000", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113010+0000", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9890000000000003e-06 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113023+0000", 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2755000000000001e-05 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113046+0000", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3496e-05 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113062+0000", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5173e-05 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113167+0000", 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010532800000000001 2024-09-20T19:30:28.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113177+0000", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7939999999999997e-06 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.116457+0000", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0032798469999999998 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.116566+0000", 2024-09-20T19:30:28.845 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010926500000000001 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.139645+0000", 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023079375999999999 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.144713+0000", 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0050679649999999998 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.846 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.148089+0000", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.003376029 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.150343+0000", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0022536740000000002 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.150436+0000", 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3171000000000006e-05 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.150512+0000", 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5999000000000002e-05 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2993 7.b 7:d7080ac6:::smithi04268048-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e366)", 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:55.113096+0000", 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 273.687882527, 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030597961999999999, 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.848 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2993 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113096+0000", 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113096+0000", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113097+0000", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4200000000000004e-07 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113101+0000", 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3470000000000001e-06 2024-09-20T19:30:28.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113116+0000", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5378000000000001e-05 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113124+0000", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3710000000000002e-06 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113248+0000", 2024-09-20T19:30:28.851 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012449999999999999 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.113257+0000", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9730000000000003e-06 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.116741+0000", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0034839430000000002 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.116799+0000", 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7343000000000003e-05 2024-09-20T19:30:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.143391+0000", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.026592405 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.143533+0000", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014246899999999999 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:55.143694+0000", 2024-09-20T19:30:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016029099999999999 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3192 7.b 7:d7080ac6:::smithi04268048-34:head [copy-from ver 313] snapc 92=[92,91,90,8f,8e,8d,8c,8b,8a,89,88,87,86,85,84,83,82,81,7f,7c,79,72,41] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.328609+0000", 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 254.47236981500001, 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020318190999999999, 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.854 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3192 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328609+0000", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328609+0000", 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328614+0000", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3299999999999998e-06 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328633+0000", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9096000000000001e-05 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328678+0000", 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5327000000000002e-05 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328703+0000", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.476e-05 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328879+0000", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017631799999999999 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.329062+0000", 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018257899999999999 2024-09-20T19:30:28.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.336838+0000", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0077760779999999996 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.346846+0000", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010008285 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.347701+0000", 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00085485400000000005 2024-09-20T19:30:28.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.348683+0000", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00098205300000000009 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.348787+0000", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000103762 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.348927+0000", 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013974899999999999 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.859 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3193 7.b 7:d7080ac6:::smithi04268048-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.328749+0000", 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 254.472229446, 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017982755, 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3193 2024-09-20T19:30:28.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328749+0000", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328749+0000", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328751+0000", 2024-09-20T19:30:28.861 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2800000000000002e-06 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328761+0000", 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.6730000000000004e-06 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328803+0000", 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.265e-05 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.862 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.328824+0000", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0869999999999998e-05 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.329370+0000", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00054556899999999998 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.329537+0000", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016721600000000001 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.863 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.346267+0000", 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01673034 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.346516+0000", 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024853499999999998 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.346732+0000", 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000215622 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.6.0:688 7.d e388/112 7:b90bdf9e:::smithi04268048-23:91 v 388'268, mlcod=384'262)", 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:17.337264+0000", 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 251.46371390799999, 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017241576000000002, 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337264+0000", 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.865 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337264+0000", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337266+0000", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.356e-06 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337274+0000", 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6889999999999993e-06 2024-09-20T19:30:28.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337289+0000", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4790000000000001e-05 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337298+0000", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0480000000000004e-06 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337421+0000", 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012250499999999999 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.337483+0000", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.2316000000000004e-05 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.354165+0000", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016681714 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.354506+0000", 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00034115799999999999 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.868 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:3232 7.4 e388/112 7:2b212c05:::smithi04268048-30:head v 388'686, mlcod=385'683)", 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:17.338916+0000", 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 251.46206242700001, 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015468658999999999, 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338916+0000", 2024-09-20T19:30:28.869 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338916+0000", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338919+0000", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4759999999999998e-06 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338937+0000", 2024-09-20T19:30:28.870 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7757e-05 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338970+0000", 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2453000000000003e-05 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.338990+0000", 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0100000000000001e-05 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.871 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.339093+0000", 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010370300000000001 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.339118+0000", 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4236999999999999e-05 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.354274+0000", 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01515682 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.872 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:17.354385+0000", 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011011299999999999 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3906 7.14 7:2924b69c:::smithi04268048-21:ab [sparse-read 0~3064801,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=5 ondisk+retry+read+known_if_redirected+supports_pool_eio e476)", 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:28:45.597537+0000", 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 103.20344161600001, 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.047163541, 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.873 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3906 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597537+0000", 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.874 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597537+0000", 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597540+0000", 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8619999999999999e-06 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597550+0000", 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0202e-05 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597563+0000", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3662000000000001e-05 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597575+0000", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1765e-05 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597654+0000", 2024-09-20T19:30:28.876 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8943999999999995e-05 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597667+0000", 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3307e-05 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.641657+0000", 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.043989085 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.877 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.641938+0000", 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028192499999999998 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.644700+0000", 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0027617890000000002 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3907 7.14 7:2924b69c:::smithi04268048-21:ab [read 0~3428694,checksum] snapc 0=[] RETRY=5 ondisk+retry+read+known_if_redirected+supports_pool_eio e476)", 2024-09-20T19:30:28.878 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:28:45.597600+0000", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 103.20337847899999, 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.048365284, 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3907 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597600+0000", 2024-09-20T19:30:28.879 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597600+0000", 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597601+0000", 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.048e-06 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.880 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597608+0000", 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5739999999999998e-06 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597616+0000", 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0069999999999997e-06 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597623+0000", 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9670000000000004e-06 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597725+0000", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000101427 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597738+0000", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3246e-05 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.644769+0000", 2024-09-20T19:30:28.882 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.047031018 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.644970+0000", 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000200779 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.645965+0000", 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00099521799999999997 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.883 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3908 7.14 7:2924b69c:::smithi04268048-21:ab [sparse-read 0~3064801] snapc 0=[] RETRY=5 ondisk+retry+read+known_if_redirected+supports_pool_eio e476)", 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:28:45.597633+0000", 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 103.203345394, 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0491799390000001, 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3908 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.884 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597633+0000", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597633+0000", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597634+0000", 2024-09-20T19:30:28.885 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2999999999999999e-07 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597637+0000", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4879999999999999e-06 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597645+0000", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3069999999999997e-06 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597651+0000", 2024-09-20T19:30:28.886 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.28e-06 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597851+0000", 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019995000000000001 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:45.597860+0000", 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4749999999999999e-06 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.887 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.646217+0000", 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.04835685 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.646412+0000", 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019486199999999999 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:46.646813+0000", 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00040089699999999999 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.888 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2378 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94817798638848 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:28:48.765131+0000", 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 100.035847527, 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.45447553800000001, 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:30:28.889 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2378 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765131+0000", 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765131+0000", 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.890 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765137+0000", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9100000000000002e-06 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765153+0000", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6656000000000001e-05 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765184+0000", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0397999999999999e-05 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765234+0000", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9867999999999998e-05 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765477+0000", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000243506 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765488+0000", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1235e-05 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:49.217787+0000", 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.452299065 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:49.218231+0000", 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00044352999999999998 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:49.219606+0000", 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013753700000000001 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.893 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2379 5.0 5.16dafda0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:28:48.765274+0000", 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 100.03570390100001, 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.44085612400000002, 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:30:28.894 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2379 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765274+0000", 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765274+0000", 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765276+0000", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.435e-06 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765282+0000", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.2570000000000004e-06 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765293+0000", 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1096e-05 2024-09-20T19:30:28.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765303+0000", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5519999999999993e-06 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765437+0000", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013423100000000001 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:48.765454+0000", 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7399999999999999e-05 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:49.206059+0000", 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.44060453799999999 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:28:49.206131+0000", 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.1614999999999998e-05 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.898 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.2.0:719 7.14 e517/480 7:2a17ba7b:::smithi04268048-18:b7 v 517'802, mlcod=515'798)", 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:25.230541+0000", 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 3.5704372549999999, 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016901099999999999, 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230541+0000", 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230541+0000", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230543+0000", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.03e-06 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230559+0000", 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5617e-05 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230584+0000", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5380000000000001e-05 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.230607+0000", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2883000000000001e-05 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.246256+0000", 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015649294000000001 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.246288+0000", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2119000000000002e-05 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.247307+0000", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010190780000000001 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:25.247442+0000", 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013469900000000001 2024-09-20T19:30:28.902 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:28.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:28.903 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:30:28.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:29.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:29.035 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[77847]: debug 2024-09-20T19:30:29.007+0000 7feaddca8640 -1 osd.4 443 log_to_monitors true 2024-09-20T19:30:29.046 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:29.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:29.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:29.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.294 INFO:tasks.rados.rados.0.smithi042.stdout:1639: done (0 left) 2024-09-20T19:30:29.294 INFO:tasks.rados.rados.0.smithi042.stdout:1640: write (excl) oid 32 current snap is 184 2024-09-20T19:30:29.295 INFO:tasks.rados.rados.0.smithi042.stdout:1640: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-20T19:30:29.300 INFO:tasks.rados.rados.0.smithi042.stdout:1640: writing smithi04268048-32 from 480832 to 950731 tid 1 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[24557]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.310 INFO:tasks.rados.rados.0.smithi042.stdout:1640: writing smithi04268048-32 from 1606334 to 2270534 tid 2 2024-09-20T19:30:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:29.318 INFO:tasks.rados.rados.0.smithi042.stdout:1640: writing smithi04268048-32 from 3018232 to 3680832 tid 3 2024-09-20T19:30:29.318 INFO:tasks.rados.rados.0.smithi042.stdout:1641: delete oid 11 current snap is 184 2024-09-20T19:30:29.322 INFO:tasks.rados.rados.0.smithi042.stdout:1640: finishing write tid 1 to smithi04268048-32 2024-09-20T19:30:29.322 INFO:tasks.rados.rados.0.smithi042.stdout:1640: finishing write tid 2 to smithi04268048-32 2024-09-20T19:30:29.324 INFO:tasks.rados.rados.0.smithi042.stdout:1641: done (1 left) 2024-09-20T19:30:29.324 INFO:tasks.rados.rados.0.smithi042.stdout:1642: snap_create 2024-09-20T19:30:29.327 INFO:tasks.rados.rados.0.smithi042.stdout:1640: finishing write tid 3 to smithi04268048-32 2024-09-20T19:30:29.327 INFO:tasks.rados.rados.0.smithi042.stdout:1640: finishing write tid 4 to smithi04268048-32 2024-09-20T19:30:29.327 INFO:tasks.rados.rados.0.smithi042.stdout:1640: finishing write tid 5 to smithi04268048-32 2024-09-20T19:30:29.327 INFO:tasks.rados.rados.0.smithi042.stdout:1640: oid 32 updating version 0 to 942 2024-09-20T19:30:29.328 INFO:tasks.rados.rados.0.smithi042.stdout:1640: oid 32 updating version 942 to 944 2024-09-20T19:30:29.328 INFO:tasks.rados.rados.0.smithi042.stdout:1640: oid 32 version 944 is already newer than 943 2024-09-20T19:30:29.328 INFO:tasks.rados.rados.0.smithi042.stdout:1640: oid 32 version 944 is already newer than 941 2024-09-20T19:30:29.328 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 944 (ObjNum 576 snap 184 seq_num 576) dirty exists 2024-09-20T19:30:29.328 INFO:tasks.rados.rados.0.smithi042.stdout:1640: left oid 32 (ObjNum 576 snap 184 seq_num 576) 2024-09-20T19:30:29.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.441 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:29.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:30:29.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:29.551 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:30:29 smithi162 bash[77847]: debug 2024-09-20T19:30:29.427+0000 7fead1a4a640 -1 osd.4 521 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[32075]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.252699+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e520: 8 total, 6 up, 8 in 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: cluster 2024-09-20T19:30:28.253954+0000 mgr.y (mgr.24491) 473 : cluster [DBG] pgmap v778: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 444 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: audit 2024-09-20T19:30:29.010857+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:29 smithi042 bash[22706]: audit 2024-09-20T19:30:29.016088+0000 mon.b (mon.1) 31 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T19:30:29.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:29.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:29.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:29.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:30:29.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:29.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:29.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:29.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:29.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:30.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:30.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:30.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:30.313 INFO:tasks.rados.rados.0.smithi042.stdout:1640: done (1 left) 2024-09-20T19:30:30.313 INFO:tasks.rados.rados.0.smithi042.stdout:1642: done (0 left) 2024-09-20T19:30:30.313 INFO:tasks.rados.rados.0.smithi042.stdout:1643: delete oid 33 current snap is 185 2024-09-20T19:30:30.323 INFO:tasks.rados.rados.0.smithi042.stdout:1643: done (0 left) 2024-09-20T19:30:30.323 INFO:tasks.rados.rados.0.smithi042.stdout:1644: write (excl) oid 38 current snap is 185 2024-09-20T19:30:30.323 INFO:tasks.rados.rados.0.smithi042.stdout:1644: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-20T19:30:30.326 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:30.335 INFO:tasks.rados.rados.0.smithi042.stdout:1644: writing smithi04268048-38 from 531253 to 943252 tid 1 2024-09-20T19:30:30.346 INFO:tasks.rados.rados.0.smithi042.stdout:1644: writing smithi04268048-38 from 1381780 to 1793307 tid 2 2024-09-20T19:30:30.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:30.358 INFO:tasks.rados.rados.0.smithi042.stdout:1644: writing smithi04268048-38 from 2300501 to 2887467 tid 3 2024-09-20T19:30:30.366 INFO:tasks.rados.rados.0.smithi042.stdout:1644: writing smithi04268048-38 from 3376974 to 3731253 tid 4 2024-09-20T19:30:30.367 INFO:tasks.rados.rados.0.smithi042.stdout:1645: snap_remove snap 164 2024-09-20T19:30:30.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:30 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.279896+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:30:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:29.285701+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e521: 8 total, 6 up, 8 in 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.293791+0000 mon.b (mon.1) 32 : audit [INF] from='osd.4 v1:172.21.15.162:6800/449450686' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.295520+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: audit 2024-09-20T19:30:29.482464+0000 mgr.y (mgr.24491) 474 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.248597+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:30 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.252564+0000 osd.7 (osd.7) 146 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:30:30.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:30.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:30.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:30:30.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:30.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:30.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:30.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:30.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.002 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:31.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:31.132 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:31.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:31.258 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1645: done (1 left) 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1646: write oid 49 current snap is 185 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 1 to smithi04268048-38 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 2 to smithi04268048-38 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 3 to smithi04268048-38 2024-09-20T19:30:31.310 INFO:tasks.rados.rados.0.smithi042.stdout:1646: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-20T19:30:31.322 INFO:tasks.rados.rados.0.smithi042.stdout:1646: writing smithi04268048-49 from 548060 to 1035308 tid 1 2024-09-20T19:30:31.333 INFO:tasks.rados.rados.0.smithi042.stdout:1646: writing smithi04268048-49 from 1440262 to 2111800 tid 2 2024-09-20T19:30:31.348 INFO:tasks.rados.rados.0.smithi042.stdout:1646: writing smithi04268048-49 from 2844277 to 3642515 tid 3 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 4 to smithi04268048-38 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 5 to smithi04268048-38 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: finishing write tid 6 to smithi04268048-38 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: oid 38 updating version 0 to 594 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: oid 38 version 594 is already newer than 593 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: oid 38 updating version 594 to 597 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: oid 38 version 597 is already newer than 595 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: oid 38 version 597 is already newer than 596 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 597 (ObjNum 579 snap 185 seq_num 579) dirty exists 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: left oid 38 (ObjNum 579 snap 185 seq_num 579) 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1644: done (1 left) 2024-09-20T19:30:31.349 INFO:tasks.rados.rados.0.smithi042.stdout:1647: delete oid 11 current snap is 185 2024-09-20T19:30:31.358 INFO:tasks.rados.rados.0.smithi042.stdout:1647: done (1 left) 2024-09-20T19:30:31.359 INFO:tasks.rados.rados.0.smithi042.stdout:1648: read oid 1 snap -1 2024-09-20T19:30:31.359 INFO:tasks.rados.rados.0.smithi042.stdout:1648: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-20T19:30:31.359 INFO:tasks.rados.rados.0.smithi042.stdout:1649: copy_from oid 12 from oid 7 current snap is 185 2024-09-20T19:30:31.359 INFO:tasks.rados.rados.0.smithi042.stdout:1650: delete oid 9 current snap is 185 2024-09-20T19:30:31.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:1650: done (3 left) 2024-09-20T19:30:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:1651: read oid 23 snap 177 2024-09-20T19:30:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:1651: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-20T19:30:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:1652: snap_create 2024-09-20T19:30:31.371 INFO:tasks.rados.rados.0.smithi042.stdout:1648: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-20T19:30:31.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[22706]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:31 smithi042 bash[32075]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:31.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:31.565 INFO:tasks.rados.rados.0.smithi042.stdout:1651: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-20T19:30:31.582 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1942670 2024-09-20T19:30:31.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.255286+0000 mgr.y (mgr.24491) 475 : cluster [DBG] pgmap v780: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 1 active+clean+snaptrim_wait, 3 down, 5 active+clean+snaptrim, 8 undersized+peered, 52 active+undersized, 6 undersized+degraded+peered, 42 active+undersized+degraded, 49 active+clean; 373 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 442 KiB/s wr, 9 op/s; 286/1155 objects degraded (24.762%); 10/1155 objects misplaced (0.866%) 2024-09-20T19:30:31.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.281482+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.312109+0000 mon.a (mon.0) 1849 : cluster [INF] osd.4 v1:172.21.15.162:6800/449450686 boot 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.312203+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:31 smithi162 bash[24557]: audit 2024-09-20T19:30:30.312944+0000 mon.a (mon.0) 1851 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:30:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1798084 2024-09-20T19:30:31.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:31.708 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1244902 2024-09-20T19:30:31.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: finishing write tid 1 to smithi04268048-49 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: finishing write tid 2 to smithi04268048-49 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: finishing write tid 3 to smithi04268048-49 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: finishing write tid 4 to smithi04268048-49 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: finishing write tid 5 to smithi04268048-49 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: oid 49 updating version 0 to 947 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: oid 49 version 947 is already newer than 946 2024-09-20T19:30:31.768 INFO:tasks.rados.rados.0.smithi042.stdout:1646: oid 49 updating version 947 to 948 2024-09-20T19:30:31.769 INFO:tasks.rados.rados.0.smithi042.stdout:1646: oid 49 version 948 is already newer than 945 2024-09-20T19:30:31.769 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 948 (ObjNum 580 snap 185 seq_num 580) dirty exists 2024-09-20T19:30:31.769 INFO:tasks.rados.rados.0.smithi042.stdout:1646: left oid 49 (ObjNum 580 snap 185 seq_num 580) 2024-09-20T19:30:31.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:31.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:31.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:32.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:32.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:32.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:32.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:32.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:32.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1646: done (4 left) 2024-09-20T19:30:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1648: done (3 left) 2024-09-20T19:30:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1651: done (2 left) 2024-09-20T19:30:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1652: done (1 left) 2024-09-20T19:30:32.305 INFO:tasks.rados.rados.0.smithi042.stdout:1653: snap_create 2024-09-20T19:30:32.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.400 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.401 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:32 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:32.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:32.544 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:32.647 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.022783+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:30.024095+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:31.308855+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:32 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.302645+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-09-20T19:30:32.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:32.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:32.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:32.876 INFO:tasks.rados.rados.0.smithi042.stdout:1649: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:30:32.878 INFO:tasks.rados.rados.0.smithi042.stdout:1649: finishing copy_from to smithi04268048-12 2024-09-20T19:30:32.878 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 626 (ObjNum 574 snap 184 seq_num 574) dirty exists 2024-09-20T19:30:32.905 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:32.925 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:32.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:33.006 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:30:33.006 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:30:33.024 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:30:33.024 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:30:33.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:33.041 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:30:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:30:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:30:33.054 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:33.062 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:30:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-20T19:30:33.079 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:30:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:30:33.099 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:30:33.099 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:30:33.117 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:30:33.117 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:30:33.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:33.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:33.223 INFO:tasks.rados.rados.0.smithi042.stdout:1649: done (1 left) 2024-09-20T19:30:33.223 INFO:tasks.rados.rados.0.smithi042.stdout:1653: done (0 left) 2024-09-20T19:30:33.223 INFO:tasks.rados.rados.0.smithi042.stdout:1654: read oid 14 snap -1 2024-09-20T19:30:33.223 INFO:tasks.rados.rados.0.smithi042.stdout:1654: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:30:33.224 INFO:tasks.rados.rados.0.smithi042.stdout:1655: snap_create 2024-09-20T19:30:33.229 INFO:tasks.rados.rados.0.smithi042.stdout:1654: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:30:33.242 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2260695 2024-09-20T19:30:33.275 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3307328 2024-09-20T19:30:33.279 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:33.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:33.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:33.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:33 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:33.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.258003+0000 mgr.y (mgr.24491) 476 : cluster [DBG] pgmap v783: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+remapped, 3 down, 6 active+clean+snaptrim, 4 undersized+peered, 39 active+undersized, 3 undersized+degraded+peered, 38 active+undersized+degraded, 72 active+clean; 354 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 53 op/s; 238/1116 objects degraded (21.326%); 8/1116 objects misplaced (0.717%); 1.8 MiB/s, 0 objects/s recovering 2024-09-20T19:30:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:32.318465+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-20T19:30:33.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: audit 2024-09-20T19:30:32.987638+0000 mon.a (mon.0) 1855 : audit [DBG] from='client.? v1:172.21.15.42:0/4292704895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.213662+0000 mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 238/1116 objects degraded (21.326%), 41 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:33 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.219492+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-09-20T19:30:33.568 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:33.671 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:33.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:33.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:33.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:33.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:34.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.090 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:30:34.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:30:34.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1654: done (1 left) 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1655: done (0 left) 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1656: read oid 45 snap 177 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1656: expect (ObjNum 514 snap 166 seq_num 514) 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1657: read oid 38 snap -1 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1657: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-20T19:30:34.222 INFO:tasks.rados.rados.0.smithi042.stdout:1658: snap_remove snap 169 2024-09-20T19:30:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:34.291 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:30:34.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:34.348 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:34.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:34.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:34.579 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:34 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.597 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:34.610 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:34.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.688861+0000 osd.0 (osd.0) 3 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.692743+0000 osd.0 (osd.0) 4 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:34 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.221832+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-09-20T19:30:34.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:34.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:34.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:34.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:34.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:35.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:35.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:35.233 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.237 INFO:tasks.rados.rados.0.smithi042.stdout:1658: done (2 left) 2024-09-20T19:30:35.237 INFO:tasks.rados.rados.0.smithi042.stdout:1659: rollback oid 48 current snap is 188 2024-09-20T19:30:35.238 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 185 2024-09-20T19:30:35.238 INFO:tasks.rados.rados.0.smithi042.stdout:1660: read oid 6 snap 176 2024-09-20T19:30:35.238 INFO:tasks.rados.rados.0.smithi042.stdout:1660: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:30:35.238 INFO:tasks.rados.rados.0.smithi042.stdout:1661: write (excl) oid 13 current snap is 188 2024-09-20T19:30:35.238 INFO:tasks.rados.rados.0.smithi042.stdout:1661: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-20T19:30:35.242 INFO:tasks.rados.rados.0.smithi042.stdout:1661: writing smithi04268048-13 from 598481 to 1311476 tid 1 2024-09-20T19:30:35.249 INFO:tasks.rados.rados.0.smithi042.stdout:1661: writing smithi04268048-13 from 1899355 to 2318220 tid 2 2024-09-20T19:30:35.259 INFO:tasks.rados.rados.0.smithi042.stdout:1661: writing smithi04268048-13 from 3093840 to 3798481 tid 3 2024-09-20T19:30:35.259 INFO:tasks.rados.rados.0.smithi042.stdout:1662: write (excl) oid 30 current snap is 188 2024-09-20T19:30:35.259 INFO:tasks.rados.rados.0.smithi042.stdout:1662: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2024-09-20T19:30:35.267 INFO:tasks.rados.rados.0.smithi042.stdout:1662: writing smithi04268048-30 from 615288 to 1403532 tid 1 2024-09-20T19:30:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:35.274 INFO:tasks.rados.rados.0.smithi042.stdout:1662: writing smithi04268048-30 from 1957837 to 2520360 tid 2 2024-09-20T19:30:35.280 INFO:tasks.rados.rados.0.smithi042.stdout:1662: writing smithi04268048-30 from 3121263 to 3680942 tid 3 2024-09-20T19:30:35.281 INFO:tasks.rados.rados.0.smithi042.stdout:1663: write (excl) oid 1 current snap is 188 2024-09-20T19:30:35.281 INFO:tasks.rados.rados.0.smithi042.stdout:1656: expect (ObjNum 514 snap 166 seq_num 514) 2024-09-20T19:30:35.309 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:35.367 INFO:tasks.rados.rados.0.smithi042.stdout:1657: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-20T19:30:35.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:35.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:35.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[32075]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:35 smithi042 bash[22706]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1659: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: finishing write tid 1 to smithi04268048-13 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1659: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1659: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 745 (ObjNum 535 snap 173 seq_num 535) dirty exists 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: finishing write tid 2 to smithi04268048-13 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: finishing write tid 3 to smithi04268048-13 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: finishing write tid 4 to smithi04268048-13 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: finishing write tid 5 to smithi04268048-13 2024-09-20T19:30:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:1661: oid 13 updating version 0 to 563 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:1661: oid 13 updating version 563 to 564 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:1661: oid 13 version 564 is already newer than 562 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:1661: oid 13 updating version 564 to 565 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 565 (ObjNum 583 snap 188 seq_num 583) dirty exists 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:1661: left oid 13 (ObjNum 583 snap 188 seq_num 583) 2024-09-20T19:30:35.469 INFO:tasks.rados.rados.0.smithi042.stdout:1660: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:30:35.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:35.529 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:35.589 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:35.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:35.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.875951+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:33.876438+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.033921+0000 osd.4 (osd.4) 3 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.035771+0000 osd.4 (osd.4) 4 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.258991+0000 mgr.y (mgr.24491) 477 : cluster [DBG] pgmap v787: 168 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 1 down, 6 active+clean+snaptrim, 4 undersized+peered, 38 active+undersized, 3 undersized+degraded+peered, 35 active+undersized+degraded, 75 active+clean; 358 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 74 op/s; 234/1116 objects degraded (20.968%); 8/1116 objects misplaced (0.717%); 2.6 MiB/s, 0 objects/s recovering 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.877185+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:34.878117+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:35 smithi162 bash[24557]: cluster 2024-09-20T19:30:35.232205+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-09-20T19:30:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.730 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:35.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: finishing write tid 1 to smithi04268048-30 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: finishing write tid 2 to smithi04268048-30 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: finishing write tid 3 to smithi04268048-30 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: finishing write tid 4 to smithi04268048-30 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: finishing write tid 5 to smithi04268048-30 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: oid 30 updating version 0 to 814 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: oid 30 updating version 814 to 816 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: oid 30 version 816 is already newer than 815 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:1662: oid 30 version 816 is already newer than 813 2024-09-20T19:30:35.731 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 816 (ObjNum 584 snap 188 seq_num 584) dirty exists 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1662: left oid 30 (ObjNum 584 snap 188 seq_num 584) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1663: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1663: writing smithi04268048-1 from 632095 to 1095588 tid 1 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1663: writing smithi04268048-1 from 1616319 to 2038853 tid 2 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1663: writing smithi04268048-1 from 2748686 to 3235990 tid 3 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1656: done (6 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1657: done (5 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1659: done (4 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1660: done (3 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1661: done (2 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1662: done (1 left) 2024-09-20T19:30:35.732 INFO:tasks.rados.rados.0.smithi042.stdout:1664: snap_remove snap 181 2024-09-20T19:30:35.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:35.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:35.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:35.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:36.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:36.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:30:36.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:36.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:36.343 INFO:tasks.rados.rados.0.smithi042.stdout:1664: done (1 left) 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1665: write oid 9 current snap is 188 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: finishing write tid 1 to smithi04268048-1 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: finishing write tid 2 to smithi04268048-1 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: finishing write tid 3 to smithi04268048-1 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: finishing write tid 4 to smithi04268048-1 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: finishing write tid 5 to smithi04268048-1 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: oid 1 updating version 0 to 613 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: oid 1 updating version 613 to 616 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: oid 1 version 616 is already newer than 614 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: oid 1 version 616 is already newer than 615 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 616 (ObjNum 585 snap 188 seq_num 585) dirty exists 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1663: left oid 1 (ObjNum 585 snap 188 seq_num 585) 2024-09-20T19:30:36.344 INFO:tasks.rados.rados.0.smithi042.stdout:1665: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-09-20T19:30:36.350 INFO:tasks.rados.rados.0.smithi042.stdout:1665: writing smithi04268048-9 from 648902 to 1303997 tid 1 2024-09-20T19:30:36.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:36.358 INFO:tasks.rados.rados.0.smithi042.stdout:1665: writing smithi04268048-9 from 2074801 to 2640993 tid 2 2024-09-20T19:30:36.365 INFO:tasks.rados.rados.0.smithi042.stdout:1665: writing smithi04268048-9 from 3176109 to 3848902 tid 3 2024-09-20T19:30:36.365 INFO:tasks.rados.rados.0.smithi042.stdout:1663: done (1 left) 2024-09-20T19:30:36.366 INFO:tasks.rados.rados.0.smithi042.stdout:1666: write oid 4 current snap is 188 2024-09-20T19:30:36.366 INFO:tasks.rados.rados.0.smithi042.stdout:1665: finishing write tid 1 to smithi04268048-9 2024-09-20T19:30:36.366 INFO:tasks.rados.rados.0.smithi042.stdout:1665: finishing write tid 2 to smithi04268048-9 2024-09-20T19:30:36.366 INFO:tasks.rados.rados.0.smithi042.stdout:1666: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-20T19:30:36.373 INFO:tasks.rados.rados.0.smithi042.stdout:1666: writing smithi04268048-4 from 665709 to 1396053 tid 1 2024-09-20T19:30:36.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:1666: writing smithi04268048-4 from 2133283 to 2559486 tid 2 2024-09-20T19:30:36.386 INFO:tasks.rados.rados.0.smithi042.stdout:1666: writing smithi04268048-4 from 3203532 to 3713380 tid 3 2024-09-20T19:30:36.386 INFO:tasks.rados.rados.0.smithi042.stdout:1667: snap_remove snap 157 2024-09-20T19:30:36.413 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:30:36.413 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:36 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.334910+0000 mon.a (mon.0) 1860 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 1 pg down) 2024-09-20T19:30:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.342374+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-09-20T19:30:36.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.353442+0000 osd.2 (osd.2) 173 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:30:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:36 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.361065+0000 osd.2 (osd.2) 174 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:30:36.604 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:36.631 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:36.733 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:36.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:36.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:36.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:36.972 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:30:36.973 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:30:36.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:37.017 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:37.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:37.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:37.195 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-20T19:30:37.195 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-09-20T19:30:37.195 DEBUG:teuthology.orchestra.run.smithi042:osd.3> sudo journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T19:30:37.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl start ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3 2024-09-20T19:30:37.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:37.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:30:37.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:37.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:37.399 INFO:tasks.rados.rados.0.smithi042.stdout:1667: done (2 left) 2024-09-20T19:30:37.399 INFO:tasks.rados.rados.0.smithi042.stdout:1668: snap_remove snap 179 2024-09-20T19:30:37.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:37.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.521 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[32075]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.521 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:37 smithi042 systemd[1]: Started Ceph osd.3 for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.522 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:37 smithi042 bash[22706]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.532 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:37.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.036406+0000 osd.4 (osd.4) 5 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:30:37.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.040751+0000 osd.4 (osd.4) 6 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:30:37.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:37 smithi162 bash[24557]: cluster 2024-09-20T19:30:36.259658+0000 mgr.y (mgr.24491) 478 : cluster [DBG] pgmap v789: 168 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+remapped, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 450 KiB/s wr, 46 op/s; 160/1083 objects degraded (14.774%); 8/1083 objects misplaced (0.739%); 8.9 MiB/s, 10 objects/s recovering 2024-09-20T19:30:37.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:37.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:37.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:37.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:38.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.054 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:38.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:38.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:38.286 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:38.313 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:38.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.420 INFO:tasks.rados.rados.0.smithi042.stdout:1668: done (2 left) 2024-09-20T19:30:38.420 INFO:tasks.rados.rados.0.smithi042.stdout:1669: read oid 2 snap -1 2024-09-20T19:30:38.421 INFO:tasks.rados.rados.0.smithi042.stdout:1669: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:30:38.421 INFO:tasks.rados.rados.0.smithi042.stdout:1670: write oid 38 current snap is 188 2024-09-20T19:30:38.421 INFO:tasks.rados.rados.0.smithi042.stdout:1665: finishing write tid 3 to smithi04268048-9 2024-09-20T19:30:38.421 INFO:tasks.rados.rados.0.smithi042.stdout:1670: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-20T19:30:38.430 INFO:tasks.rados.rados.0.smithi042.stdout:1670: writing smithi04268048-38 from 682516 to 1088109 tid 1 2024-09-20T19:30:38.440 INFO:tasks.rados.rados.0.smithi042.stdout:1670: writing smithi04268048-38 from 1791765 to 2361626 tid 2 2024-09-20T19:30:38.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:1670: writing smithi04268048-38 from 2830955 to 3552075 tid 3 2024-09-20T19:30:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:1671: rollback oid 5 current snap is 188 2024-09-20T19:30:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 167 2024-09-20T19:30:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:1665: finishing write tid 4 to smithi04268048-9 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: finishing write tid 5 to smithi04268048-9 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: oid 9 updating version 0 to 746 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: oid 9 updating version 746 to 747 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: oid 9 updating version 747 to 749 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: oid 9 version 749 is already newer than 748 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 749 (ObjNum 586 snap 188 seq_num 586) dirty exists 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: left oid 9 (ObjNum 586 snap 188 seq_num 586) 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1665: done (4 left) 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1672: write oid 45 current snap is 188 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1666: finishing write tid 1 to smithi04268048-4 2024-09-20T19:30:38.448 INFO:tasks.rados.rados.0.smithi042.stdout:1672: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-09-20T19:30:38.454 INFO:tasks.rados.rados.0.smithi042.stdout:1672: writing smithi04268048-45 from 699323 to 1180165 tid 1 2024-09-20T19:30:38.460 INFO:tasks.rados.rados.0.smithi042.stdout:1672: writing smithi04268048-45 from 1850247 to 2280119 tid 2 2024-09-20T19:30:38.467 INFO:tasks.rados.rados.0.smithi042.stdout:1672: writing smithi04268048-45 from 2858378 to 3390770 tid 3 2024-09-20T19:30:38.467 INFO:tasks.rados.rados.0.smithi042.stdout:1673: copy_from oid 20 from oid 29 current snap is 188 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: finishing write tid 2 to smithi04268048-4 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: finishing write tid 3 to smithi04268048-4 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: finishing write tid 4 to smithi04268048-4 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: finishing write tid 5 to smithi04268048-4 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: oid 4 updating version 0 to 767 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: oid 4 updating version 767 to 768 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: oid 4 updating version 768 to 769 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: oid 4 version 769 is already newer than 766 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 769 (ObjNum 587 snap 188 seq_num 587) dirty exists 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1666: left oid 4 (ObjNum 587 snap 188 seq_num 587) 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:1669: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:30:38.468 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:38.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:38 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.018864+0000 osd.4 (osd.4) 7 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.025224+0000 osd.4 (osd.4) 8 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:37.393134+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.559 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:38 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.215319+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: Degraded data redundancy: 160/1083 objects degraded (14.774%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:38.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1670: finishing write tid 1 to smithi04268048-38 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1670: finishing write tid 2 to smithi04268048-38 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1671: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1671: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 423 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1670: finishing write tid 3 to smithi04268048-38 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1670: finishing write tid 4 to smithi04268048-38 2024-09-20T19:30:38.627 INFO:tasks.rados.rados.0.smithi042.stdout:1670: finishing write tid 5 to smithi04268048-38 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: oid 38 updating version 0 to 599 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: oid 38 updating version 599 to 600 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: oid 38 updating version 600 to 601 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: oid 38 version 601 is already newer than 598 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 601 (ObjNum 588 snap 188 seq_num 588) dirty exists 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: left oid 38 (ObjNum 588 snap 188 seq_num 588) 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1672: finishing write tid 1 to smithi04268048-45 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1672: finishing write tid 2 to smithi04268048-45 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1666: done (5 left) 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1669: done (4 left) 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1670: done (3 left) 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1671: done (2 left) 2024-09-20T19:30:38.628 INFO:tasks.rados.rados.0.smithi042.stdout:1674: snap_create 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: finishing write tid 3 to smithi04268048-45 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: finishing write tid 4 to smithi04268048-45 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: finishing write tid 5 to smithi04268048-45 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: oid 45 updating version 0 to 995 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: oid 45 updating version 995 to 999 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: oid 45 updating version 999 to 1000 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: oid 45 version 1000 is already newer than 998 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1000 (ObjNum 589 snap 188 seq_num 589) dirty exists 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1672: left oid 45 (ObjNum 589 snap 188 seq_num 589) 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1673: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:1673: finishing copy_from to smithi04268048-20 2024-09-20T19:30:38.629 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 436 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:30:38.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:38.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:38.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:38.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:38.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:38.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:38.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:39.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:39.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:39.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.227 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:39.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:39.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:39.420 INFO:tasks.rados.rados.0.smithi042.stdout:1672: done (2 left) 2024-09-20T19:30:39.420 INFO:tasks.rados.rados.0.smithi042.stdout:1673: done (1 left) 2024-09-20T19:30:39.420 INFO:tasks.rados.rados.0.smithi042.stdout:1674: done (0 left) 2024-09-20T19:30:39.420 INFO:tasks.rados.rados.0.smithi042.stdout:1675: delete oid 15 current snap is 189 2024-09-20T19:30:39.429 INFO:tasks.rados.rados.0.smithi042.stdout:1675: done (0 left) 2024-09-20T19:30:39.429 INFO:tasks.rados.rados.0.smithi042.stdout:1676: snap_remove snap 167 2024-09-20T19:30:39.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:39.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:39.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.555 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[32075]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.556 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:39 smithi042 bash[22706]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:39.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:30:39.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:39.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:39.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:39.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.013971+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.015811+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.261275+0000 mgr.y (mgr.24491) 479 : cluster [DBG] pgmap v792: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1092 objects degraded (14.744%); 9/1092 objects misplaced (0.824%); 19 KiB/s, 0 objects/s recovering 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:39 smithi162 bash[24557]: cluster 2024-09-20T19:30:38.415850+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-09-20T19:30:39.854 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.880 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:39.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:40.003 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:40.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:40.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:40.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:40.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:40.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:40.429 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[113310]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:40.429 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[113310]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:40.432 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:30:40.445 INFO:tasks.rados.rados.0.smithi042.stdout:1676: done (0 left) 2024-09-20T19:30:40.445 INFO:tasks.rados.rados.0.smithi042.stdout:1677: snap_remove snap 174 2024-09-20T19:30:40.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:40.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:40.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:40.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:40 smithi162 bash[24557]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:40.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:40.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:40.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:40.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[22706]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: cluster 2024-09-20T19:30:39.418928+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.898 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: audit 2024-09-20T19:30:39.491661+0000 mgr.y (mgr.24491) 480 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:40.899 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.899 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:40 smithi042 bash[32075]: audit 2024-09-20T19:30:39.937456+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:30:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:40.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:41.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:41.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:41.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:41.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:41.305 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:41.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:41.446 INFO:tasks.rados.rados.0.smithi042.stdout:1677: done (0 left) 2024-09-20T19:30:41.446 INFO:tasks.rados.rados.0.smithi042.stdout:1678: copy_from oid 20 from oid 46 current snap is 189 2024-09-20T19:30:41.446 INFO:tasks.rados.rados.0.smithi042.stdout:1679: write (excl) oid 35 current snap is 189 2024-09-20T19:30:41.446 INFO:tasks.rados.rados.0.smithi042.stdout:1679: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-20T19:30:41.455 INFO:tasks.rados.rados.0.smithi042.stdout:1679: writing smithi04268048-35 from 732937 to 1364277 tid 1 2024-09-20T19:30:41.461 INFO:tasks.rados.rados.0.smithi042.stdout:1679: writing smithi04268048-35 from 1850858 to 2284399 tid 2 2024-09-20T19:30:41.468 INFO:tasks.rados.rados.0.smithi042.stdout:1679: writing smithi04268048-35 from 2796871 to 3468160 tid 3 2024-09-20T19:30:41.471 INFO:tasks.rados.rados.0.smithi042.stdout:1679: writing smithi04268048-35 from 3888195 to 3932937 tid 4 2024-09-20T19:30:41.471 INFO:tasks.rados.rados.0.smithi042.stdout:1680: copy_from oid 22 from oid 45 current snap is 189 2024-09-20T19:30:41.472 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 1 to smithi04268048-35 2024-09-20T19:30:41.472 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 2 to smithi04268048-35 2024-09-20T19:30:41.472 INFO:tasks.rados.rados.0.smithi042.stdout:1681: write oid 1 current snap is 189 2024-09-20T19:30:41.472 INFO:tasks.rados.rados.0.smithi042.stdout:1681: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-20T19:30:41.480 INFO:tasks.rados.rados.0.smithi042.stdout:1681: writing smithi04268048-1 from 749744 to 1456333 tid 1 2024-09-20T19:30:41.486 INFO:tasks.rados.rados.0.smithi042.stdout:1681: writing smithi04268048-1 from 1909340 to 2486539 tid 2 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1681: writing smithi04268048-1 from 3224294 to 3706855 tid 3 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1682: write oid 32 current snap is 189 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1678: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1678: finishing copy_from to smithi04268048-20 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 973 (ObjNum 531 snap 173 seq_num 531) dirty exists 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 3 to smithi04268048-35 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 4 to smithi04268048-35 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 5 to smithi04268048-35 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1679: finishing write tid 6 to smithi04268048-35 2024-09-20T19:30:41.494 INFO:tasks.rados.rados.0.smithi042.stdout:1679: oid 35 updating version 0 to 587 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1679: oid 35 updating version 587 to 589 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1679: oid 35 version 589 is already newer than 588 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1679: oid 35 updating version 589 to 591 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1679: oid 35 version 591 is already newer than 590 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 591 (ObjNum 591 snap 189 seq_num 591) dirty exists 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1679: left oid 35 (ObjNum 591 snap 189 seq_num 591) 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: finishing write tid 1 to smithi04268048-1 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1680: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:30:41.495 INFO:tasks.rados.rados.0.smithi042.stdout:1682: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-20T19:30:41.501 INFO:tasks.rados.rados.0.smithi042.stdout:1682: writing smithi04268048-32 from 766551 to 1264742 tid 1 2024-09-20T19:30:41.508 INFO:tasks.rados.rados.0.smithi042.stdout:1682: writing smithi04268048-32 from 1684175 to 2405032 tid 2 2024-09-20T19:30:41.514 INFO:tasks.rados.rados.0.smithi042.stdout:1682: writing smithi04268048-32 from 2851717 to 3545550 tid 3 2024-09-20T19:30:41.514 INFO:tasks.rados.rados.0.smithi042.stdout:1678: done (4 left) 2024-09-20T19:30:41.514 INFO:tasks.rados.rados.0.smithi042.stdout:1679: done (3 left) 2024-09-20T19:30:41.515 INFO:tasks.rados.rados.0.smithi042.stdout:1683: snap_remove snap 171 2024-09-20T19:30:41.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:41.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:41.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:41.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:41 smithi162 bash[24557]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:41.798 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:41.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[32075]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.262312+0000 mgr.y (mgr.24491) 481 : cluster [DBG] pgmap v795: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 364 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 161/1110 objects degraded (14.505%); 9/1110 objects misplaced (0.811%) 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.422862+0000 mon.a (mon.0) 1867 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: audit 2024-09-20T19:30:40.432467+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.42:0/3794117493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.812 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:41 smithi042 bash[22706]: cluster 2024-09-20T19:30:40.444044+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-09-20T19:30:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:41.918 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:41.918 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:41.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:41.949 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:41.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:42.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:42.181 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:42.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:42.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:42.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:42.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:42.458 INFO:tasks.rados.rados.0.smithi042.stdout:1683: done (3 left) 2024-09-20T19:30:42.459 INFO:tasks.rados.rados.0.smithi042.stdout:1684: write oid 43 current snap is 189 2024-09-20T19:30:42.459 INFO:tasks.rados.rados.0.smithi042.stdout:1684: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-09-20T19:30:42.467 INFO:tasks.rados.rados.0.smithi042.stdout:1684: writing smithi04268048-43 from 783358 to 1356798 tid 1 2024-09-20T19:30:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:42.475 INFO:tasks.rados.rados.0.smithi042.stdout:1684: writing smithi04268048-43 from 2026304 to 2607172 tid 2 2024-09-20T19:30:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:1684: writing smithi04268048-43 from 3279140 to 3900598 tid 3 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:1685: write oid 33 current snap is 189 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:1681: finishing write tid 2 to smithi04268048-1 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:1680: finishing copy_from to smithi04268048-22 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1000 (ObjNum 589 snap 188 seq_num 589) dirty exists 2024-09-20T19:30:42.483 INFO:tasks.rados.rados.0.smithi042.stdout:1685: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-20T19:30:42.489 INFO:tasks.rados.rados.0.smithi042.stdout:1685: writing smithi04268048-33 from 400165 to 1048854 tid 1 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1685: writing smithi04268048-33 from 1684786 to 2000165 tid 2 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1680: done (4 left) 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1686: read oid 31 snap -1 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1686: expect (ObjNum 565 snap 180 seq_num 565) 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1687: write oid 9 current snap is 189 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: finishing write tid 3 to smithi04268048-1 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: finishing write tid 4 to smithi04268048-1 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: finishing write tid 5 to smithi04268048-1 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: oid 1 updating version 0 to 621 2024-09-20T19:30:42.495 INFO:tasks.rados.rados.0.smithi042.stdout:1681: oid 1 version 621 is already newer than 619 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1681: oid 1 version 621 is already newer than 618 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1681: oid 1 version 621 is already newer than 620 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 621 (ObjNum 592 snap 189 seq_num 592) dirty exists 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1681: left oid 1 (ObjNum 592 snap 189 seq_num 592) 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: finishing write tid 1 to smithi04268048-32 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: finishing write tid 2 to smithi04268048-32 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: finishing write tid 3 to smithi04268048-32 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: finishing write tid 4 to smithi04268048-32 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: finishing write tid 5 to smithi04268048-32 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: oid 32 updating version 0 to 1015 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: oid 32 version 1015 is already newer than 1013 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: oid 32 version 1015 is already newer than 1014 2024-09-20T19:30:42.496 INFO:tasks.rados.rados.0.smithi042.stdout:1682: oid 32 version 1015 is already newer than 1012 2024-09-20T19:30:42.497 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1015 (ObjNum 593 snap 189 seq_num 593) dirty exists 2024-09-20T19:30:42.497 INFO:tasks.rados.rados.0.smithi042.stdout:1682: left oid 32 (ObjNum 593 snap 189 seq_num 593) 2024-09-20T19:30:42.497 INFO:tasks.rados.rados.0.smithi042.stdout:1684: finishing write tid 1 to smithi04268048-43 2024-09-20T19:30:42.497 INFO:tasks.rados.rados.0.smithi042.stdout:1684: finishing write tid 2 to smithi04268048-43 2024-09-20T19:30:42.497 INFO:tasks.rados.rados.0.smithi042.stdout:1687: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-20T19:30:42.502 INFO:tasks.rados.rados.0.smithi042.stdout:1687: writing smithi04268048-9 from 416972 to 1140910 tid 1 2024-09-20T19:30:42.508 INFO:tasks.rados.rados.0.smithi042.stdout:1687: writing smithi04268048-9 from 1743268 to 2016972 tid 2 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1681: done (5 left) 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1682: done (4 left) 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1688: rollback oid 12 current snap is 189 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 168 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: finishing write tid 3 to smithi04268048-43 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: finishing write tid 4 to smithi04268048-43 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: finishing write tid 5 to smithi04268048-43 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: oid 43 updating version 0 to 443 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: oid 43 version 443 is already newer than 442 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: oid 43 version 443 is already newer than 441 2024-09-20T19:30:42.509 INFO:tasks.rados.rados.0.smithi042.stdout:1684: oid 43 version 443 is already newer than 440 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 443 (ObjNum 594 snap 189 seq_num 594) dirty exists 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1684: left oid 43 (ObjNum 594 snap 189 seq_num 594) 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: finishing write tid 1 to smithi04268048-33 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: finishing write tid 2 to smithi04268048-33 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: finishing write tid 3 to smithi04268048-33 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: finishing write tid 4 to smithi04268048-33 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: oid 33 updating version 0 to 584 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: oid 33 updating version 584 to 585 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: oid 33 updating version 585 to 586 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 586 (ObjNum 595 snap 189 seq_num 595) dirty exists 2024-09-20T19:30:42.510 INFO:tasks.rados.rados.0.smithi042.stdout:1685: left oid 33 (ObjNum 595 snap 189 seq_num 595) 2024-09-20T19:30:42.511 INFO:tasks.rados.rados.0.smithi042.stdout:1684: done (4 left) 2024-09-20T19:30:42.511 INFO:tasks.rados.rados.0.smithi042.stdout:1685: done (3 left) 2024-09-20T19:30:42.511 INFO:tasks.rados.rados.0.smithi042.stdout:1689: write oid 4 current snap is 189 2024-09-20T19:30:42.511 INFO:tasks.rados.rados.0.smithi042.stdout:1689: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-20T19:30:42.516 INFO:tasks.rados.rados.0.smithi042.stdout:1689: writing smithi04268048-4 from 433779 to 1232966 tid 1 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1689: writing smithi04268048-4 from 1801750 to 2033779 tid 2 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1690: rollback oid 22 current snap is 189 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 188 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1688: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1687: finishing write tid 1 to smithi04268048-9 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1688: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1687: finishing write tid 2 to smithi04268048-9 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1688: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 629 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:30:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:1686: expect (ObjNum 565 snap 180 seq_num 565) 2024-09-20T19:30:42.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:42.621 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1412361 2024-09-20T19:30:42.696 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:42.700 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:42.710 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:42 smithi042 bash[32075]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.710 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:42 smithi042 bash[32075]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.711 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:42 smithi042 bash[22706]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.711 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:42 smithi042 bash[22706]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:42.794 INFO:tasks.rados.rados.0.smithi042.stdout:1687: finishing write tid 3 to smithi04268048-9 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1687: finishing write tid 4 to smithi04268048-9 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1687: oid 9 updating version 0 to 755 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1687: oid 9 updating version 755 to 757 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1687: oid 9 version 757 is already newer than 756 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 757 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1687: left oid 9 (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1690: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1690: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1690: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1003 (ObjNum 340 snap 109 seq_num 340) dirty exists 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1689: finishing write tid 1 to smithi04268048-4 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1689: finishing write tid 2 to smithi04268048-4 2024-09-20T19:30:42.795 INFO:tasks.rados.rados.0.smithi042.stdout:1686: done (4 left) 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1687: done (3 left) 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1688: done (2 left) 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1690: done (1 left) 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1691: write oid 31 current snap is 189 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: finishing write tid 3 to smithi04268048-4 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: finishing write tid 4 to smithi04268048-4 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: oid 4 updating version 0 to 780 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: oid 4 updating version 780 to 781 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: oid 4 version 781 is already newer than 779 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 781 (ObjNum 597 snap 189 seq_num 597) dirty exists 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1689: left oid 4 (ObjNum 597 snap 189 seq_num 597) 2024-09-20T19:30:42.796 INFO:tasks.rados.rados.0.smithi042.stdout:1691: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-20T19:30:42.800 INFO:tasks.rados.rados.0.smithi042.stdout:1691: writing smithi04268048-31 from 450586 to 925022 tid 1 2024-09-20T19:30:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:42 smithi162 bash[24557]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:42 smithi162 bash[24557]: cluster 2024-09-20T19:30:41.440696+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-09-20T19:30:42.806 INFO:tasks.rados.rados.0.smithi042.stdout:1691: writing smithi04268048-31 from 1343879 to 1932085 tid 2 2024-09-20T19:30:42.806 INFO:tasks.rados.rados.0.smithi042.stdout:1689: done (1 left) 2024-09-20T19:30:42.806 INFO:tasks.rados.rados.0.smithi042.stdout:1692: copy_from oid 32 from oid 35 current snap is 189 2024-09-20T19:30:42.806 INFO:tasks.rados.rados.0.smithi042.stdout:1693: snap_create 2024-09-20T19:30:42.813 INFO:tasks.rados.rados.0.smithi042.stdout:1691: finishing write tid 1 to smithi04268048-31 2024-09-20T19:30:42.814 INFO:tasks.rados.rados.0.smithi042.stdout:1691: finishing write tid 2 to smithi04268048-31 2024-09-20T19:30:42.815 INFO:tasks.rados.rados.0.smithi042.stdout:1691: finishing write tid 3 to smithi04268048-31 2024-09-20T19:30:42.816 INFO:tasks.rados.rados.0.smithi042.stdout:1691: finishing write tid 4 to smithi04268048-31 2024-09-20T19:30:42.816 INFO:tasks.rados.rados.0.smithi042.stdout:1691: oid 31 updating version 0 to 624 2024-09-20T19:30:42.816 INFO:tasks.rados.rados.0.smithi042.stdout:1691: oid 31 version 624 is already newer than 623 2024-09-20T19:30:42.817 INFO:tasks.rados.rados.0.smithi042.stdout:1691: oid 31 updating version 624 to 625 2024-09-20T19:30:42.817 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 625 (ObjNum 598 snap 189 seq_num 598) dirty exists 2024-09-20T19:30:42.817 INFO:tasks.rados.rados.0.smithi042.stdout:1691: left oid 31 (ObjNum 598 snap 189 seq_num 598) 2024-09-20T19:30:42.820 INFO:tasks.rados.rados.0.smithi042.stdout:1692: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:30:42.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.830 INFO:tasks.rados.rados.0.smithi042.stdout:1692: finishing copy_from to smithi04268048-32 2024-09-20T19:30:42.830 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 591 (ObjNum 591 snap 189 seq_num 591) dirty exists 2024-09-20T19:30:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:42.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:42.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:42.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:43.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:43.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:43.210 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:1691: done (2 left) 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:1692: done (1 left) 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:1693: done (0 left) 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:1694: rollback oid 5 current snap is 190 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 185 2024-09-20T19:30:43.233 INFO:tasks.rados.rados.0.smithi042.stdout:1695: snap_remove snap 187 2024-09-20T19:30:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:43.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:43.372 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:43.475 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.487 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[32075]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:43 smithi042 bash[22706]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:30:43.502 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:43.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:43.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:42.263695+0000 mgr.y (mgr.24491) 482 : cluster [DBG] pgmap v798: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 380 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 164/1119 objects degraded (14.656%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:42.458683+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: audit 2024-09-20T19:30:42.760459+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: audit 2024-09-20T19:30:42.761908+0000 mon.a (mon.0) 1873 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:43.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:43.217604+0000 mon.a (mon.0) 1874 : cluster [WRN] Health check update: Degraded data redundancy: 164/1119 objects degraded (14.656%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:43.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:43 smithi162 bash[24557]: cluster 2024-09-20T19:30:43.232994+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-09-20T19:30:43.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:43.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:43.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:43.864 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:43.892 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:43.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:43.955 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:43.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:44.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:44.149 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:44.236 INFO:tasks.rados.rados.0.smithi042.stdout:1695: done (1 left) 2024-09-20T19:30:44.237 INFO:tasks.rados.rados.0.smithi042.stdout:1696: snap_remove snap 182 2024-09-20T19:30:44.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.273 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:44.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.403 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:44.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:44.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:44.596 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-20T19:30:44.596 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:44.596 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:30:44.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:44.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:44.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:44.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:44.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:44.906 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:30:44.906 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-20T19:30:44.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:44.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:45.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:45.054 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:45.157 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.169 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T19:30:45.169 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-20T19:30:45.169 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T19:30:45.169 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:30:45.170 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:44 smithi042 bash[113310]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-20T19:30:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:45.236 INFO:tasks.rados.rados.0.smithi042.stdout:1696: done (1 left) 2024-09-20T19:30:45.236 INFO:tasks.rados.rados.0.smithi042.stdout:1697: snap_create 2024-09-20T19:30:45.237 INFO:tasks.rados.rados.0.smithi042.stdout:1694: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:30:45.237 INFO:tasks.rados.rados.0.smithi042.stdout:1694: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:30:45.237 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 425 (ObjNum 522 snap 169 seq_num 522) dirty exists 2024-09-20T19:30:45.289 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:45.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.301 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:45 smithi162 bash[24557]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:45.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[22706]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.234338+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.265046+0000 mgr.y (mgr.24491) 483 : cluster [DBG] pgmap v802: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 14 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:45.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.916117+0000 osd.4 (osd.4) 11 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:30:45.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:45 smithi042 bash[32075]: cluster 2024-09-20T19:30:44.918237+0000 osd.4 (osd.4) 12 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:30:45.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:45.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:45.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:45.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:45.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:45.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.975 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:45.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:46.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:46.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:46.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:46.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:46.247 INFO:tasks.rados.rados.0.smithi042.stdout:1694: done (1 left) 2024-09-20T19:30:46.247 INFO:tasks.rados.rados.0.smithi042.stdout:1697: done (0 left) 2024-09-20T19:30:46.247 INFO:tasks.rados.rados.0.smithi042.stdout:1698: write oid 22 current snap is 191 2024-09-20T19:30:46.247 INFO:tasks.rados.rados.0.smithi042.stdout:1698: seq_num 599 ranges {467393=549685,1802361=265032} 2024-09-20T19:30:46.259 INFO:tasks.rados.rados.0.smithi042.stdout:1698: writing smithi04268048-22 from 467393 to 1017078 tid 1 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1698: writing smithi04268048-22 from 1802361 to 2067393 tid 2 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1699: copy_from oid 28 from oid 34 current snap is 191 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1698: finishing write tid 1 to smithi04268048-22 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1700: read oid 20 snap 184 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1700: expect (ObjNum 566 snap 180 seq_num 566) 2024-09-20T19:30:46.275 INFO:tasks.rados.rados.0.smithi042.stdout:1701: read oid 13 snap -1 2024-09-20T19:30:46.276 INFO:tasks.rados.rados.0.smithi042.stdout:1701: expect (ObjNum 583 snap 188 seq_num 583) 2024-09-20T19:30:46.276 INFO:tasks.rados.rados.0.smithi042.stdout:1702: read oid 27 snap -1 2024-09-20T19:30:46.276 INFO:tasks.rados.rados.0.smithi042.stdout:1702: expect deleted 2024-09-20T19:30:46.276 INFO:tasks.rados.rados.0.smithi042.stdout:1703: delete oid 16 current snap is 191 2024-09-20T19:30:46.287 INFO:tasks.rados.rados.0.smithi042.stdout:1702: done (5 left) 2024-09-20T19:30:46.287 INFO:tasks.rados.rados.0.smithi042.stdout:1703: done (4 left) 2024-09-20T19:30:46.287 INFO:tasks.rados.rados.0.smithi042.stdout:1704: delete oid 45 current snap is 191 2024-09-20T19:30:46.297 INFO:tasks.rados.rados.0.smithi042.stdout:1700: expect (ObjNum 566 snap 180 seq_num 566) 2024-09-20T19:30:46.298 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:46.342 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:46.353 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:46.353 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:46 smithi042 bash[32075]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.354 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:46 smithi042 bash[32075]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:46 smithi042 bash[22706]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:46 smithi042 bash[22706]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:46.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:46.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:46 smithi162 bash[24557]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:46 smithi162 bash[24557]: cluster 2024-09-20T19:30:45.236120+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-09-20T19:30:46.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:46.505 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1281570 2024-09-20T19:30:46.553 INFO:tasks.rados.rados.0.smithi042.stdout:1698: finishing write tid 2 to smithi04268048-22 2024-09-20T19:30:46.553 INFO:tasks.rados.rados.0.smithi042.stdout:1698: finishing write tid 3 to smithi04268048-22 2024-09-20T19:30:46.553 INFO:tasks.rados.rados.0.smithi042.stdout:1701: expect (ObjNum 583 snap 188 seq_num 583) 2024-09-20T19:30:46.553 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:46.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:46.644 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2903806 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1698: finishing write tid 4 to smithi04268048-22 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1698: oid 22 updating version 0 to 1006 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1698: oid 22 version 1006 is already newer than 1005 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1698: oid 22 version 1006 is already newer than 1004 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1006 (ObjNum 599 snap 191 seq_num 599) dirty exists 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1698: left oid 22 (ObjNum 599 snap 191 seq_num 599) 2024-09-20T19:30:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:1699: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1699: finishing copy_from to smithi04268048-28 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 581 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1698: done (4 left) 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1699: done (3 left) 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1700: done (2 left) 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1701: done (1 left) 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1704: done (0 left) 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1705: write (excl) oid 3 current snap is 191 2024-09-20T19:30:46.659 INFO:tasks.rados.rados.0.smithi042.stdout:1705: seq_num 602 ranges {517814=491785,1577807=540007} 2024-09-20T19:30:46.674 INFO:tasks.rados.rados.0.smithi042.stdout:1705: writing smithi04268048-3 from 517814 to 1009599 tid 1 2024-09-20T19:30:46.694 INFO:tasks.rados.rados.0.smithi042.stdout:1705: writing smithi04268048-3 from 1577807 to 2117814 tid 2 2024-09-20T19:30:46.694 INFO:tasks.rados.rados.0.smithi042.stdout:1706: snap_remove snap 189 2024-09-20T19:30:46.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:46.755 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:46.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:46.887 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:46.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:46.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:47.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:47.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:47.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:47.182 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:30:47.183 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:30:47.200 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:30:47.201 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:30:47.217 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:30:47.218 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:30:47.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T19:30:47.239 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:30:47.239 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:30:47.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.263 INFO:tasks.rados.rados.0.smithi042.stdout:1706: done (1 left) 2024-09-20T19:30:47.263 INFO:tasks.rados.rados.0.smithi042.stdout:1707: read oid 6 snap -1 2024-09-20T19:30:47.263 INFO:tasks.rados.rados.0.smithi042.stdout:1707: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:30:47.263 INFO:tasks.rados.rados.0.smithi042.stdout:1708: snap_remove snap 184 2024-09-20T19:30:47.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.266 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.266 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.266 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[22706]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.273 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:47.277 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:30:47.277 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:30:47.303 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:30:47.303 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:30:47.325 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:30:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:30:47.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:47.464 INFO:teuthology.orchestra.run.smithi042.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-20T19:30:47.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:47.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:47 smithi162 bash[24557]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: cluster 2024-09-20T19:30:46.241199+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: cluster 2024-09-20T19:30:46.267131+0000 mgr.y (mgr.24491) 484 : cluster [DBG] pgmap v805: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 90 active+clean; 379 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 165/1119 objects degraded (14.745%); 9/1119 objects misplaced (0.804%) 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:47 smithi042 bash[32075]: audit 2024-09-20T19:30:47.163631+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.42:0/4069144802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:30:47.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:47.630 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:47.701 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:30:47.701 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:30:47.701 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:30:47.701 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:30:47.701 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:30:47.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:47.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:47.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:47.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:47.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:48.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:48.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:48.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:48.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:1708: done (2 left) 2024-09-20T19:30:48.287 INFO:tasks.rados.rados.0.smithi042.stdout:1709: write oid 21 current snap is 191 2024-09-20T19:30:48.288 INFO:tasks.rados.rados.0.smithi042.stdout:1709: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-20T19:30:48.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:48 smithi162 bash[24557]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:48 smithi162 bash[24557]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:48 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:48 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:48.301 INFO:tasks.rados.rados.0.smithi042.stdout:1709: writing smithi04268048-21 from 534621 to 1101655 tid 1 2024-09-20T19:30:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:1709: writing smithi04268048-21 from 1636289 to 2134621 tid 2 2024-09-20T19:30:48.315 INFO:tasks.rados.rados.0.smithi042.stdout:1710: write (excl) oid 29 current snap is 191 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: finishing write tid 1 to smithi04268048-3 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: finishing write tid 2 to smithi04268048-3 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: finishing write tid 3 to smithi04268048-3 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: finishing write tid 4 to smithi04268048-3 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: oid 3 updating version 0 to 696 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: oid 3 updating version 696 to 698 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: oid 3 version 698 is already newer than 697 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 698 (ObjNum 602 snap 191 seq_num 602) dirty exists 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1705: left oid 3 (ObjNum 602 snap 191 seq_num 602) 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:1707: expect (ObjNum 534 snap 173 seq_num 534) 2024-09-20T19:30:48.316 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:48.404 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:48.430 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:48.527 INFO:tasks.rados.rados.0.smithi042.stdout:1709: finishing write tid 1 to smithi04268048-21 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: finishing write tid 2 to smithi04268048-21 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: finishing write tid 3 to smithi04268048-21 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: finishing write tid 4 to smithi04268048-21 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: oid 21 updating version 0 to 836 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: oid 21 updating version 836 to 837 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: oid 21 updating version 837 to 838 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 838 (ObjNum 603 snap 191 seq_num 603) dirty exists 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1709: left oid 21 (ObjNum 603 snap 191 seq_num 603) 2024-09-20T19:30:48.528 INFO:tasks.rados.rados.0.smithi042.stdout:1710: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-20T19:30:48.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:48.547 INFO:tasks.rados.rados.0.smithi042.stdout:1710: writing smithi04268048-29 from 551428 to 1193711 tid 1 2024-09-20T19:30:48.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:48.563 INFO:tasks.rados.rados.0.smithi042.stdout:1710: writing smithi04268048-29 from 1694771 to 2151428 tid 2 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1705: done (3 left) 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1707: done (2 left) 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1709: done (1 left) 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1711: read oid 13 snap -1 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1711: expect (ObjNum 583 snap 188 seq_num 583) 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1712: write oid 50 current snap is 191 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1710: finishing write tid 1 to smithi04268048-29 2024-09-20T19:30:48.564 INFO:tasks.rados.rados.0.smithi042.stdout:1712: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-20T19:30:48.584 INFO:tasks.rados.rados.0.smithi042.stdout:1712: writing smithi04268048-50 from 568235 to 1285767 tid 1 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1712: writing smithi04268048-50 from 1753253 to 2168235 tid 2 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1713: read oid 30 snap -1 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1713: expect (ObjNum 584 snap 188 seq_num 584) 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1714: write oid 11 current snap is 191 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1710: finishing write tid 2 to smithi04268048-29 2024-09-20T19:30:48.598 INFO:tasks.rados.rados.0.smithi042.stdout:1710: finishing write tid 3 to smithi04268048-29 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1710: finishing write tid 4 to smithi04268048-29 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1710: oid 29 updating version 0 to 602 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1710: oid 29 updating version 602 to 603 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1710: oid 29 updating version 603 to 604 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 604 (ObjNum 604 snap 191 seq_num 604) dirty exists 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1710: left oid 29 (ObjNum 604 snap 191 seq_num 604) 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:1711: expect (ObjNum 583 snap 188 seq_num 583) 2024-09-20T19:30:48.599 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:48.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[32075]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[32075]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[22706]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[22706]: cluster 2024-09-20T19:30:47.263136+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-09-20T19:30:48.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:48 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.219896+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: Degraded data redundancy: 165/1119 objects degraded (14.745%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:48.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:48.720 INFO:tasks.rados.rados.0.smithi042.stdout:1712: finishing write tid 1 to smithi04268048-50 2024-09-20T19:30:48.720 INFO:tasks.rados.rados.0.smithi042.stdout:1712: finishing write tid 2 to smithi04268048-50 2024-09-20T19:30:48.720 INFO:tasks.rados.rados.0.smithi042.stdout:1712: finishing write tid 3 to smithi04268048-50 2024-09-20T19:30:48.720 INFO:tasks.rados.rados.0.smithi042.stdout:1712: finishing write tid 4 to smithi04268048-50 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:1712: oid 50 updating version 0 to 765 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:1712: oid 50 updating version 765 to 766 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:1712: oid 50 updating version 766 to 767 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 767 (ObjNum 605 snap 191 seq_num 605) dirty exists 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:1712: left oid 50 (ObjNum 605 snap 191 seq_num 605) 2024-09-20T19:30:48.721 INFO:tasks.rados.rados.0.smithi042.stdout:1713: expect (ObjNum 584 snap 188 seq_num 584) 2024-09-20T19:30:48.789 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:48.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:48.830 INFO:tasks.rados.rados.0.smithi042.stdout:1714: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-20T19:30:48.838 INFO:tasks.rados.rados.0.smithi042.stdout:1714: writing smithi04268048-11 from 585042 to 1377823 tid 1 2024-09-20T19:30:48.844 INFO:tasks.rados.rados.0.smithi042.stdout:1714: writing smithi04268048-11 from 2095382 to 2185042 tid 2 2024-09-20T19:30:48.845 INFO:tasks.rados.rados.0.smithi042.stdout:1710: done (4 left) 2024-09-20T19:30:48.845 INFO:tasks.rados.rados.0.smithi042.stdout:1711: done (3 left) 2024-09-20T19:30:48.845 INFO:tasks.rados.rados.0.smithi042.stdout:1712: done (2 left) 2024-09-20T19:30:48.845 INFO:tasks.rados.rados.0.smithi042.stdout:1713: done (1 left) 2024-09-20T19:30:48.845 INFO:tasks.rados.rados.0.smithi042.stdout:1715: snap_remove snap 186 2024-09-20T19:30:48.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.948 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:48.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:49.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:49.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:49.288 INFO:tasks.rados.rados.0.smithi042.stdout:1715: done (1 left) 2024-09-20T19:30:49.288 INFO:tasks.rados.rados.0.smithi042.stdout:1716: write (excl) oid 30 current snap is 191 2024-09-20T19:30:49.288 INFO:tasks.rados.rados.0.smithi042.stdout:1716: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-20T19:30:49.307 INFO:tasks.rados.rados.0.smithi042.stdout:1716: writing smithi04268048-30 from 601849 to 1069879 tid 1 2024-09-20T19:30:49.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:49.326 INFO:tasks.rados.rados.0.smithi042.stdout:1716: writing smithi04268048-30 from 1753864 to 2201849 tid 2 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1717: snap_create 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: finishing write tid 1 to smithi04268048-11 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: finishing write tid 2 to smithi04268048-11 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: finishing write tid 3 to smithi04268048-11 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: finishing write tid 4 to smithi04268048-11 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: oid 11 updating version 0 to 768 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: oid 11 updating version 768 to 770 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: oid 11 version 770 is already newer than 769 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 770 (ObjNum 606 snap 191 seq_num 606) dirty exists 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1714: left oid 11 (ObjNum 606 snap 191 seq_num 606) 2024-09-20T19:30:49.327 INFO:tasks.rados.rados.0.smithi042.stdout:1716: finishing write tid 1 to smithi04268048-30 2024-09-20T19:30:49.331 INFO:tasks.rados.rados.0.smithi042.stdout:1716: finishing write tid 2 to smithi04268048-30 2024-09-20T19:30:49.331 INFO:tasks.rados.rados.0.smithi042.stdout:1716: finishing write tid 3 to smithi04268048-30 2024-09-20T19:30:49.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:49.332 INFO:tasks.rados.rados.0.smithi042.stdout:1716: finishing write tid 4 to smithi04268048-30 2024-09-20T19:30:49.332 INFO:tasks.rados.rados.0.smithi042.stdout:1716: oid 30 updating version 0 to 844 2024-09-20T19:30:49.332 INFO:tasks.rados.rados.0.smithi042.stdout:1716: oid 30 version 844 is already newer than 842 2024-09-20T19:30:49.332 INFO:tasks.rados.rados.0.smithi042.stdout:1716: oid 30 version 844 is already newer than 843 2024-09-20T19:30:49.333 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 844 (ObjNum 607 snap 191 seq_num 607) dirty exists 2024-09-20T19:30:49.333 INFO:tasks.rados.rados.0.smithi042.stdout:1716: left oid 30 (ObjNum 607 snap 191 seq_num 607) 2024-09-20T19:30:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:49.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:49.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:49 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:49 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:49 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:49 smithi162 bash[24557]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:49.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:49.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:49.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:49.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[32075]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.268920+0000 mgr.y (mgr.24491) 485 : cluster [DBG] pgmap v807: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1128 objects degraded (14.894%); 9/1128 objects misplaced (0.798%) 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:49 smithi042 bash[22706]: cluster 2024-09-20T19:30:48.277199+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-09-20T19:30:49.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:49.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:49.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:49.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:49.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:49.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:50.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:50.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:50.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:50.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:50.312 INFO:tasks.rados.rados.0.smithi042.stdout:1714: done (2 left) 2024-09-20T19:30:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1716: done (1 left) 2024-09-20T19:30:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1717: done (0 left) 2024-09-20T19:30:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1718: delete oid 20 current snap is 192 2024-09-20T19:30:50.323 INFO:tasks.rados.rados.0.smithi042.stdout:1718: done (0 left) 2024-09-20T19:30:50.323 INFO:tasks.rados.rados.0.smithi042.stdout:1719: snap_create 2024-09-20T19:30:50.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.352 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:50 smithi162 bash[24557]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:50.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:50.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:50.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:50.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[32075]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.287770+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-09-20T19:30:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: audit 2024-09-20T19:30:49.502792+0000 mgr.y (mgr.24491) 486 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:30:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.900303+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:30:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:50 smithi042 bash[22706]: cluster 2024-09-20T19:30:49.903136+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:30:50.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T19:30:50.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T19:30:50.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:50.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:50.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:30:50.891 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-20T19:30:50.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:30:50.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:51.013 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:51.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:51.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:51.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:51.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1719: done (0 left) 2024-09-20T19:30:51.306 INFO:tasks.rados.rados.0.smithi042.stdout:1720: snap_remove snap 154 2024-09-20T19:30:51.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:51.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:51.400 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[114337]: debug 2024-09-20T19:30:51.193+0000 7f6787e3b640 1 -- 172.21.15.42:0/437452270 <== mon.2 v1:172.21.15.42:6790/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 2481702606 0 0) 0x563e23e68960 con 0x563e24c54000 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[32075]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[32075]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[32075]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[32075]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[22706]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[22706]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[22706]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.401 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:51 smithi042 bash[22706]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:51.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:51.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:51 smithi162 bash[24557]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:51 smithi162 bash[24557]: cluster 2024-09-20T19:30:50.270728+0000 mgr.y (mgr.24491) 487 : cluster [DBG] pgmap v810: 168 pgs: 2 active+clean+remapped, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 91 active+clean; 389 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 3.5 MiB/s wr, 19 op/s; 168/1131 objects degraded (14.854%); 9/1131 objects misplaced (0.796%) 2024-09-20T19:30:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:51 smithi162 bash[24557]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:51 smithi162 bash[24557]: cluster 2024-09-20T19:30:50.306827+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-09-20T19:30:51.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:51.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:51.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:51.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:51.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:51.922 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:51.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:52.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:52.155 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:52.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:52.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:52.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:1720: done (0 left) 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:1721: rollback oid 11 current snap is 193 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 190 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:1722: read oid 8 snap -1 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:1722: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:30:52.317 INFO:tasks.rados.rados.0.smithi042.stdout:1723: delete oid 48 current snap is 193 2024-09-20T19:30:52.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:52.333 INFO:tasks.rados.rados.0.smithi042.stdout:1722: expect (ObjNum 495 snap 160 seq_num 495) 2024-09-20T19:30:52.412 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1739145 2024-09-20T19:30:52.417 INFO:tasks.rados.rados.0.smithi042.stdout:1721: finishing rollback tid 0 to smithi04268048-11 2024-09-20T19:30:52.417 INFO:tasks.rados.rados.0.smithi042.stdout:1721: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:30:52.417 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 777 (ObjNum 289 snap 0 seq_num 174164285) dirty dne 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1721: done (2 left) 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1722: done (1 left) 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1723: done (0 left) 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1724: rollback oid 27 current snap is 193 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 190 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1725: write oid 2 current snap is 193 2024-09-20T19:30:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:1725: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-20T19:30:52.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:52.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:52 smithi042 bash[32075]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:52 smithi042 bash[32075]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:52 smithi042 bash[22706]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:52 smithi042 bash[22706]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.435 INFO:tasks.rados.rados.0.smithi042.stdout:1725: writing smithi04268048-2 from 652270 to 1062400 tid 1 2024-09-20T19:30:52.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:52.451 INFO:tasks.rados.rados.0.smithi042.stdout:1725: writing smithi04268048-2 from 1529310 to 2023177 tid 2 2024-09-20T19:30:52.451 INFO:tasks.rados.rados.0.smithi042.stdout:1726: read oid 14 snap -1 2024-09-20T19:30:52.451 INFO:tasks.rados.rados.0.smithi042.stdout:1726: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:30:52.451 INFO:tasks.rados.rados.0.smithi042.stdout:1727: snap_remove snap 191 2024-09-20T19:30:52.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:52.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:52 smithi162 bash[24557]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:52 smithi162 bash[24557]: cluster 2024-09-20T19:30:51.303472+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-09-20T19:30:52.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:52.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:52.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:30:52.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:52.929 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:52 smithi042 bash[114337]: debug 2024-09-20T19:30:52.565+0000 7f678a6a2640 -1 Falling back to public interface 2024-09-20T19:30:52.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:52.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:52.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:53.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:53.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:53.192 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:53.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1727: done (3 left) 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1728: write (excl) oid 39 current snap is 193 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1724: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 683 (ObjNum 323009 snap 0 seq_num 1415033760) dirty dne 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: finishing write tid 1 to smithi04268048-2 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: finishing write tid 2 to smithi04268048-2 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: finishing write tid 3 to smithi04268048-2 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: finishing write tid 4 to smithi04268048-2 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: oid 2 updating version 0 to 588 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: oid 2 updating version 588 to 589 2024-09-20T19:30:53.227 INFO:tasks.rados.rados.0.smithi042.stdout:1725: oid 2 version 589 is already newer than 587 2024-09-20T19:30:53.228 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 589 (ObjNum 610 snap 193 seq_num 610) dirty exists 2024-09-20T19:30:53.228 INFO:tasks.rados.rados.0.smithi042.stdout:1725: left oid 2 (ObjNum 610 snap 193 seq_num 610) 2024-09-20T19:30:53.228 INFO:tasks.rados.rados.0.smithi042.stdout:1726: expect (ObjNum 217 snap 65 seq_num 217) 2024-09-20T19:30:53.228 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:53.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:53.329 INFO:tasks.rados.rados.0.smithi042.stdout:1728: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-20T19:30:53.335 INFO:tasks.rados.rados.0.smithi042.stdout:1728: writing smithi04268048-39 from 669077 to 1154456 tid 1 2024-09-20T19:30:53.342 INFO:tasks.rados.rados.0.smithi042.stdout:1728: writing smithi04268048-39 from 1587792 to 2269077 tid 2 2024-09-20T19:30:53.342 INFO:tasks.rados.rados.0.smithi042.stdout:1724: done (3 left) 2024-09-20T19:30:53.342 INFO:tasks.rados.rados.0.smithi042.stdout:1725: done (2 left) 2024-09-20T19:30:53.342 INFO:tasks.rados.rados.0.smithi042.stdout:1726: done (1 left) 2024-09-20T19:30:53.343 INFO:tasks.rados.rados.0.smithi042.stdout:1729: read oid 47 snap -1 2024-09-20T19:30:53.343 INFO:tasks.rados.rados.0.smithi042.stdout:1729: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:53.343 INFO:tasks.rados.rados.0.smithi042.stdout:1730: delete oid 48 current snap is 193 2024-09-20T19:30:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:1730: done (2 left) 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:1731: rollback oid 13 current snap is 193 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 177 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:1732: rollback oid 8 current snap is 193 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 188 2024-09-20T19:30:53.345 INFO:tasks.rados.rados.0.smithi042.stdout:1733: delete oid 31 current snap is 193 2024-09-20T19:30:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:53.349 INFO:tasks.rados.rados.0.smithi042.stdout:1731: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:30:53.350 INFO:tasks.rados.rados.0.smithi042.stdout:1731: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:30:53.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 567 (ObjNum 129 snap 0 seq_num 1415956992) dirty dne 2024-09-20T19:30:53.357 INFO:tasks.rados.rados.0.smithi042.stdout:1728: finishing write tid 1 to smithi04268048-39 2024-09-20T19:30:53.357 INFO:tasks.rados.rados.0.smithi042.stdout:1728: finishing write tid 2 to smithi04268048-39 2024-09-20T19:30:53.358 INFO:tasks.rados.rados.0.smithi042.stdout:1728: finishing write tid 3 to smithi04268048-39 2024-09-20T19:30:53.359 INFO:tasks.rados.rados.0.smithi042.stdout:1728: finishing write tid 4 to smithi04268048-39 2024-09-20T19:30:53.359 INFO:tasks.rados.rados.0.smithi042.stdout:1728: oid 39 updating version 0 to 587 2024-09-20T19:30:53.360 INFO:tasks.rados.rados.0.smithi042.stdout:1728: oid 39 version 587 is already newer than 586 2024-09-20T19:30:53.360 INFO:tasks.rados.rados.0.smithi042.stdout:1728: oid 39 version 587 is already newer than 585 2024-09-20T19:30:53.360 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 587 (ObjNum 611 snap 193 seq_num 611) dirty exists 2024-09-20T19:30:53.360 INFO:tasks.rados.rados.0.smithi042.stdout:1728: left oid 39 (ObjNum 611 snap 193 seq_num 611) 2024-09-20T19:30:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:1732: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:30:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:1732: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:30:53.362 INFO:tasks.rados.rados.0.smithi042.stdout:1729: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:53.362 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 284438 2024-09-20T19:30:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 105310 2024-09-20T19:30:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:53.447 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1732: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 629 (ObjNum 495 snap 160 seq_num 495) dirty exists 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1728: done (4 left) 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1729: done (3 left) 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1731: done (2 left) 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1732: done (1 left) 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1733: done (0 left) 2024-09-20T19:30:53.448 INFO:tasks.rados.rados.0.smithi042.stdout:1734: snap_remove snap 188 2024-09-20T19:30:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:53 smithi162 bash[24557]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:53.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:53.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[32075]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:52.272989+0000 mgr.y (mgr.24491) 488 : cluster [DBG] pgmap v813: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 391 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:52.305193+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-09-20T19:30:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:53.220907+0000 mon.a (mon.0) 1886 : cluster [WRN] Health check update: Degraded data redundancy: 169/1137 objects degraded (14.864%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:53 smithi042 bash[22706]: cluster 2024-09-20T19:30:53.226635+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-09-20T19:30:53.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:53.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:53.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:53.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:53.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:53.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:53.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:54.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:54.123 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:54.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:54.252 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:54.331 INFO:tasks.rados.rados.0.smithi042.stdout:1734: done (0 left) 2024-09-20T19:30:54.331 INFO:tasks.rados.rados.0.smithi042.stdout:1735: read oid 46 snap -1 2024-09-20T19:30:54.331 INFO:tasks.rados.rados.0.smithi042.stdout:1735: expect (ObjNum 531 snap 173 seq_num 531) 2024-09-20T19:30:54.331 INFO:tasks.rados.rados.0.smithi042.stdout:1736: delete oid 7 current snap is 193 2024-09-20T19:30:54.343 INFO:tasks.rados.rados.0.smithi042.stdout:1735: expect (ObjNum 531 snap 173 seq_num 531) 2024-09-20T19:30:54.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:54.476 INFO:tasks.rados.rados.0.smithi042.stdout:1735: done (1 left) 2024-09-20T19:30:54.476 INFO:tasks.rados.rados.0.smithi042.stdout:1736: done (0 left) 2024-09-20T19:30:54.476 INFO:tasks.rados.rados.0.smithi042.stdout:1737: write oid 4 current snap is 193 2024-09-20T19:30:54.476 INFO:tasks.rados.rados.0.smithi042.stdout:1737: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-20T19:30:54.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:1737: writing smithi04268048-4 from 736305 to 1522680 tid 1 2024-09-20T19:30:54.489 INFO:tasks.rados.rados.0.smithi042.stdout:1737: writing smithi04268048-4 from 2105367 to 2336305 tid 2 2024-09-20T19:30:54.489 INFO:tasks.rados.rados.0.smithi042.stdout:1738: read oid 20 snap -1 2024-09-20T19:30:54.490 INFO:tasks.rados.rados.0.smithi042.stdout:1738: expect deleted 2024-09-20T19:30:54.490 INFO:tasks.rados.rados.0.smithi042.stdout:1739: read oid 37 snap -1 2024-09-20T19:30:54.490 INFO:tasks.rados.rados.0.smithi042.stdout:1739: expect (ObjNum 497 snap 160 seq_num 497) 2024-09-20T19:30:54.490 INFO:tasks.rados.rados.0.smithi042.stdout:1740: snap_create 2024-09-20T19:30:54.494 INFO:tasks.rados.rados.0.smithi042.stdout:1739: expect (ObjNum 497 snap 160 seq_num 497) 2024-09-20T19:30:54.507 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:54.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:54.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:54.647 INFO:tasks.rados.rados.0.smithi042.stdout:1737: finishing write tid 1 to smithi04268048-4 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: finishing write tid 2 to smithi04268048-4 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: finishing write tid 3 to smithi04268048-4 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: finishing write tid 4 to smithi04268048-4 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: oid 4 updating version 0 to 799 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: oid 4 version 799 is already newer than 798 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: oid 4 updating version 799 to 800 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 800 (ObjNum 615 snap 193 seq_num 615) dirty exists 2024-09-20T19:30:54.648 INFO:tasks.rados.rados.0.smithi042.stdout:1737: left oid 4 (ObjNum 615 snap 193 seq_num 615) 2024-09-20T19:30:54.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:54.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:54.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:54.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:54.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:55.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:55.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:55.235 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:55.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:55.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1737: done (3 left) 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1738: done (2 left) 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1739: done (1 left) 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1740: done (0 left) 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1741: copy_from oid 39 from oid 23 current snap is 194 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1742: copy_from oid 32 from oid 43 current snap is 194 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1743: write oid 44 current snap is 194 2024-09-20T19:30:55.328 INFO:tasks.rados.rados.0.smithi042.stdout:1743: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-20T19:30:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:30:55.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[22706]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.337 INFO:tasks.rados.rados.0.smithi042.stdout:1743: writing smithi04268048-44 from 753112 to 1331089 tid 1 2024-09-20T19:30:55.344 INFO:tasks.rados.rados.0.smithi042.stdout:1743: writing smithi04268048-44 from 1880202 to 2353112 tid 2 2024-09-20T19:30:55.344 INFO:tasks.rados.rados.0.smithi042.stdout:1744: rollback oid 25 current snap is 194 2024-09-20T19:30:55.344 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 176 2024-09-20T19:30:55.344 INFO:tasks.rados.rados.0.smithi042.stdout:1745: write oid 36 current snap is 194 2024-09-20T19:30:55.344 INFO:tasks.rados.rados.0.smithi042.stdout:1745: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-20T19:30:55.355 INFO:tasks.rados.rados.0.smithi042.stdout:1745: writing smithi04268048-36 from 769919 to 1423145 tid 1 2024-09-20T19:30:55.361 INFO:tasks.rados.rados.0.smithi042.stdout:1745: writing smithi04268048-36 from 1938684 to 2369919 tid 2 2024-09-20T19:30:55.361 INFO:tasks.rados.rados.0.smithi042.stdout:1746: copy_from oid 17 from oid 18 current snap is 194 2024-09-20T19:30:55.361 INFO:tasks.rados.rados.0.smithi042.stdout:1743: finishing write tid 1 to smithi04268048-44 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1741: finishing copy_from racing read to smithi04268048-39 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1742: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: finishing write tid 2 to smithi04268048-44 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1747: delete oid 5 current snap is 194 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: finishing write tid 3 to smithi04268048-44 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: finishing write tid 4 to smithi04268048-44 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: oid 44 updating version 0 to 1012 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: oid 44 version 1012 is already newer than 1011 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: oid 44 version 1012 is already newer than 1010 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1012 (ObjNum 616 snap 194 seq_num 616) dirty exists 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1743: left oid 44 (ObjNum 616 snap 194 seq_num 616) 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:1744: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:30:55.362 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 724 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:30:55.363 INFO:tasks.rados.rados.0.smithi042.stdout:1741: finishing copy_from to smithi04268048-39 2024-09-20T19:30:55.363 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 695 (ObjNum 563 snap 180 seq_num 563) dirty exists 2024-09-20T19:30:55.363 INFO:tasks.rados.rados.0.smithi042.stdout:1746: finishing copy_from to smithi04268048-17 2024-09-20T19:30:55.363 INFO:tasks.rados.rados.0.smithi042.stdout:1746: got expected ENOENT (src dne) 2024-09-20T19:30:55.364 INFO:tasks.rados.rados.0.smithi042.stdout:1746: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1742: finishing copy_from to smithi04268048-32 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 443 (ObjNum 594 snap 189 seq_num 594) dirty exists 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1741: done (6 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1742: done (5 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1743: done (4 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1744: done (3 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1746: done (2 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1747: done (1 left) 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1748: read oid 22 snap 168 2024-09-20T19:30:55.365 INFO:tasks.rados.rados.0.smithi042.stdout:1748: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:30:55.366 INFO:tasks.rados.rados.0.smithi042.stdout:1749: write (excl) oid 12 current snap is 194 2024-09-20T19:30:55.366 INFO:tasks.rados.rados.0.smithi042.stdout:1749: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-20T19:30:55.369 INFO:tasks.rados.rados.0.smithi042.stdout:1749: writing smithi04268048-12 from 403533 to 807257 tid 1 2024-09-20T19:30:55.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:55.377 INFO:tasks.rados.rados.0.smithi042.stdout:1749: writing smithi04268048-12 from 1539295 to 2191496 tid 2 2024-09-20T19:30:55.377 INFO:tasks.rados.rados.0.smithi042.stdout:1750: read oid 28 snap -1 2024-09-20T19:30:55.377 INFO:tasks.rados.rados.0.smithi042.stdout:1750: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:30:55.378 INFO:tasks.rados.rados.0.smithi042.stdout:1751: snap_remove snap 185 2024-09-20T19:30:55.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.384 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:55 smithi162 bash[24557]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.397 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:30:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:30:55.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:55.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:55.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:55.752 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:55.778 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:55.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:55.884 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: cluster 2024-09-20T19:30:54.274631+0000 mgr.y (mgr.24491) 489 : cluster [DBG] pgmap v816: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 387 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.2 MiB/s wr, 16 op/s; 169/1137 objects degraded (14.864%); 11/1137 objects misplaced (0.967%) 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: cluster 2024-09-20T19:30:54.327559+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: audit 2024-09-20T19:30:55.220471+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.42:0/25218147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.885 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:55 smithi042 bash[32075]: audit 2024-09-20T19:30:55.221029+0000 mon.a (mon.0) 1889 : audit [INF] from='client.25448 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:30:55.891 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:30:55.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:55.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:56.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:56.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:56.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:56.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:56.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:56.346 INFO:tasks.rados.rados.0.smithi042.stdout:1751: done (4 left) 2024-09-20T19:30:56.347 INFO:tasks.rados.rados.0.smithi042.stdout:1752: read oid 48 snap -1 2024-09-20T19:30:56.347 INFO:tasks.rados.rados.0.smithi042.stdout:1752: expect deleted 2024-09-20T19:30:56.347 INFO:tasks.rados.rados.0.smithi042.stdout:1753: write (excl) oid 20 current snap is 194 2024-09-20T19:30:56.347 INFO:tasks.rados.rados.0.smithi042.stdout:1753: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-20T19:30:56.352 INFO:tasks.rados.rados.0.smithi042.stdout:1753: writing smithi04268048-20 from 420340 to 899313 tid 1 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1753: writing smithi04268048-20 from 1597777 to 2109989 tid 2 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1754: snap_create 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1745: finishing write tid 1 to smithi04268048-36 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1745: finishing write tid 2 to smithi04268048-36 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1745: finishing write tid 3 to smithi04268048-36 2024-09-20T19:30:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1745: finishing write tid 4 to smithi04268048-36 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1745: oid 36 updating version 0 to 564 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1745: oid 36 version 564 is already newer than 563 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1745: oid 36 version 564 is already newer than 562 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 564 (ObjNum 617 snap 194 seq_num 617) dirty exists 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1745: left oid 36 (ObjNum 617 snap 194 seq_num 617) 2024-09-20T19:30:56.360 INFO:tasks.rados.rados.0.smithi042.stdout:1748: expect (ObjNum 340 snap 109 seq_num 340) 2024-09-20T19:30:56.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:56.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:56.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:30:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:30:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[32075]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[22706]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.472 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:56.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:56.536 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:56.620 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:56 smithi042 bash[114337]: debug 2024-09-20T19:30:56.617+0000 7f678a6a2640 -1 osd.3 479 log_to_monitors true 2024-09-20T19:30:56.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: finishing write tid 1 to smithi04268048-12 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: finishing write tid 2 to smithi04268048-12 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: finishing write tid 3 to smithi04268048-12 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: finishing write tid 4 to smithi04268048-12 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: oid 12 updating version 0 to 630 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: oid 12 updating version 630 to 631 2024-09-20T19:30:56.641 INFO:tasks.rados.rados.0.smithi042.stdout:1749: oid 12 updating version 631 to 632 2024-09-20T19:30:56.642 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 632 (ObjNum 619 snap 194 seq_num 619) dirty exists 2024-09-20T19:30:56.642 INFO:tasks.rados.rados.0.smithi042.stdout:1749: left oid 12 (ObjNum 619 snap 194 seq_num 619) 2024-09-20T19:30:56.642 INFO:tasks.rados.rados.0.smithi042.stdout:1750: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:30:56.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: cluster 2024-09-20T19:30:55.320165+0000 mon.a (mon.0) 1890 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:30:56.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: audit 2024-09-20T19:30:55.321801+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25448 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:30:56.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:56 smithi162 bash[24557]: cluster 2024-09-20T19:30:55.327699+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-09-20T19:30:56.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:56.751 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1357702 2024-09-20T19:30:56.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:56.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: finishing write tid 1 to smithi04268048-20 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: finishing write tid 2 to smithi04268048-20 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: finishing write tid 3 to smithi04268048-20 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: finishing write tid 4 to smithi04268048-20 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: oid 20 updating version 0 to 977 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: oid 20 version 977 is already newer than 976 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:1753: oid 20 version 977 is already newer than 975 2024-09-20T19:30:56.794 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 977 (ObjNum 620 snap 194 seq_num 620) dirty exists 2024-09-20T19:30:56.795 INFO:tasks.rados.rados.0.smithi042.stdout:1753: left oid 20 (ObjNum 620 snap 194 seq_num 620) 2024-09-20T19:30:56.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:56.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:57.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:57.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:57.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:57.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1745: done (6 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1748: done (5 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1749: done (4 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1750: done (3 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1752: done (2 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1753: done (1 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1754: done (0 left) 2024-09-20T19:30:57.362 INFO:tasks.rados.rados.0.smithi042.stdout:1755: snap_remove snap 192 2024-09-20T19:30:57.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:57.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:57 smithi162 bash[24557]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:57.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.562 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:57.662 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[114337]: debug 2024-09-20T19:30:57.529+0000 7f678244c640 -1 osd.3 549 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:30:57.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[32075]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: cluster 2024-09-20T19:30:56.275619+0000 mgr.y (mgr.24491) 490 : cluster [DBG] pgmap v819: 168 pgs: 2 active+clean+remapped, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 93 active+clean; 376 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s; 169/1131 objects degraded (14.943%); 11/1131 objects misplaced (0.973%) 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: cluster 2024-09-20T19:30:56.342906+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.663 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:57 smithi042 bash[22706]: audit 2024-09-20T19:30:56.624524+0000 mon.a (mon.0) 1894 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:30:57.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:57.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:57.795 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:57.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:57.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:58.001 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:30:58.001 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:30:58.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1019 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 43] snapc 2f=[2f,2e,2d,2c,2a,26,24] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 465.81052508699997, 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032249699999999999, 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1019 2024-09-20T19:30:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188455+0000", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188459+0000", 2024-09-20T19:30:58.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8380000000000004e-06 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188472+0000", 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3171e-05 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188489+0000", 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7516e-05 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188503+0000", 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3641e-05 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188623+0000", 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000120104 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.188745+0000", 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000121839 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.201923+0000", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013178337 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.210593+0000", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0086699889999999995 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220372+0000", 2024-09-20T19:30:58.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.009778241 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220589+0000", 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021783800000000001 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220647+0000", 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7151000000000002e-05 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:12.220705+0000", 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8035000000000003e-05 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1222 7.8 7:17ea136f:::smithi04268048-32:head [copy-from ver 147] snapc 39=[39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 441.67457809500002, 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.039637587000000002, 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.008 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1222 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324402+0000", 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324404+0000", 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.73e-06 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324413+0000", 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7429999999999997e-06 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324432+0000", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8723e-05 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324454+0000", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1727000000000001e-05 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.326930+0000", 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0024762009999999999 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327029+0000", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9409999999999999e-05 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.344328+0000", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017298289000000001 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363895+0000", 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019566842000000001 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363943+0000", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8529999999999998e-05 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363959+0000", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6246e-05 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363999+0000", 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9836000000000003e-05 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.364039+0000", 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0309999999999999e-05 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1223 7.8 7:17ea136f:::smithi04268048-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 441.67450265500003, 2024-09-20T19:30:58.014 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.039355411999999999, 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1223 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324477+0000", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324479+0000", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.643e-06 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324523+0000", 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4363000000000002e-05 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324553+0000", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9856999999999999e-05 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324571+0000", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8246000000000001e-05 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327203+0000", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0026315039999999998 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:58.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327246+0000", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3305e-05 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363693+0000", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036446495000000002 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363768+0000", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5630000000000006e-05 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363833+0000", 2024-09-20T19:30:58.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4369000000000007e-05 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1231 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 27] snapc 39=[39,33,30] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 441.67434284000001, 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.045744076000000002, 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.019 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1231 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324637+0000", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324640+0000", 2024-09-20T19:30:58.020 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4569999999999999e-06 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324652+0000", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2007e-05 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324678+0000", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6080000000000001e-05 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324710+0000", 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2492e-05 2024-09-20T19:30:58.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327276+0000", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002565496 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327356+0000", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0316999999999997e-05 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.357039+0000", 2024-09-20T19:30:58.022 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.029683095 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.364449+0000", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0074094540000000002 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.367262+0000", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0028136179999999999 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370298+0000", 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0030354420000000002 2024-09-20T19:30:58.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370351+0000", 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3581000000000001e-05 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.370381+0000", 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0037000000000001e-05 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:1232 7.8 7:1695abb3:::smithi04268048-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-20T19:30:58.024 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 441.67424510400002, 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.038895403000000002, 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 1232 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.025 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324735+0000", 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324737+0000", 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8870000000000001e-06 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324746+0000", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8899999999999996e-06 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324771+0000", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5136000000000001e-05 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.324788+0000", 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7410000000000001e-05 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327489+0000", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0027011679999999999 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.327534+0000", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5117000000000002e-05 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363387+0000", 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.035852123 2024-09-20T19:30:58.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363496+0000", 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000109545 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:23:36.363630+0000", 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000134127 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.029 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2718 7.8 7:1517470f:::smithi04268048-49:head [copy-from ver 311] snapc 77=[77,76,73,72,70,6d,6c,69,67,5e,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 327.67802780199997, 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032791556999999999, 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2718 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.030 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320952+0000", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320958+0000", 2024-09-20T19:30:58.031 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8930000000000002e-06 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.320980+0000", 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2240000000000001e-05 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321019+0000", 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.871e-05 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.032 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321042+0000", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.281e-05 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321179+0000", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000137317 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321322+0000", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014273899999999999 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.033 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.339340+0000", 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.018018031 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.351735+0000", 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012394794000000001 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.352003+0000", 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000267951 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353665+0000", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016625170000000001 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353706+0000", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0837000000000002e-05 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.353744+0000", 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7718e-05 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:2719 7.8 7:1517470f:::smithi04268048-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e341)", 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 327.677871853, 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.029709101000000002, 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2719 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321108+0000", 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321110+0000", 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1469999999999999e-06 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321120+0000", 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9129999999999998e-06 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321145+0000", 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4833000000000001e-05 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321163+0000", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7773000000000001e-05 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321650+0000", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048751600000000001 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.321706+0000", 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5578999999999998e-05 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350549+0000", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028842689000000001 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350676+0000", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012729199999999999 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:25:30.350817+0000", 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000141359 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3207 7.8 7:1b5f1bb3:::smithi04268048-46:head [copy-from ver 541] snapc 92=[92,91,90,8f,8e,8d,8c,8b,8a,89,88,87,86,85,84,83,82,81,7f,7c,79,72,41] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 283.64018905900002, 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028733953999999999, 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3207 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358791+0000", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358797+0000", 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7100000000000004e-06 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358816+0000", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9830000000000002e-05 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358878+0000", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1472999999999999e-05 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358903+0000", 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.51e-05 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359022+0000", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000119246 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359152+0000", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000130075 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.372285+0000", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013132437 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383793+0000", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011508107 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387396+0000", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0036035619999999998 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387434+0000", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7360000000000001e-05 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387484+0000", 2024-09-20T19:30:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9815000000000003e-05 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.387525+0000", 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1239000000000003e-05 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3208 7.8 7:1b5f1bb3:::smithi04268048-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e385)", 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 283.64002266300002, 2024-09-20T19:30:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.024280572, 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3208 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.047 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358957+0000", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358960+0000", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.216e-06 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358968+0000", 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7539999999999995e-06 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.358988+0000", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9530000000000001e-05 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359004+0000", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5943e-05 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359429+0000", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00042558400000000001 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:58.049 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.359586+0000", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000156489 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.382994+0000", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023408279000000001 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383137+0000", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014323499999999999 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:14.383238+0000", 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100542 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "rep_scrubmap(5.6 e73 from shard 5)", 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:18.261620+0000", 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 279.73735997099999, 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.027365707, 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.051 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.261620+0000", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.261620+0000", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.261623+0000", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9969999999999999e-06 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.261637+0000", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3953e-05 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.288546+0000", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.026908873999999999 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.288585+0000", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8893000000000002e-05 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.288877+0000", 2024-09-20T19:30:58.053 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000292358 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.288909+0000", 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2042999999999998e-05 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:18.288986+0000", 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6588999999999995e-05 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3461 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 337] snapc 9b=[9b,9a,99,98,97,94,8f,8d,8b,8a,88,87,86,84,83,81,7f,79,72] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:26:36.198197+0000", 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 261.80078251499998, 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.026914662999999998, 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3461 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.055 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198197+0000", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198197+0000", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198201+0000", 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.298e-06 2024-09-20T19:30:58.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198211+0000", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.944e-06 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198228+0000", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.7283e-05 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198239+0000", 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0686999999999999e-05 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198344+0000", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000105544 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.198824+0000", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00048001000000000001 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.208746+0000", 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0099218850000000001 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.218444+0000", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0096980580000000007 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.219823+0000", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013788089999999999 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.224992+0000", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0051686850000000001 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.059 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.225056+0000", 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4109000000000006e-05 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:26:36.225112+0000", 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6351000000000003e-05 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3748 7.8 7:1695abb3:::smithi04268048-45:head [copy-from ver 577] snapc a7=[a7,a6,a5,a4,a3,a2,a1,a0,9f,9d,9c,98,8b,8a,88,81,72] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:27:02.162317+0000", 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 235.83666249800001, 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030778878999999999, 2024-09-20T19:30:58.060 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3748 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162317+0000", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.061 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162317+0000", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162318+0000", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3600000000000003e-07 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162323+0000", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.5090000000000004e-06 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162331+0000", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7139999999999999e-06 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162338+0000", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.2500000000000003e-06 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162506+0000", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000168166 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.162510+0000", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9199999999999997e-06 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.166892+0000", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0043819499999999999 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.167015+0000", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012334299999999999 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.174950+0000", 2024-09-20T19:30:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0079354119999999993 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.181051+0000", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0061004049999999997 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.188853+0000", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0078024710000000001 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.192959+0000", 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0041061129999999998 2024-09-20T19:30:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.193029+0000", 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9638e-05 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:27:02.193096+0000", 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7352000000000002e-05 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(osd.5.44:13 7.13 7:cf582c87:::smithi04268048-12:head [assert-version v436,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:09.881060+0000", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 48.117919956999998, 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.90253167599999995, 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881060+0000", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881060+0000", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881063+0000", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.366e-06 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881083+0000", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9508e-05 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881104+0000", 2024-09-20T19:30:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0727999999999999e-05 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881114+0000", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.047e-05 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881193+0000", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9058000000000001e-05 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.881202+0000", 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9160000000000007e-06 2024-09-20T19:30:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:10.780728+0000", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.89952559399999998 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:10.780927+0000", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019898 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:10.783592+0000", 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0026650559999999998 2024-09-20T19:30:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4225 7.8 7:1517470f:::smithi04268048-49:head [write 548060~487248] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.323218+0000", 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.675761790999999, 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.066340851000000006, 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4225 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323218+0000", 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323218+0000", 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323221+0000", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5890000000000001e-06 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323528+0000", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000306884 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323544+0000", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5897e-05 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323555+0000", 2024-09-20T19:30:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1717e-05 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323632+0000", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6747000000000002e-05 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.323644+0000", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1857e-05 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350412+0000", 2024-09-20T19:30:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.026768610000000002 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for degraded object", 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350958+0000", 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00054596600000000005 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.381337+0000", 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030378898000000001 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.381470+0000", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000133041 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.381748+0000", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027786400000000001 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.387829+0000", 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060807190000000001 2024-09-20T19:30:58.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388023+0000", 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000193616 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389209+0000", 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001186274 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.077 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389330+0000", 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012149799999999999 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389559+0000", 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000228674 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.078 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4226 7.8 7:1517470f:::smithi04268048-49:head [write 1440262~671538] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.333853+0000", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.665126663999999, 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.057998964, 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4226 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.079 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.333853+0000", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.333853+0000", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.333856+0000", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8600000000000001e-06 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.080 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.334360+0000", 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000503704 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.334374+0000", 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4066000000000001e-05 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.334381+0000", 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7059999999999998e-06 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.081 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.334451+0000", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0344000000000003e-05 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.334470+0000", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9113000000000001e-05 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350996+0000", 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016525782999999999 2024-09-20T19:30:58.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for degraded object", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351053+0000", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6900000000000001e-05 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.383315+0000", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.032262129000000001 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.383576+0000", 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00026137000000000003 2024-09-20T19:30:58.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.384085+0000", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00050852299999999998 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388031+0000", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039464340000000004 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388145+0000", 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000113896 2024-09-20T19:30:58.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.391400+0000", 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0032545249999999999 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.391536+0000", 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000136212 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.085 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.391852+0000", 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031639899999999998 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4227 7.8 7:1517470f:::smithi04268048-49:head [write 2844277~798238] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.349643+0000", 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.649336810000001, 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.045840877000000002, 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.086 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4227 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.349643+0000", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.349643+0000", 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.349645+0000", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2819999999999999e-06 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350196+0000", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00055058600000000004 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350211+0000", 2024-09-20T19:30:58.088 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4803e-05 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350224+0000", 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3439e-05 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351268+0000", 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010436040000000001 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.089 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for degraded object", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351317+0000", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8766e-05 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.386039+0000", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.034722063999999997 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.386178+0000", 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013934099999999999 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.386368+0000", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000190371 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389143+0000", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002774257 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.391959+0000", 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0028160170000000001 2024-09-20T19:30:58.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.394894+0000", 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0029351799999999999 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.395064+0000", 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017003499999999999 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.092 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.395484+0000", 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00042013199999999998 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4228 7.8 7:1517470f:::smithi04268048-49:head [setxattr _header (58) in=7b,truncate 3748060] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.350249+0000", 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.64873094, 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.045797006000000001, 2024-09-20T19:30:58.093 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4228 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350249+0000", 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350249+0000", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350251+0000", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6559999999999999e-06 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350259+0000", 2024-09-20T19:30:58.095 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8110000000000003e-06 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350269+0000", 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0913e-05 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350276+0000", 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3149999999999997e-06 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.096 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351387+0000", 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011114440000000001 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for degraded object", 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351567+0000", 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000180213 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388225+0000", 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036657164999999999 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.097 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388328+0000", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000103535 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388541+0000", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021274999999999999 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389586+0000", 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010452320000000001 2024-09-20T19:30:58.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.392030+0000", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0024438419999999999 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.395607+0000", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0035775619999999998 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.395735+0000", 2024-09-20T19:30:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000128002 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.396046+0000", 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031056600000000002 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4229 7.8 7:1517470f:::smithi04268048-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.350307+0000", 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 26.648673372000001, 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.046227559000000001, 2024-09-20T19:30:58.100 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4229 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350307+0000", 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.101 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350307+0000", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350308+0000", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.615e-06 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350315+0000", 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7469999999999999e-06 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.102 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350330+0000", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5285e-05 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.350343+0000", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3200000000000001e-05 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351613+0000", 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001269589 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.103 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for degraded object", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.351662+0000", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.9440000000000001e-05 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.388977+0000", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037314934000000001 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.389122+0000", 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014460300000000001 2024-09-20T19:30:58.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.396173+0000", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070509420000000001 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.396293+0000", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000119998 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.396534+0000", 2024-09-20T19:30:58.105 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024120599999999999 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4471 7.8 7:17ea136f:::smithi04268048-32:head [copy-from ver 443] snapc c3=[c3,c2,c0,bb,b3,b2,aa] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:55.328872+0000", 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 2.6701078699999998, 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036610351999999999, 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.106 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4471 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328872+0000", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328872+0000", 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328876+0000", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1030000000000004e-06 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328890+0000", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3329000000000001e-05 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328911+0000", 2024-09-20T19:30:58.108 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1518999999999998e-05 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328924+0000", 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2516e-05 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.329009+0000", 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5562000000000005e-05 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.109 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.329015+0000", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.19e-06 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.333856+0000", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0048409270000000001 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.334152+0000", 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029605499999999998 2024-09-20T19:30:58.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 4,7", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.342877+0000", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0087245710000000004 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.355383+0000", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012506096 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.362459+0000", 2024-09-20T19:30:58.111 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070759229999999996 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.365339+0000", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0028798650000000001 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.365417+0000", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8251999999999995e-05 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.365482+0000", 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.5444000000000006e-05 2024-09-20T19:30:58.112 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4472 7.8 7:17ea136f:::smithi04268048-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e547)", 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:55.328952+0000", 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 2.6700281050000001, 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023724536000000001, 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:30:58.113 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4472 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328952+0000", 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328952+0000", 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.114 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328953+0000", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1999999999999998e-07 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328957+0000", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4329999999999997e-06 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328973+0000", 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6059999999999999e-05 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.328981+0000", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3520000000000001e-06 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.329066+0000", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5524000000000003e-05 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.329070+0000", 2024-09-20T19:30:58.116 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.1250000000000003e-06 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.334503+0000", 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0054329349999999998 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.334615+0000", 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011167000000000001 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:30:58.117 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.352386+0000", 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017771248999999999 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.352544+0000", 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015774800000000001 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:55.352676+0000", 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013252000000000001 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.119 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:30:58.119 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:30:58.119 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:30:58.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:58.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:58.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:58.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:58.373 INFO:tasks.rados.rados.0.smithi042.stdout:1755: done (0 left) 2024-09-20T19:30:58.373 INFO:tasks.rados.rados.0.smithi042.stdout:1756: write (excl) oid 30 current snap is 195 2024-09-20T19:30:58.373 INFO:tasks.rados.rados.0.smithi042.stdout:1756: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-20T19:30:58.378 INFO:tasks.rados.rados.0.smithi042.stdout:1756: writing smithi04268048-30 from 437147 to 991369 tid 1 2024-09-20T19:30:58.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:58.386 INFO:tasks.rados.rados.0.smithi042.stdout:1756: writing smithi04268048-30 from 1656259 to 2312129 tid 2 2024-09-20T19:30:58.387 INFO:tasks.rados.rados.0.smithi042.stdout:1757: rollback oid 50 current snap is 195 2024-09-20T19:30:58.387 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 168 2024-09-20T19:30:58.387 INFO:tasks.rados.rados.0.smithi042.stdout:1758: write oid 10 current snap is 195 2024-09-20T19:30:58.387 INFO:tasks.rados.rados.0.smithi042.stdout:1758: seq_num 622 ranges {453954=629471,1598388=515881} 2024-09-20T19:30:58.392 INFO:tasks.rados.rados.0.smithi042.stdout:1758: writing smithi04268048-10 from 453954 to 1083425 tid 1 2024-09-20T19:30:58.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:58 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.399 INFO:tasks.rados.rados.0.smithi042.stdout:1758: writing smithi04268048-10 from 1598388 to 2114269 tid 2 2024-09-20T19:30:58.399 INFO:tasks.rados.rados.0.smithi042.stdout:1759: copy_from oid 22 from oid 42 current snap is 195 2024-09-20T19:30:58.399 INFO:tasks.rados.rados.0.smithi042.stdout:1760: copy_from oid 35 from oid 24 current snap is 195 2024-09-20T19:30:58.399 INFO:tasks.rados.rados.0.smithi042.stdout:1761: read oid 36 snap -1 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1761: expect (ObjNum 617 snap 194 seq_num 617) 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1762: read oid 47 snap 176 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1762: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1763: read oid 3 snap -1 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1763: expect (ObjNum 602 snap 191 seq_num 602) 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1764: write oid 12 current snap is 195 2024-09-20T19:30:58.400 INFO:tasks.rados.rados.0.smithi042.stdout:1764: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-20T19:30:58.406 INFO:tasks.rados.rados.0.smithi042.stdout:1764: writing smithi04268048-12 from 470761 to 1175481 tid 1 2024-09-20T19:30:58.413 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:58.414 INFO:tasks.rados.rados.0.smithi042.stdout:1764: writing smithi04268048-12 from 1656870 to 2316409 tid 2 2024-09-20T19:30:58.415 INFO:tasks.rados.rados.0.smithi042.stdout:1765: read oid 32 snap -1 2024-09-20T19:30:58.415 INFO:tasks.rados.rados.0.smithi042.stdout:1765: expect (ObjNum 594 snap 189 seq_num 594) 2024-09-20T19:30:58.415 INFO:tasks.rados.rados.0.smithi042.stdout:1766: snap_create 2024-09-20T19:30:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:58.417 INFO:tasks.rados.rados.0.smithi042.stdout:1763: expect (ObjNum 602 snap 191 seq_num 602) 2024-09-20T19:30:58.417 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:58.494 INFO:tasks.rados.rados.0.smithi042.stdout:1762: expect (ObjNum 483 snap 158 seq_num 483) 2024-09-20T19:30:58.512 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:58.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.349966+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.354781+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-09-20T19:30:58.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.356238+0000 mon.a (mon.0) 1897 : audit [INF] from='osd.3 v1:172.21.15.42:6813/1278675080' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]} : dispatch 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.786679+0000 mon.a (mon.0) 1898 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: audit 2024-09-20T19:30:57.789605+0000 mon.a (mon.0) 1899 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:58 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.222479+0000 mon.a (mon.0) 1900 : cluster [WRN] Health check update: Degraded data redundancy: 169/1131 objects degraded (14.943%), 31 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-09-20T19:30:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:1757: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:30:58.567 INFO:tasks.rados.rados.0.smithi042.stdout:1761: expect (ObjNum 617 snap 194 seq_num 617) 2024-09-20T19:30:58.567 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:58.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:58.691 INFO:tasks.rados.rados.0.smithi042.stdout:1757: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:30:58.691 INFO:tasks.rados.rados.0.smithi042.stdout:1757: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:30:58.691 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 785 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: finishing write tid 1 to smithi04268048-12 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: finishing write tid 2 to smithi04268048-12 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: finishing write tid 3 to smithi04268048-12 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: finishing write tid 4 to smithi04268048-12 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: oid 12 updating version 0 to 635 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: oid 12 version 635 is already newer than 633 2024-09-20T19:30:58.692 INFO:tasks.rados.rados.0.smithi042.stdout:1764: oid 12 version 635 is already newer than 634 2024-09-20T19:30:58.693 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 635 (ObjNum 623 snap 195 seq_num 623) dirty exists 2024-09-20T19:30:58.693 INFO:tasks.rados.rados.0.smithi042.stdout:1764: left oid 12 (ObjNum 623 snap 195 seq_num 623) 2024-09-20T19:30:58.693 INFO:tasks.rados.rados.0.smithi042.stdout:1765: expect (ObjNum 594 snap 189 seq_num 594) 2024-09-20T19:30:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3799850 2024-09-20T19:30:58.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:58.788 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3613584 2024-09-20T19:30:58.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:58.798 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:30:58.867 INFO:tasks.rados.rados.0.smithi042.stdout:1760: finishing copy_from racing read to smithi04268048-35 2024-09-20T19:30:58.867 INFO:tasks.rados.rados.0.smithi042.stdout:1760: finishing copy_from to smithi04268048-35 2024-09-20T19:30:58.867 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 716 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:30:58.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:58.918 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:30:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:58.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:30:59.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:59.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:30:59.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T19:30:59.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:59.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:59.376 INFO:tasks.rados.rados.0.smithi042.stdout:1757: done (10 left) 2024-09-20T19:30:59.376 INFO:tasks.rados.rados.0.smithi042.stdout:1760: done (9 left) 2024-09-20T19:30:59.376 INFO:tasks.rados.rados.0.smithi042.stdout:1761: done (8 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1762: done (7 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1763: done (6 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1764: done (5 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1765: done (4 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1766: done (3 left) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1767: read oid 35 snap -1 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1767: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:30:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:1768: snap_remove snap 193 2024-09-20T19:30:59.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[32075]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:30:59 smithi042 bash[22706]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:30:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.276477+0000 mgr.y (mgr.24491) 491 : cluster [DBG] pgmap v822: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 36 active+undersized, 31 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 174/1095 objects degraded (15.890%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.352034+0000 mon.a (mon.0) 1901 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.372425+0000 mon.a (mon.0) 1902 : cluster [INF] osd.3 v1:172.21.15.42:6813/1278675080 boot 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: cluster 2024-09-20T19:30:58.372496+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[24557]: audit 2024-09-20T19:30:58.372789+0000 mon.a (mon.0) 1904 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:30:59.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:59.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:30:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:30:59.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:59.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:30:59.693 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:30:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:30:59.719 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:30:59.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:59.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:30:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:30:59.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:30:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:30:59.928 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.950 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:30:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:30:59.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:00.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:00.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:00.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:00.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:00.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:00.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:00.380 INFO:tasks.rados.rados.0.smithi042.stdout:1768: done (4 left) 2024-09-20T19:31:00.380 INFO:tasks.rados.rados.0.smithi042.stdout:1769: snap_remove snap 195 2024-09-20T19:31:00.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:00.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:00.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:00.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[22706]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:00 smithi042 bash[32075]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:00.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.589995+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:57.591057+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: cluster 2024-09-20T19:30:59.374893+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:00 smithi162 bash[24557]: audit 2024-09-20T19:30:59.511672+0000 mgr.y (mgr.24491) 492 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:00.835 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:00.860 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:00.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:00.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:01.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:01.117 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:01.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:01.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:01.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:01.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:01.389 INFO:tasks.rados.rados.0.smithi042.stdout:1769: done (4 left) 2024-09-20T19:31:01.389 INFO:tasks.rados.rados.0.smithi042.stdout:1770: write (excl) oid 40 current snap is 196 2024-09-20T19:31:01.389 INFO:tasks.rados.rados.0.smithi042.stdout:1770: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-20T19:31:01.389 INFO:tasks.rados.rados.0.smithi042.stdout:1770: writing smithi04268048-40 from 487568 to 983890 tid 1 2024-09-20T19:31:01.392 INFO:tasks.rados.rados.0.smithi042.stdout:1770: writing smithi04268048-40 from 1431705 to 1951255 tid 2 2024-09-20T19:31:01.401 INFO:tasks.rados.rados.0.smithi042.stdout:1767: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:31:01.431 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 578047 2024-09-20T19:31:01.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:01 smithi162 bash[24557]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.500 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: finishing write tid 1 to smithi04268048-10 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: finishing write tid 2 to smithi04268048-10 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: finishing write tid 3 to smithi04268048-10 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: finishing write tid 4 to smithi04268048-10 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: oid 10 updating version 0 to 607 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: oid 10 version 607 is already newer than 605 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: oid 10 version 607 is already newer than 606 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 607 (ObjNum 622 snap 195 seq_num 622) dirty exists 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1758: left oid 10 (ObjNum 622 snap 195 seq_num 622) 2024-09-20T19:31:01.504 INFO:tasks.rados.rados.0.smithi042.stdout:1759: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1759: finishing copy_from to smithi04268048-22 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 712 (ObjNum 543 snap 177 seq_num 543) dirty exists 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: finishing write tid 1 to smithi04268048-30 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: finishing write tid 2 to smithi04268048-30 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1758: done (4 left) 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1759: done (3 left) 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1767: done (2 left) 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: finishing write tid 3 to smithi04268048-30 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: finishing write tid 4 to smithi04268048-30 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: oid 30 updating version 0 to 855 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: oid 30 updating version 855 to 856 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: oid 30 version 856 is already newer than 854 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 856 (ObjNum 621 snap 195 seq_num 621) dirty exists 2024-09-20T19:31:01.505 INFO:tasks.rados.rados.0.smithi042.stdout:1756: left oid 30 (ObjNum 621 snap 195 seq_num 621) 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: finishing write tid 1 to smithi04268048-40 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: finishing write tid 2 to smithi04268048-40 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: finishing write tid 3 to smithi04268048-40 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: finishing write tid 4 to smithi04268048-40 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: oid 40 updating version 0 to 587 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: oid 40 updating version 587 to 589 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: oid 40 version 589 is already newer than 588 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 589 (ObjNum 624 snap 196 seq_num 624) dirty exists 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: left oid 40 (ObjNum 624 snap 196 seq_num 624) 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1771: read oid 17 snap -1 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1771: expect (ObjNum 514 snap 166 seq_num 514) 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1756: done (2 left) 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1770: done (1 left) 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1772: read oid 28 snap 177 2024-09-20T19:31:01.506 INFO:tasks.rados.rados.0.smithi042.stdout:1772: expect deleted 2024-09-20T19:31:01.507 INFO:tasks.rados.rados.0.smithi042.stdout:1773: copy_from oid 37 from oid 9 current snap is 196 2024-09-20T19:31:01.507 INFO:tasks.rados.rados.0.smithi042.stdout:1774: snap_remove snap 176 2024-09-20T19:31:01.603 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:01.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[32075]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:00.278521+0000 mgr.y (mgr.24491) 493 : cluster [DBG] pgmap v825: 168 pgs: 1 active+clean+remapped, 1 active+clean+remapped+snaptrim_wait, 7 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 33 active+undersized, 27 active+undersized+degraded, 92 active+clean; 335 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 158/1095 objects degraded (14.429%); 8/1095 objects misplaced (0.731%) 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:00.376728+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: audit 2024-09-20T19:31:01.334176+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:01 smithi042 bash[22706]: cluster 2024-09-20T19:31:01.378177+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-20T19:31:01.722 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:31:01.722 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:31:01.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:01.741 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:31:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:31:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:01.759 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:31:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:31:01.777 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:31:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:31:01.805 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:31:01.805 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:31:01.824 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:31:01.824 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:31:01.841 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:31:01.842 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:31:01.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:01.859 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:31:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:31:01.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:01.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:01.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:01.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:02.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:02.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:02.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:02.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:02.382 INFO:tasks.rados.rados.0.smithi042.stdout:1774: done (3 left) 2024-09-20T19:31:02.382 INFO:tasks.rados.rados.0.smithi042.stdout:1775: read oid 36 snap 190 2024-09-20T19:31:02.382 INFO:tasks.rados.rados.0.smithi042.stdout:1775: expect (ObjNum 480 snap 158 seq_num 480) 2024-09-20T19:31:02.382 INFO:tasks.rados.rados.0.smithi042.stdout:1776: rollback oid 29 current snap is 196 2024-09-20T19:31:02.383 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 168 2024-09-20T19:31:02.383 INFO:tasks.rados.rados.0.smithi042.stdout:1777: read oid 49 snap -1 2024-09-20T19:31:02.383 INFO:tasks.rados.rados.0.smithi042.stdout:1777: expect (ObjNum 580 snap 185 seq_num 580) 2024-09-20T19:31:02.383 INFO:tasks.rados.rados.0.smithi042.stdout:1778: snap_remove snap 194 2024-09-20T19:31:02.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:02.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:02.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:02.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:02.635 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:02 smithi162 bash[24557]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:02 smithi162 bash[24557]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:02 smithi162 bash[24557]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:02 smithi162 bash[24557]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:02.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[32075]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[32075]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[32075]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[32075]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[22706]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[22706]: audit 2024-09-20T19:31:01.704093+0000 mon.a (mon.0) 1909 : audit [DBG] from='client.? v1:172.21.15.42:0/2830433296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[22706]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:02 smithi042 bash[22706]: cluster 2024-09-20T19:31:02.381325+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-20T19:31:02.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:02.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:02.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:02.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:02.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:03.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:03.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:03.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:03.238 INFO:tasks.rados.rados.0.smithi042.stdout:1778: done (6 left) 2024-09-20T19:31:03.238 INFO:tasks.rados.rados.0.smithi042.stdout:1779: write oid 42 current snap is 196 2024-09-20T19:31:03.239 INFO:tasks.rados.rados.0.smithi042.stdout:1779: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-20T19:31:03.249 INFO:tasks.rados.rados.0.smithi042.stdout:1779: writing smithi04268048-42 from 504375 to 1075946 tid 1 2024-09-20T19:31:03.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:03.259 INFO:tasks.rados.rados.0.smithi042.stdout:1779: writing smithi04268048-42 from 1490187 to 2153395 tid 2 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:1780: read oid 25 snap -1 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:1780: expect deleted 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:1781: rollback oid 38 current snap is 196 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 177 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:1773: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:1771: expect (ObjNum 514 snap 166 seq_num 514) 2024-09-20T19:31:03.260 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:03.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:03.302 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:03.361 INFO:tasks.rados.rados.0.smithi042.stdout:1773: finishing copy_from to smithi04268048-37 2024-09-20T19:31:03.361 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 757 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:31:03.361 INFO:tasks.rados.rados.0.smithi042.stdout:1775: expect (ObjNum 480 snap 158 seq_num 480) 2024-09-20T19:31:03.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:03.436 INFO:tasks.rados.rados.0.smithi042.stdout:1777: expect (ObjNum 580 snap 185 seq_num 580) 2024-09-20T19:31:03.436 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:03.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:03.537 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:03.542 INFO:tasks.rados.rados.0.smithi042.stdout:1776: finishing rollback tid 0 to smithi04268048-29 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1776: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1776: finishing rollback tid 2 to smithi04268048-29 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 610 (ObjNum 471 snap 152 seq_num 471) dirty exists 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1771: done (8 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1772: done (7 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1773: done (6 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1775: done (5 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1776: done (4 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1777: done (3 left) 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1782: write (excl) oid 8 current snap is 196 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1779: finishing write tid 1 to smithi04268048-42 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1779: finishing write tid 2 to smithi04268048-42 2024-09-20T19:31:03.543 INFO:tasks.rados.rados.0.smithi042.stdout:1782: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-20T19:31:03.549 INFO:tasks.rados.rados.0.smithi042.stdout:1782: writing smithi04268048-8 from 521182 to 1168002 tid 1 2024-09-20T19:31:03.556 INFO:tasks.rados.rados.0.smithi042.stdout:1782: writing smithi04268048-8 from 1948669 to 2471888 tid 2 2024-09-20T19:31:03.556 INFO:tasks.rados.rados.0.smithi042.stdout:1783: snap_remove snap 190 2024-09-20T19:31:03.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:03.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:03.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:03 smithi162 bash[24557]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:03.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[22706]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:02.280405+0000 mgr.y (mgr.24491) 494 : cluster [DBG] pgmap v828: 168 pgs: 1 active+clean+remapped, 7 peering, 6 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 15 active+undersized, 19 active+undersized+degraded, 112 active+clean; 331 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s; 87/1080 objects degraded (8.056%); 8/1080 objects misplaced (0.741%); 7.2 MiB/s, 8 objects/s recovering 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:03.224600+0000 mon.a (mon.0) 1911 : cluster [WRN] Health check update: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:03 smithi042 bash[32075]: cluster 2024-09-20T19:31:03.236955+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-20T19:31:03.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:03.799 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:03.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:03.929 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:03.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:04.032 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:04.046 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:04.061 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:04.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:04.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:04.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:04.316 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:04.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1783: done (4 left) 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1784: read oid 21 snap -1 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1784: expect (ObjNum 603 snap 191 seq_num 603) 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1785: snap_create 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1779: finishing write tid 3 to smithi04268048-42 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1779: finishing write tid 4 to smithi04268048-42 2024-09-20T19:31:04.429 INFO:tasks.rados.rados.0.smithi042.stdout:1779: oid 42 updating version 0 to 810 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1779: oid 42 version 810 is already newer than 808 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1779: oid 42 version 810 is already newer than 809 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 810 (ObjNum 625 snap 196 seq_num 625) dirty exists 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1779: left oid 42 (ObjNum 625 snap 196 seq_num 625) 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1781: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1781: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1781: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 613 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: finishing write tid 1 to smithi04268048-8 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: finishing write tid 2 to smithi04268048-8 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: finishing write tid 3 to smithi04268048-8 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: finishing write tid 4 to smithi04268048-8 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: oid 8 updating version 0 to 640 2024-09-20T19:31:04.430 INFO:tasks.rados.rados.0.smithi042.stdout:1782: oid 8 version 640 is already newer than 638 2024-09-20T19:31:04.431 INFO:tasks.rados.rados.0.smithi042.stdout:1782: oid 8 version 640 is already newer than 639 2024-09-20T19:31:04.431 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 640 (ObjNum 626 snap 196 seq_num 626) dirty exists 2024-09-20T19:31:04.431 INFO:tasks.rados.rados.0.smithi042.stdout:1782: left oid 8 (ObjNum 626 snap 196 seq_num 626) 2024-09-20T19:31:04.446 INFO:tasks.rados.rados.0.smithi042.stdout:1784: expect (ObjNum 603 snap 191 seq_num 603) 2024-09-20T19:31:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:04.479 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:04.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:04.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:04.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:04.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:04 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:04 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:04 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:04 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:04.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.834 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:31:04.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:04.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.410951+0000 mon.a (mon.0) 1913 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/1080 objects degraded (8.056%), 19 pgs degraded, 15 pgs undersized) 2024-09-20T19:31:04.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.846 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:04 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.427506+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-20T19:31:04.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:04.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:05.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:05.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:05.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:05.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:05.437 INFO:tasks.rados.rados.0.smithi042.stdout:1779: done (5 left) 2024-09-20T19:31:05.437 INFO:tasks.rados.rados.0.smithi042.stdout:1780: done (4 left) 2024-09-20T19:31:05.437 INFO:tasks.rados.rados.0.smithi042.stdout:1781: done (3 left) 2024-09-20T19:31:05.438 INFO:tasks.rados.rados.0.smithi042.stdout:1782: done (2 left) 2024-09-20T19:31:05.438 INFO:tasks.rados.rados.0.smithi042.stdout:1784: done (1 left) 2024-09-20T19:31:05.438 INFO:tasks.rados.rados.0.smithi042.stdout:1785: done (0 left) 2024-09-20T19:31:05.438 INFO:tasks.rados.rados.0.smithi042.stdout:1786: write oid 38 current snap is 197 2024-09-20T19:31:05.438 INFO:tasks.rados.rados.0.smithi042.stdout:1786: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-20T19:31:05.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:05.453 INFO:tasks.rados.rados.0.smithi042.stdout:1786: writing smithi04268048-38 from 537989 to 1260058 tid 1 2024-09-20T19:31:05.477 INFO:tasks.rados.rados.0.smithi042.stdout:1786: writing smithi04268048-38 from 1890798 to 2537989 tid 2 2024-09-20T19:31:05.477 INFO:tasks.rados.rados.0.smithi042.stdout:1787: write oid 25 current snap is 197 2024-09-20T19:31:05.477 INFO:tasks.rados.rados.0.smithi042.stdout:1786: finishing write tid 1 to smithi04268048-38 2024-09-20T19:31:05.477 INFO:tasks.rados.rados.0.smithi042.stdout:1787: seq_num 628 ranges {554796=797318,1949280=526888} 2024-09-20T19:31:05.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:05.485 INFO:tasks.rados.rados.0.smithi042.stdout:1787: writing smithi04268048-25 from 554796 to 1352114 tid 1 2024-09-20T19:31:05.492 INFO:tasks.rados.rados.0.smithi042.stdout:1787: writing smithi04268048-25 from 1949280 to 2476168 tid 2 2024-09-20T19:31:05.492 INFO:tasks.rados.rados.0.smithi042.stdout:1788: snap_create 2024-09-20T19:31:05.492 INFO:tasks.rados.rados.0.smithi042.stdout:1786: finishing write tid 2 to smithi04268048-38 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:1786: finishing write tid 3 to smithi04268048-38 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:1786: finishing write tid 4 to smithi04268048-38 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:1786: oid 38 updating version 0 to 616 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:1786: oid 38 version 616 is already newer than 615 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:1786: oid 38 version 616 is already newer than 614 2024-09-20T19:31:05.493 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 616 (ObjNum 627 snap 197 seq_num 627) dirty exists 2024-09-20T19:31:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:1786: left oid 38 (ObjNum 627 snap 197 seq_num 627) 2024-09-20T19:31:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:1787: finishing write tid 1 to smithi04268048-25 2024-09-20T19:31:05.496 INFO:tasks.rados.rados.0.smithi042.stdout:1787: finishing write tid 2 to smithi04268048-25 2024-09-20T19:31:05.496 INFO:tasks.rados.rados.0.smithi042.stdout:1787: finishing write tid 3 to smithi04268048-25 2024-09-20T19:31:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:1787: finishing write tid 4 to smithi04268048-25 2024-09-20T19:31:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:1787: oid 25 updating version 0 to 769 2024-09-20T19:31:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:1787: oid 25 updating version 769 to 770 2024-09-20T19:31:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:1787: oid 25 version 770 is already newer than 768 2024-09-20T19:31:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 770 (ObjNum 628 snap 197 seq_num 628) dirty exists 2024-09-20T19:31:05.498 INFO:tasks.rados.rados.0.smithi042.stdout:1787: left oid 25 (ObjNum 628 snap 197 seq_num 628) 2024-09-20T19:31:05.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.592 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:05 smithi162 bash[24557]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.602 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:05.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:05.710 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:05.710 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:05.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:05.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[22706]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:04.281207+0000 mgr.y (mgr.24491) 495 : cluster [DBG] pgmap v831: 168 pgs: 7 peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 140 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 20 op/s; 14 MiB/s, 26 objects/s recovering 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.249375+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.250472+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:05 smithi042 bash[32075]: cluster 2024-09-20T19:31:05.432035+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-20T19:31:05.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:05.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:06.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:06.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:06.158 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:06.158 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:06.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:06.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:06.279 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:06.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:06.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:06.480 INFO:tasks.rados.rados.0.smithi042.stdout:1786: done (2 left) 2024-09-20T19:31:06.480 INFO:tasks.rados.rados.0.smithi042.stdout:1787: done (1 left) 2024-09-20T19:31:06.480 INFO:tasks.rados.rados.0.smithi042.stdout:1788: done (0 left) 2024-09-20T19:31:06.480 INFO:tasks.rados.rados.0.smithi042.stdout:1789: write (excl) oid 6 current snap is 198 2024-09-20T19:31:06.480 INFO:tasks.rados.rados.0.smithi042.stdout:1789: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-20T19:31:06.485 INFO:tasks.rados.rados.0.smithi042.stdout:1789: writing smithi04268048-6 from 571603 to 1044170 tid 1 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1789: writing smithi04268048-6 from 1607762 to 2278308 tid 2 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1790: read oid 27 snap -1 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1790: expect deleted 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1791: read oid 9 snap -1 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1791: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1792: copy_from oid 26 from oid 48 current snap is 198 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1793: read oid 39 snap -1 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1793: expect (ObjNum 563 snap 180 seq_num 563) 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1794: write (excl) oid 2 current snap is 198 2024-09-20T19:31:06.493 INFO:tasks.rados.rados.0.smithi042.stdout:1794: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-09-20T19:31:06.499 INFO:tasks.rados.rados.0.smithi042.stdout:1794: writing smithi04268048-2 from 588410 to 1136226 tid 1 2024-09-20T19:31:06.505 INFO:tasks.rados.rados.0.smithi042.stdout:1794: writing smithi04268048-2 from 1666244 to 2080448 tid 2 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1794: writing smithi04268048-2 from 2484603 to 2588410 tid 3 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1795: copy_from oid 8 from oid 22 current snap is 198 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1789: finishing write tid 1 to smithi04268048-6 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1790: done (6 left) 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1796: read oid 41 snap 196 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1796: expect deleted 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1797: copy_from oid 5 from oid 33 current snap is 198 2024-09-20T19:31:06.509 INFO:tasks.rados.rados.0.smithi042.stdout:1791: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:31:06.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T19:31:06.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:06.613 INFO:tasks.rados.rados.0.smithi042.stdout:1789: finishing write tid 2 to smithi04268048-6 2024-09-20T19:31:06.614 INFO:tasks.rados.rados.0.smithi042.stdout:1793: expect (ObjNum 563 snap 180 seq_num 563) 2024-09-20T19:31:06.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:06.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:06.690 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3036735 2024-09-20T19:31:06.766 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2796842 2024-09-20T19:31:06.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:06.784 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 643882 2024-09-20T19:31:06.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: finishing write tid 3 to smithi04268048-6 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: finishing write tid 4 to smithi04268048-6 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: oid 6 updating version 0 to 644 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: oid 6 version 644 is already newer than 643 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: oid 6 version 644 is already newer than 642 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 644 (ObjNum 629 snap 198 seq_num 629) dirty exists 2024-09-20T19:31:06.876 INFO:tasks.rados.rados.0.smithi042.stdout:1789: left oid 6 (ObjNum 629 snap 198 seq_num 629) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1792: finishing copy_from to smithi04268048-26 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1792: got expected ENOENT (src dne) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1792: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1794: finishing write tid 1 to smithi04268048-2 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1794: finishing write tid 2 to smithi04268048-2 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1789: done (7 left) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1791: done (6 left) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1792: done (5 left) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1793: done (4 left) 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1798: write (excl) oid 7 current snap is 198 2024-09-20T19:31:06.877 INFO:tasks.rados.rados.0.smithi042.stdout:1798: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-20T19:31:06.883 INFO:tasks.rados.rados.0.smithi042.stdout:1798: writing smithi04268048-7 from 605217 to 1344635 tid 1 2024-09-20T19:31:06.890 INFO:tasks.rados.rados.0.smithi042.stdout:1798: writing smithi04268048-7 from 2124726 to 2605217 tid 2 2024-09-20T19:31:06.890 INFO:tasks.rados.rados.0.smithi042.stdout:1799: write oid 18 current snap is 198 2024-09-20T19:31:06.890 INFO:tasks.rados.rados.0.smithi042.stdout:1799: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-20T19:31:06.896 INFO:tasks.rados.rados.0.smithi042.stdout:1799: writing smithi04268048-18 from 622024 to 1036691 tid 1 2024-09-20T19:31:06.903 INFO:tasks.rados.rados.0.smithi042.stdout:1799: writing smithi04268048-18 from 1783208 to 2201081 tid 2 2024-09-20T19:31:06.903 INFO:tasks.rados.rados.0.smithi042.stdout:1800: read oid 11 snap -1 2024-09-20T19:31:06.903 INFO:tasks.rados.rados.0.smithi042.stdout:1800: expect deleted 2024-09-20T19:31:06.903 INFO:tasks.rados.rados.0.smithi042.stdout:1801: rollback oid 27 current snap is 198 2024-09-20T19:31:06.903 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 197 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: finishing write tid 3 to smithi04268048-2 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1796: done (7 left) 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1802: snap_create 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: finishing write tid 4 to smithi04268048-2 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: finishing write tid 5 to smithi04268048-2 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: oid 2 updating version 0 to 593 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: oid 2 version 593 is already newer than 591 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: oid 2 version 593 is already newer than 592 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: oid 2 version 593 is already newer than 590 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 593 (ObjNum 630 snap 198 seq_num 630) dirty exists 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1794: left oid 2 (ObjNum 630 snap 198 seq_num 630) 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1797: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1797: finishing copy_from to smithi04268048-5 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 586 (ObjNum 595 snap 189 seq_num 595) dirty exists 2024-09-20T19:31:06.904 INFO:tasks.rados.rados.0.smithi042.stdout:1795: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1795: finishing copy_from to smithi04268048-8 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 712 (ObjNum 543 snap 177 seq_num 543) dirty exists 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: finishing write tid 1 to smithi04268048-7 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: finishing write tid 2 to smithi04268048-7 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: finishing write tid 3 to smithi04268048-7 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: finishing write tid 4 to smithi04268048-7 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: oid 7 updating version 0 to 629 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: oid 7 updating version 629 to 630 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: oid 7 version 630 is already newer than 628 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 630 (ObjNum 631 snap 198 seq_num 631) dirty exists 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1798: left oid 7 (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:31:06.905 INFO:tasks.rados.rados.0.smithi042.stdout:1799: finishing write tid 1 to smithi04268048-18 2024-09-20T19:31:06.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1801: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: finishing write tid 2 to smithi04268048-18 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: finishing write tid 3 to smithi04268048-18 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: finishing write tid 4 to smithi04268048-18 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: oid 18 updating version 0 to 863 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: oid 18 version 863 is already newer than 862 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: oid 18 version 863 is already newer than 861 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 863 (ObjNum 632 snap 198 seq_num 632) dirty exists 2024-09-20T19:31:06.907 INFO:tasks.rados.rados.0.smithi042.stdout:1799: left oid 18 (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:06.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:06.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:07.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:07.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:07.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:07.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:07.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:07.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:07.497 INFO:tasks.rados.rados.0.smithi042.stdout:1794: done (7 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1795: done (6 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1797: done (5 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1798: done (4 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1799: done (3 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1800: done (2 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1801: done (1 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1802: done (0 left) 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1803: write (excl) oid 26 current snap is 199 2024-09-20T19:31:07.498 INFO:tasks.rados.rados.0.smithi042.stdout:1803: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-20T19:31:07.499 INFO:tasks.rados.rados.0.smithi042.stdout:1803: writing smithi04268048-26 from 638831 to 1128747 tid 1 2024-09-20T19:31:07.510 INFO:tasks.rados.rados.0.smithi042.stdout:1803: writing smithi04268048-26 from 1841690 to 2519574 tid 2 2024-09-20T19:31:07.510 INFO:tasks.rados.rados.0.smithi042.stdout:1804: write (excl) oid 49 current snap is 199 2024-09-20T19:31:07.510 INFO:tasks.rados.rados.0.smithi042.stdout:1803: finishing write tid 1 to smithi04268048-26 2024-09-20T19:31:07.510 INFO:tasks.rados.rados.0.smithi042.stdout:1804: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-20T19:31:07.517 INFO:tasks.rados.rados.0.smithi042.stdout:1804: writing smithi04268048-49 from 655638 to 1220803 tid 1 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1804: writing smithi04268048-49 from 1900172 to 2321714 tid 2 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1805: delete oid 20 current snap is 199 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1803: finishing write tid 2 to smithi04268048-26 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1803: finishing write tid 3 to smithi04268048-26 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1803: finishing write tid 4 to smithi04268048-26 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1803: oid 26 updating version 0 to 632 2024-09-20T19:31:07.524 INFO:tasks.rados.rados.0.smithi042.stdout:1803: oid 26 version 632 is already newer than 631 2024-09-20T19:31:07.525 INFO:tasks.rados.rados.0.smithi042.stdout:1803: oid 26 updating version 632 to 633 2024-09-20T19:31:07.525 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 633 (ObjNum 633 snap 199 seq_num 633) dirty exists 2024-09-20T19:31:07.525 INFO:tasks.rados.rados.0.smithi042.stdout:1803: left oid 26 (ObjNum 633 snap 199 seq_num 633) 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1803: done (2 left) 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1805: done (1 left) 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1806: read oid 41 snap -1 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1806: expect deleted 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1807: rollback oid 44 current snap is 199 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 177 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1808: read oid 48 snap -1 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1808: expect deleted 2024-09-20T19:31:07.527 INFO:tasks.rados.rados.0.smithi042.stdout:1809: snap_create 2024-09-20T19:31:07.529 INFO:tasks.rados.rados.0.smithi042.stdout:1804: finishing write tid 1 to smithi04268048-49 2024-09-20T19:31:07.529 INFO:tasks.rados.rados.0.smithi042.stdout:1804: finishing write tid 2 to smithi04268048-49 2024-09-20T19:31:07.529 INFO:tasks.rados.rados.0.smithi042.stdout:1804: finishing write tid 3 to smithi04268048-49 2024-09-20T19:31:07.529 INFO:tasks.rados.rados.0.smithi042.stdout:1804: finishing write tid 4 to smithi04268048-49 2024-09-20T19:31:07.530 INFO:tasks.rados.rados.0.smithi042.stdout:1804: oid 49 updating version 0 to 1062 2024-09-20T19:31:07.530 INFO:tasks.rados.rados.0.smithi042.stdout:1804: oid 49 updating version 1062 to 1063 2024-09-20T19:31:07.530 INFO:tasks.rados.rados.0.smithi042.stdout:1804: oid 49 updating version 1063 to 1064 2024-09-20T19:31:07.530 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1064 (ObjNum 634 snap 199 seq_num 634) dirty exists 2024-09-20T19:31:07.530 INFO:tasks.rados.rados.0.smithi042.stdout:1804: left oid 49 (ObjNum 634 snap 199 seq_num 634) 2024-09-20T19:31:07.531 INFO:tasks.rados.rados.0.smithi042.stdout:1807: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:31:07.531 INFO:tasks.rados.rados.0.smithi042.stdout:1807: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:31:07.532 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1017 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:31:07.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:07.561 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:07.575 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T19:31:07.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:07 smithi162 bash[24557]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[32075]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.281705+0000 mgr.y (mgr.24491) 496 : cluster [DBG] pgmap v834: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 329 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 2 op/s; 7.0 MiB/s, 19 objects/s recovering 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.477887+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.523568+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: cluster 2024-09-20T19:31:06.524415+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.918356+0000 mon.a (mon.0) 1917 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.923952+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.979068+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:07 smithi042 bash[22706]: audit 2024-09-20T19:31:06.983461+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:07.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:07.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:07.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:07.936 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:07.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:08.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:08.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:08.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:08.192 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:08.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:08.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:08.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1804: done (4 left) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1806: done (3 left) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1807: done (2 left) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1808: done (1 left) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1809: done (0 left) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1810: read oid 46 snap -1 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1810: expect (ObjNum 531 snap 173 seq_num 531) 2024-09-20T19:31:08.507 INFO:tasks.rados.rados.0.smithi042.stdout:1811: snap_create 2024-09-20T19:31:08.512 INFO:tasks.rados.rados.0.smithi042.stdout:1810: expect (ObjNum 531 snap 173 seq_num 531) 2024-09-20T19:31:08.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:08.569 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:08.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:08.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:08.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[22706]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.704 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:08 smithi042 bash[32075]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: cluster 2024-09-20T19:31:07.489980+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.604338+0000 mon.a (mon.0) 1922 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.605092+0000 mon.a (mon.0) 1923 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.610207+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.772265+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25447 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:08 smithi162 bash[24557]: audit 2024-09-20T19:31:07.777012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? v1:172.21.15.42:0/2175015310' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:08.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:08.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:08.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:08.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:09.073 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:09.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:09.231 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:09.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:09.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:09.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:09.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:09.504 INFO:tasks.rados.rados.0.smithi042.stdout:1810: done (1 left) 2024-09-20T19:31:09.505 INFO:tasks.rados.rados.0.smithi042.stdout:1811: done (0 left) 2024-09-20T19:31:09.505 INFO:tasks.rados.rados.0.smithi042.stdout:1812: snap_remove snap 196 2024-09-20T19:31:09.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:09.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.612 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:09 smithi162 bash[24557]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:09.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:09.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[32075]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.743 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.282449+0000 mgr.y (mgr.24491) 497 : cluster [DBG] pgmap v837: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 310 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 23 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.486900+0000 mon.a (mon.0) 1926 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: audit 2024-09-20T19:31:08.498599+0000 mon.a (mon.0) 1927 : audit [INF] from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.744 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:09 smithi042 bash[22706]: cluster 2024-09-20T19:31:08.506575+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-20T19:31:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:09.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:09.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:09.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:09.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:10.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:10.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:10.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:10.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:10.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:10.400 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:10.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:10.521 INFO:tasks.rados.rados.0.smithi042.stdout:1812: done (0 left) 2024-09-20T19:31:10.522 INFO:tasks.rados.rados.0.smithi042.stdout:1813: read oid 29 snap -1 2024-09-20T19:31:10.522 INFO:tasks.rados.rados.0.smithi042.stdout:1813: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:10.522 INFO:tasks.rados.rados.0.smithi042.stdout:1814: snap_remove snap 177 2024-09-20T19:31:10.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:10.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:10.667 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:10.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:10.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:10 smithi162 bash[24557]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:10 smithi162 bash[24557]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:10 smithi162 bash[24557]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:10 smithi162 bash[24557]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.798 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:10.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:10.927 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[22706]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[22706]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[22706]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[22706]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[32075]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[32075]: cluster 2024-09-20T19:31:09.504601+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-20T19:31:10.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[32075]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:10 smithi042 bash[32075]: audit 2024-09-20T19:31:09.519842+0000 mgr.y (mgr.24491) 498 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:10.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:10.992 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:11.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:11.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:11.162 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:11.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:11.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:11.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:11.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:31:11.529 INFO:tasks.rados.rados.0.smithi042.stdout:1814: done (1 left) 2024-09-20T19:31:11.529 INFO:tasks.rados.rados.0.smithi042.stdout:1815: copy_from oid 47 from oid 40 current snap is 201 2024-09-20T19:31:11.529 INFO:tasks.rados.rados.0.smithi042.stdout:1816: snap_create 2024-09-20T19:31:11.529 INFO:tasks.rados.rados.0.smithi042.stdout:1813: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:11.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:11.560 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1576182 2024-09-20T19:31:11.572 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1677165 2024-09-20T19:31:11.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:11.683 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 859148 2024-09-20T19:31:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:11.746 INFO:tasks.rados.rados.0.smithi042.stdout:1815: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:31:11.746 INFO:tasks.rados.rados.0.smithi042.stdout:1815: finishing copy_from to smithi04268048-47 2024-09-20T19:31:11.746 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 589 (ObjNum 624 snap 196 seq_num 624) dirty exists 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:11 smithi162 bash[24557]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:11.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:11.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[32075]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.208275+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.209448+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.283219+0000 mgr.y (mgr.24491) 499 : cluster [DBG] pgmap v840: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 311 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:11 smithi042 bash[22706]: cluster 2024-09-20T19:31:10.519190+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-20T19:31:11.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:11.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:11.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:12.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:12.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:12.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:12.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:12.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:12.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:12.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:12.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:12.534 INFO:tasks.rados.rados.0.smithi042.stdout:1813: done (2 left) 2024-09-20T19:31:12.534 INFO:tasks.rados.rados.0.smithi042.stdout:1815: done (1 left) 2024-09-20T19:31:12.534 INFO:tasks.rados.rados.0.smithi042.stdout:1816: done (0 left) 2024-09-20T19:31:12.535 INFO:tasks.rados.rados.0.smithi042.stdout:1817: snap_create 2024-09-20T19:31:12.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:12.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:12.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:12 smithi042 bash[32075]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:12 smithi042 bash[32075]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:12 smithi042 bash[22706]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:12 smithi042 bash[22706]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:12.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:12 smithi162 bash[24557]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:12 smithi162 bash[24557]: cluster 2024-09-20T19:31:11.527378+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-20T19:31:12.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:12.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:12.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:12.994 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:12.994 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:13.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:13.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:13.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:13.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:13.242 INFO:tasks.rados.rados.0.smithi042.stdout:1817: done (0 left) 2024-09-20T19:31:13.242 INFO:tasks.rados.rados.0.smithi042.stdout:1818: snap_create 2024-09-20T19:31:13.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:13.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:13.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:13.524 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:13.627 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[22706]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:13 smithi042 bash[32075]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:13.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:13.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:12.284704+0000 mgr.y (mgr.24491) 500 : cluster [DBG] pgmap v843: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 288 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:12.534583+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: audit 2024-09-20T19:31:12.825830+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: audit 2024-09-20T19:31:12.827066+0000 mon.a (mon.0) 1934 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.786 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:13 smithi162 bash[24557]: cluster 2024-09-20T19:31:13.241071+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-20T19:31:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:13.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T19:31:13.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:13.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:13.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:14.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:14.040 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:14.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:14.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:14.176 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:31:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:31:14.194 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:31:14.194 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:31:14.211 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:31:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:31:14.228 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:31:14.228 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:31:14.241 INFO:tasks.rados.rados.0.smithi042.stdout:1818: done (0 left) 2024-09-20T19:31:14.241 INFO:tasks.rados.rados.0.smithi042.stdout:1819: snap_remove snap 203 2024-09-20T19:31:14.253 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:31:14.253 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:31:14.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:14.274 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:31:14.274 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:31:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:14.302 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:31:14.302 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:31:14.323 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:31:14.323 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:31:14.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:14.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:14.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:14.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[32075]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[32075]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[32075]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[32075]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[22706]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[22706]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[22706]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:14 smithi042 bash[22706]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.682 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:14.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:14.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:14 smithi162 bash[24557]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:14 smithi162 bash[24557]: audit 2024-09-20T19:31:14.156504+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.42:0/4079628668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:14.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:14 smithi162 bash[24557]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:14 smithi162 bash[24557]: cluster 2024-09-20T19:31:14.239492+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-20T19:31:14.809 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:14.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:14.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:15.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:15.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:15.173 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:15.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:15.243 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T19:31:15.252 INFO:tasks.rados.rados.0.smithi042.stdout:1819: done (0 left) 2024-09-20T19:31:15.252 INFO:tasks.rados.rados.0.smithi042.stdout:1820: snap_remove snap 199 2024-09-20T19:31:15.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:15.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:15.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:15.460 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:15.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:15.586 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:15.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:15.700 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:15 smithi162 bash[24557]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:15.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:15.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[32075]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: cluster 2024-09-20T19:31:14.285182+0000 mgr.y (mgr.24491) 501 : cluster [DBG] pgmap v847: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 276 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 899 KiB/s wr, 8 op/s 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: audit 2024-09-20T19:31:14.673832+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:31:15.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: audit 2024-09-20T19:31:15.243337+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.42:0/1030307583' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:31:15.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:15 smithi042 bash[22706]: cluster 2024-09-20T19:31:15.248680+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-20T19:31:15.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:15.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:15.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:15.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:16.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T19:31:16.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:16.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:16.256 INFO:tasks.rados.rados.0.smithi042.stdout:1820: done (0 left) 2024-09-20T19:31:16.256 INFO:tasks.rados.rados.0.smithi042.stdout:1821: read oid 7 snap 198 2024-09-20T19:31:16.256 INFO:tasks.rados.rados.0.smithi042.stdout:1821: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:31:16.256 INFO:tasks.rados.rados.0.smithi042.stdout:1822: delete oid 14 current snap is 204 2024-09-20T19:31:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:1822: done (1 left) 2024-09-20T19:31:16.265 INFO:tasks.rados.rados.0.smithi042.stdout:1823: snap_remove snap 200 2024-09-20T19:31:16.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:16.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:16.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:16.429 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:16.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:16.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:16.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:16.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:16.758 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:16.891 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:16.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:16.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:17.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:17.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:17.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:17.179 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:17.265 INFO:tasks.rados.rados.0.smithi042.stdout:1823: done (1 left) 2024-09-20T19:31:17.265 INFO:tasks.rados.rados.0.smithi042.stdout:1824: delete oid 49 current snap is 204 2024-09-20T19:31:17.265 INFO:tasks.rados.rados.0.smithi042.stdout:1821: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:31:17.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:17.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:31:17.308 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:17.311 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[32075]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.311 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[32075]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.311 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[32075]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.311 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[32075]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.315 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1008272 2024-09-20T19:31:17.338 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:17.368 INFO:tasks.rados.rados.0.smithi042.stdout:1821: done (1 left) 2024-09-20T19:31:17.368 INFO:tasks.rados.rados.0.smithi042.stdout:1824: done (0 left) 2024-09-20T19:31:17.368 INFO:tasks.rados.rados.0.smithi042.stdout:1825: delete oid 39 current snap is 204 2024-09-20T19:31:17.376 INFO:tasks.rados.rados.0.smithi042.stdout:1825: done (0 left) 2024-09-20T19:31:17.376 INFO:tasks.rados.rados.0.smithi042.stdout:1826: delete oid 16 current snap is 204 2024-09-20T19:31:17.382 INFO:tasks.rados.rados.0.smithi042.stdout:1826: done (0 left) 2024-09-20T19:31:17.382 INFO:tasks.rados.rados.0.smithi042.stdout:1827: read oid 42 snap -1 2024-09-20T19:31:17.382 INFO:tasks.rados.rados.0.smithi042.stdout:1827: expect (ObjNum 625 snap 196 seq_num 625) 2024-09-20T19:31:17.382 INFO:tasks.rados.rados.0.smithi042.stdout:1828: snap_create 2024-09-20T19:31:17.391 INFO:tasks.rados.rados.0.smithi042.stdout:1827: expect (ObjNum 625 snap 196 seq_num 625) 2024-09-20T19:31:17.412 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:17.432 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:17.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:17.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:17 smithi162 bash[24557]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:17 smithi162 bash[24557]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:17 smithi162 bash[24557]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:17 smithi162 bash[24557]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:17.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[22706]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[22706]: cluster 2024-09-20T19:31:16.253090+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-20T19:31:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[22706]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:17 smithi042 bash[22706]: cluster 2024-09-20T19:31:16.285585+0000 mgr.y (mgr.24491) 502 : cluster [DBG] pgmap v850: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 273 MiB data, 889 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 2 op/s 2024-09-20T19:31:17.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:17.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:17.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:17.959 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:17.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:18.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:18.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:18.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:18.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:18.245 INFO:tasks.rados.rados.0.smithi042.stdout:1827: done (1 left) 2024-09-20T19:31:18.245 INFO:tasks.rados.rados.0.smithi042.stdout:1828: done (0 left) 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1829: read oid 39 snap 202 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1829: expect (ObjNum 563 snap 180 seq_num 563) 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1830: read oid 38 snap -1 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1830: expect (ObjNum 627 snap 197 seq_num 627) 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1831: write oid 34 current snap is 205 2024-09-20T19:31:18.246 INFO:tasks.rados.rados.0.smithi042.stdout:1831: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-20T19:31:18.270 INFO:tasks.rados.rados.0.smithi042.stdout:1831: writing smithi04268048-34 from 756480 to 1489492 tid 1 2024-09-20T19:31:18.283 INFO:tasks.rados.rados.0.smithi042.stdout:1831: writing smithi04268048-34 from 2134711 to 2567260 tid 2 2024-09-20T19:31:18.283 INFO:tasks.rados.rados.0.smithi042.stdout:1832: rollback oid 50 current snap is 205 2024-09-20T19:31:18.283 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 202 2024-09-20T19:31:18.283 INFO:tasks.rados.rados.0.smithi042.stdout:1830: expect (ObjNum 627 snap 197 seq_num 627) 2024-09-20T19:31:18.283 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:18.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:18.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:18 smithi162 bash[24557]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:18 smithi162 bash[24557]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:18 smithi162 bash[24557]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:18 smithi162 bash[24557]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:18.417 INFO:tasks.rados.rados.0.smithi042.stdout:1829: expect (ObjNum 563 snap 180 seq_num 563) 2024-09-20T19:31:18.449 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:18.496 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3315830 2024-09-20T19:31:18.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1694575 2024-09-20T19:31:18.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:18.596 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 660222 2024-09-20T19:31:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:18.625 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:18.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[32075]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[32075]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[32075]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[32075]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[22706]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[22706]: cluster 2024-09-20T19:31:17.263742+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-20T19:31:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[22706]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:18 smithi042 bash[22706]: cluster 2024-09-20T19:31:18.243682+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-20T19:31:18.685 INFO:tasks.rados.rados.0.smithi042.stdout:1831: finishing write tid 1 to smithi04268048-34 2024-09-20T19:31:18.685 INFO:tasks.rados.rados.0.smithi042.stdout:1831: finishing write tid 2 to smithi04268048-34 2024-09-20T19:31:18.685 INFO:tasks.rados.rados.0.smithi042.stdout:1829: done (3 left) 2024-09-20T19:31:18.685 INFO:tasks.rados.rados.0.smithi042.stdout:1830: done (2 left) 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1833: copy_from oid 24 from oid 5 current snap is 205 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1832: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: finishing write tid 3 to smithi04268048-34 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1832: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: finishing write tid 4 to smithi04268048-34 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: oid 34 updating version 0 to 718 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: oid 34 updating version 718 to 719 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: oid 34 version 719 is already newer than 717 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 719 (ObjNum 640 snap 205 seq_num 640) dirty exists 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1831: left oid 34 (ObjNum 640 snap 205 seq_num 640) 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:1832: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:31:18.686 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 807 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1831: done (2 left) 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1832: done (1 left) 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1834: copy_from oid 3 from oid 16 current snap is 205 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1835: read oid 27 snap 168 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1835: expect deleted 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1836: write oid 22 current snap is 205 2024-09-20T19:31:18.687 INFO:tasks.rados.rados.0.smithi042.stdout:1836: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-20T19:31:18.693 INFO:tasks.rados.rados.0.smithi042.stdout:1836: writing smithi04268048-22 from 773287 to 1181548 tid 1 2024-09-20T19:31:18.701 INFO:tasks.rados.rados.0.smithi042.stdout:1836: writing smithi04268048-22 from 1793193 to 2369400 tid 2 2024-09-20T19:31:18.701 INFO:tasks.rados.rados.0.smithi042.stdout:1837: snap_remove snap 201 2024-09-20T19:31:18.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:18.829 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:18.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:18.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:18.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:18.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:19.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:19.113 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:19.216 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:19.271 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:31:19.277 INFO:tasks.rados.rados.0.smithi042.stdout:1837: done (4 left) 2024-09-20T19:31:19.277 INFO:tasks.rados.rados.0.smithi042.stdout:1838: copy_from oid 38 from oid 29 current snap is 205 2024-09-20T19:31:19.277 INFO:tasks.rados.rados.0.smithi042.stdout:1839: snap_remove snap 198 2024-09-20T19:31:19.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:19.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:19.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[32075]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[32075]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[32075]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.488 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[32075]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[22706]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[22706]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[22706]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:19 smithi042 bash[22706]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.503 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:19.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:19.617 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:19.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:19 smithi162 bash[24557]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:19 smithi162 bash[24557]: cluster 2024-09-20T19:31:18.286460+0000 mgr.y (mgr.24491) 503 : cluster [DBG] pgmap v853: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 245 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:31:19.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:19 smithi162 bash[24557]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:19 smithi162 bash[24557]: audit 2024-09-20T19:31:18.852826+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:31:19.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:19.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:19.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:19.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:19.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:19.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:20.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:20.149 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:20.252 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:20.285 INFO:tasks.rados.rados.0.smithi042.stdout:1839: done (5 left) 2024-09-20T19:31:20.285 INFO:tasks.rados.rados.0.smithi042.stdout:1840: write oid 42 current snap is 205 2024-09-20T19:31:20.285 INFO:tasks.rados.rados.0.smithi042.stdout:1840: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-20T19:31:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:20.293 INFO:tasks.rados.rados.0.smithi042.stdout:1840: writing smithi04268048-42 from 790094 to 1273604 tid 1 2024-09-20T19:31:20.299 INFO:tasks.rados.rados.0.smithi042.stdout:1840: writing smithi04268048-42 from 1851675 to 2287893 tid 2 2024-09-20T19:31:20.299 INFO:tasks.rados.rados.0.smithi042.stdout:1841: snap_remove snap 197 2024-09-20T19:31:20.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:20.432 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:31:20.432 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:31:20.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:31:20.432 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2310 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704965712000003, 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078455948, 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2310 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730444+0000", 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6960000000000001e-06 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730468+0000", 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4110000000000001e-05 2024-09-20T19:31:20.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730484+0000", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6342e-05 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730499+0000", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5379999999999998e-05 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.734975+0000", 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044754510000000001 2024-09-20T19:31:20.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.734992+0000", 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7053000000000001e-05 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782595+0000", 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047602575 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782717+0000", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012212200000000001 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808530+0000", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025813250999999999 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808636+0000", 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010632300000000001 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808897+0000", 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000260645 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2321 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704887764999995, 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078792609, 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2321 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730521+0000", 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.623e-06 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730528+0000", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.08e-06 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730545+0000", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7880000000000002e-05 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730561+0000", 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5353e-05 2024-09-20T19:31:20.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735052+0000", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044915340000000002 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735069+0000", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6889e-05 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782803+0000", 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0477337369999999 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782886+0000", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2651999999999994e-05 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808959+0000", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026073354999999999 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809062+0000", 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010259000000000001 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809312+0000", 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024991599999999998 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2331 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704829119999999, 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0791174560000001, 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.446 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2331 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730579+0000", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.367e-06 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730586+0000", 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6660000000000002e-06 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730600+0000", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4643e-05 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730624+0000", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3506000000000002e-05 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735238+0000", 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046146870000000001 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735257+0000", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9015999999999999e-05 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782943+0000", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047685296 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783055+0000", 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112483 2024-09-20T19:31:20.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809397+0000", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026341956 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809500+0000", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102439 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809695+0000", 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019539700000000001 2024-09-20T19:31:20.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2341 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704767292, 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0794075430000001, 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.452 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2341 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730641+0000", 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3e-06 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730647+0000", 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3559999999999998e-06 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730661+0000", 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4433999999999999e-05 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730675+0000", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3866000000000001e-05 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735322+0000", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046470920000000002 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735338+0000", 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5906999999999999e-05 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783131+0000", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0477926209999999 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783244+0000", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112748 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809759+0000", 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026515001 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809860+0000", 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100936 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810047+0000", 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018728199999999999 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.457 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2351 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704712682999997, 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0797808179999999, 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2351 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730696+0000", 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5659999999999999e-06 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730701+0000", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9370000000000001e-06 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730720+0000", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8488999999999999e-05 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730733+0000", 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3403e-05 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735398+0000", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046646819999999999 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735415+0000", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6852000000000001e-05 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783331+0000", 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0479157459999999 2024-09-20T19:31:20.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783426+0000", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5086000000000004e-05 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810119+0000", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026693063 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810237+0000", 2024-09-20T19:31:20.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011857399999999999 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810475+0000", 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023741999999999999 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2361 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704658182000003, 2024-09-20T19:31:20.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0800758349999999, 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2361 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:31:20.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730750+0000", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4470000000000001e-06 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730761+0000", 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.151e-05 2024-09-20T19:31:20.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730776+0000", 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4629e-05 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730789+0000", 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.342e-05 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735477+0000", 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046872600000000004 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735492+0000", 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5483999999999999e-05 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783503+0000", 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048011147 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783561+0000", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7921000000000003e-05 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810544+0000", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026982448999999999 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810641+0000", 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7238999999999995e-05 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810824+0000", 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332900000000001 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2371 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.704601937000007, 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080392268, 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.469 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2371 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730806+0000", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4130000000000001e-06 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730812+0000", 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7130000000000002e-06 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730827+0000", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.482e-05 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730840+0000", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3239e-05 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735549+0000", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047094379999999998 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735565+0000", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5903e-05 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783599+0000", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0480339780000001 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783748+0000", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014867099999999999 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810884+0000", 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027136239999999999 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810960+0000", 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5442e-05 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811197+0000", 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023741100000000001 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.474 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2386 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.703293446999993, 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.07942407, 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2386 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732116+0000", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8930000000000001e-06 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732128+0000", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1545999999999999e-05 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733339+0000", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012117230000000001 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733364+0000", 2024-09-20T19:31:20.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4851999999999999e-05 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735635+0000", 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022705329999999999 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735651+0000", 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6252999999999998e-05 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783859+0000", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0482082370000001 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783959+0000", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9746000000000003e-05 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811276+0000", 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027317174999999999 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811366+0000", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9417000000000001e-05 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811537+0000", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171695 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2396 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.702018573999993, 2024-09-20T19:31:20.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078525953, 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2396 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733390+0000", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3329999999999999e-06 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733400+0000", 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5570000000000001e-06 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733417+0000", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6739999999999999e-05 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733431+0000", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4112e-05 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735711+0000", 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022797160000000002 2024-09-20T19:31:20.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735727+0000", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-05 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784043+0000", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0483154619999999 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784182+0000", 2024-09-20T19:31:20.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013889600000000001 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811601+0000", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027418898000000001 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811709+0000", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010836300000000001 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811914+0000", 2024-09-20T19:31:20.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020494600000000001 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2406 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701959814999995, 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078899925, 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2406 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733448+0000", 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.423e-06 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733459+0000", 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0285999999999999e-05 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733475+0000", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6563e-05 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733488+0000", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3139000000000001e-05 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735846+0000", 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023578169999999999 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735863+0000", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6549e-05 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784317+0000", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0484547749999999 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784360+0000", 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2880000000000003e-05 2024-09-20T19:31:20.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811994+0000", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027633993999999999 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812107+0000", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113169 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812347+0000", 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023933 2024-09-20T19:31:20.491 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2416 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701900566999996, 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0792861229999999, 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.492 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2416 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733508+0000", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.457e-06 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733514+0000", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6220000000000003e-06 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733528+0000", 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3983e-05 2024-09-20T19:31:20.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733540+0000", 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.206e-05 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735933+0000", 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023926220000000001 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735949+0000", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6552999999999999e-05 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784410+0000", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0484610160000001 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784474+0000", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3885000000000002e-05 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812437+0000", 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027962655999999999 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812571+0000", 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134232 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812792+0000", 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022103699999999999 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2426 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701851406000003, 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0795227549999999, 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2426 2024-09-20T19:31:20.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733557+0000", 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.384e-06 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733563+0000", 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2859999999999996e-06 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733578+0000", 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4693e-05 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733591+0000", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.346e-05 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736007+0000", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024159749999999999 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736024+0000", 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6952e-05 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784553+0000", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048529026 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784649+0000", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6232999999999998e-05 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812877+0000", 2024-09-20T19:31:20.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028227498 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812961+0000", 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4085999999999994e-05 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813078+0000", 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117162 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2436 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701800390000002, 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.079874939, 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2436 2024-09-20T19:31:20.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733607+0000", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.223e-06 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733614+0000", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.336e-06 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733628+0000", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4157e-05 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733641+0000", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3363000000000001e-05 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736084+0000", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024431499999999998 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736101+0000", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6039999999999999e-05 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784727+0000", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048626016 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784794+0000", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7896000000000002e-05 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813130+0000", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028335289 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813213+0000", 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3163000000000003e-05 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813481+0000", 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000268306 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2446 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:31:20.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701749319000001, 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080036373, 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2446 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:31:20.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733659+0000", 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8500000000000001e-06 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733666+0000", 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5599999999999999e-06 2024-09-20T19:31:20.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733681+0000", 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5288999999999999e-05 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733696+0000", 2024-09-20T19:31:20.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4707e-05 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736162+0000", 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024662260000000002 2024-09-20T19:31:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736178+0000", 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6422999999999998e-05 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784966+0000", 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048787887 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785132+0000", 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016522100000000001 2024-09-20T19:31:20.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813533+0000", 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02840123 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813583+0000", 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0760999999999997e-05 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813694+0000", 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110219 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2456 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701695960999999, 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080351998, 2024-09-20T19:31:20.519 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2456 2024-09-20T19:31:20.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733712+0000", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.204e-06 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733735+0000", 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759e-05 2024-09-20T19:31:20.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733752+0000", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6963e-05 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733765+0000", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3716999999999999e-05 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736238+0000", 2024-09-20T19:31:20.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024724230000000001 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736254+0000", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6245000000000001e-05 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785216+0000", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0489618709999999 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785294+0000", 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.852e-05 2024-09-20T19:31:20.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813766+0000", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028471195000000001 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813907+0000", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014175 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814063+0000", 2024-09-20T19:31:20.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155351 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2466 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701624186000004, 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0804979480000001, 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2466 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733784+0000", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733791+0000", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733807+0000", 2024-09-20T19:31:20.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6022999999999999e-05 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733821+0000", 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4059999999999999e-05 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736314+0000", 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002493655 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736330+0000", 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5650000000000001e-05 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785356+0000", 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049026096 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785471+0000", 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114505 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814112+0000", 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02864152 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814164+0000", 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2030999999999997e-05 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814280+0000", 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011619499999999999 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2476 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701570693999997, 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0808984619999999, 2024-09-20T19:31:20.532 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2476 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733837+0000", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3060000000000001e-06 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733843+0000", 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8869999999999997e-06 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733857+0000", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4146e-05 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733871+0000", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3366e-05 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736389+0000", 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025186549999999999 2024-09-20T19:31:20.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736406+0000", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6500000000000001e-05 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785587+0000", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049180741 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785739+0000", 2024-09-20T19:31:20.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152609 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814343+0000", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028604258 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814462+0000", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118318 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814734+0000", 2024-09-20T19:31:20.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027267600000000002 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2486 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701521851999999, 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081119497, 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.538 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2486 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733886+0000", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2729999999999999e-06 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733892+0000", 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7429999999999999e-06 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733905+0000", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3653e-05 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733919+0000", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3626e-05 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736467+0000", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025480009999999998 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736483+0000", 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6273000000000002e-05 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785878+0000", 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049395021 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785995+0000", 2024-09-20T19:31:20.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011697300000000001 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814815+0000", 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028819918999999999 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814875+0000", 2024-09-20T19:31:20.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0208999999999999e-05 2024-09-20T19:31:20.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815004+0000", 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128806 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2496 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.701473336999996, 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0813325680000001, 2024-09-20T19:31:20.546 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2496 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733935+0000", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2559999999999999e-06 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733941+0000", 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9429999999999997e-06 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733954+0000", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3890000000000001e-05 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733968+0000", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3569e-05 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743265+0000", 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092973120000000003 2024-09-20T19:31:20.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743287+0000", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1846000000000001e-05 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.786078+0000", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0427906039999999 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.786224+0000", 2024-09-20T19:31:20.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014672700000000001 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815055+0000", 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028830130999999998 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815112+0000", 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7178e-05 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815266+0000", 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154112 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2543 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94817798633088 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.695630804999993, 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080228567, 2024-09-20T19:31:20.552 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2543 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739777+0000", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4729999999999999e-06 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739783+0000", 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9599999999999997e-06 2024-09-20T19:31:20.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739798+0000", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4493e-05 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739811+0000", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3468999999999999e-05 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743681+0000", 2024-09-20T19:31:20.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038696569999999999 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743698+0000", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7110000000000001e-05 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.787209+0000", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0435105220000001 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.787321+0000", 2024-09-20T19:31:20.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112125 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.817383+0000", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030062286000000001 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.817453+0000", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0094999999999999e-05 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.820004+0000", 2024-09-20T19:31:20.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025513770000000001 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:20.558 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:31:20.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.561 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:20 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.269479+0000 mon.a (mon.0) 1944 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: audit 2024-09-20T19:31:19.271376+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? v1:172.21.15.42:0/3357106466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.276312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: audit 2024-09-20T19:31:19.528103+0000 mgr.y (mgr.24491) 504 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.543330+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.546433+0000 osd.3 (osd.3) 4 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.573235+0000 osd.0 (osd.0) 7 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:31:20.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.575073+0000 osd.0 (osd.0) 8 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.804640+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.806365+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:20 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.277550+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-20T19:31:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:20.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.720 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:20.823 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:20.851 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:20.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:20.981 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:20.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:21.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:21.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:21.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T19:31:21.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:21.301 INFO:tasks.rados.rados.0.smithi042.stdout:1841: done (6 left) 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1842: snap_create 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1834: finishing copy_from to smithi04268048-3 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1834: got expected ENOENT (src dne) 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1834: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1836: finishing write tid 1 to smithi04268048-22 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1833: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1833: finishing copy_from to smithi04268048-24 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 586 (ObjNum 595 snap 189 seq_num 595) dirty exists 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1836: finishing write tid 2 to smithi04268048-22 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1836: finishing write tid 3 to smithi04268048-22 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1836: finishing write tid 4 to smithi04268048-22 2024-09-20T19:31:21.302 INFO:tasks.rados.rados.0.smithi042.stdout:1836: oid 22 updating version 0 to 1009 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1836: oid 22 version 1009 is already newer than 1008 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1836: oid 22 version 1009 is already newer than 1007 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1009 (ObjNum 641 snap 205 seq_num 641) dirty exists 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1836: left oid 22 (ObjNum 641 snap 205 seq_num 641) 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1838: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1838: finishing copy_from to smithi04268048-38 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 610 (ObjNum 471 snap 152 seq_num 471) dirty exists 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: finishing write tid 1 to smithi04268048-42 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: finishing write tid 2 to smithi04268048-42 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: finishing write tid 3 to smithi04268048-42 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: finishing write tid 4 to smithi04268048-42 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: oid 42 updating version 0 to 832 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: oid 42 version 832 is already newer than 831 2024-09-20T19:31:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:1840: oid 42 version 832 is already newer than 830 2024-09-20T19:31:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 832 (ObjNum 642 snap 205 seq_num 642) dirty exists 2024-09-20T19:31:21.304 INFO:tasks.rados.rados.0.smithi042.stdout:1840: left oid 42 (ObjNum 642 snap 205 seq_num 642) 2024-09-20T19:31:21.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:21.361 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.361 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.361 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.361 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.361 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.362 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.363 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.363 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.363 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.363 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:21 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:21.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:21.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.348304+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:19.351130+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.287096+0000 mgr.y (mgr.24491) 505 : cluster [DBG] pgmap v856: 168 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 155 active+clean; 247 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 488 KiB/s wr, 4 op/s 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.819354+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.491 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:21 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.821588+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:31:21.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:21.605 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:21.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:21.737 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:21.749 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:21.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:21.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:21.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:21.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:21.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:22.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:22.260 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:22.272 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:31:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1833: done (6 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1834: done (5 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1835: done (4 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1836: done (3 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1838: done (2 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1840: done (1 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1842: done (0 left) 2024-09-20T19:31:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:1843: read oid 7 snap 202 2024-09-20T19:31:22.309 INFO:tasks.rados.rados.0.smithi042.stdout:1843: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:31:22.309 INFO:tasks.rados.rados.0.smithi042.stdout:1844: write (excl) oid 25 current snap is 206 2024-09-20T19:31:22.309 INFO:tasks.rados.rados.0.smithi042.stdout:1844: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-20T19:31:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:1844: writing smithi04268048-25 from 406901 to 965660 tid 1 2024-09-20T19:31:22.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: writing smithi04268048-25 from 1393804 to 1973680 tid 2 2024-09-20T19:31:22.325 INFO:tasks.rados.rados.0.smithi042.stdout:1844: writing smithi04268048-25 from 2608396 to 2806901 tid 3 2024-09-20T19:31:22.326 INFO:tasks.rados.rados.0.smithi042.stdout:1845: write oid 33 current snap is 206 2024-09-20T19:31:22.326 INFO:tasks.rados.rados.0.smithi042.stdout:1845: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-20T19:31:22.331 INFO:tasks.rados.rados.0.smithi042.stdout:1845: writing smithi04268048-33 from 423708 to 1057716 tid 1 2024-09-20T19:31:22.338 INFO:tasks.rados.rados.0.smithi042.stdout:1845: writing smithi04268048-33 from 1852286 to 2292173 tid 2 2024-09-20T19:31:22.342 INFO:tasks.rados.rados.0.smithi042.stdout:1845: writing smithi04268048-33 from 2752172 to 2823708 tid 3 2024-09-20T19:31:22.342 INFO:tasks.rados.rados.0.smithi042.stdout:1846: write (excl) oid 17 current snap is 206 2024-09-20T19:31:22.342 INFO:tasks.rados.rados.0.smithi042.stdout:1846: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-20T19:31:22.348 INFO:tasks.rados.rados.0.smithi042.stdout:1846: writing smithi04268048-17 from 440515 to 1149772 tid 1 2024-09-20T19:31:22.356 INFO:tasks.rados.rados.0.smithi042.stdout:1846: writing smithi04268048-17 from 1910768 to 2494313 tid 2 2024-09-20T19:31:22.356 INFO:tasks.rados.rados.0.smithi042.stdout:1847: snap_remove snap 168 2024-09-20T19:31:22.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:22.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:22 smithi162 bash[24557]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:22.644 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:22.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[22706]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.342457+0000 osd.1 (osd.1) 199 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.343878+0000 osd.1 (osd.1) 200 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.479845+0000 osd.2 (osd.2) 177 : cluster [DBG] 7.e scrub starts 2024-09-20T19:31:22.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:20.483979+0000 osd.2 (osd.2) 178 : cluster [DBG] 7.e scrub ok 2024-09-20T19:31:22.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:22 smithi042 bash[32075]: cluster 2024-09-20T19:31:21.300208+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-20T19:31:22.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:22.768 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:22.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:22.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:22.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:22.931 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:22.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:23.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:23.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:23.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.189 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:23.249 INFO:tasks.rados.rados.0.smithi042.stdout:1847: done (4 left) 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:1848: copy_from oid 45 from oid 20 current snap is 206 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:1849: rollback oid 27 current snap is 206 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 27 to 205 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:1850: copy_from oid 29 from oid 31 current snap is 206 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:1851: snap_create 2024-09-20T19:31:23.250 INFO:tasks.rados.rados.0.smithi042.stdout:1843: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:31:23.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:23.317 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2498132 2024-09-20T19:31:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:23.319 INFO:tasks.rados.rados.0.smithi042.stdout:1844: finishing write tid 1 to smithi04268048-25 2024-09-20T19:31:23.319 INFO:tasks.rados.rados.0.smithi042.stdout:1844: finishing write tid 2 to smithi04268048-25 2024-09-20T19:31:23.319 INFO:tasks.rados.rados.0.smithi042.stdout:1844: finishing write tid 3 to smithi04268048-25 2024-09-20T19:31:23.319 INFO:tasks.rados.rados.0.smithi042.stdout:1844: finishing write tid 4 to smithi04268048-25 2024-09-20T19:31:23.319 INFO:tasks.rados.rados.0.smithi042.stdout:1844: finishing write tid 5 to smithi04268048-25 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: oid 25 updating version 0 to 778 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: oid 25 version 778 is already newer than 777 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: oid 25 version 778 is already newer than 776 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: oid 25 updating version 778 to 779 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 779 (ObjNum 643 snap 206 seq_num 643) dirty exists 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1844: left oid 25 (ObjNum 643 snap 206 seq_num 643) 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: finishing write tid 1 to smithi04268048-33 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: finishing write tid 2 to smithi04268048-33 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: finishing write tid 3 to smithi04268048-33 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: finishing write tid 4 to smithi04268048-33 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: finishing write tid 5 to smithi04268048-33 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: oid 33 updating version 0 to 597 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: oid 33 version 597 is already newer than 596 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: oid 33 version 597 is already newer than 594 2024-09-20T19:31:23.320 INFO:tasks.rados.rados.0.smithi042.stdout:1845: oid 33 version 597 is already newer than 595 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 597 (ObjNum 644 snap 206 seq_num 644) dirty exists 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1845: left oid 33 (ObjNum 644 snap 206 seq_num 644) 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: finishing write tid 1 to smithi04268048-17 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: finishing write tid 2 to smithi04268048-17 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: finishing write tid 3 to smithi04268048-17 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: finishing write tid 4 to smithi04268048-17 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: oid 17 updating version 0 to 700 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: oid 17 updating version 700 to 701 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: oid 17 version 701 is already newer than 699 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 701 (ObjNum 645 snap 206 seq_num 645) dirty exists 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1846: left oid 17 (ObjNum 645 snap 206 seq_num 645) 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1850: finishing copy_from to smithi04268048-29 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1850: got expected ENOENT (src dne) 2024-09-20T19:31:23.321 INFO:tasks.rados.rados.0.smithi042.stdout:1850: finishing copy_from racing read to smithi04268048-29 2024-09-20T19:31:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:1848: finishing copy_from to smithi04268048-45 2024-09-20T19:31:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:1848: got expected ENOENT (src dne) 2024-09-20T19:31:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:1848: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:31:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:1849: finishing rollback tid 1 to smithi04268048-27 2024-09-20T19:31:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 0 (ObjNum 353 snap 0 seq_num 3047886384) dirty dne 2024-09-20T19:31:23.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:23 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:23.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:23.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:23.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.288098+0000 mgr.y (mgr.24491) 506 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 237 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-20T19:31:23.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.297167+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.298725+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.306966+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.590146+0000 osd.0 (osd.0) 9 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:22.595422+0000 osd.0 (osd.0) 10 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:23 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.242743+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-20T19:31:23.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:23.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:23.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:23.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:23.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:24.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:24.096 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:24.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:24.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1843: done (7 left) 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1844: done (6 left) 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1845: done (5 left) 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1846: done (4 left) 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1848: done (3 left) 2024-09-20T19:31:24.260 INFO:tasks.rados.rados.0.smithi042.stdout:1849: done (2 left) 2024-09-20T19:31:24.261 INFO:tasks.rados.rados.0.smithi042.stdout:1850: done (1 left) 2024-09-20T19:31:24.261 INFO:tasks.rados.rados.0.smithi042.stdout:1851: done (0 left) 2024-09-20T19:31:24.261 INFO:tasks.rados.rados.0.smithi042.stdout:1852: copy_from oid 31 from oid 28 current snap is 207 2024-09-20T19:31:24.261 INFO:tasks.rados.rados.0.smithi042.stdout:1853: copy_from oid 43 from oid 8 current snap is 207 2024-09-20T19:31:24.261 INFO:tasks.rados.rados.0.smithi042.stdout:1854: snap_remove snap 202 2024-09-20T19:31:24.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:24.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:24.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:24.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:24.591 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:24.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:24.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:24.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:24.869 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:24.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:24.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:25.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:25.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:25.261 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1854: done (2 left) 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1855: delete oid 33 current snap is 207 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1852: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1852: finishing copy_from to smithi04268048-31 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 581 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1853: finishing copy_from racing read to smithi04268048-43 2024-09-20T19:31:25.269 INFO:tasks.rados.rados.0.smithi042.stdout:1853: finishing copy_from to smithi04268048-43 2024-09-20T19:31:25.270 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 712 (ObjNum 543 snap 177 seq_num 543) dirty exists 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1852: done (2 left) 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1853: done (1 left) 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1855: done (0 left) 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1856: read oid 18 snap -1 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1856: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:25.274 INFO:tasks.rados.rados.0.smithi042.stdout:1857: delete oid 31 current snap is 207 2024-09-20T19:31:25.277 INFO:tasks.rados.rados.0.smithi042.stdout:1857: done (1 left) 2024-09-20T19:31:25.277 INFO:tasks.rados.rados.0.smithi042.stdout:1858: write oid 43 current snap is 207 2024-09-20T19:31:25.277 INFO:tasks.rados.rados.0.smithi042.stdout:1858: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-20T19:31:25.287 INFO:tasks.rados.rados.0.smithi042.stdout:1858: writing smithi04268048-43 from 490936 to 1142293 tid 1 2024-09-20T19:31:25.295 INFO:tasks.rados.rados.0.smithi042.stdout:1858: writing smithi04268048-43 from 1686214 to 2133439 tid 2 2024-09-20T19:31:25.299 INFO:tasks.rados.rados.0.smithi042.stdout:1858: writing smithi04268048-43 from 2861864 to 2890936 tid 3 2024-09-20T19:31:25.299 INFO:tasks.rados.rados.0.smithi042.stdout:1859: delete oid 42 current snap is 207 2024-09-20T19:31:25.300 INFO:tasks.rados.rados.0.smithi042.stdout:1856: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:25.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:25.369 INFO:tasks.rados.rados.0.smithi042.stdout:1858: finishing write tid 1 to smithi04268048-43 2024-09-20T19:31:25.369 INFO:tasks.rados.rados.0.smithi042.stdout:1858: finishing write tid 2 to smithi04268048-43 2024-09-20T19:31:25.369 INFO:tasks.rados.rados.0.smithi042.stdout:1858: finishing write tid 3 to smithi04268048-43 2024-09-20T19:31:25.369 INFO:tasks.rados.rados.0.smithi042.stdout:1858: finishing write tid 4 to smithi04268048-43 2024-09-20T19:31:25.369 INFO:tasks.rados.rados.0.smithi042.stdout:1856: done (2 left) 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1859: done (1 left) 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1860: read oid 22 snap -1 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1860: expect (ObjNum 641 snap 205 seq_num 641) 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1861: read oid 48 snap -1 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1861: expect deleted 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1862: rollback oid 33 current snap is 207 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 206 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: finishing write tid 5 to smithi04268048-43 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: oid 43 updating version 0 to 715 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: oid 43 version 715 is already newer than 714 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: oid 43 version 715 is already newer than 713 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: oid 43 updating version 715 to 716 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 716 (ObjNum 648 snap 207 seq_num 648) dirty exists 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: left oid 43 (ObjNum 648 snap 207 seq_num 648) 2024-09-20T19:31:25.370 INFO:tasks.rados.rados.0.smithi042.stdout:1858: done (3 left) 2024-09-20T19:31:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:1863: delete oid 32 current snap is 207 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1861: done (3 left) 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1863: done (2 left) 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1864: read oid 13 snap -1 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1864: expect deleted 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1865: snap_create 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1862: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:1862: finishing rollback tid 2 to smithi04268048-33 2024-09-20T19:31:25.372 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 600 (ObjNum 644 snap 206 seq_num 644) dirty exists 2024-09-20T19:31:25.373 INFO:tasks.rados.rados.0.smithi042.stdout:1860: expect (ObjNum 641 snap 205 seq_num 641) 2024-09-20T19:31:25.373 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 55682 2024-09-20T19:31:25.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:25.417 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[22706]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:25 smithi042 bash[32075]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.461 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:25.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.519061+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:23.521542+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:24.258506+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: cluster 2024-09-20T19:31:24.289069+0000 mgr.y (mgr.24491) 507 : cluster [DBG] pgmap v862: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 217 MiB data, 828 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:25 smithi162 bash[24557]: audit 2024-09-20T19:31:25.082144+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:31:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:25.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:25.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:25.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:25.839 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:31:25.840 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:31:25.857 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:31:25.857 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:31:25.866 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:25.875 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:31:25.875 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:31:25.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:25.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:25.893 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:31:25.893 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:31:25.914 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:31:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:31:25.935 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:31:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:31:25.956 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:31:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:31:25.977 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:31:25.978 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:31:25.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:26.008 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:26.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:26.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.148 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:26.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:26.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:26.279 INFO:tasks.rados.rados.0.smithi042.stdout:1860: done (3 left) 2024-09-20T19:31:26.279 INFO:tasks.rados.rados.0.smithi042.stdout:1862: done (2 left) 2024-09-20T19:31:26.279 INFO:tasks.rados.rados.0.smithi042.stdout:1864: done (1 left) 2024-09-20T19:31:26.279 INFO:tasks.rados.rados.0.smithi042.stdout:1865: done (0 left) 2024-09-20T19:31:26.279 INFO:tasks.rados.rados.0.smithi042.stdout:1866: snap_remove snap 206 2024-09-20T19:31:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:26.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.395 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[32075]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:26 smithi042 bash[22706]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:26.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T19:31:26.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: audit 2024-09-20T19:31:25.260938+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.42:0/1866841441' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.266736+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:26 smithi162 bash[24557]: audit 2024-09-20T19:31:25.820536+0000 mon.a (mon.0) 1955 : audit [DBG] from='client.? v1:172.21.15.42:0/2203789619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:26.536 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:26.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:26.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:26.766 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:26.891 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:26.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:27.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:27.051 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:27.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:27.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.289 INFO:tasks.rados.rados.0.smithi042.stdout:1866: done (0 left) 2024-09-20T19:31:27.289 INFO:tasks.rados.rados.0.smithi042.stdout:1867: delete oid 22 current snap is 208 2024-09-20T19:31:27.298 INFO:tasks.rados.rados.0.smithi042.stdout:1867: done (0 left) 2024-09-20T19:31:27.298 INFO:tasks.rados.rados.0.smithi042.stdout:1868: read oid 18 snap 205 2024-09-20T19:31:27.298 INFO:tasks.rados.rados.0.smithi042.stdout:1868: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:27.298 INFO:tasks.rados.rados.0.smithi042.stdout:1869: snap_remove snap 207 2024-09-20T19:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:27.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:27 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:27.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.556 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.356243+0000 osd.1 (osd.1) 203 : cluster [DBG] 5.c scrub starts 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:25.357596+0000 osd.1 (osd.1) 204 : cluster [DBG] 5.c scrub ok 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.276377+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.557 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:27 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.289680+0000 mgr.y (mgr.24491) 508 : cluster [DBG] pgmap v865: 168 pgs: 2 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 157 active+clean; 208 MiB data, 815 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:31:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:27.638 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:27.638 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:27.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:27.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:27.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:27.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:28.002 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:28.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.093 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:28.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:28.248 INFO:tasks.rados.rados.0.smithi042.stdout:1869: done (1 left) 2024-09-20T19:31:28.248 INFO:tasks.rados.rados.0.smithi042.stdout:1870: read oid 3 snap -1 2024-09-20T19:31:28.248 INFO:tasks.rados.rados.0.smithi042.stdout:1870: expect (ObjNum 602 snap 191 seq_num 602) 2024-09-20T19:31:28.248 INFO:tasks.rados.rados.0.smithi042.stdout:1871: write oid 20 current snap is 208 2024-09-20T19:31:28.248 INFO:tasks.rados.rados.0.smithi042.stdout:1871: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-09-20T19:31:28.256 INFO:tasks.rados.rados.0.smithi042.stdout:1871: writing smithi04268048-20 from 558164 to 1110517 tid 1 2024-09-20T19:31:28.266 INFO:tasks.rados.rados.0.smithi042.stdout:1871: writing smithi04268048-20 from 1803789 to 2258352 tid 2 2024-09-20T19:31:28.272 INFO:tasks.rados.rados.0.smithi042.stdout:1871: writing smithi04268048-20 from 2855203 to 2958164 tid 3 2024-09-20T19:31:28.272 INFO:tasks.rados.rados.0.smithi042.stdout:1872: copy_from oid 30 from oid 22 current snap is 208 2024-09-20T19:31:28.272 INFO:tasks.rados.rados.0.smithi042.stdout:1868: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:28.284 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2115590 2024-09-20T19:31:28.326 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.332 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1911607 2024-09-20T19:31:28.340 INFO:tasks.rados.rados.0.smithi042.stdout:1870: expect (ObjNum 602 snap 191 seq_num 602) 2024-09-20T19:31:28.340 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:28.349 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:28.350 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.351 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.352 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:28 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:28.363 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 355810 2024-09-20T19:31:28.406 INFO:tasks.rados.rados.0.smithi042.stdout:1871: finishing write tid 1 to smithi04268048-20 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: finishing write tid 2 to smithi04268048-20 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: finishing write tid 3 to smithi04268048-20 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: finishing write tid 4 to smithi04268048-20 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: finishing write tid 5 to smithi04268048-20 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: oid 20 updating version 0 to 982 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: oid 20 version 982 is already newer than 979 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: oid 20 version 982 is already newer than 980 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: oid 20 version 982 is already newer than 981 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 982 (ObjNum 652 snap 208 seq_num 652) dirty exists 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1871: left oid 20 (ObjNum 652 snap 208 seq_num 652) 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1872: finishing copy_from to smithi04268048-30 2024-09-20T19:31:28.407 INFO:tasks.rados.rados.0.smithi042.stdout:1872: got expected ENOENT (src dne) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1872: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1868: done (3 left) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1870: done (2 left) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1871: done (1 left) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1872: done (0 left) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1873: read oid 45 snap 204 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1873: expect deleted 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1874: read oid 29 snap -1 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1874: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1875: rollback oid 24 current snap is 208 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 204 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1876: rollback oid 43 current snap is 208 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 204 2024-09-20T19:31:28.408 INFO:tasks.rados.rados.0.smithi042.stdout:1877: write (excl) oid 33 current snap is 208 2024-09-20T19:31:28.409 INFO:tasks.rados.rados.0.smithi042.stdout:1877: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-20T19:31:28.426 INFO:tasks.rados.rados.0.smithi042.stdout:1877: writing smithi04268048-33 from 574971 to 1202573 tid 1 2024-09-20T19:31:28.447 INFO:tasks.rados.rados.0.smithi042.stdout:1877: writing smithi04268048-33 from 1862271 to 2460492 tid 2 2024-09-20T19:31:28.447 INFO:tasks.rados.rados.0.smithi042.stdout:1878: write oid 16 current snap is 208 2024-09-20T19:31:28.447 INFO:tasks.rados.rados.0.smithi042.stdout:1878: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-20T19:31:28.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T19:31:28.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:28.456 INFO:tasks.rados.rados.0.smithi042.stdout:1878: writing smithi04268048-16 from 591778 to 1010982 tid 1 2024-09-20T19:31:28.466 INFO:tasks.rados.rados.0.smithi042.stdout:1878: writing smithi04268048-16 from 1637106 to 2378985 tid 2 2024-09-20T19:31:28.472 INFO:tasks.rados.rados.0.smithi042.stdout:1878: writing smithi04268048-16 from 2910049 to 2991778 tid 3 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1879: delete oid 28 current snap is 208 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1876: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1876: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1876: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 719 (ObjNum 594 snap 189 seq_num 594) dirty exists 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1875: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1875: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1875: finishing rollback tid 2 to smithi04268048-24 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 720 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:1874: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:28.473 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.547690+0000 osd.2 (osd.2) 181 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:26.549375+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:27.288355+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: audit 2024-09-20T19:31:27.841646+0000 mon.a (mon.0) 1958 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:28 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.244843+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-20T19:31:28.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:28.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:28.631 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 17274 2024-09-20T19:31:28.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:28.705 INFO:tasks.rados.rados.0.smithi042.stdout:1877: finishing write tid 1 to smithi04268048-33 2024-09-20T19:31:28.705 INFO:tasks.rados.rados.0.smithi042.stdout:1877: finishing write tid 2 to smithi04268048-33 2024-09-20T19:31:28.705 INFO:tasks.rados.rados.0.smithi042.stdout:1877: finishing write tid 3 to smithi04268048-33 2024-09-20T19:31:28.705 INFO:tasks.rados.rados.0.smithi042.stdout:1877: finishing write tid 4 to smithi04268048-33 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1877: oid 33 updating version 0 to 603 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1877: oid 33 version 603 is already newer than 602 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1877: oid 33 version 603 is already newer than 601 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 603 (ObjNum 653 snap 208 seq_num 653) dirty exists 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1877: left oid 33 (ObjNum 653 snap 208 seq_num 653) 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: finishing write tid 1 to smithi04268048-16 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: finishing write tid 2 to smithi04268048-16 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: finishing write tid 3 to smithi04268048-16 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: finishing write tid 4 to smithi04268048-16 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: finishing write tid 5 to smithi04268048-16 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: oid 16 updating version 0 to 722 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: oid 16 updating version 722 to 723 2024-09-20T19:31:28.706 INFO:tasks.rados.rados.0.smithi042.stdout:1878: oid 16 updating version 723 to 724 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1878: oid 16 version 724 is already newer than 721 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 724 (ObjNum 654 snap 208 seq_num 654) dirty exists 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1878: left oid 16 (ObjNum 654 snap 208 seq_num 654) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1873: done (6 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1874: done (5 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1875: done (4 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1876: done (3 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1877: done (2 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1878: done (1 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1879: done (0 left) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1880: read oid 50 snap -1 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1880: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:31:28.707 INFO:tasks.rados.rados.0.smithi042.stdout:1881: snap_create 2024-09-20T19:31:28.716 INFO:tasks.rados.rados.0.smithi042.stdout:1880: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:31:28.716 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:28.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:28.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:28.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:28.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:29.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:29.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:29.296 INFO:tasks.rados.rados.0.smithi042.stdout:1880: done (1 left) 2024-09-20T19:31:29.296 INFO:tasks.rados.rados.0.smithi042.stdout:1881: done (0 left) 2024-09-20T19:31:29.296 INFO:tasks.rados.rados.0.smithi042.stdout:1882: snap_create 2024-09-20T19:31:29.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:31:29.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:29.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:29.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:29 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.472 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:29.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.072123+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.073027+0000 osd.1 (osd.1) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:29 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.290280+0000 mgr.y (mgr.24491) 509 : cluster [DBG] pgmap v868: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 142 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:31:29.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:29.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:29.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:29.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:29.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.997 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:29.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:30.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:30.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:30.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:30.257 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:30.296 INFO:tasks.rados.rados.0.smithi042.stdout:1882: done (0 left) 2024-09-20T19:31:30.296 INFO:tasks.rados.rados.0.smithi042.stdout:1883: write oid 17 current snap is 210 2024-09-20T19:31:30.296 INFO:tasks.rados.rados.0.smithi042.stdout:1883: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-20T19:31:30.301 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:31:30.304 INFO:tasks.rados.rados.0.smithi042.stdout:1883: writing smithi04268048-17 from 625392 to 1195094 tid 1 2024-09-20T19:31:30.312 INFO:tasks.rados.rados.0.smithi042.stdout:1883: writing smithi04268048-17 from 1637717 to 2383265 tid 2 2024-09-20T19:31:30.316 INFO:tasks.rados.rados.0.smithi042.stdout:1883: writing smithi04268048-17 from 2848542 to 3025392 tid 3 2024-09-20T19:31:30.316 INFO:tasks.rados.rados.0.smithi042.stdout:1884: rollback oid 10 current snap is 210 2024-09-20T19:31:30.316 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 209 2024-09-20T19:31:30.316 INFO:tasks.rados.rados.0.smithi042.stdout:1885: read oid 35 snap -1 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1885: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1886: read oid 6 snap -1 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1886: expect (ObjNum 629 snap 198 seq_num 629) 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1887: rollback oid 50 current snap is 210 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 209 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1888: write (excl) oid 36 current snap is 210 2024-09-20T19:31:30.317 INFO:tasks.rados.rados.0.smithi042.stdout:1888: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-20T19:31:30.324 INFO:tasks.rados.rados.0.smithi042.stdout:1888: writing smithi04268048-36 from 642199 to 1287150 tid 1 2024-09-20T19:31:30.331 INFO:tasks.rados.rados.0.smithi042.stdout:1888: writing smithi04268048-36 from 1696199 to 2301758 tid 2 2024-09-20T19:31:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1888: writing smithi04268048-36 from 2992318 to 3042199 tid 3 2024-09-20T19:31:30.336 INFO:tasks.rados.rados.0.smithi042.stdout:1889: read oid 18 snap -1 2024-09-20T19:31:30.337 INFO:tasks.rados.rados.0.smithi042.stdout:1889: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:30.337 INFO:tasks.rados.rados.0.smithi042.stdout:1890: write (excl) oid 2 current snap is 210 2024-09-20T19:31:30.337 INFO:tasks.rados.rados.0.smithi042.stdout:1890: seq_num 658 ranges {659006=720200,2154681=749217} 2024-09-20T19:31:30.345 INFO:tasks.rados.rados.0.smithi042.stdout:1890: writing smithi04268048-2 from 659006 to 1379206 tid 1 2024-09-20T19:31:30.354 INFO:tasks.rados.rados.0.smithi042.stdout:1890: writing smithi04268048-2 from 2154681 to 2903898 tid 2 2024-09-20T19:31:30.354 INFO:tasks.rados.rados.0.smithi042.stdout:1891: delete oid 47 current snap is 210 2024-09-20T19:31:30.354 INFO:tasks.rados.rados.0.smithi042.stdout:1885: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:31:30.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.370 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:30 smithi162 bash[24557]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:30.391 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:30.464 INFO:tasks.rados.rados.0.smithi042.stdout:1884: finishing rollback tid 0 to smithi04268048-10 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1884: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1884: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 646 (ObjNum 622 snap 195 seq_num 622) dirty exists 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1887: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1887: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1887: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 821 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:31:30.465 INFO:tasks.rados.rados.0.smithi042.stdout:1886: expect (ObjNum 629 snap 198 seq_num 629) 2024-09-20T19:31:30.483 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2238195 2024-09-20T19:31:30.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.513 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1883: finishing write tid 1 to smithi04268048-17 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1883: finishing write tid 2 to smithi04268048-17 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1884: done (8 left) 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1885: done (7 left) 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1886: done (6 left) 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1887: done (5 left) 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1891: done (4 left) 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1892: rollback oid 28 current snap is 210 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 204 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1888: finishing write tid 1 to smithi04268048-36 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1883: finishing write tid 3 to smithi04268048-17 2024-09-20T19:31:30.614 INFO:tasks.rados.rados.0.smithi042.stdout:1883: finishing write tid 4 to smithi04268048-17 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: finishing write tid 5 to smithi04268048-17 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: oid 17 updating version 0 to 702 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: oid 17 updating version 702 to 705 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: oid 17 version 705 is already newer than 703 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: oid 17 version 705 is already newer than 704 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 705 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1883: left oid 17 (ObjNum 656 snap 210 seq_num 656) 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1888: finishing write tid 2 to smithi04268048-36 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1888: finishing write tid 3 to smithi04268048-36 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1888: finishing write tid 4 to smithi04268048-36 2024-09-20T19:31:30.615 INFO:tasks.rados.rados.0.smithi042.stdout:1889: expect (ObjNum 632 snap 198 seq_num 632) 2024-09-20T19:31:30.616 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:30.619 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 943016 2024-09-20T19:31:30.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[22706]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.340044+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:28.342204+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.295729+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.527767+0000 osd.0 (osd.0) 11 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.639 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.529427+0000 osd.0 (osd.0) 12 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.536176+0000 mgr.y (mgr.24491) 510 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.952681+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.42:0/144144316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.640 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:30 smithi042 bash[32075]: audit 2024-09-20T19:31:29.953124+0000 mon.a (mon.0) 1961 : audit [INF] from='client.25499 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:30.640 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:30.640 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: finishing write tid 5 to smithi04268048-36 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: oid 36 updating version 0 to 567 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: oid 36 version 567 is already newer than 566 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: oid 36 version 567 is already newer than 565 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: oid 36 updating version 567 to 568 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 568 (ObjNum 657 snap 210 seq_num 657) dirty exists 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1888: left oid 36 (ObjNum 657 snap 210 seq_num 657) 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1890: finishing write tid 1 to smithi04268048-2 2024-09-20T19:31:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:1890: finishing write tid 2 to smithi04268048-2 2024-09-20T19:31:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:1890: finishing write tid 3 to smithi04268048-2 2024-09-20T19:31:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:1883: done (4 left) 2024-09-20T19:31:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:1888: done (3 left) 2024-09-20T19:31:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:1889: done (2 left) 2024-09-20T19:31:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:1893: snap_remove snap 209 2024-09-20T19:31:30.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.767 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:30.870 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:30.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:31.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:31.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:31.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:31.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:31.259 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1893: done (2 left) 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1894: snap_create 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1890: finishing write tid 4 to smithi04268048-2 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1890: oid 2 updating version 0 to 604 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1890: oid 2 updating version 604 to 605 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1890: oid 2 updating version 605 to 606 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 606 (ObjNum 658 snap 210 seq_num 658) dirty exists 2024-09-20T19:31:31.350 INFO:tasks.rados.rados.0.smithi042.stdout:1890: left oid 2 (ObjNum 658 snap 210 seq_num 658) 2024-09-20T19:31:31.351 INFO:tasks.rados.rados.0.smithi042.stdout:1892: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:31:31.351 INFO:tasks.rados.rados.0.smithi042.stdout:1892: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:31:31.351 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 584 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:31:31.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:31.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:31.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:31.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[32075]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.534 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:31 smithi042 bash[22706]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:31.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:31.773 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:31.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:31.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.373432+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.374745+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.551680+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.b scrub starts 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:29.553125+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.b scrub ok 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.289872+0000 mon.a (mon.0) 1962 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.289894+0000 mon.a (mon.0) 1963 : cluster [INF] Cluster is now healthy 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.290887+0000 mgr.y (mgr.24491) 511 : cluster [DBG] pgmap v870: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 149 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: audit 2024-09-20T19:31:30.291706+0000 mon.a (mon.0) 1964 : audit [INF] from='client.25499 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:31 smithi162 bash[24557]: cluster 2024-09-20T19:31:30.296196+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-20T19:31:31.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:31.933 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:31.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:32.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:32.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:32.296 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:32.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:32.359 INFO:tasks.rados.rados.0.smithi042.stdout:1890: done (2 left) 2024-09-20T19:31:32.359 INFO:tasks.rados.rados.0.smithi042.stdout:1892: done (1 left) 2024-09-20T19:31:32.359 INFO:tasks.rados.rados.0.smithi042.stdout:1894: done (0 left) 2024-09-20T19:31:32.359 INFO:tasks.rados.rados.0.smithi042.stdout:1895: snap_remove snap 210 2024-09-20T19:31:32.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:32.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.568 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:32 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:32.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.349492+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.387023+0000 osd.1 (osd.1) 211 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.390005+0000 osd.1 (osd.1) 212 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:31:32.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.603694+0000 osd.3 (osd.3) 5 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:31:32.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:32 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.606803+0000 osd.3 (osd.3) 6 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:31:32.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:32.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:32.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:32.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:32.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:32.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:33.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:33.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:33.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:33.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:33.250 INFO:tasks.rados.rados.0.smithi042.stdout:1895: done (0 left) 2024-09-20T19:31:33.250 INFO:tasks.rados.rados.0.smithi042.stdout:1896: read oid 25 snap -1 2024-09-20T19:31:33.250 INFO:tasks.rados.rados.0.smithi042.stdout:1896: expect (ObjNum 643 snap 206 seq_num 643) 2024-09-20T19:31:33.250 INFO:tasks.rados.rados.0.smithi042.stdout:1897: snap_create 2024-09-20T19:31:33.251 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:33.261 INFO:tasks.rados.rados.0.smithi042.stdout:1896: expect (ObjNum 643 snap 206 seq_num 643) 2024-09-20T19:31:33.261 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 280474 2024-09-20T19:31:33.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:31:33.302 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:33.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:33.343 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:33.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:33.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.465 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:33 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:33.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:33.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:33.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.523511+0000 osd.2 (osd.2) 185 : cluster [DBG] 6.f scrub starts 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:31.525728+0000 osd.2 (osd.2) 186 : cluster [DBG] 6.f scrub ok 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.292233+0000 mgr.y (mgr.24491) 512 : cluster [DBG] pgmap v873: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 146 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 22 op/s 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.357975+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.721 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:33 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.248088+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-20T19:31:33.736 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:33.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:33.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:33.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:33.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:33.998 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:34.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:34.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T19:31:34.255 INFO:tasks.rados.rados.0.smithi042.stdout:1896: done (1 left) 2024-09-20T19:31:34.255 INFO:tasks.rados.rados.0.smithi042.stdout:1897: done (0 left) 2024-09-20T19:31:34.255 INFO:tasks.rados.rados.0.smithi042.stdout:1898: write (excl) oid 6 current snap is 212 2024-09-20T19:31:34.255 INFO:tasks.rados.rados.0.smithi042.stdout:1898: seq_num 660 ranges {692620=470698,1755292=752886} 2024-09-20T19:31:34.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:34.261 INFO:tasks.rados.rados.0.smithi042.stdout:1898: writing smithi04268048-6 from 692620 to 1163318 tid 1 2024-09-20T19:31:34.269 INFO:tasks.rados.rados.0.smithi042.stdout:1898: writing smithi04268048-6 from 1755292 to 2508178 tid 2 2024-09-20T19:31:34.269 INFO:tasks.rados.rados.0.smithi042.stdout:1899: write oid 16 current snap is 212 2024-09-20T19:31:34.269 INFO:tasks.rados.rados.0.smithi042.stdout:1899: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-20T19:31:34.276 INFO:tasks.rados.rados.0.smithi042.stdout:1899: writing smithi04268048-16 from 709427 to 1255374 tid 1 2024-09-20T19:31:34.283 INFO:tasks.rados.rados.0.smithi042.stdout:1899: writing smithi04268048-16 from 1813774 to 2426671 tid 2 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1899: writing smithi04268048-16 from 2985657 to 3109427 tid 3 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1900: delete oid 29 current snap is 212 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: finishing write tid 1 to smithi04268048-6 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: finishing write tid 2 to smithi04268048-6 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: finishing write tid 3 to smithi04268048-6 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: finishing write tid 4 to smithi04268048-6 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: oid 6 updating version 0 to 715 2024-09-20T19:31:34.288 INFO:tasks.rados.rados.0.smithi042.stdout:1898: oid 6 version 715 is already newer than 714 2024-09-20T19:31:34.289 INFO:tasks.rados.rados.0.smithi042.stdout:1898: oid 6 updating version 715 to 716 2024-09-20T19:31:34.289 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 716 (ObjNum 660 snap 212 seq_num 660) dirty exists 2024-09-20T19:31:34.289 INFO:tasks.rados.rados.0.smithi042.stdout:1898: left oid 6 (ObjNum 660 snap 212 seq_num 660) 2024-09-20T19:31:34.289 INFO:tasks.rados.rados.0.smithi042.stdout:1899: finishing write tid 1 to smithi04268048-16 2024-09-20T19:31:34.289 INFO:tasks.rados.rados.0.smithi042.stdout:1899: finishing write tid 2 to smithi04268048-16 2024-09-20T19:31:34.290 INFO:tasks.rados.rados.0.smithi042.stdout:1899: finishing write tid 3 to smithi04268048-16 2024-09-20T19:31:34.292 INFO:tasks.rados.rados.0.smithi042.stdout:1899: finishing write tid 4 to smithi04268048-16 2024-09-20T19:31:34.292 INFO:tasks.rados.rados.0.smithi042.stdout:1899: finishing write tid 5 to smithi04268048-16 2024-09-20T19:31:34.292 INFO:tasks.rados.rados.0.smithi042.stdout:1899: oid 16 updating version 0 to 726 2024-09-20T19:31:34.292 INFO:tasks.rados.rados.0.smithi042.stdout:1899: oid 16 updating version 726 to 728 2024-09-20T19:31:34.292 INFO:tasks.rados.rados.0.smithi042.stdout:1899: oid 16 version 728 is already newer than 725 2024-09-20T19:31:34.293 INFO:tasks.rados.rados.0.smithi042.stdout:1899: oid 16 version 728 is already newer than 727 2024-09-20T19:31:34.293 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 728 (ObjNum 661 snap 212 seq_num 661) dirty exists 2024-09-20T19:31:34.293 INFO:tasks.rados.rados.0.smithi042.stdout:1899: left oid 16 (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:31:34.293 INFO:tasks.rados.rados.0.smithi042.stdout:1898: done (2 left) 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1899: done (1 left) 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1900: done (0 left) 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1901: rollback oid 25 current snap is 212 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 211 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1902: read oid 27 snap -1 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1902: expect deleted 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1903: rollback oid 11 current snap is 212 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 205 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1904: rollback oid 48 current snap is 212 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 208 2024-09-20T19:31:34.294 INFO:tasks.rados.rados.0.smithi042.stdout:1905: snap_create 2024-09-20T19:31:34.295 INFO:tasks.rados.rados.0.smithi042.stdout:1903: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:31:34.295 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:1901: finishing rollback tid 0 to smithi04268048-25 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:1901: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:1904: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 0 (ObjNum 129 snap 0 seq_num 1415356720) dirty dne 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:1901: finishing rollback tid 2 to smithi04268048-25 2024-09-20T19:31:34.299 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 787 (ObjNum 643 snap 206 seq_num 643) dirty exists 2024-09-20T19:31:34.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.371 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.372 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.373 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.373 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:34 smithi162 bash[24557]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:34.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.513 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:34.616 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:34.628 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[32075]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.545724+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:32.549147+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.453568+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.455576+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.779772+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:33.780944+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: audit 2024-09-20T19:31:33.824400+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: audit 2024-09-20T19:31:34.246408+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.42:0/2679862117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:34 smithi042 bash[22706]: cluster 2024-09-20T19:31:34.253124+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-20T19:31:34.643 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:34.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:34.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:34.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:34.915 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:34.915 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:35.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:35.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:35.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1901: done (4 left) 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1902: done (3 left) 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1903: done (2 left) 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1904: done (1 left) 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1905: done (0 left) 2024-09-20T19:31:35.260 INFO:tasks.rados.rados.0.smithi042.stdout:1906: delete oid 3 current snap is 213 2024-09-20T19:31:35.276 INFO:tasks.rados.rados.0.smithi042.stdout:1906: done (0 left) 2024-09-20T19:31:35.276 INFO:tasks.rados.rados.0.smithi042.stdout:1907: copy_from oid 49 from oid 11 current snap is 213 2024-09-20T19:31:35.276 INFO:tasks.rados.rados.0.smithi042.stdout:1908: write (excl) oid 40 current snap is 213 2024-09-20T19:31:35.276 INFO:tasks.rados.rados.0.smithi042.stdout:1908: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-20T19:31:35.282 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:35.283 INFO:tasks.rados.rados.0.smithi042.stdout:1908: writing smithi04268048-40 from 759848 to 1247895 tid 1 2024-09-20T19:31:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.285 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T19:31:35.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:35.301 INFO:tasks.rados.rados.0.smithi042.stdout:1908: writing smithi04268048-40 from 1989220 to 2749444 tid 2 2024-09-20T19:31:35.301 INFO:tasks.rados.rados.0.smithi042.stdout:1909: write oid 27 current snap is 213 2024-09-20T19:31:35.301 INFO:tasks.rados.rados.0.smithi042.stdout:1909: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-20T19:31:35.313 INFO:tasks.rados.rados.0.smithi042.stdout:1909: writing smithi04268048-27 from 776655 to 1339951 tid 1 2024-09-20T19:31:35.325 INFO:tasks.rados.rados.0.smithi042.stdout:1909: writing smithi04268048-27 from 2047702 to 2667937 tid 2 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1909: writing smithi04268048-27 from 3095349 to 3176655 tid 3 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1910: delete oid 46 current snap is 213 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1907: finishing copy_from to smithi04268048-49 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1907: got expected ENOENT (src dne) 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1907: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1908: finishing write tid 1 to smithi04268048-40 2024-09-20T19:31:35.330 INFO:tasks.rados.rados.0.smithi042.stdout:1908: finishing write tid 2 to smithi04268048-40 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: finishing write tid 3 to smithi04268048-40 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: finishing write tid 4 to smithi04268048-40 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: oid 40 updating version 0 to 591 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: oid 40 version 591 is already newer than 590 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: oid 40 updating version 591 to 592 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 592 (ObjNum 664 snap 213 seq_num 664) dirty exists 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1908: left oid 40 (ObjNum 664 snap 213 seq_num 664) 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1909: finishing write tid 1 to smithi04268048-27 2024-09-20T19:31:35.331 INFO:tasks.rados.rados.0.smithi042.stdout:1909: finishing write tid 2 to smithi04268048-27 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1909: finishing write tid 3 to smithi04268048-27 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1907: done (3 left) 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1908: done (2 left) 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1910: done (1 left) 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1911: rollback oid 15 current snap is 213 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 211 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1909: finishing write tid 4 to smithi04268048-27 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1912: snap_create 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1909: finishing write tid 5 to smithi04268048-27 2024-09-20T19:31:35.333 INFO:tasks.rados.rados.0.smithi042.stdout:1909: oid 27 updating version 0 to 851 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:1909: oid 27 updating version 851 to 853 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:1909: oid 27 version 853 is already newer than 852 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:1909: oid 27 version 853 is already newer than 850 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 853 (ObjNum 665 snap 213 seq_num 665) dirty exists 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:1909: left oid 27 (ObjNum 665 snap 213 seq_num 665) 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:1911: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:31:35.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 0 (ObjNum 113 snap 0 seq_num 1415329120) dirty dne 2024-09-20T19:31:35.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:35.409 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:35.512 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:35.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:35.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[32075]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[32075]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[22706]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[22706]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:35 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.745 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:35.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:35.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:35 smithi162 bash[24557]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:35 smithi162 bash[24557]: cluster 2024-09-20T19:31:34.292678+0000 mgr.y (mgr.24491) 513 : cluster [DBG] pgmap v877: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 131 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:35.790 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:35 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:35 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.260154+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-20T19:31:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:35.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T19:31:35.893 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:35.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:36.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:36.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:36.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:36.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:36.266 INFO:tasks.rados.rados.0.smithi042.stdout:1909: done (2 left) 2024-09-20T19:31:36.266 INFO:tasks.rados.rados.0.smithi042.stdout:1911: done (1 left) 2024-09-20T19:31:36.267 INFO:tasks.rados.rados.0.smithi042.stdout:1912: done (0 left) 2024-09-20T19:31:36.267 INFO:tasks.rados.rados.0.smithi042.stdout:1913: snap_remove snap 208 2024-09-20T19:31:36.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:36.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:36.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:36.432 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:36.535 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:36.546 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:36 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.564252+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.565320+0000 osd.3 (osd.3) 8 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.804931+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.e scrub starts 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.807090+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.e scrub ok 2024-09-20T19:31:36.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:36 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.265775+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-20T19:31:36.560 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:36.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:36.689 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:36.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:36.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:36.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:36.952 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:36.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:37.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:37.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:37.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:37.275 INFO:tasks.rados.rados.0.smithi042.stdout:1913: done (0 left) 2024-09-20T19:31:37.275 INFO:tasks.rados.rados.0.smithi042.stdout:1914: rollback oid 40 current snap is 214 2024-09-20T19:31:37.275 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 204 2024-09-20T19:31:37.275 INFO:tasks.rados.rados.0.smithi042.stdout:1915: snap_create 2024-09-20T19:31:37.288 INFO:tasks.rados.rados.0.smithi042.stdout:1914: finishing rollback tid 0 to smithi04268048-40 2024-09-20T19:31:37.294 INFO:tasks.rados.rados.0.smithi042.stdout:1914: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:31:37.294 INFO:tasks.rados.rados.0.smithi042.stdout:1914: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:31:37.294 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 595 (ObjNum 624 snap 196 seq_num 624) dirty exists 2024-09-20T19:31:37.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:37.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:37.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:37.482 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:31:37.482 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:31:37.501 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:31:37.502 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:31:37.519 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:31:37.519 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:31:37.535 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:37.536 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:31:37.537 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:31:37.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:37.557 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:31:37.558 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:31:37.579 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:31:37.579 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:31:37.600 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:31:37.600 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:31:37.622 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:31:37.622 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:31:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:37 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.566802+0000 osd.2 (osd.2) 189 : cluster [DBG] 2.e scrub starts 2024-09-20T19:31:37.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:35.568621+0000 osd.2 (osd.2) 190 : cluster [DBG] 2.e scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.293397+0000 mgr.y (mgr.24491) 514 : cluster [DBG] pgmap v880: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 132 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.525781+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.528315+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.547798+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.551724+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.805663+0000 osd.6 (osd.6) 144 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:31:37.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.808081+0000 osd.6 (osd.6) 145 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:31:37.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:37 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.273979+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-20T19:31:37.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:37.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:37.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:37.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:38.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:38.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:38.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:38.256 INFO:tasks.rados.rados.0.smithi042.stdout:1914: done (1 left) 2024-09-20T19:31:38.256 INFO:tasks.rados.rados.0.smithi042.stdout:1915: done (0 left) 2024-09-20T19:31:38.257 INFO:tasks.rados.rados.0.smithi042.stdout:1916: write (excl) oid 5 current snap is 215 2024-09-20T19:31:38.257 INFO:tasks.rados.rados.0.smithi042.stdout:1916: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-20T19:31:38.269 INFO:tasks.rados.rados.0.smithi042.stdout:1916: writing smithi04268048-5 from 410269 to 1124063 tid 1 2024-09-20T19:31:38.280 INFO:tasks.rados.rados.0.smithi042.stdout:1916: writing smithi04268048-5 from 1648313 to 2155864 tid 2 2024-09-20T19:31:38.292 INFO:tasks.rados.rados.0.smithi042.stdout:1916: writing smithi04268048-5 from 2917489 to 3210269 tid 3 2024-09-20T19:31:38.292 INFO:tasks.rados.rados.0.smithi042.stdout:1916: finishing write tid 1 to smithi04268048-5 2024-09-20T19:31:38.292 INFO:tasks.rados.rados.0.smithi042.stdout:1917: snap_create 2024-09-20T19:31:38.293 INFO:tasks.rados.rados.0.smithi042.stdout:1916: finishing write tid 2 to smithi04268048-5 2024-09-20T19:31:38.296 INFO:tasks.rados.rados.0.smithi042.stdout:1916: finishing write tid 3 to smithi04268048-5 2024-09-20T19:31:38.297 INFO:tasks.rados.rados.0.smithi042.stdout:1916: finishing write tid 4 to smithi04268048-5 2024-09-20T19:31:38.297 INFO:tasks.rados.rados.0.smithi042.stdout:1916: finishing write tid 5 to smithi04268048-5 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:1916: oid 5 updating version 0 to 760 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:1916: oid 5 updating version 760 to 761 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:1916: oid 5 version 761 is already newer than 759 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:1916: oid 5 updating version 761 to 762 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 762 (ObjNum 667 snap 215 seq_num 667) dirty exists 2024-09-20T19:31:38.298 INFO:tasks.rados.rados.0.smithi042.stdout:1916: left oid 5 (ObjNum 667 snap 215 seq_num 667) 2024-09-20T19:31:38.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:38.329 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:38.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:38.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:38 smithi162 bash[24557]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:38.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[32075]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.573532+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:36.575799+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: audit 2024-09-20T19:31:37.464076+0000 mon.a (mon.0) 1975 : audit [DBG] from='client.? v1:172.21.15.42:0/3283385839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.500409+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:37.502353+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:38 smithi042 bash[22706]: cluster 2024-09-20T19:31:38.254849+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-20T19:31:38.598 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:38.701 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:38.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:38.852 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:38.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:38.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:38.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:39.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:39.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:39.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:39.275 INFO:tasks.rados.rados.0.smithi042.stdout:1916: done (1 left) 2024-09-20T19:31:39.275 INFO:tasks.rados.rados.0.smithi042.stdout:1917: done (0 left) 2024-09-20T19:31:39.276 INFO:tasks.rados.rados.0.smithi042.stdout:1918: delete oid 5 current snap is 216 2024-09-20T19:31:39.289 INFO:tasks.rados.rados.0.smithi042.stdout:1918: done (0 left) 2024-09-20T19:31:39.289 INFO:tasks.rados.rados.0.smithi042.stdout:1919: write oid 18 current snap is 216 2024-09-20T19:31:39.289 INFO:tasks.rados.rados.0.smithi042.stdout:1919: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-09-20T19:31:39.298 INFO:tasks.rados.rados.0.smithi042.stdout:1919: writing smithi04268048-18 from 443883 to 908175 tid 1 2024-09-20T19:31:39.314 INFO:tasks.rados.rados.0.smithi042.stdout:1919: writing smithi04268048-18 from 1365277 to 1876497 tid 2 2024-09-20T19:31:39.334 INFO:tasks.rados.rados.0.smithi042.stdout:1919: writing smithi04268048-18 from 2572335 to 3243883 tid 3 2024-09-20T19:31:39.334 INFO:tasks.rados.rados.0.smithi042.stdout:1920: write oid 14 current snap is 216 2024-09-20T19:31:39.334 INFO:tasks.rados.rados.0.smithi042.stdout:1920: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-20T19:31:39.346 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:39.347 INFO:tasks.rados.rados.0.smithi042.stdout:1920: writing smithi04268048-14 from 460690 to 1116584 tid 1 2024-09-20T19:31:39.359 INFO:tasks.rados.rados.0.smithi042.stdout:1920: writing smithi04268048-14 from 1540112 to 2311343 tid 2 2024-09-20T19:31:39.367 INFO:tasks.rados.rados.0.smithi042.stdout:1920: writing smithi04268048-14 from 2716111 to 3255889 tid 3 2024-09-20T19:31:39.367 INFO:tasks.rados.rados.0.smithi042.stdout:1921: delete oid 23 current snap is 216 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: finishing write tid 1 to smithi04268048-18 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: finishing write tid 2 to smithi04268048-18 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: finishing write tid 3 to smithi04268048-18 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: finishing write tid 4 to smithi04268048-18 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: finishing write tid 5 to smithi04268048-18 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: oid 18 updating version 0 to 880 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: oid 18 version 880 is already newer than 879 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: oid 18 updating version 880 to 881 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: oid 18 updating version 881 to 882 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 882 (ObjNum 669 snap 216 seq_num 669) dirty exists 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1919: left oid 18 (ObjNum 669 snap 216 seq_num 669) 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1920: finishing write tid 1 to smithi04268048-14 2024-09-20T19:31:39.368 INFO:tasks.rados.rados.0.smithi042.stdout:1920: finishing write tid 2 to smithi04268048-14 2024-09-20T19:31:39.372 INFO:tasks.rados.rados.0.smithi042.stdout:1919: done (2 left) 2024-09-20T19:31:39.372 INFO:tasks.rados.rados.0.smithi042.stdout:1921: done (1 left) 2024-09-20T19:31:39.372 INFO:tasks.rados.rados.0.smithi042.stdout:1922: write oid 45 current snap is 216 2024-09-20T19:31:39.372 INFO:tasks.rados.rados.0.smithi042.stdout:1922: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-20T19:31:39.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:39.377 INFO:tasks.rados.rados.0.smithi042.stdout:1922: writing smithi04268048-45 from 494304 to 900696 tid 1 2024-09-20T19:31:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:39.385 INFO:tasks.rados.rados.0.smithi042.stdout:1922: writing smithi04268048-45 from 1540723 to 2315623 tid 2 2024-09-20T19:31:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:1922: writing smithi04268048-45 from 2770957 to 3294304 tid 3 2024-09-20T19:31:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:1923: snap_remove snap 211 2024-09-20T19:31:39.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:39 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:38.293933+0000 mgr.y (mgr.24491) 515 : cluster [DBG] pgmap v883: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.212793+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.a scrub starts 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.214427+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.a scrub ok 2024-09-20T19:31:39.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.273788+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-20T19:31:39.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:39.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:39.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:39.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.763 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:39.866 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:39.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:39.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:39.998 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:40.026 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:40.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:40.151 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:40.158 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:40.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:40.289 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:40.391 INFO:tasks.rados.rados.0.smithi042.stdout:1923: done (2 left) 2024-09-20T19:31:40.391 INFO:tasks.rados.rados.0.smithi042.stdout:1924: delete oid 46 current snap is 216 2024-09-20T19:31:40.391 INFO:tasks.rados.rados.0.smithi042.stdout:1920: finishing write tid 3 to smithi04268048-14 2024-09-20T19:31:40.391 INFO:tasks.rados.rados.0.smithi042.stdout:1920: finishing write tid 4 to smithi04268048-14 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: finishing write tid 5 to smithi04268048-14 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: oid 14 updating version 0 to 764 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: oid 14 updating version 764 to 765 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: oid 14 updating version 765 to 767 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: oid 14 version 767 is already newer than 766 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 767 (ObjNum 670 snap 216 seq_num 670) dirty exists 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1920: left oid 14 (ObjNum 670 snap 216 seq_num 670) 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: finishing write tid 1 to smithi04268048-45 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: finishing write tid 2 to smithi04268048-45 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: finishing write tid 3 to smithi04268048-45 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: finishing write tid 4 to smithi04268048-45 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: finishing write tid 5 to smithi04268048-45 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: oid 45 updating version 0 to 1093 2024-09-20T19:31:40.392 INFO:tasks.rados.rados.0.smithi042.stdout:1922: oid 45 updating version 1093 to 1094 2024-09-20T19:31:40.393 INFO:tasks.rados.rados.0.smithi042.stdout:1922: oid 45 version 1094 is already newer than 1091 2024-09-20T19:31:40.393 INFO:tasks.rados.rados.0.smithi042.stdout:1922: oid 45 version 1094 is already newer than 1092 2024-09-20T19:31:40.393 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1094 (ObjNum 672 snap 216 seq_num 672) dirty exists 2024-09-20T19:31:40.393 INFO:tasks.rados.rados.0.smithi042.stdout:1922: left oid 45 (ObjNum 672 snap 216 seq_num 672) 2024-09-20T19:31:40.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:40.412 INFO:tasks.rados.rados.0.smithi042.stdout:1920: done (2 left) 2024-09-20T19:31:40.412 INFO:tasks.rados.rados.0.smithi042.stdout:1922: done (1 left) 2024-09-20T19:31:40.412 INFO:tasks.rados.rados.0.smithi042.stdout:1924: done (0 left) 2024-09-20T19:31:40.412 INFO:tasks.rados.rados.0.smithi042.stdout:1925: write oid 21 current snap is 216 2024-09-20T19:31:40.412 INFO:tasks.rados.rados.0.smithi042.stdout:1925: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-20T19:31:40.418 INFO:tasks.rados.rados.0.smithi042.stdout:1925: writing smithi04268048-21 from 527918 to 1084808 tid 1 2024-09-20T19:31:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:40.426 INFO:tasks.rados.rados.0.smithi042.stdout:1925: writing smithi04268048-21 from 1657687 to 2436256 tid 2 2024-09-20T19:31:40.431 INFO:tasks.rados.rados.0.smithi042.stdout:1925: writing smithi04268048-21 from 3225803 to 3327918 tid 3 2024-09-20T19:31:40.431 INFO:tasks.rados.rados.0.smithi042.stdout:1926: write (excl) oid 46 current snap is 216 2024-09-20T19:31:40.432 INFO:tasks.rados.rados.0.smithi042.stdout:1925: finishing write tid 1 to smithi04268048-21 2024-09-20T19:31:40.432 INFO:tasks.rados.rados.0.smithi042.stdout:1926: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-09-20T19:31:40.438 INFO:tasks.rados.rados.0.smithi042.stdout:1926: writing smithi04268048-46 from 544725 to 1176864 tid 1 2024-09-20T19:31:40.444 INFO:tasks.rados.rados.0.smithi042.stdout:1926: writing smithi04268048-46 from 1716169 to 2238396 tid 2 2024-09-20T19:31:40.450 INFO:tasks.rados.rados.0.smithi042.stdout:1926: writing smithi04268048-46 from 2736873 to 3249364 tid 3 2024-09-20T19:31:40.450 INFO:tasks.rados.rados.0.smithi042.stdout:1927: snap_remove snap 215 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.525 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.525 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:40 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:40.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:40.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:31:40.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:40.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: audit 2024-09-20T19:31:39.547308+0000 mgr.y (mgr.24491) 516 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:40.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.878442+0000 osd.7 (osd.7) 149 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:31:40.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:40 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.884429+0000 osd.7 (osd.7) 150 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:31:40.667 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:40.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:40.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:40.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:40.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:40.935 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:40.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:41.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:41.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:41.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:41.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:1927: done (2 left) 2024-09-20T19:31:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:1928: rollback oid 29 current snap is 216 2024-09-20T19:31:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 214 2024-09-20T19:31:41.388 INFO:tasks.rados.rados.0.smithi042.stdout:1929: snap_create 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: finishing write tid 2 to smithi04268048-21 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: finishing write tid 3 to smithi04268048-21 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: finishing write tid 4 to smithi04268048-21 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: finishing write tid 5 to smithi04268048-21 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: oid 21 updating version 0 to 888 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: oid 21 updating version 888 to 889 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: oid 21 version 889 is already newer than 887 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: oid 21 version 889 is already newer than 886 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 889 (ObjNum 674 snap 216 seq_num 674) dirty exists 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1925: left oid 21 (ObjNum 674 snap 216 seq_num 674) 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1926: finishing write tid 1 to smithi04268048-46 2024-09-20T19:31:41.389 INFO:tasks.rados.rados.0.smithi042.stdout:1926: finishing write tid 2 to smithi04268048-46 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: finishing write tid 3 to smithi04268048-46 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: finishing write tid 4 to smithi04268048-46 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: finishing write tid 5 to smithi04268048-46 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: oid 46 updating version 0 to 1038 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: oid 46 version 1038 is already newer than 1035 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: oid 46 version 1038 is already newer than 1036 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: oid 46 version 1038 is already newer than 1037 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1038 (ObjNum 675 snap 216 seq_num 675) dirty exists 2024-09-20T19:31:41.390 INFO:tasks.rados.rados.0.smithi042.stdout:1926: left oid 46 (ObjNum 675 snap 216 seq_num 675) 2024-09-20T19:31:41.397 INFO:tasks.rados.rados.0.smithi042.stdout:1928: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:31:41.397 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 650 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:31:41.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:41.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:41.569 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.569 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.569 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.569 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.570 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:41 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.543375+0000 osd.2 (osd.2) 193 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:39.546436+0000 osd.2 (osd.2) 194 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.294524+0000 mgr.y (mgr.24491) 517 : cluster [DBG] pgmap v885: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 126 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.389346+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.416853+0000 osd.3 (osd.3) 13 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:31:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:41 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.419572+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:31:41.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:41.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:41.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:41.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:41.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:41.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:42.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:42.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:42.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:42.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1176 7.d 7:b90bdf9e:::smithi04268048-23:head [copy-from ver 162] snapc 39=[39,38,37,36,33,30,2c] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 490.09770566700001, 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028448029, 2024-09-20T19:31:42.338 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1176 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240931+0000", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240937+0000", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4999999999999999e-06 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.240956+0000", 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9012999999999999e-05 2024-09-20T19:31:42.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241002+0000", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5838999999999999e-05 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241025+0000", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3869000000000001e-05 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241233+0000", 2024-09-20T19:31:42.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020778999999999999 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.241494+0000", 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000260824 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.255282+0000", 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013787785 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.266475+0000", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011193602 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.267752+0000", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001276447 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269090+0000", 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001338253 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269191+0000", 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010124399999999999 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:32.269379+0000", 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187863 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.344 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2132 7.d 7:baa4e37b:::smithi04268048-17:head [copy-from ver 407] snapc 5f=[5f,5e,5d,5a,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 416.09711901899999, 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034554666999999997, 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2132 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241518+0000", 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241522+0000", 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4150000000000001e-06 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241538+0000", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5338000000000001e-05 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241570+0000", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2070999999999998e-05 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241585+0000", 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4982e-05 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241798+0000", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021353400000000001 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241819+0000", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0426000000000001e-05 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.246976+0000", 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051576520000000004 2024-09-20T19:31:42.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.247111+0000", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134358 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.257327+0000", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010216182000000001 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.266016+0000", 2024-09-20T19:31:42.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0086890890000000005 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.273265+0000", 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072488439999999999 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.275923+0000", 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026584500000000001 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.276003+0000", 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9417000000000002e-05 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.276073+0000", 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9908999999999998e-05 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.351 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2355 7.c 7:33675e62:::smithi04268048-1:head [copy-from ver 311] snapc 6c=[6c,6b,6a,69,68,67,66,65,63,62,5e,5d,5a,57,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 397.23887973000001, 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035881936000000003, 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2355 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099757+0000", 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099760+0000", 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.106e-06 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099770+0000", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3549999999999994e-06 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099788+0000", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8496999999999999e-05 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099798+0000", 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0020000000000001e-05 2024-09-20T19:31:42.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099977+0000", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017844000000000001 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.099997+0000", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0628999999999998e-05 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105625+0000", 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056274300000000001 2024-09-20T19:31:42.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.105761+0000", 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136654 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.121616+0000", 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015854995 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.132728+0000", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011111653000000001 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.132861+0000", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133237 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135569+0000", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002707845 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135607+0000", 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8127999999999998e-05 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:05.135639+0000", 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1946999999999998e-05 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2586 7.d 7:b90bdf9e:::smithi04268048-23:head [rollback 69] snapc 74=[74,73,72,70,6d,6c,69,67,63,5e,5d,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:31:42.358 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 379.23745652999997, 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028973180000000001, 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2586 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101180+0000", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101181+0000", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1099999999999995e-07 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101184+0000", 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9390000000000002e-06 2024-09-20T19:31:42.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101191+0000", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.905e-06 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.101194+0000", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5640000000000001e-06 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.113927+0000", 2024-09-20T19:31:42.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012732581 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.114158+0000", 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023086 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:23.130154+0000", 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015995720000000001 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3145 7.c 7:3b97d3fe:::smithi04268048-6:head [copy-from ver 496] snapc 90=[90,8f,8e,8d,8c,8b,8a,89,88,87,86,85,84,83,82,81,7f,7c,79,72,41] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 330.058606199, 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030932989000000001, 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3145 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280031+0000", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280037+0000", 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1569999999999997e-06 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280055+0000", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8068999999999999e-05 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280092+0000", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7367000000000003e-05 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280111+0000", 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8732999999999998e-05 2024-09-20T19:31:42.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280202+0000", 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1125000000000006e-05 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.280219+0000", 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6418000000000001e-05 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283343+0000", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031240460000000001 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.283476+0000", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013334299999999999 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.294947+0000", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011470721999999999 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.309429+0000", 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014482116999999999 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310736+0000", 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013068419999999999 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310763+0000", 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6913000000000001e-05 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310873+0000", 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010998200000000001 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:12.310964+0000", 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1155000000000004e-05 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3546 7.d 7:b3515764:::smithi04268048-3:head [copy-from ver 568] snapc 9f=[9f,9e,9d,9c,9b,99,98,94,8d,8b,8a,88,87,86,84,81,7f,72] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-20T19:31:42.369 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 297.043445029, 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033137336000000003, 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3546 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295192+0000", 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295197+0000", 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5999999999999997e-06 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295222+0000", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4366e-05 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295259+0000", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6973e-05 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295278+0000", 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9063e-05 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295461+0000", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018272000000000001 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295477+0000", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6799999999999998e-05 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.303293+0000", 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078156179999999999 2024-09-20T19:31:42.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.303631+0000", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033842200000000001 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.315863+0000", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012231367 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.327364+0000", 2024-09-20T19:31:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011501711 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328001+0000", 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063663100000000002 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328115+0000", 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011380800000000001 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.328329+0000", 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021425700000000001 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3547 7.d 7:b3515764:::smithi04268048-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e416)", 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 297.04332044099999, 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031928874000000003, 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.376 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3547 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295316+0000", 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295318+0000", 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295324+0000", 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8499999999999999e-06 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295336+0000", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1963e-05 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295345+0000", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6330000000000008e-06 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295580+0000", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023457999999999999 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:31:42.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.295600+0000", 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0346000000000001e-05 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.304072+0000", 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084717909999999993 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.304244+0000", 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017183500000000001 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.326603+0000", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022359176000000001 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.326840+0000", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023657300000000001 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:45.327245+0000", 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040576700000000002 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.381 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4005 7.d 7:b3515764:::smithi04268048-3:head [stat,delete] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.808274+0000", 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.530363340999997, 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030574449, 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4005 2024-09-20T19:31:42.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808274+0000", 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808274+0000", 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808276+0000", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4870000000000001e-06 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808285+0000", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0839999999999998e-06 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808304+0000", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8442999999999999e-05 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808316+0000", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2914999999999999e-05 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808498+0000", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000181569 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810265+0000", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017670120000000001 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.833066+0000", 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022801305000000001 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.833282+0000", 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021598 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.833792+0000", 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051012799999999999 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.836811+0000", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030190310000000001 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.836882+0000", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0520999999999998e-05 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.838382+0000", 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001499872 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.838497+0000", 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011506300000000001 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.838848+0000", 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035103899999999998 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4022 7.18 7:1b5f1bb3:::smithi04268048-46:head [write 443850~564366,stat] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.388 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.870550+0000", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.468086597999999, 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92715243400000003, 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4022 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.870550+0000", 2024-09-20T19:31:42.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.870550+0000", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.870553+0000", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.872635+0000", 2024-09-20T19:31:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020824820000000001 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.872657+0000", 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1843000000000001e-05 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.872671+0000", 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4432e-05 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.872770+0000", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8148999999999998e-05 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.872782+0000", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1904e-05 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.788514+0000", 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91573266499999995 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.788940+0000", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042619400000000002 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.789570+0000", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062983099999999997 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.793050+0000", 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034793409999999999 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.793085+0000", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5293000000000002e-05 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.797400+0000", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043148639999999999 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.797491+0000", 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1051000000000006e-05 2024-09-20T19:31:42.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.797703+0000", 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021196999999999999 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4023 7.18 7:1b5f1bb3:::smithi04268048-46:head [write 1634861~732856] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.880432+0000", 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.458205174, 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.919251912, 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.395 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4023 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.880432+0000", 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.880432+0000", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.880435+0000", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9220000000000001e-06 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.881174+0000", 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073959700000000004 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.881192+0000", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8046e-05 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.881201+0000", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6349999999999995e-06 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.881325+0000", 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012406599999999999 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.881355+0000", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9709e-05 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.793110+0000", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91175492000000002 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.793205+0000", 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5409999999999996e-05 2024-09-20T19:31:42.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.793354+0000", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014887799999999999 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799033+0000", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056793039999999996 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799061+0000", 2024-09-20T19:31:42.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7398000000000001e-05 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799193+0000", 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132459 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799328+0000", 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013537900000000001 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799684+0000", 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000355189 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4024 7.18 7:1b5f1bb3:::smithi04268048-46:head [write 2770646~473204] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.885964+0000", 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.452673246000003, 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.914601306, 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4024 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.885964+0000", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.885964+0000", 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.885975+0000", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1708999999999999e-05 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886421+0000", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044576199999999999 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886445+0000", 2024-09-20T19:31:42.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4029999999999999e-05 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886461+0000", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.56e-05 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886564+0000", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103249 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886585+0000", 2024-09-20T19:31:42.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1145000000000002e-05 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.794283+0000", 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90769811899999997 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.794645+0000", 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036140199999999998 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:31:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.795242+0000", 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059761899999999997 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799074+0000", 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038312889999999999 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799949+0000", 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087546499999999997 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800082+0000", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013263899999999999 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800213+0000", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013162300000000001 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800565+0000", 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035165499999999998 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4025 7.18 7:1b5f1bb3:::smithi04268048-46:head [setxattr _header (58) in=7b,truncate 3243850] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.886513+0000", 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.452124022999996, 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91475257600000004, 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4025 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886513+0000", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886513+0000", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886515+0000", 2024-09-20T19:31:42.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.826e-06 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886523+0000", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1349999999999992e-06 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886533+0000", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8640000000000008e-06 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886541+0000", 2024-09-20T19:31:42.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4489999999999999e-06 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886638+0000", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6668000000000003e-05 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886655+0000", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7104000000000001e-05 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.796652+0000", 2024-09-20T19:31:42.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90999677499999998 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.796816+0000", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016411099999999999 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.797013+0000", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000197412 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.799833+0000", 2024-09-20T19:31:42.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028197140000000001 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800707+0000", 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087432799999999995 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800862+0000", 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154322 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.800986+0000", 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012422600000000001 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.801265+0000", 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027964199999999998 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4026 7.18 7:1b5f1bb3:::smithi04268048-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e501)", 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:09.886559+0000", 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.452078291000007, 2024-09-20T19:31:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91632666299999999, 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4026 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886559+0000", 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886559+0000", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886560+0000", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7199999999999997e-07 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886564+0000", 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2039999999999999e-06 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886573+0000", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9609999999999994e-06 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886580+0000", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0999999999999998e-06 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886698+0000", 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011858000000000001 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.886715+0000", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6524999999999998e-05 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.801695+0000", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91498025800000005 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.801917+0000", 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022189599999999999 2024-09-20T19:31:42.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:10.802885+0000", 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096816700000000003 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2593 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94817798629632 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.311077+0000", 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.027559443000001, 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0769074089999999, 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.420 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2593 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311077+0000", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311077+0000", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311081+0000", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7390000000000001e-06 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311090+0000", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1179999999999998e-06 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311106+0000", 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.607e-05 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311119+0000", 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2269000000000001e-05 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311215+0000", 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5985999999999999e-05 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311224+0000", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2380000000000003e-06 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.323878+0000", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012654331 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.323886+0000", 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5270000000000001e-06 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.370253+0000", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.046367343 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.370493+0000", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023962300000000001 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.387985+0000", 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017492165 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.425 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2594 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94817801929728 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.311150+0000", 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.027486729000003, 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.07646036, 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:31:42.426 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2594 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311150+0000", 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311150+0000", 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311151+0000", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.066e-06 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311156+0000", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8500000000000002e-06 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311164+0000", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6639999999999998e-06 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311171+0000", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7950000000000002e-06 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311286+0000", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115927 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.311294+0000", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0620000000000003e-06 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.330255+0000", 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018961143999999999 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.330272+0000", 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6971000000000001e-05 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.369462+0000", 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0391901589999999 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.369778+0000", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031649100000000002 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.387611+0000", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017832231 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4520 7.19 7:9b44618a:::smithi04268048-36:head [read 0~2369919,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.405173+0000", 2024-09-20T19:31:42.431 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.933463465000003, 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030133007999999999, 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4520 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405173+0000", 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405173+0000", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405179+0000", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1540000000000001e-06 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405194+0000", 2024-09-20T19:31:42.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5770999999999999e-05 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405238+0000", 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3949999999999998e-05 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405261+0000", 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2467999999999999e-05 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.406828+0000", 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001567414 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407050+0000", 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221898 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.432206+0000", 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025156265000000001 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.432282+0000", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5632999999999999e-05 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435306+0000", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030244550000000001 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4521 7.19 7:9b44618a:::smithi04268048-36:head [sparse-read 0~1770676] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.405291+0000", 2024-09-20T19:31:42.436 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.933345473999999, 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030247862, 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4521 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405291+0000", 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405291+0000", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405293+0000", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.08e-06 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405302+0000", 2024-09-20T19:31:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1049999999999995e-06 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405319+0000", 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7170999999999999e-05 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405334+0000", 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5393999999999999e-05 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407122+0000", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017878099999999999 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407307+0000", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000184718 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435354+0000", 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028046862999999998 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435423+0000", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9058000000000002e-05 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435539+0000", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116663 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4522 7.19 7:9b44618a:::smithi04268048-36:head [sparse-read 413056~1770676] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.405354+0000", 2024-09-20T19:31:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.933283009999997, 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030371591999999999, 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4522 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405354+0000", 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405354+0000", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405355+0000", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5510000000000001e-06 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405369+0000", 2024-09-20T19:31:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3179999999999999e-05 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405384+0000", 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5845000000000002e-05 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405398+0000", 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3188e-05 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407355+0000", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019569850000000001 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407513+0000", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015880199999999999 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435575+0000", 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028061195000000001 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435632+0000", 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7351e-05 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.435725+0000", 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3494999999999998e-05 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4737 7.c 7:38c1396f:::smithi04268048-31:head [copy-from ver 581] snapc d0=[d0,cf,ce,cc] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-20T19:31:42.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:24.265909+0000", 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.0727279, 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029091457000000001, 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4737 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265909+0000", 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265909+0000", 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265913+0000", 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6660000000000001e-06 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265925+0000", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2275e-05 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265946+0000", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1401000000000001e-05 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265962+0000", 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6039999999999999e-05 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266034+0000", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1734000000000001e-05 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266121+0000", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7218000000000004e-05 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.282616+0000", 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016494372 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294416+0000", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011800566 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294806+0000", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038948200000000002 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294831+0000", 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5649000000000001e-05 2024-09-20T19:31:42.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294921+0000", 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9503e-05 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.295000+0000", 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9550999999999998e-05 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4738 7.c 7:38c1396f:::smithi04268048-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e576)", 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:24.265988+0000", 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.072648726000001, 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028361796000000002, 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4738 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265988+0000", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265988+0000", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265989+0000", 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.24e-06 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.265995+0000", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1769999999999997e-06 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266007+0000", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2904e-05 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266017+0000", 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4280000000000002e-06 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266310+0000", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000293367 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.266357+0000", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7079000000000001e-05 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294194+0000", 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027836487 2024-09-20T19:31:42.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294270+0000", 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6173000000000004e-05 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294350+0000", 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9940999999999999e-05 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:31:42.457 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:31:42.458 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:42.458 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1925: done (3 left) 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1926: done (2 left) 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1928: done (1 left) 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1929: done (0 left) 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1930: read oid 41 snap -1 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1930: expect deleted 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1931: rollback oid 24 current snap is 217 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 213 2024-09-20T19:31:42.460 INFO:tasks.rados.rados.0.smithi042.stdout:1932: read oid 11 snap -1 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1932: expect deleted 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1933: rollback oid 29 current snap is 217 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 216 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1934: read oid 38 snap -1 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1934: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1935: read oid 16 snap 212 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1935: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1936: delete oid 22 current snap is 217 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1932: done (6 left) 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1936: done (5 left) 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1937: read oid 1 snap -1 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1937: expect (ObjNum 592 snap 189 seq_num 592) 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1938: write oid 49 current snap is 217 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1938: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-20T19:31:42.461 INFO:tasks.rados.rados.0.smithi042.stdout:1938: writing smithi04268048-49 from 578339 to 1077329 tid 1 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1938: writing smithi04268048-49 from 1833133 to 2359029 tid 2 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1938: writing smithi04268048-49 from 2791719 to 3378339 tid 3 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1939: rollback oid 3 current snap is 217 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 213 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1931: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1931: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1930: done (7 left) 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1940: snap_create 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1933: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 650 (ObjNum 177 snap 0 seq_num 1415014272) dirty dne 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1938: finishing write tid 1 to smithi04268048-49 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:1935: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:31:42.462 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:42.522 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:42 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.571 INFO:tasks.rados.rados.0.smithi042.stdout:1931: finishing rollback tid 2 to smithi04268048-24 2024-09-20T19:31:42.571 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 731 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:31:42.571 INFO:tasks.rados.rados.0.smithi042.stdout:1934: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:42.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:42.609 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.537333+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.c scrub starts 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:40.539583+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.c scrub ok 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.386332+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.451837+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.454758+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: audit 2024-09-20T19:31:42.347227+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.379076+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: audit 2024-09-20T19:31:42.380618+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? v1:172.21.15.42:0/3898265825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:42 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.387888+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-20T19:31:42.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:42.716 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 847784 2024-09-20T19:31:42.735 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:42.777 INFO:tasks.rados.rados.0.smithi042.stdout:1938: finishing write tid 2 to smithi04268048-49 2024-09-20T19:31:42.777 INFO:tasks.rados.rados.0.smithi042.stdout:1937: expect (ObjNum 592 snap 189 seq_num 592) 2024-09-20T19:31:42.777 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:42.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:42.875 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1666002 2024-09-20T19:31:42.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:42.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:42.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1939: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 706 (ObjNum 33 snap 0 seq_num 1415368320) dirty dne 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: finishing write tid 3 to smithi04268048-49 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: finishing write tid 4 to smithi04268048-49 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: finishing write tid 5 to smithi04268048-49 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: oid 49 updating version 0 to 1098 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: oid 49 version 1098 is already newer than 1095 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: oid 49 version 1098 is already newer than 1097 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:1938: oid 49 version 1098 is already newer than 1096 2024-09-20T19:31:43.017 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1098 (ObjNum 677 snap 217 seq_num 677) dirty exists 2024-09-20T19:31:43.018 INFO:tasks.rados.rados.0.smithi042.stdout:1938: left oid 49 (ObjNum 677 snap 217 seq_num 677) 2024-09-20T19:31:43.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:43.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:31:43.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1931: done (7 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1933: done (6 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1934: done (5 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1935: done (4 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1937: done (3 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1938: done (2 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1939: done (1 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1940: done (0 left) 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1941: read oid 29 snap -1 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1941: expect deleted 2024-09-20T19:31:43.253 INFO:tasks.rados.rados.0.smithi042.stdout:1942: write (excl) oid 13 current snap is 218 2024-09-20T19:31:43.254 INFO:tasks.rados.rados.0.smithi042.stdout:1942: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-20T19:31:43.259 INFO:tasks.rados.rados.0.smithi042.stdout:1942: writing smithi04268048-13 from 595146 to 1169385 tid 1 2024-09-20T19:31:43.260 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:43.268 INFO:tasks.rados.rados.0.smithi042.stdout:1942: writing smithi04268048-13 from 1891615 to 2561169 tid 2 2024-09-20T19:31:43.273 INFO:tasks.rados.rados.0.smithi042.stdout:1942: writing smithi04268048-13 from 3219142 to 3395146 tid 3 2024-09-20T19:31:43.273 INFO:tasks.rados.rados.0.smithi042.stdout:1943: write (excl) oid 31 current snap is 218 2024-09-20T19:31:43.273 INFO:tasks.rados.rados.0.smithi042.stdout:1943: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-09-20T19:31:43.280 INFO:tasks.rados.rados.0.smithi042.stdout:1943: writing smithi04268048-31 from 611953 to 1261441 tid 1 2024-09-20T19:31:43.287 INFO:tasks.rados.rados.0.smithi042.stdout:1943: writing smithi04268048-31 from 1950097 to 2479662 tid 2 2024-09-20T19:31:43.293 INFO:tasks.rados.rados.0.smithi042.stdout:1943: writing smithi04268048-31 from 3246565 to 3411953 tid 3 2024-09-20T19:31:43.293 INFO:tasks.rados.rados.0.smithi042.stdout:1944: write (excl) oid 34 current snap is 218 2024-09-20T19:31:43.293 INFO:tasks.rados.rados.0.smithi042.stdout:1944: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-20T19:31:43.300 INFO:tasks.rados.rados.0.smithi042.stdout:1944: writing smithi04268048-34 from 628760 to 1353497 tid 1 2024-09-20T19:31:43.308 INFO:tasks.rados.rados.0.smithi042.stdout:1944: writing smithi04268048-34 from 1892226 to 2565449 tid 2 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1944: writing smithi04268048-34 from 3157635 to 3428760 tid 3 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1945: read oid 28 snap 217 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1945: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1946: snap_create 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: finishing write tid 1 to smithi04268048-13 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: finishing write tid 2 to smithi04268048-13 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: finishing write tid 3 to smithi04268048-13 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: finishing write tid 4 to smithi04268048-13 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: finishing write tid 5 to smithi04268048-13 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: oid 13 updating version 0 to 569 2024-09-20T19:31:43.313 INFO:tasks.rados.rados.0.smithi042.stdout:1942: oid 13 version 569 is already newer than 568 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1942: oid 13 updating version 569 to 570 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1942: oid 13 updating version 570 to 571 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 571 (ObjNum 678 snap 218 seq_num 678) dirty exists 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1942: left oid 13 (ObjNum 678 snap 218 seq_num 678) 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: finishing write tid 1 to smithi04268048-31 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: finishing write tid 2 to smithi04268048-31 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: finishing write tid 3 to smithi04268048-31 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: finishing write tid 4 to smithi04268048-31 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: finishing write tid 5 to smithi04268048-31 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: oid 31 updating version 0 to 719 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: oid 31 updating version 719 to 720 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: oid 31 version 720 is already newer than 718 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:1943: oid 31 version 720 is already newer than 717 2024-09-20T19:31:43.314 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 720 (ObjNum 679 snap 218 seq_num 679) dirty exists 2024-09-20T19:31:43.315 INFO:tasks.rados.rados.0.smithi042.stdout:1943: left oid 31 (ObjNum 679 snap 218 seq_num 679) 2024-09-20T19:31:43.315 INFO:tasks.rados.rados.0.smithi042.stdout:1944: finishing write tid 1 to smithi04268048-34 2024-09-20T19:31:43.315 INFO:tasks.rados.rados.0.smithi042.stdout:1944: finishing write tid 2 to smithi04268048-34 2024-09-20T19:31:43.316 INFO:tasks.rados.rados.0.smithi042.stdout:1944: finishing write tid 3 to smithi04268048-34 2024-09-20T19:31:43.317 INFO:tasks.rados.rados.0.smithi042.stdout:1945: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:31:43.364 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.386 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:31:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:1944: finishing write tid 4 to smithi04268048-34 2024-09-20T19:31:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:1944: finishing write tid 5 to smithi04268048-34 2024-09-20T19:31:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:1944: oid 34 updating version 0 to 721 2024-09-20T19:31:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:1944: oid 34 version 721 is already newer than 720 2024-09-20T19:31:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:1944: oid 34 updating version 721 to 723 2024-09-20T19:31:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:1944: oid 34 version 723 is already newer than 722 2024-09-20T19:31:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 723 (ObjNum 680 snap 218 seq_num 680) dirty exists 2024-09-20T19:31:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:1944: left oid 34 (ObjNum 680 snap 218 seq_num 680) 2024-09-20T19:31:43.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:43.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:43.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[22706]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.629 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:43 smithi042 bash[32075]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.642 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:43.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.496864+0000 osd.2 (osd.2) 197 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:31:43.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:41.498227+0000 osd.2 (osd.2) 198 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:42.295812+0000 mgr.y (mgr.24491) 518 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 128 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: audit 2024-09-20T19:31:42.858620+0000 mon.a (mon.0) 1984 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:43 smithi162 bash[24557]: cluster 2024-09-20T19:31:43.252488+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-20T19:31:43.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.901 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:43.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:44.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:44.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:44.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T19:31:44.263 INFO:tasks.rados.rados.0.smithi042.stdout:1941: done (5 left) 2024-09-20T19:31:44.263 INFO:tasks.rados.rados.0.smithi042.stdout:1942: done (4 left) 2024-09-20T19:31:44.264 INFO:tasks.rados.rados.0.smithi042.stdout:1943: done (3 left) 2024-09-20T19:31:44.264 INFO:tasks.rados.rados.0.smithi042.stdout:1944: done (2 left) 2024-09-20T19:31:44.264 INFO:tasks.rados.rados.0.smithi042.stdout:1945: done (1 left) 2024-09-20T19:31:44.264 INFO:tasks.rados.rados.0.smithi042.stdout:1946: done (0 left) 2024-09-20T19:31:44.264 INFO:tasks.rados.rados.0.smithi042.stdout:1947: snap_create 2024-09-20T19:31:44.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:44.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.423 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:44.526 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:44.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:44 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.124021+0000 mon.c (mon.2) 85 : audit [INF] from='client.? v1:172.21.15.42:0/594131831' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.124443+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25523 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: audit 2024-09-20T19:31:44.251782+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25523 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:44 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.259075+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-20T19:31:44.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:44.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:44.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:44.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:44.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:45.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:45.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:45.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:45.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:45.269 INFO:tasks.rados.rados.0.smithi042.stdout:1947: done (0 left) 2024-09-20T19:31:45.269 INFO:tasks.rados.rados.0.smithi042.stdout:1948: snap_create 2024-09-20T19:31:45.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:45.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T19:31:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:45.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:45.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:45 smithi162 bash[24557]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:45.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:45.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[22706]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.296541+0000 mgr.y (mgr.24491) 519 : cluster [DBG] pgmap v892: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 140 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.614934+0000 osd.7 (osd.7) 151 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:44.615996+0000 osd.7 (osd.7) 152 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:45 smithi042 bash[32075]: cluster 2024-09-20T19:31:45.268354+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-20T19:31:45.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:45.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:45.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:45.848 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:31:45.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:45.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:45.977 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:45.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:46.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:46.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:46.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:46.268 INFO:tasks.rados.rados.0.smithi042.stdout:1948: done (0 left) 2024-09-20T19:31:46.268 INFO:tasks.rados.rados.0.smithi042.stdout:1949: read oid 38 snap -1 2024-09-20T19:31:46.268 INFO:tasks.rados.rados.0.smithi042.stdout:1949: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:46.268 INFO:tasks.rados.rados.0.smithi042.stdout:1950: write oid 18 current snap is 221 2024-09-20T19:31:46.268 INFO:tasks.rados.rados.0.smithi042.stdout:1950: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-20T19:31:46.281 INFO:tasks.rados.rados.0.smithi042.stdout:1950: writing smithi04268048-18 from 645567 to 1445553 tid 1 2024-09-20T19:31:46.290 INFO:tasks.rados.rados.0.smithi042.stdout:1950: writing smithi04268048-18 from 1950708 to 2483942 tid 2 2024-09-20T19:31:46.299 INFO:tasks.rados.rados.0.smithi042.stdout:1950: writing smithi04268048-18 from 3185058 to 3445567 tid 3 2024-09-20T19:31:46.299 INFO:tasks.rados.rados.0.smithi042.stdout:1951: rollback oid 2 current snap is 221 2024-09-20T19:31:46.299 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 204 2024-09-20T19:31:46.299 INFO:tasks.rados.rados.0.smithi042.stdout:1949: expect (ObjNum 471 snap 152 seq_num 471) 2024-09-20T19:31:46.320 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1731479 2024-09-20T19:31:46.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:46.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:46.412 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3450989 2024-09-20T19:31:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:46.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.492 INFO:tasks.rados.rados.0.smithi042.stdout:1950: finishing write tid 1 to smithi04268048-18 2024-09-20T19:31:46.492 INFO:tasks.rados.rados.0.smithi042.stdout:1950: finishing write tid 2 to smithi04268048-18 2024-09-20T19:31:46.492 INFO:tasks.rados.rados.0.smithi042.stdout:1950: finishing write tid 3 to smithi04268048-18 2024-09-20T19:31:46.492 INFO:tasks.rados.rados.0.smithi042.stdout:1950: finishing write tid 4 to smithi04268048-18 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: finishing write tid 5 to smithi04268048-18 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: oid 18 updating version 0 to 895 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: oid 18 updating version 895 to 898 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: oid 18 version 898 is already newer than 897 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: oid 18 version 898 is already newer than 896 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 898 (ObjNum 681 snap 221 seq_num 681) dirty exists 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: left oid 18 (ObjNum 681 snap 221 seq_num 681) 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1951: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1951: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1951: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 609 (ObjNum 630 snap 198 seq_num 630) dirty exists 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1949: done (2 left) 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1950: done (1 left) 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1951: done (0 left) 2024-09-20T19:31:46.493 INFO:tasks.rados.rados.0.smithi042.stdout:1952: write oid 42 current snap is 221 2024-09-20T19:31:46.494 INFO:tasks.rados.rados.0.smithi042.stdout:1952: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-20T19:31:46.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:1952: writing smithi04268048-42 from 662374 to 1137609 tid 1 2024-09-20T19:31:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:46.510 INFO:tasks.rados.rados.0.smithi042.stdout:1952: writing smithi04268048-42 from 1609190 to 2286082 tid 2 2024-09-20T19:31:46.519 INFO:tasks.rados.rados.0.smithi042.stdout:1952: writing smithi04268048-42 from 2812481 to 3436582 tid 3 2024-09-20T19:31:46.519 INFO:tasks.rados.rados.0.smithi042.stdout:1953: snap_remove snap 220 2024-09-20T19:31:46.555 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:46.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:46.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:46.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:46.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:46.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:46.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:47.013 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:47.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:47.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:47.247 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1953: done (1 left) 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1954: copy_from oid 15 from oid 35 current snap is 221 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1952: finishing write tid 1 to smithi04268048-42 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1952: finishing write tid 2 to smithi04268048-42 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1952: finishing write tid 3 to smithi04268048-42 2024-09-20T19:31:47.279 INFO:tasks.rados.rados.0.smithi042.stdout:1952: finishing write tid 4 to smithi04268048-42 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: finishing write tid 5 to smithi04268048-42 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: oid 42 updating version 0 to 858 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: oid 42 version 858 is already newer than 856 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: oid 42 updating version 858 to 859 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: oid 42 version 859 is already newer than 857 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 859 (ObjNum 682 snap 221 seq_num 682) dirty exists 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: left oid 42 (ObjNum 682 snap 221 seq_num 682) 2024-09-20T19:31:47.280 INFO:tasks.rados.rados.0.smithi042.stdout:1952: done (1 left) 2024-09-20T19:31:47.281 INFO:tasks.rados.rados.0.smithi042.stdout:1955: write oid 13 current snap is 221 2024-09-20T19:31:47.281 INFO:tasks.rados.rados.0.smithi042.stdout:1955: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-20T19:31:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:47.286 INFO:tasks.rados.rados.0.smithi042.stdout:1955: writing smithi04268048-13 from 679181 to 1229665 tid 1 2024-09-20T19:31:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:47.293 INFO:tasks.rados.rados.0.smithi042.stdout:1955: writing smithi04268048-13 from 1667672 to 2204575 tid 2 2024-09-20T19:31:47.300 INFO:tasks.rados.rados.0.smithi042.stdout:1955: writing smithi04268048-13 from 2839904 to 3275277 tid 3 2024-09-20T19:31:47.300 INFO:tasks.rados.rados.0.smithi042.stdout:1956: snap_create 2024-09-20T19:31:47.300 INFO:tasks.rados.rados.0.smithi042.stdout:1955: finishing write tid 1 to smithi04268048-13 2024-09-20T19:31:47.300 INFO:tasks.rados.rados.0.smithi042.stdout:1954: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:31:47.306 INFO:tasks.rados.rados.0.smithi042.stdout:1955: finishing write tid 2 to smithi04268048-13 2024-09-20T19:31:47.310 INFO:tasks.rados.rados.0.smithi042.stdout:1954: finishing copy_from to smithi04268048-15 2024-09-20T19:31:47.310 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 716 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:31:47.310 INFO:tasks.rados.rados.0.smithi042.stdout:1955: finishing write tid 3 to smithi04268048-13 2024-09-20T19:31:47.310 INFO:tasks.rados.rados.0.smithi042.stdout:1955: finishing write tid 4 to smithi04268048-13 2024-09-20T19:31:47.310 INFO:tasks.rados.rados.0.smithi042.stdout:1955: finishing write tid 5 to smithi04268048-13 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:1955: oid 13 updating version 0 to 572 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:1955: oid 13 updating version 572 to 573 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:1955: oid 13 updating version 573 to 575 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:1955: oid 13 version 575 is already newer than 574 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 575 (ObjNum 683 snap 221 seq_num 683) dirty exists 2024-09-20T19:31:47.311 INFO:tasks.rados.rados.0.smithi042.stdout:1955: left oid 13 (ObjNum 683 snap 221 seq_num 683) 2024-09-20T19:31:47.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:47.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:47 smithi162 bash[24557]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:47 smithi162 bash[24557]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:47 smithi162 bash[24557]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:47 smithi162 bash[24557]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:47.523 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:47.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[32075]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[32075]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[32075]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[32075]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[22706]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[22706]: cluster 2024-09-20T19:31:46.267296+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-20T19:31:47.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[22706]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:47 smithi042 bash[22706]: cluster 2024-09-20T19:31:46.297104+0000 mgr.y (mgr.24491) 520 : cluster [DBG] pgmap v895: 168 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 159 active+clean; 139 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T19:31:47.544 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:47.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:47.675 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:47.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:47.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:47.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:47.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:47.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:47.986 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:48.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:48.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:48.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:48.185 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:31:48.186 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:31:48.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:48.203 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:31:48.204 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:31:48.222 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:31:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:31:48.240 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:31:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:31:48.260 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:31:48.261 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:31:48.283 INFO:tasks.rados.rados.0.smithi042.stdout:1954: done (2 left) 2024-09-20T19:31:48.283 INFO:tasks.rados.rados.0.smithi042.stdout:1955: done (1 left) 2024-09-20T19:31:48.283 INFO:tasks.rados.rados.0.smithi042.stdout:1956: done (0 left) 2024-09-20T19:31:48.283 INFO:tasks.rados.rados.0.smithi042.stdout:1957: delete oid 30 current snap is 222 2024-09-20T19:31:48.289 INFO:tasks.rados.rados.0.smithi042.stdout:1957: done (0 left) 2024-09-20T19:31:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:1958: read oid 11 snap 205 2024-09-20T19:31:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:1958: expect deleted 2024-09-20T19:31:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:1959: write (excl) oid 34 current snap is 222 2024-09-20T19:31:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:1959: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-20T19:31:48.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:48.296 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:31:48.296 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:31:48.296 INFO:tasks.rados.rados.0.smithi042.stdout:1959: writing smithi04268048-34 from 712795 to 1130130 tid 1 2024-09-20T19:31:48.304 INFO:tasks.rados.rados.0.smithi042.stdout:1959: writing smithi04268048-34 from 1784636 to 2325208 tid 2 2024-09-20T19:31:48.311 INFO:tasks.rados.rados.0.smithi042.stdout:1959: writing smithi04268048-34 from 2894750 to 3352667 tid 3 2024-09-20T19:31:48.311 INFO:tasks.rados.rados.0.smithi042.stdout:1960: snap_remove snap 219 2024-09-20T19:31:48.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:48.322 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:31:48.322 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:31:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:48.340 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:31:48.341 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:31:48.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:48.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[32075]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[32075]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[22706]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:48 smithi042 bash[22706]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:48.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:48 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:48 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.278351+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-20T19:31:48.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:48 smithi162 bash[24557]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:48 smithi162 bash[24557]: audit 2024-09-20T19:31:48.166522+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.42:0/2118099903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:31:48.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:48.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:48.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:48.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:48.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:48.962 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:48.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:49.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:49.094 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:49.197 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:49.290 INFO:tasks.rados.rados.0.smithi042.stdout:1960: done (2 left) 2024-09-20T19:31:49.290 INFO:tasks.rados.rados.0.smithi042.stdout:1961: read oid 23 snap -1 2024-09-20T19:31:49.290 INFO:tasks.rados.rados.0.smithi042.stdout:1961: expect deleted 2024-09-20T19:31:49.290 INFO:tasks.rados.rados.0.smithi042.stdout:1962: rollback oid 49 current snap is 222 2024-09-20T19:31:49.290 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 49 to 214 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: finishing write tid 1 to smithi04268048-34 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: finishing write tid 2 to smithi04268048-34 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: finishing write tid 3 to smithi04268048-34 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: finishing write tid 4 to smithi04268048-34 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: finishing write tid 5 to smithi04268048-34 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: oid 34 updating version 0 to 726 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: oid 34 updating version 726 to 727 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: oid 34 version 727 is already newer than 724 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: oid 34 version 727 is already newer than 725 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 727 (ObjNum 685 snap 222 seq_num 685) dirty exists 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: left oid 34 (ObjNum 685 snap 222 seq_num 685) 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1958: done (3 left) 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1959: done (2 left) 2024-09-20T19:31:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:1963: copy_from oid 41 from oid 32 current snap is 222 2024-09-20T19:31:49.292 INFO:tasks.rados.rados.0.smithi042.stdout:1964: snap_remove snap 213 2024-09-20T19:31:49.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[22706]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:49 smithi042 bash[32075]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.818449+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:47.819064+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:48.279783+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[24557]: cluster 2024-09-20T19:31:48.297708+0000 mgr.y (mgr.24491) 521 : cluster [DBG] pgmap v898: 168 pgs: 2 peering, 2 active+clean+snaptrim, 164 active+clean; 152 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 737 KiB/s, 2 objects/s recovering 2024-09-20T19:31:49.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:49.666 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:49.666 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:49.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:49.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:49.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:49.841 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:49.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:49.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:49.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:50.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:50.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:50.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:50.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:50.303 INFO:tasks.rados.rados.0.smithi042.stdout:1964: done (3 left) 2024-09-20T19:31:50.303 INFO:tasks.rados.rados.0.smithi042.stdout:1965: delete oid 31 current snap is 222 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:1963: finishing copy_from to smithi04268048-41 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:1963: got expected ENOENT (src dne) 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:1963: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:1962: finishing rollback tid 0 to smithi04268048-49 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:1962: finishing rollback tid 1 to smithi04268048-49 2024-09-20T19:31:50.304 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1101 (ObjNum 33 snap 0 seq_num 173138053) dirty dne 2024-09-20T19:31:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1961: done (3 left) 2024-09-20T19:31:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1962: done (2 left) 2024-09-20T19:31:50.313 INFO:tasks.rados.rados.0.smithi042.stdout:1963: done (1 left) 2024-09-20T19:31:50.314 INFO:tasks.rados.rados.0.smithi042.stdout:1965: done (0 left) 2024-09-20T19:31:50.314 INFO:tasks.rados.rados.0.smithi042.stdout:1966: write (excl) oid 47 current snap is 222 2024-09-20T19:31:50.314 INFO:tasks.rados.rados.0.smithi042.stdout:1966: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-09-20T19:31:50.321 INFO:tasks.rados.rados.0.smithi042.stdout:1966: writing smithi04268048-47 from 746409 to 1314242 tid 1 2024-09-20T19:31:50.328 INFO:tasks.rados.rados.0.smithi042.stdout:1966: writing smithi04268048-47 from 1901600 to 2445841 tid 2 2024-09-20T19:31:50.335 INFO:tasks.rados.rados.0.smithi042.stdout:1966: writing smithi04268048-47 from 3065949 to 3546409 tid 3 2024-09-20T19:31:50.335 INFO:tasks.rados.rados.0.smithi042.stdout:1967: write (excl) oid 45 current snap is 222 2024-09-20T19:31:50.336 INFO:tasks.rados.rados.0.smithi042.stdout:1967: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-20T19:31:50.343 INFO:tasks.rados.rados.0.smithi042.stdout:1967: writing smithi04268048-45 from 763216 to 1406298 tid 1 2024-09-20T19:31:50.350 INFO:tasks.rados.rados.0.smithi042.stdout:1967: writing smithi04268048-45 from 1843729 to 2531628 tid 2 2024-09-20T19:31:50.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:50.356 INFO:tasks.rados.rados.0.smithi042.stdout:1967: writing smithi04268048-45 from 3260666 to 3563216 tid 3 2024-09-20T19:31:50.356 INFO:tasks.rados.rados.0.smithi042.stdout:1968: write (excl) oid 5 current snap is 222 2024-09-20T19:31:50.356 INFO:tasks.rados.rados.0.smithi042.stdout:1966: finishing write tid 1 to smithi04268048-47 2024-09-20T19:31:50.356 INFO:tasks.rados.rados.0.smithi042.stdout:1968: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-20T19:31:50.364 INFO:tasks.rados.rados.0.smithi042.stdout:1968: writing smithi04268048-5 from 780023 to 1498354 tid 1 2024-09-20T19:31:50.370 INFO:tasks.rados.rados.0.smithi042.stdout:1968: writing smithi04268048-5 from 1902211 to 2450121 tid 2 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1968: writing smithi04268048-5 from 3004442 to 3580023 tid 3 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1969: delete oid 8 current snap is 222 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: finishing write tid 2 to smithi04268048-47 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: finishing write tid 3 to smithi04268048-47 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: finishing write tid 4 to smithi04268048-47 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: finishing write tid 5 to smithi04268048-47 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: oid 47 updating version 0 to 861 2024-09-20T19:31:50.377 INFO:tasks.rados.rados.0.smithi042.stdout:1966: oid 47 version 861 is already newer than 860 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1966: oid 47 updating version 861 to 863 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1966: oid 47 version 863 is already newer than 862 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 863 (ObjNum 687 snap 222 seq_num 687) dirty exists 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1966: left oid 47 (ObjNum 687 snap 222 seq_num 687) 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: finishing write tid 1 to smithi04268048-45 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: finishing write tid 2 to smithi04268048-45 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: finishing write tid 3 to smithi04268048-45 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: finishing write tid 4 to smithi04268048-45 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: finishing write tid 5 to smithi04268048-45 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: oid 45 updating version 0 to 1104 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: oid 45 version 1104 is already newer than 1103 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: oid 45 updating version 1104 to 1105 2024-09-20T19:31:50.378 INFO:tasks.rados.rados.0.smithi042.stdout:1967: oid 45 updating version 1105 to 1106 2024-09-20T19:31:50.379 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1106 (ObjNum 688 snap 222 seq_num 688) dirty exists 2024-09-20T19:31:50.379 INFO:tasks.rados.rados.0.smithi042.stdout:1967: left oid 45 (ObjNum 688 snap 222 seq_num 688) 2024-09-20T19:31:50.379 INFO:tasks.rados.rados.0.smithi042.stdout:1968: finishing write tid 1 to smithi04268048-5 2024-09-20T19:31:50.379 INFO:tasks.rados.rados.0.smithi042.stdout:1968: finishing write tid 2 to smithi04268048-5 2024-09-20T19:31:50.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:50.380 INFO:tasks.rados.rados.0.smithi042.stdout:1966: done (3 left) 2024-09-20T19:31:50.380 INFO:tasks.rados.rados.0.smithi042.stdout:1967: done (2 left) 2024-09-20T19:31:50.380 INFO:tasks.rados.rados.0.smithi042.stdout:1969: done (1 left) 2024-09-20T19:31:50.380 INFO:tasks.rados.rados.0.smithi042.stdout:1970: write oid 3 current snap is 222 2024-09-20T19:31:50.380 INFO:tasks.rados.rados.0.smithi042.stdout:1970: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-20T19:31:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:50.385 INFO:tasks.rados.rados.0.smithi042.stdout:1970: writing smithi04268048-3 from 413637 to 882466 tid 1 2024-09-20T19:31:50.392 INFO:tasks.rados.rados.0.smithi042.stdout:1970: writing smithi04268048-3 from 1619175 to 2054401 tid 2 2024-09-20T19:31:50.399 INFO:tasks.rados.rados.0.smithi042.stdout:1970: writing smithi04268048-3 from 2542935 to 3301190 tid 3 2024-09-20T19:31:50.399 INFO:tasks.rados.rados.0.smithi042.stdout:1971: snap_create 2024-09-20T19:31:50.399 INFO:tasks.rados.rados.0.smithi042.stdout:1968: finishing write tid 3 to smithi04268048-5 2024-09-20T19:31:50.399 INFO:tasks.rados.rados.0.smithi042.stdout:1968: finishing write tid 4 to smithi04268048-5 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: finishing write tid 5 to smithi04268048-5 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: oid 5 updating version 0 to 771 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: oid 5 version 771 is already newer than 769 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: oid 5 version 771 is already newer than 768 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: oid 5 version 771 is already newer than 770 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 771 (ObjNum 689 snap 222 seq_num 689) dirty exists 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1968: left oid 5 (ObjNum 689 snap 222 seq_num 689) 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1970: finishing write tid 1 to smithi04268048-3 2024-09-20T19:31:50.400 INFO:tasks.rados.rados.0.smithi042.stdout:1970: finishing write tid 2 to smithi04268048-3 2024-09-20T19:31:50.404 INFO:tasks.rados.rados.0.smithi042.stdout:1970: finishing write tid 3 to smithi04268048-3 2024-09-20T19:31:50.404 INFO:tasks.rados.rados.0.smithi042.stdout:1970: finishing write tid 4 to smithi04268048-3 2024-09-20T19:31:50.404 INFO:tasks.rados.rados.0.smithi042.stdout:1970: finishing write tid 5 to smithi04268048-3 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:1970: oid 3 updating version 0 to 709 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:1970: oid 3 updating version 709 to 711 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:1970: oid 3 version 711 is already newer than 710 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:1970: oid 3 version 711 is already newer than 708 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 711 (ObjNum 691 snap 222 seq_num 691) dirty exists 2024-09-20T19:31:50.405 INFO:tasks.rados.rados.0.smithi042.stdout:1970: left oid 3 (ObjNum 691 snap 222 seq_num 691) 2024-09-20T19:31:50.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:50.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:50 smithi162 bash[24557]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:50 smithi162 bash[24557]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:50 smithi162 bash[24557]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:50 smithi162 bash[24557]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:50.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:31:50.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:50.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[22706]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[22706]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[22706]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[22706]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[32075]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[32075]: cluster 2024-09-20T19:31:49.288769+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[32075]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:50 smithi042 bash[32075]: audit 2024-09-20T19:31:49.556364+0000 mgr.y (mgr.24491) 522 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:31:50.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:50.743 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:50.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:50.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:50.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:51.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:51.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:51.131 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:51.244 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:31:51.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.310 INFO:tasks.rados.rados.0.smithi042.stdout:1968: done (2 left) 2024-09-20T19:31:51.310 INFO:tasks.rados.rados.0.smithi042.stdout:1970: done (1 left) 2024-09-20T19:31:51.310 INFO:tasks.rados.rados.0.smithi042.stdout:1971: done (0 left) 2024-09-20T19:31:51.311 INFO:tasks.rados.rados.0.smithi042.stdout:1972: copy_from oid 34 from oid 46 current snap is 223 2024-09-20T19:31:51.311 INFO:tasks.rados.rados.0.smithi042.stdout:1973: copy_from oid 5 from oid 37 current snap is 223 2024-09-20T19:31:51.311 INFO:tasks.rados.rados.0.smithi042.stdout:1974: snap_create 2024-09-20T19:31:51.322 INFO:tasks.rados.rados.0.smithi042.stdout:1973: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:31:51.325 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.325 INFO:tasks.rados.rados.0.smithi042.stdout:1972: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:31:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:51.338 INFO:tasks.rados.rados.0.smithi042.stdout:1973: finishing copy_from to smithi04268048-5 2024-09-20T19:31:51.338 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 757 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:31:51.341 INFO:tasks.rados.rados.0.smithi042.stdout:1972: finishing copy_from to smithi04268048-34 2024-09-20T19:31:51.341 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1038 (ObjNum 675 snap 216 seq_num 675) dirty exists 2024-09-20T19:31:51.429 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[22706]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[22706]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[22706]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[22706]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[32075]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[32075]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[32075]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:51 smithi042 bash[32075]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:51.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:51.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:51 smithi162 bash[24557]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:51 smithi162 bash[24557]: cluster 2024-09-20T19:31:50.298304+0000 mgr.y (mgr.24491) 523 : cluster [DBG] pgmap v901: 168 pgs: 1 peering, 2 active+clean+snaptrim, 165 active+clean; 152 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s; 2.3 MiB/s, 9 objects/s recovering 2024-09-20T19:31:51.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:51 smithi162 bash[24557]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:51 smithi162 bash[24557]: cluster 2024-09-20T19:31:50.302628+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-20T19:31:51.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:51.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:51.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:51.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:51.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:51.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:51.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:52.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:52.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:52.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:52.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:52.312 INFO:tasks.rados.rados.0.smithi042.stdout:1972: done (2 left) 2024-09-20T19:31:52.312 INFO:tasks.rados.rados.0.smithi042.stdout:1973: done (1 left) 2024-09-20T19:31:52.312 INFO:tasks.rados.rados.0.smithi042.stdout:1974: done (0 left) 2024-09-20T19:31:52.312 INFO:tasks.rados.rados.0.smithi042.stdout:1975: snap_create 2024-09-20T19:31:52.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:52.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:52.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:52.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:52 smithi162 bash[24557]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:52 smithi162 bash[24557]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:52.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:52.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:52 smithi042 bash[32075]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:52 smithi042 bash[32075]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:52 smithi042 bash[22706]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:52 smithi042 bash[22706]: cluster 2024-09-20T19:31:51.307482+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-20T19:31:52.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:52.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:52.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:52.880 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:52.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:52.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:53.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:53.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:53.128 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:53.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:53.268 INFO:tasks.rados.rados.0.smithi042.stdout:1975: done (0 left) 2024-09-20T19:31:53.268 INFO:tasks.rados.rados.0.smithi042.stdout:1976: snap_create 2024-09-20T19:31:53.268 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T19:31:53.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[22706]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:53 smithi042 bash[32075]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:52.299755+0000 mgr.y (mgr.24491) 524 : cluster [DBG] pgmap v903: 168 pgs: 1 peering, 4 active+clean+snaptrim, 163 active+clean; 168 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.1 MiB/s wr, 10 op/s; 1.6 MiB/s, 6 objects/s recovering 2024-09-20T19:31:53.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.498 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:52.310766+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.018150+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.42:0/868862171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.018577+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25535 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: audit 2024-09-20T19:31:53.252817+0000 mon.a (mon.0) 1998 : audit [INF] from='client.25535 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.499 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:53 smithi162 bash[24557]: cluster 2024-09-20T19:31:53.261355+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-20T19:31:53.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:53.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:53.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:53.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:53.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:54.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:54.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:54.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:54.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T19:31:54.255 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:31:54.263 INFO:tasks.rados.rados.0.smithi042.stdout:1976: done (0 left) 2024-09-20T19:31:54.263 INFO:tasks.rados.rados.0.smithi042.stdout:1977: write oid 19 current snap is 226 2024-09-20T19:31:54.263 INFO:tasks.rados.rados.0.smithi042.stdout:1977: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-20T19:31:54.269 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:54.271 INFO:tasks.rados.rados.0.smithi042.stdout:1977: writing smithi04268048-19 from 430444 to 1090875 tid 1 2024-09-20T19:31:54.280 INFO:tasks.rados.rados.0.smithi042.stdout:1977: writing smithi04268048-19 from 1677657 to 2372894 tid 2 2024-09-20T19:31:54.288 INFO:tasks.rados.rados.0.smithi042.stdout:1977: writing smithi04268048-19 from 2970358 to 3539885 tid 3 2024-09-20T19:31:54.288 INFO:tasks.rados.rados.0.smithi042.stdout:1978: snap_remove snap 217 2024-09-20T19:31:54.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:54.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:54.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:54 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:54.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: audit 2024-09-20T19:31:53.392618+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.253195+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: audit 2024-09-20T19:31:54.255354+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.42:0/1547215236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.539 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:54 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.262204+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-20T19:31:54.552 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:54.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.685 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:54.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:54.816 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:54.919 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:54.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:55.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:55.180 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:55.207 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1978: done (1 left) 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1979: read oid 26 snap -1 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1979: expect (ObjNum 633 snap 199 seq_num 633) 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1980: delete oid 47 current snap is 226 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1977: finishing write tid 1 to smithi04268048-19 2024-09-20T19:31:55.269 INFO:tasks.rados.rados.0.smithi042.stdout:1977: finishing write tid 2 to smithi04268048-19 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: finishing write tid 3 to smithi04268048-19 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: finishing write tid 4 to smithi04268048-19 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: finishing write tid 5 to smithi04268048-19 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: oid 19 updating version 0 to 586 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: oid 19 updating version 586 to 587 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: oid 19 version 587 is already newer than 585 2024-09-20T19:31:55.270 INFO:tasks.rados.rados.0.smithi042.stdout:1977: oid 19 updating version 587 to 588 2024-09-20T19:31:55.271 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 588 (ObjNum 692 snap 226 seq_num 692) dirty exists 2024-09-20T19:31:55.271 INFO:tasks.rados.rados.0.smithi042.stdout:1977: left oid 19 (ObjNum 692 snap 226 seq_num 692) 2024-09-20T19:31:55.285 INFO:tasks.rados.rados.0.smithi042.stdout:1977: done (2 left) 2024-09-20T19:31:55.285 INFO:tasks.rados.rados.0.smithi042.stdout:1980: done (1 left) 2024-09-20T19:31:55.285 INFO:tasks.rados.rados.0.smithi042.stdout:1981: read oid 16 snap 204 2024-09-20T19:31:55.285 INFO:tasks.rados.rados.0.smithi042.stdout:1981: expect deleted 2024-09-20T19:31:55.285 INFO:tasks.rados.rados.0.smithi042.stdout:1982: delete oid 33 current snap is 226 2024-09-20T19:31:55.293 INFO:tasks.rados.rados.0.smithi042.stdout:1979: expect (ObjNum 633 snap 199 seq_num 633) 2024-09-20T19:31:55.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:55.368 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[32075]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[32075]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[32075]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.429 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[22706]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[22706]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:55 smithi042 bash[22706]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.434 INFO:tasks.rados.rados.0.smithi042.stdout:1979: done (2 left) 2024-09-20T19:31:55.434 INFO:tasks.rados.rados.0.smithi042.stdout:1981: done (1 left) 2024-09-20T19:31:55.435 INFO:tasks.rados.rados.0.smithi042.stdout:1982: done (0 left) 2024-09-20T19:31:55.435 INFO:tasks.rados.rados.0.smithi042.stdout:1983: write oid 26 current snap is 226 2024-09-20T19:31:55.435 INFO:tasks.rados.rados.0.smithi042.stdout:1983: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-20T19:31:55.441 INFO:tasks.rados.rados.0.smithi042.stdout:1983: writing smithi04268048-26 from 480865 to 967043 tid 1 2024-09-20T19:31:55.444 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:55.448 INFO:tasks.rados.rados.0.smithi042.stdout:1983: writing smithi04268048-26 from 1453103 to 1895667 tid 2 2024-09-20T19:31:55.458 INFO:tasks.rados.rados.0.smithi042.stdout:1983: writing smithi04268048-26 from 2652627 to 3172323 tid 3 2024-09-20T19:31:55.458 INFO:tasks.rados.rados.0.smithi042.stdout:1984: write (excl) oid 34 current snap is 226 2024-09-20T19:31:55.458 INFO:tasks.rados.rados.0.smithi042.stdout:1983: finishing write tid 1 to smithi04268048-26 2024-09-20T19:31:55.459 INFO:tasks.rados.rados.0.smithi042.stdout:1983: finishing write tid 2 to smithi04268048-26 2024-09-20T19:31:55.459 INFO:tasks.rados.rados.0.smithi042.stdout:1984: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-20T19:31:55.466 INFO:tasks.rados.rados.0.smithi042.stdout:1984: writing smithi04268048-34 from 497672 to 1059099 tid 1 2024-09-20T19:31:55.470 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:55.474 INFO:tasks.rados.rados.0.smithi042.stdout:1984: writing smithi04268048-34 from 1795232 to 2497807 tid 2 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1984: writing smithi04268048-34 from 2963697 to 3694665 tid 3 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1985: read oid 9 snap -1 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1985: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1986: copy_from oid 21 from oid 8 current snap is 226 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1983: finishing write tid 3 to smithi04268048-26 2024-09-20T19:31:55.482 INFO:tasks.rados.rados.0.smithi042.stdout:1987: snap_create 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: finishing write tid 4 to smithi04268048-26 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: finishing write tid 5 to smithi04268048-26 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: oid 26 updating version 0 to 637 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: oid 26 version 637 is already newer than 635 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: oid 26 version 637 is already newer than 636 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: oid 26 version 637 is already newer than 634 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 637 (ObjNum 695 snap 226 seq_num 695) dirty exists 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1983: left oid 26 (ObjNum 695 snap 226 seq_num 695) 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1984: finishing write tid 1 to smithi04268048-34 2024-09-20T19:31:55.483 INFO:tasks.rados.rados.0.smithi042.stdout:1984: finishing write tid 2 to smithi04268048-34 2024-09-20T19:31:55.484 INFO:tasks.rados.rados.0.smithi042.stdout:1986: finishing copy_from to smithi04268048-21 2024-09-20T19:31:55.484 INFO:tasks.rados.rados.0.smithi042.stdout:1986: got expected ENOENT (src dne) 2024-09-20T19:31:55.484 INFO:tasks.rados.rados.0.smithi042.stdout:1986: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:31:55.486 INFO:tasks.rados.rados.0.smithi042.stdout:1985: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:31:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 839172 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: finishing write tid 3 to smithi04268048-34 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: finishing write tid 4 to smithi04268048-34 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: finishing write tid 5 to smithi04268048-34 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: oid 34 updating version 0 to 1041 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: oid 34 version 1041 is already newer than 1040 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: oid 34 updating version 1041 to 1042 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: oid 34 version 1042 is already newer than 1039 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1042 (ObjNum 696 snap 226 seq_num 696) dirty exists 2024-09-20T19:31:55.557 INFO:tasks.rados.rados.0.smithi042.stdout:1984: left oid 34 (ObjNum 696 snap 226 seq_num 696) 2024-09-20T19:31:55.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:55.584 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:55 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.584 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:55 smithi162 bash[24557]: cluster 2024-09-20T19:31:54.300225+0000 mgr.y (mgr.24491) 525 : cluster [DBG] pgmap v907: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:31:55.585 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:55 smithi162 bash[24557]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.585 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:55 smithi162 bash[24557]: cluster 2024-09-20T19:31:55.266428+0000 mon.a (mon.0) 2004 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-20T19:31:55.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:55.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:55.727 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:55.831 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:55.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:55.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:55.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:56.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:56.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:56.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:56.335 INFO:tasks.rados.rados.0.smithi042.stdout:1983: done (4 left) 2024-09-20T19:31:56.335 INFO:tasks.rados.rados.0.smithi042.stdout:1984: done (3 left) 2024-09-20T19:31:56.335 INFO:tasks.rados.rados.0.smithi042.stdout:1985: done (2 left) 2024-09-20T19:31:56.335 INFO:tasks.rados.rados.0.smithi042.stdout:1986: done (1 left) 2024-09-20T19:31:56.336 INFO:tasks.rados.rados.0.smithi042.stdout:1987: done (0 left) 2024-09-20T19:31:56.336 INFO:tasks.rados.rados.0.smithi042.stdout:1988: rollback oid 17 current snap is 227 2024-09-20T19:31:56.336 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 225 2024-09-20T19:31:56.336 INFO:tasks.rados.rados.0.smithi042.stdout:1989: delete oid 19 current snap is 227 2024-09-20T19:31:56.343 INFO:tasks.rados.rados.0.smithi042.stdout:1989: done (1 left) 2024-09-20T19:31:56.343 INFO:tasks.rados.rados.0.smithi042.stdout:1990: write (excl) oid 38 current snap is 227 2024-09-20T19:31:56.344 INFO:tasks.rados.rados.0.smithi042.stdout:1990: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-20T19:31:56.351 INFO:tasks.rados.rados.0.smithi042.stdout:1990: writing smithi04268048-38 from 531286 to 1243211 tid 1 2024-09-20T19:31:56.351 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:56.359 INFO:tasks.rados.rados.0.smithi042.stdout:1990: writing smithi04268048-38 from 1912196 to 2618440 tid 2 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1990: writing smithi04268048-38 from 3018543 to 3731286 tid 3 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1991: rollback oid 19 current snap is 227 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 225 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1988: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1988: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1992: write oid 25 current snap is 227 2024-09-20T19:31:56.366 INFO:tasks.rados.rados.0.smithi042.stdout:1988: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:31:56.367 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 714 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-20T19:31:56.367 INFO:tasks.rados.rados.0.smithi042.stdout:1990: finishing write tid 1 to smithi04268048-38 2024-09-20T19:31:56.367 INFO:tasks.rados.rados.0.smithi042.stdout:1990: finishing write tid 2 to smithi04268048-38 2024-09-20T19:31:56.367 INFO:tasks.rados.rados.0.smithi042.stdout:1992: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-20T19:31:56.374 INFO:tasks.rados.rados.0.smithi042.stdout:1992: writing smithi04268048-25 from 548093 to 1335267 tid 1 2024-09-20T19:31:56.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:56.380 INFO:tasks.rados.rados.0.smithi042.stdout:1992: writing smithi04268048-25 from 1970678 to 2420580 tid 2 2024-09-20T19:31:56.388 INFO:tasks.rados.rados.0.smithi042.stdout:1992: writing smithi04268048-25 from 3045966 to 3727103 tid 3 2024-09-20T19:31:56.388 INFO:tasks.rados.rados.0.smithi042.stdout:1988: done (3 left) 2024-09-20T19:31:56.388 INFO:tasks.rados.rados.0.smithi042.stdout:1993: rollback oid 13 current snap is 227 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 212 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1991: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 589 (ObjNum 5 snap 0 seq_num 304) dirty dne 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: finishing write tid 3 to smithi04268048-38 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: finishing write tid 4 to smithi04268048-38 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: finishing write tid 5 to smithi04268048-38 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: oid 38 updating version 0 to 661 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: oid 38 version 661 is already newer than 660 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: oid 38 updating version 661 to 663 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: oid 38 version 663 is already newer than 662 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 663 (ObjNum 698 snap 227 seq_num 698) dirty exists 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: left oid 38 (ObjNum 698 snap 227 seq_num 698) 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1992: finishing write tid 1 to smithi04268048-25 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1992: finishing write tid 2 to smithi04268048-25 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1990: done (3 left) 2024-09-20T19:31:56.389 INFO:tasks.rados.rados.0.smithi042.stdout:1991: done (2 left) 2024-09-20T19:31:56.390 INFO:tasks.rados.rados.0.smithi042.stdout:1994: write oid 19 current snap is 227 2024-09-20T19:31:56.390 INFO:tasks.rados.rados.0.smithi042.stdout:1994: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-09-20T19:31:56.394 INFO:tasks.rados.rados.0.smithi042.stdout:1994: writing smithi04268048-19 from 564900 to 1143676 tid 1 2024-09-20T19:31:56.401 INFO:tasks.rados.rados.0.smithi042.stdout:1994: writing smithi04268048-19 from 1629160 to 2339073 tid 2 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1994: writing smithi04268048-19 from 3073389 to 3565798 tid 3 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1995: snap_create 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1993: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1993: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 577 (ObjNum 97 snap 0 seq_num 1415970624) dirty dne 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: finishing write tid 3 to smithi04268048-25 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: finishing write tid 4 to smithi04268048-25 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: finishing write tid 5 to smithi04268048-25 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: oid 25 updating version 0 to 795 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: oid 25 updating version 795 to 796 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: oid 25 version 796 is already newer than 794 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: oid 25 version 796 is already newer than 793 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 796 (ObjNum 699 snap 227 seq_num 699) dirty exists 2024-09-20T19:31:56.409 INFO:tasks.rados.rados.0.smithi042.stdout:1992: left oid 25 (ObjNum 699 snap 227 seq_num 699) 2024-09-20T19:31:56.410 INFO:tasks.rados.rados.0.smithi042.stdout:1994: finishing write tid 1 to smithi04268048-19 2024-09-20T19:31:56.410 INFO:tasks.rados.rados.0.smithi042.stdout:1994: finishing write tid 2 to smithi04268048-19 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: finishing write tid 3 to smithi04268048-19 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: finishing write tid 4 to smithi04268048-19 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: finishing write tid 5 to smithi04268048-19 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: oid 19 updating version 0 to 590 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: oid 19 updating version 590 to 593 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: oid 19 version 593 is already newer than 592 2024-09-20T19:31:56.412 INFO:tasks.rados.rados.0.smithi042.stdout:1994: oid 19 version 593 is already newer than 591 2024-09-20T19:31:56.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 593 (ObjNum 700 snap 227 seq_num 700) dirty exists 2024-09-20T19:31:56.413 INFO:tasks.rados.rados.0.smithi042.stdout:1994: left oid 19 (ObjNum 700 snap 227 seq_num 700) 2024-09-20T19:31:56.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:56.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:56.608 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:56.620 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:31:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:31:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:31:56.636 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:56.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:56.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:56.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:56.981 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:31:56.981 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:31:56.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:57.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:57.180 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:31:57.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:57.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:57.346 INFO:tasks.rados.rados.0.smithi042.stdout:1992: done (3 left) 2024-09-20T19:31:57.346 INFO:tasks.rados.rados.0.smithi042.stdout:1993: done (2 left) 2024-09-20T19:31:57.346 INFO:tasks.rados.rados.0.smithi042.stdout:1994: done (1 left) 2024-09-20T19:31:57.346 INFO:tasks.rados.rados.0.smithi042.stdout:1995: done (0 left) 2024-09-20T19:31:57.346 INFO:tasks.rados.rados.0.smithi042.stdout:1996: write (excl) oid 3 current snap is 228 2024-09-20T19:31:57.347 INFO:tasks.rados.rados.0.smithi042.stdout:1996: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-20T19:31:57.354 INFO:tasks.rados.rados.0.smithi042.stdout:1996: writing smithi04268048-3 from 581707 to 1235732 tid 1 2024-09-20T19:31:57.360 INFO:tasks.rados.rados.0.smithi042.stdout:1996: writing smithi04268048-3 from 1687642 to 2141213 tid 2 2024-09-20T19:31:57.369 INFO:tasks.rados.rados.0.smithi042.stdout:1996: writing smithi04268048-3 from 2700812 to 3404493 tid 3 2024-09-20T19:31:57.369 INFO:tasks.rados.rados.0.smithi042.stdout:1997: snap_create 2024-09-20T19:31:57.373 INFO:tasks.rados.rados.0.smithi042.stdout:1996: finishing write tid 1 to smithi04268048-3 2024-09-20T19:31:57.373 INFO:tasks.rados.rados.0.smithi042.stdout:1996: finishing write tid 2 to smithi04268048-3 2024-09-20T19:31:57.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: finishing write tid 3 to smithi04268048-3 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: finishing write tid 4 to smithi04268048-3 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: finishing write tid 5 to smithi04268048-3 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: oid 3 updating version 0 to 715 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: oid 3 updating version 715 to 716 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: oid 3 updating version 716 to 718 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: oid 3 version 718 is already newer than 717 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 718 (ObjNum 701 snap 228 seq_num 701) dirty exists 2024-09-20T19:31:57.376 INFO:tasks.rados.rados.0.smithi042.stdout:1996: left oid 3 (ObjNum 701 snap 228 seq_num 701) 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[22706]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.385 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.386 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:57 smithi042 bash[32075]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:57.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.300932+0000 mgr.y (mgr.24491) 526 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 172 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 942 KiB/s wr, 2 op/s 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.333386+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.375251+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:31:57.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.513 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:57 smithi162 bash[24557]: cluster 2024-09-20T19:31:56.375851+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:31:57.528 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:31:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:31:57.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:57.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:31:57.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:57.769 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:57.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:31:57.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:57.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:57.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:58.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:58.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:58.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:58.266 INFO:tasks.rados.rados.0.smithi042.stdout:1996: done (1 left) 2024-09-20T19:31:58.266 INFO:tasks.rados.rados.0.smithi042.stdout:1997: done (0 left) 2024-09-20T19:31:58.266 INFO:tasks.rados.rados.0.smithi042.stdout:1998: delete oid 48 current snap is 229 2024-09-20T19:31:58.271 INFO:tasks.rados.rados.0.smithi042.stdout:1998: done (0 left) 2024-09-20T19:31:58.271 INFO:tasks.rados.rados.0.smithi042.stdout:1999: copy_from oid 21 from oid 40 current snap is 229 2024-09-20T19:31:58.271 INFO:tasks.rados.rados.0.smithi042.stdout:2000: rollback oid 24 current snap is 229 2024-09-20T19:31:58.271 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 225 2024-09-20T19:31:58.271 INFO:tasks.rados.rados.0.smithi042.stdout:2001: rollback oid 32 current snap is 229 2024-09-20T19:31:58.272 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 223 2024-09-20T19:31:58.272 INFO:tasks.rados.rados.0.smithi042.stdout:2002: write (excl) oid 25 current snap is 229 2024-09-20T19:31:58.272 INFO:tasks.rados.rados.0.smithi042.stdout:2002: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-09-20T19:31:58.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:58.278 INFO:tasks.rados.rados.0.smithi042.stdout:2002: writing smithi04268048-25 from 615321 to 1019844 tid 1 2024-09-20T19:31:58.286 INFO:tasks.rados.rados.0.smithi042.stdout:2002: writing smithi04268048-25 from 1804606 to 2261846 tid 2 2024-09-20T19:31:58.295 INFO:tasks.rados.rados.0.smithi042.stdout:2002: writing smithi04268048-25 from 2755658 to 3481883 tid 3 2024-09-20T19:31:58.295 INFO:tasks.rados.rados.0.smithi042.stdout:2003: rollback oid 23 current snap is 229 2024-09-20T19:31:58.295 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 222 2024-09-20T19:31:58.295 INFO:tasks.rados.rados.0.smithi042.stdout:2000: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:31:58.295 INFO:tasks.rados.rados.0.smithi042.stdout:2004: snap_remove snap 214 2024-09-20T19:31:58.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:58.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:58.436 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:31:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:31:58.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:58 smithi162 bash[24557]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.565 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:58.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[32075]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: cluster 2024-09-20T19:31:57.344860+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: audit 2024-09-20T19:31:57.871724+0000 mon.a (mon.0) 2007 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:31:58.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:58 smithi042 bash[22706]: cluster 2024-09-20T19:31:58.263410+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-20T19:31:58.693 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:31:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:31:58.796 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:31:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:58.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:58.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:31:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:58.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:31:59.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:59.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:31:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:31:59.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:59.289 INFO:tasks.rados.rados.0.smithi042.stdout:2004: done (5 left) 2024-09-20T19:31:59.289 INFO:tasks.rados.rados.0.smithi042.stdout:2005: snap_remove snap 228 2024-09-20T19:31:59.317 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:59.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:59.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[32075]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[32075]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[32075]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[32075]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[22706]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.586 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[22706]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[22706]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:31:59 smithi042 bash[22706]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:31:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:31:59.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:31:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:31:59.718 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:59 smithi162 bash[24557]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.718 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:59 smithi162 bash[24557]: cluster 2024-09-20T19:31:58.301600+0000 mgr.y (mgr.24491) 527 : cluster [DBG] pgmap v913: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 185 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 834 KiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:31:59.718 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:59 smithi162 bash[24557]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.718 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:31:59 smithi162 bash[24557]: cluster 2024-09-20T19:31:59.287663+0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-20T19:31:59.733 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:59.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:31:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:31:59.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:31:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:31:59.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:00.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:00.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:00.251 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:00.284 INFO:tasks.rados.rados.0.smithi042.stdout:2005: done (5 left) 2024-09-20T19:32:00.285 INFO:tasks.rados.rados.0.smithi042.stdout:2006: read oid 20 snap -1 2024-09-20T19:32:00.285 INFO:tasks.rados.rados.0.smithi042.stdout:2006: expect (ObjNum 652 snap 208 seq_num 652) 2024-09-20T19:32:00.286 INFO:tasks.rados.rados.0.smithi042.stdout:2007: copy_from oid 26 from oid 42 current snap is 229 2024-09-20T19:32:00.286 INFO:tasks.rados.rados.0.smithi042.stdout:2008: read oid 22 snap -1 2024-09-20T19:32:00.286 INFO:tasks.rados.rados.0.smithi042.stdout:2008: expect deleted 2024-09-20T19:32:00.286 INFO:tasks.rados.rados.0.smithi042.stdout:2009: write oid 5 current snap is 229 2024-09-20T19:32:00.286 INFO:tasks.rados.rados.0.smithi042.stdout:2009: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-20T19:32:00.293 INFO:tasks.rados.rados.0.smithi042.stdout:2009: writing smithi04268048-5 from 632128 to 1111900 tid 1 2024-09-20T19:32:00.300 INFO:tasks.rados.rados.0.smithi042.stdout:2009: writing smithi04268048-5 from 1746735 to 2347633 tid 2 2024-09-20T19:32:00.308 INFO:tasks.rados.rados.0.smithi042.stdout:2009: writing smithi04268048-5 from 3066728 to 3720578 tid 3 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2010: rollback oid 36 current snap is 229 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 36 to 205 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2000: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2000: finishing rollback tid 2 to smithi04268048-24 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 734 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2003: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 707 (ObjNum 33 snap 0 seq_num 1390889016) dirty dne 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2002: finishing write tid 1 to smithi04268048-25 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2002: finishing write tid 2 to smithi04268048-25 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2001: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1088 (ObjNum 257 snap 0 seq_num 1415436656) dirty dne 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:1999: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2002: finishing write tid 3 to smithi04268048-25 2024-09-20T19:32:00.309 INFO:tasks.rados.rados.0.smithi042.stdout:2000: done (9 left) 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2001: done (8 left) 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2003: done (7 left) 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2011: snap_create 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: finishing write tid 4 to smithi04268048-25 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: finishing write tid 5 to smithi04268048-25 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: oid 25 updating version 0 to 800 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: oid 25 version 800 is already newer than 799 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: oid 25 updating version 800 to 802 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: oid 25 version 802 is already newer than 801 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 802 (ObjNum 703 snap 229 seq_num 703) dirty exists 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2002: left oid 25 (ObjNum 703 snap 229 seq_num 703) 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:1999: finishing copy_from to smithi04268048-21 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 595 (ObjNum 624 snap 196 seq_num 624) dirty exists 2024-09-20T19:32:00.310 INFO:tasks.rados.rados.0.smithi042.stdout:2006: expect (ObjNum 652 snap 208 seq_num 652) 2024-09-20T19:32:00.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:00 smithi162 bash[24557]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:00 smithi162 bash[24557]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.374 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:00.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: finishing write tid 1 to smithi04268048-5 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: finishing write tid 2 to smithi04268048-5 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: finishing write tid 3 to smithi04268048-5 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: finishing write tid 4 to smithi04268048-5 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: finishing write tid 5 to smithi04268048-5 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: oid 5 updating version 0 to 772 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: oid 5 updating version 772 to 773 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: oid 5 updating version 773 to 774 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: oid 5 updating version 774 to 775 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 775 (ObjNum 704 snap 229 seq_num 704) dirty exists 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2009: left oid 5 (ObjNum 704 snap 229 seq_num 704) 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2010: finishing rollback tid 0 to smithi04268048-36 2024-09-20T19:32:00.408 INFO:tasks.rados.rados.0.smithi042.stdout:2010: finishing rollback tid 1 to smithi04268048-36 2024-09-20T19:32:00.409 INFO:tasks.rados.rados.0.smithi042.stdout:2010: finishing rollback tid 2 to smithi04268048-36 2024-09-20T19:32:00.409 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 571 (ObjNum 617 snap 194 seq_num 617) dirty exists 2024-09-20T19:32:00.409 INFO:tasks.rados.rados.0.smithi042.stdout:2007: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:32:00.409 INFO:tasks.rados.rados.0.smithi042.stdout:2007: finishing copy_from to smithi04268048-26 2024-09-20T19:32:00.409 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 859 (ObjNum 682 snap 221 seq_num 682) dirty exists 2024-09-20T19:32:00.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[32075]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[32075]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[22706]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[22706]: audit 2024-09-20T19:31:59.570188+0000 mgr.y (mgr.24491) 528 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:00.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:00.284550+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-20T19:32:00.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:00.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:00.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:00.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:00.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:00.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:00.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:01.090 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:32:01.090 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:32:01.109 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:32:01.109 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:32:01.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:01.126 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:32:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:32:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:01.144 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:32:01.144 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:32:01.165 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:32:01.165 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:32:01.186 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:32:01.186 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:32:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:32:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:32:01.229 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:32:01.229 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:32:01.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:01.260 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:1999: done (7 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2002: done (6 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2006: done (5 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2007: done (4 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2008: done (3 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2009: done (2 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2010: done (1 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2011: done (0 left) 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2012: read oid 25 snap -1 2024-09-20T19:32:01.296 INFO:tasks.rados.rados.0.smithi042.stdout:2012: expect (ObjNum 703 snap 229 seq_num 703) 2024-09-20T19:32:01.297 INFO:tasks.rados.rados.0.smithi042.stdout:2013: delete oid 48 current snap is 230 2024-09-20T19:32:01.315 INFO:tasks.rados.rados.0.smithi042.stdout:2012: expect (ObjNum 703 snap 229 seq_num 703) 2024-09-20T19:32:01.315 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 358122 2024-09-20T19:32:01.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:01.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:01.406 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:01.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:01.499 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 34877 2024-09-20T19:32:01.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:01.590 INFO:tasks.rados.rados.0.smithi042.stdout:2012: done (1 left) 2024-09-20T19:32:01.591 INFO:tasks.rados.rados.0.smithi042.stdout:2013: done (0 left) 2024-09-20T19:32:01.591 INFO:tasks.rados.rados.0.smithi042.stdout:2014: write (excl) oid 45 current snap is 230 2024-09-20T19:32:01.591 INFO:tasks.rados.rados.0.smithi042.stdout:2014: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-09-20T19:32:01.607 INFO:tasks.rados.rados.0.smithi042.stdout:2014: writing smithi04268048-45 from 665742 to 1296012 tid 1 2024-09-20T19:32:01.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:01.623 INFO:tasks.rados.rados.0.smithi042.stdout:2014: writing smithi04268048-45 from 1863699 to 2468266 tid 2 2024-09-20T19:32:01.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:01 smithi162 bash[24557]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.640 INFO:tasks.rados.rados.0.smithi042.stdout:2014: writing smithi04268048-45 from 3121574 to 3797968 tid 3 2024-09-20T19:32:01.640 INFO:tasks.rados.rados.0.smithi042.stdout:2015: snap_create 2024-09-20T19:32:01.640 INFO:tasks.rados.rados.0.smithi042.stdout:2014: finishing write tid 1 to smithi04268048-45 2024-09-20T19:32:01.640 INFO:tasks.rados.rados.0.smithi042.stdout:2014: finishing write tid 2 to smithi04268048-45 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: finishing write tid 3 to smithi04268048-45 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: finishing write tid 4 to smithi04268048-45 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: finishing write tid 5 to smithi04268048-45 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: oid 45 updating version 0 to 1116 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: oid 45 version 1116 is already newer than 1114 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: oid 45 version 1116 is already newer than 1113 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: oid 45 version 1116 is already newer than 1115 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1116 (ObjNum 706 snap 230 seq_num 706) dirty exists 2024-09-20T19:32:01.645 INFO:tasks.rados.rados.0.smithi042.stdout:2014: left oid 45 (ObjNum 706 snap 230 seq_num 706) 2024-09-20T19:32:01.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[32075]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: cluster 2024-09-20T19:32:00.302175+0000 mgr.y (mgr.24491) 529 : cluster [DBG] pgmap v916: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 187 MiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: audit 2024-09-20T19:32:01.071703+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.42:0/1304276359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:01 smithi042 bash[22706]: cluster 2024-09-20T19:32:01.294715+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-20T19:32:01.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:01.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:01.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:01.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:02.008 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:02.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:02.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:02.353 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T19:32:02.359 INFO:tasks.rados.rados.0.smithi042.stdout:2014: done (1 left) 2024-09-20T19:32:02.359 INFO:tasks.rados.rados.0.smithi042.stdout:2015: done (0 left) 2024-09-20T19:32:02.359 INFO:tasks.rados.rados.0.smithi042.stdout:2016: write oid 47 current snap is 231 2024-09-20T19:32:02.359 INFO:tasks.rados.rados.0.smithi042.stdout:2016: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-20T19:32:02.369 INFO:tasks.rados.rados.0.smithi042.stdout:2016: writing smithi04268048-47 from 682549 to 1388068 tid 1 2024-09-20T19:32:02.377 INFO:tasks.rados.rados.0.smithi042.stdout:2016: writing smithi04268048-47 from 1922181 to 2386759 tid 2 2024-09-20T19:32:02.387 INFO:tasks.rados.rados.0.smithi042.stdout:2016: writing smithi04268048-47 from 3148997 to 3636663 tid 3 2024-09-20T19:32:02.387 INFO:tasks.rados.rados.0.smithi042.stdout:2017: delete oid 1 current snap is 231 2024-09-20T19:32:02.387 INFO:tasks.rados.rados.0.smithi042.stdout:2016: finishing write tid 1 to smithi04268048-47 2024-09-20T19:32:02.387 INFO:tasks.rados.rados.0.smithi042.stdout:2016: finishing write tid 2 to smithi04268048-47 2024-09-20T19:32:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2016: finishing write tid 3 to smithi04268048-47 2024-09-20T19:32:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2016: finishing write tid 4 to smithi04268048-47 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: finishing write tid 5 to smithi04268048-47 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: oid 47 updating version 0 to 866 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: oid 47 updating version 866 to 868 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: oid 47 version 868 is already newer than 867 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: oid 47 updating version 868 to 869 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 869 (ObjNum 707 snap 231 seq_num 707) dirty exists 2024-09-20T19:32:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2016: left oid 47 (ObjNum 707 snap 231 seq_num 707) 2024-09-20T19:32:02.397 INFO:tasks.rados.rados.0.smithi042.stdout:2016: done (1 left) 2024-09-20T19:32:02.397 INFO:tasks.rados.rados.0.smithi042.stdout:2017: done (0 left) 2024-09-20T19:32:02.397 INFO:tasks.rados.rados.0.smithi042.stdout:2018: read oid 16 snap -1 2024-09-20T19:32:02.397 INFO:tasks.rados.rados.0.smithi042.stdout:2018: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:32:02.397 INFO:tasks.rados.rados.0.smithi042.stdout:2019: rollback oid 17 current snap is 231 2024-09-20T19:32:02.398 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 225 2024-09-20T19:32:02.398 INFO:tasks.rados.rados.0.smithi042.stdout:2020: write oid 10 current snap is 231 2024-09-20T19:32:02.398 INFO:tasks.rados.rados.0.smithi042.stdout:2020: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-20T19:32:02.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.409 INFO:tasks.rados.rados.0.smithi042.stdout:2020: writing smithi04268048-10 from 716163 to 1288533 tid 1 2024-09-20T19:32:02.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:02 smithi162 bash[24557]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:02 smithi162 bash[24557]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2020: writing smithi04268048-10 from 2039145 to 2507392 tid 2 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2020: writing smithi04268048-10 from 3203843 to 3830406 tid 3 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2021: rollback oid 25 current snap is 231 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 212 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2019: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2019: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2019: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 721 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:2018: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:32:02.428 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:02.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:02.477 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:02.525 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:02.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:02.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: finishing write tid 1 to smithi04268048-10 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: finishing write tid 2 to smithi04268048-10 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2021: finishing rollback tid 0 to smithi04268048-25 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2021: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2021: finishing rollback tid 2 to smithi04268048-25 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 811 (ObjNum 643 snap 206 seq_num 643) dirty exists 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: finishing write tid 3 to smithi04268048-10 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: finishing write tid 4 to smithi04268048-10 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: finishing write tid 5 to smithi04268048-10 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: oid 10 updating version 0 to 670 2024-09-20T19:32:02.573 INFO:tasks.rados.rados.0.smithi042.stdout:2020: oid 10 version 670 is already newer than 669 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2020: oid 10 version 670 is already newer than 667 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2020: oid 10 version 670 is already newer than 668 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 670 (ObjNum 709 snap 231 seq_num 709) dirty exists 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2020: left oid 10 (ObjNum 709 snap 231 seq_num 709) 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2018: done (3 left) 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2019: done (2 left) 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2020: done (1 left) 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2021: done (0 left) 2024-09-20T19:32:02.574 INFO:tasks.rados.rados.0.smithi042.stdout:2022: delete oid 24 current snap is 231 2024-09-20T19:32:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2022: done (0 left) 2024-09-20T19:32:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2023: read oid 4 snap -1 2024-09-20T19:32:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2023: expect (ObjNum 615 snap 193 seq_num 615) 2024-09-20T19:32:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2024: snap_remove snap 226 2024-09-20T19:32:02.625 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:32:02.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:02.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:02 smithi042 bash[22706]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:02 smithi042 bash[22706]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:02 smithi042 bash[32075]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:02 smithi042 bash[32075]: audit 2024-09-20T19:32:01.624454+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:32:02.681 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:02.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:02.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:02.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:02.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:03.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:03.170 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:03.253 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T19:32:03.263 INFO:tasks.rados.rados.0.smithi042.stdout:2024: done (1 left) 2024-09-20T19:32:03.263 INFO:tasks.rados.rados.0.smithi042.stdout:2025: snap_remove snap 225 2024-09-20T19:32:03.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:03.326 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:03.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:03.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:03.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:03.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[32075]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:03 smithi042 bash[22706]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.687 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:02.303535+0000 mgr.y (mgr.24491) 530 : cluster [DBG] pgmap v918: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 192 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: audit 2024-09-20T19:32:02.352979+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? v1:172.21.15.42:0/3376657164' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:02.358515+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:03 smithi162 bash[24557]: cluster 2024-09-20T19:32:03.262700+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-20T19:32:03.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:03.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:03.843 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:03.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:03.975 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:03.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:04.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:04.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:04.208 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:04.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:32:04.236 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2025: done (1 left) 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2026: read oid 10 snap 229 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2026: expect (ObjNum 622 snap 195 seq_num 622) 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2027: read oid 23 snap -1 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2027: expect deleted 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2028: delete oid 33 current snap is 231 2024-09-20T19:32:04.267 INFO:tasks.rados.rados.0.smithi042.stdout:2023: expect (ObjNum 615 snap 193 seq_num 615) 2024-09-20T19:32:04.283 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1719759 2024-09-20T19:32:04.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:04.350 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:04.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:04.397 INFO:tasks.rados.rados.0.smithi042.stdout:2026: expect (ObjNum 622 snap 195 seq_num 622) 2024-09-20T19:32:04.397 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:04.401 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:32:04.401 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:32:04.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:32:04.401 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:32:04.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:941 7.5 7:a6db2e09:::smithi04268048-9:head [copy-from ver 86] snapc 2b=[2b,2a,29,26,24,22,20,10] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 541.35739480400002, 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028695786000000001, 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 941 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.402 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046479+0000", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046484+0000", 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.284e-06 2024-09-20T19:32:04.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046498+0000", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4596000000000001e-05 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046531+0000", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3235999999999998e-05 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046549+0000", 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7540000000000001e-05 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046728+0000", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017878600000000001 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046750+0000", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2566e-05 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053318+0000", 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065672559999999996 2024-09-20T19:32:04.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053582+0000", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026449500000000001 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.064732+0000", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011149676000000001 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073497+0000", 2024-09-20T19:32:04.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087651080000000006 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073683+0000", 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018656599999999999 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.074975+0000", 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012920679999999999 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.075094+0000", 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118807 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.075175+0000", 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0802000000000003e-05 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:942 7.5 7:a6db2e09:::smithi04268048-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e194)", 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 541.35729411900002, 2024-09-20T19:32:04.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026852612000000001, 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 942 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046580+0000", 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046581+0000", 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1370000000000001e-06 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046587+0000", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4060000000000004e-06 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046597+0000", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046606+0000", 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7620000000000001e-06 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046884+0000", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027780399999999998 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.046895+0000", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1147e-05 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053916+0000", 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070209030000000002 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.053994+0000", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8177999999999996e-05 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.072860+0000", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018865990999999999 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073105+0000", 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024436800000000002 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:03.073433+0000", 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032792200000000002 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:968 7.5 7:adc64224:::smithi04268048-47:head [copy-from ver 27] snapc 2c=[2c,2a,29,26,24,20] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.35080714599997, 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037789484999999998, 2024-09-20T19:32:04.414 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 968 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053067+0000", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053071+0000", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1049999999999997e-06 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053084+0000", 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632e-05 2024-09-20T19:32:04.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053112+0000", 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8535999999999999e-05 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053124+0000", 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2041999999999999e-05 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053212+0000", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7688999999999993e-05 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053223+0000", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1336e-05 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.056699+0000", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034760009999999998 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.056921+0000", 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022168900000000001 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.070665+0000", 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013744361 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.077380+0000", 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067147810000000004 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.079412+0000", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020316359999999999 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090764+0000", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011352704 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090804+0000", 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9332000000000002e-05 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.090856+0000", 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2641e-05 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:969 7.5 7:adc64224:::smithi04268048-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.35072362000005, 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024003268000000001, 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.421 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 969 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053150+0000", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053151+0000", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4099999999999997e-07 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053156+0000", 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5970000000000002e-06 2024-09-20T19:32:04.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053166+0000", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0023e-05 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053173+0000", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8739999999999997e-06 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053297+0000", 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123628 2024-09-20T19:32:04.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.053301+0000", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.639e-06 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.057222+0000", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039206919999999999 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.057310+0000", 2024-09-20T19:32:04.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8491000000000004e-05 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.076556+0000", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019245532999999999 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.076819+0000", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026316700000000002 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:23:07.077154+0000", 2024-09-20T19:32:04.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033468300000000001 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1899 7.5 7:a66d8c87:::smithi04268048-11:head [copy-from ver 152] snapc 51=[51,50,4f,4e,4c,4b,49,46,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 467.30226663600001, 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036523195000000001, 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1899 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101607+0000", 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101610+0000", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8270000000000002e-06 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101619+0000", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5839999999999995e-06 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101633+0000", 2024-09-20T19:32:04.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3728e-05 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101643+0000", 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.013e-05 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101762+0000", 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011932 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101777+0000", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107805+0000", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006028299 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107879+0000", 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4240999999999996e-05 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.122444+0000", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014564638 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137584+0000", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015140278 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137772+0000", 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018759700000000001 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.137945+0000", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017279700000000001 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.138045+0000", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010033499999999999 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.138131+0000", 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5514999999999997e-05 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.433 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:1900 7.5 7:a66d8c87:::smithi04268048-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 467.30220627, 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031304029999999997, 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1900 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101668+0000", 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101669+0000", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0610000000000001e-06 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101673+0000", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.664e-06 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101680+0000", 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9979999999999997e-06 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101688+0000", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1289999999999996e-06 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101896+0000", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000208092 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.101900+0000", 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3889999999999998e-06 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.107987+0000", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060870589999999997 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.108015+0000", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7495e-05 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132810+0000", 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024794817 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132889+0000", 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9204999999999998e-05 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:17.132972+0000", 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3121000000000006e-05 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.439 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2130 7.5 7:a66d8c87:::smithi04268048-11:head [copy-from ver 340] snapc 5f=[5f,5e,5d,5a,57,55,4f,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 438.16231691299998, 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023236039999999999, 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2130 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241557+0000", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241561+0000", 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7340000000000002e-06 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241580+0000", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8797999999999999e-05 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241600+0000", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0602999999999999e-05 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241612+0000", 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2043e-05 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241732+0000", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119838 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.241849+0000", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116729 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.254378+0000", 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012528977 2024-09-20T19:32:04.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.261386+0000", 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070081910000000004 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.262140+0000", 2024-09-20T19:32:04.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075371699999999999 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264625+0000", 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002485377 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264731+0000", 2024-09-20T19:32:04.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106048 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:24:46.264793+0000", 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1984999999999997e-05 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2314 7.5 7:adc64224:::smithi04268048-47:head [copy-from ver 414] snapc 69=[69,68,67,66,65,63,62,5e,5d,5a,57,41,3e] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:32:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 422.20205577899998, 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034760207000000001, 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2314 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:32:04.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201818+0000", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201824+0000", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6869999999999999e-06 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201842+0000", 2024-09-20T19:32:04.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8196999999999999e-05 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201880+0000", 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8219000000000001e-05 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201900+0000", 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9259e-05 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202083+0000", 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000182907 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202105+0000", 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2745e-05 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.205942+0000", 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038370309999999999 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206023+0000", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0311999999999999e-05 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.218794+0000", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012771803999999999 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.233763+0000", 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014968560000000001 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236138+0000", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023753680000000001 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236390+0000", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025173900000000001 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236494+0000", 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010427799999999999 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.236579+0000", 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4101e-05 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2315 7.5 7:adc64224:::smithi04268048-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 422.20193607099998, 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029275061000000002, 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2315 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201938+0000", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201940+0000", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6199999999999999e-06 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201954+0000", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4117000000000001e-05 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.201999+0000", 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5701999999999997e-05 2024-09-20T19:32:04.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202016+0000", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6693000000000001e-05 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202220+0000", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020356000000000001 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.202243+0000", 2024-09-20T19:32:04.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.304e-05 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206174+0000", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039308399999999997 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.206306+0000", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013202099999999999 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.230853+0000", 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024547365000000002 2024-09-20T19:32:04.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.231025+0000", 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017165 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:02.231213+0000", 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018845299999999999 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.458 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2703 7.5 7:acb1cf35:::smithi04268048-42:head [copy-from ver 393] snapc 77=[77,76,75,73,72,70,6d,6c,69,67,5e,5a,57,41] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 395.01104872500002, 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025102850999999999, 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2703 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392825+0000", 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392852+0000", 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6718999999999999e-05 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392870+0000", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8e-05 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392902+0000", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1838999999999998e-05 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392918+0000", 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6456e-05 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393043+0000", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012482500000000001 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393277+0000", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000233983 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.404416+0000", 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011138677 2024-09-20T19:32:04.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417344+0000", 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012927925999999999 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417615+0000", 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027092099999999998 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417726+0000", 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111159 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417838+0000", 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111965 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.417928+0000", 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0381000000000001e-05 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:2704 7.5 7:acb1cf35:::smithi04268048-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 395.01092116500001, 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023701936, 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2704 2024-09-20T19:32:04.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392953+0000", 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392954+0000", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.387e-06 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392960+0000", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392971+0000", 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1413000000000001e-05 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.392980+0000", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8799999999999997e-06 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393747+0000", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076712099999999995 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.393847+0000", 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100008 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416276+0000", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022428547 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416466+0000", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190204 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:25:29.416655+0000", 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018893299999999999 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3508 7.5 7:aa23a6e3:::smithi04268048-4:head [copy-from ver 556] snapc 9d=[9d,9c,9b,99,98,97,94,8f,8d,8b,8a,88,87,86,84,83,81,7f,72] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 324.17028969799998, 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042870617, 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3508 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233584+0000", 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233590+0000", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4070000000000001e-06 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233609+0000", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9473000000000001e-05 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233653+0000", 2024-09-20T19:32:04.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3251999999999999e-05 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233677+0000", 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4773000000000002e-05 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233825+0000", 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014807700000000001 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233841+0000", 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.519e-05 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.246998+0000", 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013157515999999999 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.247358+0000", 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036032499999999999 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.261838+0000", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014479984 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273713+0000", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011874641999999999 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276154+0000", 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024411060000000002 2024-09-20T19:32:04.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276267+0000", 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011283200000000001 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.276455+0000", 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018804 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.476 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:3509 7.5 7:aa23a6e3:::smithi04268048-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e411)", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 324.170135507, 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039850053000000003, 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3509 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233739+0000", 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233741+0000", 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233749+0000", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.089e-06 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233769+0000", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0307000000000001e-05 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233785+0000", 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.647e-05 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233929+0000", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014325000000000001 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.233939+0000", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0317e-05 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.247951+0000", 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014011961999999999 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.248125+0000", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000174272 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273141+0000", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025015565999999999 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273308+0000", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000167088 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:26:40.273589+0000", 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000280735 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2380 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94817798636544 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 195.63266295099999, 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45087614300000001, 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:32:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2380 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771211+0000", 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771215+0000", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2169999999999996e-06 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771228+0000", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3073e-05 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771260+0000", 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1272999999999997e-05 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771276+0000", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6733000000000001e-05 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771453+0000", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017652400000000001 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:48.771485+0000", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2057000000000002e-05 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.219402+0000", 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.447917073 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.219684+0000", 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028212799999999998 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:28:49.222087+0000", 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024030649999999998 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4194 7.15 7:adc64224:::smithi04268048-47:head [read 773382~1344399,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e516)", 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:24.226336+0000", 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.177538404, 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021549948999999999, 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4194 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226336+0000", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226336+0000", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226337+0000", 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7430000000000001e-06 2024-09-20T19:32:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226348+0000", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0984e-05 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226366+0000", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7425e-05 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226379+0000", 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3037000000000001e-05 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226751+0000", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000372259 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.226778+0000", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7141999999999999e-05 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.245689+0000", 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018910762000000001 2024-09-20T19:32:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.245909+0000", 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021955999999999999 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:24.247886+0000", 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019770370000000001 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2595 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94817798637696 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.315839+0000", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.088035400999999, 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0757560559999999, 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2595 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315839+0000", 2024-09-20T19:32:04.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315839+0000", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315842+0000", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.54e-06 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315853+0000", 2024-09-20T19:32:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078e-05 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315883+0000", 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9998999999999999e-05 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.315907+0000", 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4185999999999998e-05 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.316230+0000", 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000323097 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.316260+0000", 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9839e-05 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.386595+0000", 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0703350190000001 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.386795+0000", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020023600000000001 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.391595+0000", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047993599999999999 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4509 7.5 7:a0447796:::smithi04268048-50:head [append 0~0] snapc c4=[c4,c3,c0,b3,b2,aa] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.392669+0000", 2024-09-20T19:32:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.011204677999999, 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050244150000000001, 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4509 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392669+0000", 2024-09-20T19:32:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392669+0000", 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392675+0000", 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6030000000000004e-06 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392691+0000", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6096999999999999e-05 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392725+0000", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4345999999999999e-05 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.392753+0000", 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7212e-05 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405289+0000", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012536136 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405428+0000", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013873100000000001 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.405722+0000", 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029399500000000002 2024-09-20T19:32:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.412697+0000", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069750749999999999 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.432749+0000", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020052575 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.442759+0000", 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010010043 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.442841+0000", 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1567e-05 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.442914+0000", 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2769999999999996e-05 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4510 7.5 7:a0447796:::smithi04268048-50:head [rollback aa] snapc c4=[c4,c3,c0,b3,b2,aa] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:32:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.394560+0000", 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.009314015000001, 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053452280999999997, 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4510 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394560+0000", 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394560+0000", 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394562+0000", 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4040000000000002e-06 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394572+0000", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6029999999999994e-06 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394590+0000", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7438000000000001e-05 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394604+0000", 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4171999999999999e-05 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.406697+0000", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012092952000000001 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.406813+0000", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011644700000000001 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.407085+0000", 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027212299999999998 2024-09-20T19:32:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.412711+0000", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056259689999999998 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.432827+0000", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020115772000000001 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.447921+0000", 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015093646 2024-09-20T19:32:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.447969+0000", 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8610999999999999e-05 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.448012+0000", 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3143999999999999e-05 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4511 7.5 7:a0447796:::smithi04268048-50:head [append 0~0] snapc c4=[c4,c3,c0,b3,b2,aa] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:58.394627+0000", 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.009247395000003, 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053559111999999999, 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4511 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394627+0000", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394627+0000", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394628+0000", 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2720000000000001e-06 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.394634+0000", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2600000000000002e-06 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.397287+0000", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026530970000000001 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.397318+0000", 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1046000000000001e-05 2024-09-20T19:32:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.408643+0000", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011324809 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.408896+0000", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025292100000000003 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,2", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.409256+0000", 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000359448 2024-09-20T19:32:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.412721+0000", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034654629999999998 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.432880+0000", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020159028999999998 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.448072+0000", 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015191895 2024-09-20T19:32:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.448127+0000", 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4654999999999998e-05 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:58.448186+0000", 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9216999999999999e-05 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4993 7.11 7:892846d2:::smithi04268048-15:head [copy-from ver 716] snapc de=[dd,dc,db,da,d8,d7,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:47.284875+0000", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.118998862000002, 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030791948, 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4993 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284875+0000", 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284875+0000", 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284880+0000", 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6709999999999998e-06 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284894+0000", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4440999999999999e-05 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284926+0000", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1338999999999999e-05 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.284941+0000", 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4955e-05 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.285132+0000", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019170499999999999 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.285153+0000", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1060000000000002e-05 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.289041+0000", 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038877109999999999 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.289296+0000", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025519099999999999 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.297872+0000", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085760449999999992 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.306628+0000", 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087551790000000001 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.315218+0000", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085903230000000004 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.315461+0000", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024289400000000001 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.315581+0000", 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011988 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:47.315667+0000", 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6553999999999998e-05 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:04.521 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:32:04.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:04.522 INFO:tasks.rados.rados.0.smithi042.stdout:2023: done (3 left) 2024-09-20T19:32:04.522 INFO:tasks.rados.rados.0.smithi042.stdout:2026: done (2 left) 2024-09-20T19:32:04.522 INFO:tasks.rados.rados.0.smithi042.stdout:2027: done (1 left) 2024-09-20T19:32:04.523 INFO:tasks.rados.rados.0.smithi042.stdout:2028: done (0 left) 2024-09-20T19:32:04.523 INFO:tasks.rados.rados.0.smithi042.stdout:2029: snap_remove snap 230 2024-09-20T19:32:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:04.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:04.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:04.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:04.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:04.937 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:04.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:05.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:05.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:05.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:32:05.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:05.272 INFO:tasks.rados.rados.0.smithi042.stdout:2029: done (0 left) 2024-09-20T19:32:05.272 INFO:tasks.rados.rados.0.smithi042.stdout:2030: snap_create 2024-09-20T19:32:05.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:05 smithi162 bash[24557]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:05 smithi162 bash[24557]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:05 smithi162 bash[24557]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.302 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:05 smithi162 bash[24557]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:05.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[32075]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[32075]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[32075]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[32075]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[22706]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[22706]: cluster 2024-09-20T19:32:04.264532+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-20T19:32:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[22706]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:05 smithi042 bash[22706]: cluster 2024-09-20T19:32:04.304103+0000 mgr.y (mgr.24491) 531 : cluster [DBG] pgmap v922: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 200 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:32:05.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:05.554 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:05.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:05.813 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.840 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:05.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:05.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:05.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:06.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.103 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:06.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:06.276 INFO:tasks.rados.rados.0.smithi042.stdout:2030: done (0 left) 2024-09-20T19:32:06.276 INFO:tasks.rados.rados.0.smithi042.stdout:2031: write (excl) oid 10 current snap is 232 2024-09-20T19:32:06.276 INFO:tasks.rados.rados.0.smithi042.stdout:2031: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-20T19:32:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:32:06.285 INFO:tasks.rados.rados.0.smithi042.stdout:2031: writing smithi04268048-10 from 766584 to 1564701 tid 1 2024-09-20T19:32:06.291 INFO:tasks.rados.rados.0.smithi042.stdout:2031: writing smithi04268048-10 from 2098238 to 2713812 tid 2 2024-09-20T19:32:06.298 INFO:tasks.rados.rados.0.smithi042.stdout:2031: writing smithi04268048-10 from 3169759 to 3630138 tid 3 2024-09-20T19:32:06.298 INFO:tasks.rados.rados.0.smithi042.stdout:2032: delete oid 13 current snap is 232 2024-09-20T19:32:06.298 INFO:tasks.rados.rados.0.smithi042.stdout:2031: finishing write tid 1 to smithi04268048-10 2024-09-20T19:32:06.302 INFO:tasks.rados.rados.0.smithi042.stdout:2031: finishing write tid 2 to smithi04268048-10 2024-09-20T19:32:06.303 INFO:tasks.rados.rados.0.smithi042.stdout:2032: done (1 left) 2024-09-20T19:32:06.303 INFO:tasks.rados.rados.0.smithi042.stdout:2033: delete oid 12 current snap is 232 2024-09-20T19:32:06.307 INFO:tasks.rados.rados.0.smithi042.stdout:2031: finishing write tid 3 to smithi04268048-10 2024-09-20T19:32:06.307 INFO:tasks.rados.rados.0.smithi042.stdout:2031: finishing write tid 4 to smithi04268048-10 2024-09-20T19:32:06.308 INFO:tasks.rados.rados.0.smithi042.stdout:2033: done (1 left) 2024-09-20T19:32:06.308 INFO:tasks.rados.rados.0.smithi042.stdout:2034: snap_create 2024-09-20T19:32:06.308 INFO:tasks.rados.rados.0.smithi042.stdout:2031: finishing write tid 5 to smithi04268048-10 2024-09-20T19:32:06.308 INFO:tasks.rados.rados.0.smithi042.stdout:2031: oid 10 updating version 0 to 680 2024-09-20T19:32:06.308 INFO:tasks.rados.rados.0.smithi042.stdout:2031: oid 10 updating version 680 to 681 2024-09-20T19:32:06.309 INFO:tasks.rados.rados.0.smithi042.stdout:2031: oid 10 updating version 681 to 683 2024-09-20T19:32:06.309 INFO:tasks.rados.rados.0.smithi042.stdout:2031: oid 10 version 683 is already newer than 682 2024-09-20T19:32:06.309 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 683 (ObjNum 712 snap 232 seq_num 712) dirty exists 2024-09-20T19:32:06.309 INFO:tasks.rados.rados.0.smithi042.stdout:2031: left oid 10 (ObjNum 712 snap 232 seq_num 712) 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.337 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[32075]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.338 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:06 smithi042 bash[22706]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:06.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:06.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:06.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: cluster 2024-09-20T19:32:05.271856+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-20T19:32:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: audit 2024-09-20T19:32:05.601721+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.42:0/3489111457' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:06 smithi162 bash[24557]: audit 2024-09-20T19:32:05.602092+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25559 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:06.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:06.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:06.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.756 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:06.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:06.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:06.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:06.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:07.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:07.149 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:07.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:07.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:07.307 INFO:tasks.rados.rados.0.smithi042.stdout:2031: done (1 left) 2024-09-20T19:32:07.307 INFO:tasks.rados.rados.0.smithi042.stdout:2034: done (0 left) 2024-09-20T19:32:07.307 INFO:tasks.rados.rados.0.smithi042.stdout:2035: snap_create 2024-09-20T19:32:07.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:07 smithi162 bash[24557]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:07.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[22706]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.267782+0000 mon.a (mon.0) 2019 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: audit 2024-09-20T19:32:06.269504+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.274791+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:07 smithi042 bash[32075]: cluster 2024-09-20T19:32:06.304652+0000 mgr.y (mgr.24491) 532 : cluster [DBG] pgmap v925: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 203 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T19:32:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:07.648 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:07.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:07.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:08.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:08.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:08.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:08.169 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:08.270 INFO:tasks.rados.rados.0.smithi042.stdout:2035: done (0 left) 2024-09-20T19:32:08.270 INFO:tasks.rados.rados.0.smithi042.stdout:2036: write oid 27 current snap is 234 2024-09-20T19:32:08.270 INFO:tasks.rados.rados.0.smithi042.stdout:2036: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-20T19:32:08.272 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:08.284 INFO:tasks.rados.rados.0.smithi042.stdout:2036: writing smithi04268048-27 from 417005 to 1157222 tid 1 2024-09-20T19:32:08.292 INFO:tasks.rados.rados.0.smithi042.stdout:2036: writing smithi04268048-27 from 1590037 to 2017005 tid 2 2024-09-20T19:32:08.292 INFO:tasks.rados.rados.0.smithi042.stdout:2037: snap_remove snap 233 2024-09-20T19:32:08.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:08.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.418 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.419 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:08 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:08.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: cluster 2024-09-20T19:32:07.299618+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-20T19:32:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: audit 2024-09-20T19:32:08.048133+0000 mon.a (mon.0) 2023 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:32:08.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:08 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.267803+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-20T19:32:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:08.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:08.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:08.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:08.812 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:08.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:08.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:09.041 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:09.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:09.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:09.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:09.231 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:32:09.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[32075]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.308 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.308 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.310 INFO:tasks.rados.rados.0.smithi042.stdout:2037: done (1 left) 2024-09-20T19:32:09.310 INFO:tasks.rados.rados.0.smithi042.stdout:2038: write (excl) oid 11 current snap is 234 2024-09-20T19:32:09.310 INFO:tasks.rados.rados.0.smithi042.stdout:2036: finishing write tid 1 to smithi04268048-27 2024-09-20T19:32:09.310 INFO:tasks.rados.rados.0.smithi042.stdout:2038: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-20T19:32:09.314 INFO:tasks.rados.rados.0.smithi042.stdout:2038: writing smithi04268048-11 from 433812 to 849278 tid 1 2024-09-20T19:32:09.322 INFO:tasks.rados.rados.0.smithi042.stdout:2038: writing smithi04268048-11 from 1648519 to 2033812 tid 2 2024-09-20T19:32:09.322 INFO:tasks.rados.rados.0.smithi042.stdout:2036: finishing write tid 2 to smithi04268048-27 2024-09-20T19:32:09.322 INFO:tasks.rados.rados.0.smithi042.stdout:2036: finishing write tid 3 to smithi04268048-27 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: finishing write tid 4 to smithi04268048-27 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: oid 27 updating version 0 to 873 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: oid 27 updating version 873 to 874 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: oid 27 updating version 874 to 875 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 875 (ObjNum 715 snap 234 seq_num 715) dirty exists 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: left oid 27 (ObjNum 715 snap 234 seq_num 715) 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2036: done (1 left) 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2039: write oid 26 current snap is 234 2024-09-20T19:32:09.323 INFO:tasks.rados.rados.0.smithi042.stdout:2039: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-20T19:32:09.328 INFO:tasks.rados.rados.0.smithi042.stdout:2039: writing smithi04268048-26 from 450619 to 941334 tid 1 2024-09-20T19:32:09.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:09.335 INFO:tasks.rados.rados.0.smithi042.stdout:2039: writing smithi04268048-26 from 1590648 to 2050619 tid 2 2024-09-20T19:32:09.335 INFO:tasks.rados.rados.0.smithi042.stdout:2040: write oid 31 current snap is 234 2024-09-20T19:32:09.335 INFO:tasks.rados.rados.0.smithi042.stdout:2040: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-20T19:32:09.342 INFO:tasks.rados.rados.0.smithi042.stdout:2040: writing smithi04268048-31 from 467426 to 1033390 tid 1 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2040: writing smithi04268048-31 from 1649130 to 2067426 tid 2 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2041: copy_from oid 19 from oid 43 current snap is 234 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2038: finishing write tid 1 to smithi04268048-11 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2038: finishing write tid 2 to smithi04268048-11 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2038: finishing write tid 3 to smithi04268048-11 2024-09-20T19:32:09.348 INFO:tasks.rados.rados.0.smithi042.stdout:2038: finishing write tid 4 to smithi04268048-11 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2038: oid 11 updating version 0 to 828 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2038: oid 11 version 828 is already newer than 826 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2038: oid 11 version 828 is already newer than 827 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 828 (ObjNum 716 snap 234 seq_num 716) dirty exists 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2038: left oid 11 (ObjNum 716 snap 234 seq_num 716) 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: finishing write tid 1 to smithi04268048-26 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: finishing write tid 2 to smithi04268048-26 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: finishing write tid 3 to smithi04268048-26 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: finishing write tid 4 to smithi04268048-26 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: oid 26 updating version 0 to 862 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: oid 26 version 862 is already newer than 861 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: oid 26 version 862 is already newer than 860 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 862 (ObjNum 717 snap 234 seq_num 717) dirty exists 2024-09-20T19:32:09.349 INFO:tasks.rados.rados.0.smithi042.stdout:2039: left oid 26 (ObjNum 717 snap 234 seq_num 717) 2024-09-20T19:32:09.350 INFO:tasks.rados.rados.0.smithi042.stdout:2038: done (3 left) 2024-09-20T19:32:09.350 INFO:tasks.rados.rados.0.smithi042.stdout:2039: done (2 left) 2024-09-20T19:32:09.350 INFO:tasks.rados.rados.0.smithi042.stdout:2042: snap_remove snap 232 2024-09-20T19:32:09.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.305242+0000 mgr.y (mgr.24491) 533 : cluster [DBG] pgmap v928: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 207 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.784816+0000 mon.a (mon.0) 2025 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[24557]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:09.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:09.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:09.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.786264+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:32:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:09 smithi042 bash[22706]: audit 2024-09-20T19:32:08.797189+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:09.690 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:09.734 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:09.734 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:09.735 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:09.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:09.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:09.846 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:09.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:09.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:09.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:10.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:10.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:10.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:10.236 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:10.306 INFO:tasks.rados.rados.0.smithi042.stdout:2042: done (2 left) 2024-09-20T19:32:10.306 INFO:tasks.rados.rados.0.smithi042.stdout:2043: snap_create 2024-09-20T19:32:10.306 INFO:tasks.rados.rados.0.smithi042.stdout:2040: finishing write tid 1 to smithi04268048-31 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: finishing write tid 2 to smithi04268048-31 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: finishing write tid 3 to smithi04268048-31 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: finishing write tid 4 to smithi04268048-31 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: oid 31 updating version 0 to 725 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: oid 31 updating version 725 to 726 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: oid 31 version 726 is already newer than 724 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 726 (ObjNum 718 snap 234 seq_num 718) dirty exists 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2040: left oid 31 (ObjNum 718 snap 234 seq_num 718) 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2041: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:2041: finishing copy_from to smithi04268048-19 2024-09-20T19:32:10.307 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 719 (ObjNum 594 snap 189 seq_num 594) dirty exists 2024-09-20T19:32:10.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:10.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:10.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:10.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:10 smithi162 bash[24557]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[22706]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.549114+0000 osd.2 (osd.2) 199 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:08.554864+0000 osd.2 (osd.2) 200 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.304367+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.334232+0000 osd.0 (osd.0) 17 : cluster [DBG] 7.f scrub starts 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.344672+0000 osd.0 (osd.0) 18 : cluster [DBG] 7.f scrub ok 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.565306+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.566792+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: audit 2024-09-20T19:32:09.580417+0000 mgr.y (mgr.24491) 534 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.607574+0000 osd.5 (osd.5) 181 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:10 smithi042 bash[32075]: cluster 2024-09-20T19:32:09.614893+0000 osd.5 (osd.5) 182 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:32:10.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:10.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:10.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:10.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:10.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:11.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:11.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:11.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:11.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:11.319 INFO:tasks.rados.rados.0.smithi042.stdout:2040: done (2 left) 2024-09-20T19:32:11.319 INFO:tasks.rados.rados.0.smithi042.stdout:2041: done (1 left) 2024-09-20T19:32:11.319 INFO:tasks.rados.rados.0.smithi042.stdout:2043: done (0 left) 2024-09-20T19:32:11.320 INFO:tasks.rados.rados.0.smithi042.stdout:2044: snap_remove snap 231 2024-09-20T19:32:11.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:11 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:11.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:11.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:11.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.305006+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.305807+0000 mgr.y (mgr.24491) 535 : cluster [DBG] pgmap v931: 168 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 160 active+clean; 208 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.329084+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.330779+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.438719+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.c scrub starts 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.443989+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.c scrub ok 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.637079+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.638667+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.782971+0000 osd.7 (osd.7) 153 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:11 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.788436+0000 osd.7 (osd.7) 154 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:32:11.765 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:11.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:11.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:11.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:12.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:12.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:12.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:12.282 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.316 INFO:tasks.rados.rados.0.smithi042.stdout:2044: done (0 left) 2024-09-20T19:32:12.316 INFO:tasks.rados.rados.0.smithi042.stdout:2045: write (excl) oid 39 current snap is 235 2024-09-20T19:32:12.316 INFO:tasks.rados.rados.0.smithi042.stdout:2045: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-20T19:32:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:12.323 INFO:tasks.rados.rados.0.smithi042.stdout:2045: writing smithi04268048-39 from 484233 to 1125446 tid 1 2024-09-20T19:32:12.329 INFO:tasks.rados.rados.0.smithi042.stdout:2045: writing smithi04268048-39 from 1707612 to 2084233 tid 2 2024-09-20T19:32:12.329 INFO:tasks.rados.rados.0.smithi042.stdout:2046: copy_from oid 26 from oid 1 current snap is 235 2024-09-20T19:32:12.329 INFO:tasks.rados.rados.0.smithi042.stdout:2047: delete oid 35 current snap is 235 2024-09-20T19:32:12.344 INFO:tasks.rados.rados.0.smithi042.stdout:2046: finishing copy_from to smithi04268048-26 2024-09-20T19:32:12.344 INFO:tasks.rados.rados.0.smithi042.stdout:2046: got expected ENOENT (src dne) 2024-09-20T19:32:12.344 INFO:tasks.rados.rados.0.smithi042.stdout:2046: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:32:12.348 INFO:tasks.rados.rados.0.smithi042.stdout:2046: done (2 left) 2024-09-20T19:32:12.348 INFO:tasks.rados.rados.0.smithi042.stdout:2047: done (1 left) 2024-09-20T19:32:12.349 INFO:tasks.rados.rados.0.smithi042.stdout:2048: copy_from oid 41 from oid 7 current snap is 235 2024-09-20T19:32:12.349 INFO:tasks.rados.rados.0.smithi042.stdout:2049: snap_create 2024-09-20T19:32:12.353 INFO:tasks.rados.rados.0.smithi042.stdout:2045: finishing write tid 1 to smithi04268048-39 2024-09-20T19:32:12.353 INFO:tasks.rados.rados.0.smithi042.stdout:2045: finishing write tid 2 to smithi04268048-39 2024-09-20T19:32:12.354 INFO:tasks.rados.rados.0.smithi042.stdout:2045: finishing write tid 3 to smithi04268048-39 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:2045: finishing write tid 4 to smithi04268048-39 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:2045: oid 39 updating version 0 to 698 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:2045: oid 39 version 698 is already newer than 697 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:2045: oid 39 updating version 698 to 699 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 699 (ObjNum 719 snap 235 seq_num 719) dirty exists 2024-09-20T19:32:12.355 INFO:tasks.rados.rados.0.smithi042.stdout:2045: left oid 39 (ObjNum 719 snap 235 seq_num 719) 2024-09-20T19:32:12.369 INFO:tasks.rados.rados.0.smithi042.stdout:2048: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:32:12.371 INFO:tasks.rados.rados.0.smithi042.stdout:2048: finishing copy_from to smithi04268048-41 2024-09-20T19:32:12.371 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 630 (ObjNum 631 snap 198 seq_num 631) dirty exists 2024-09-20T19:32:12.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:12 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:12.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:12.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.797679+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:10.800339+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.317153+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.367903+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.370677+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:32:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.591958+0000 osd.5 (osd.5) 185 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:32:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.594896+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:32:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:12 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.315672+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-20T19:32:12.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:12.829 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:12.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:12.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:13.035 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:32:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:32:13.055 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:32:13.055 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:32:13.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:13.072 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:32:13.073 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:32:13.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:13.108 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:32:13.108 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:32:13.129 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:32:13.129 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:32:13.150 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:32:13.151 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:32:13.151 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:32:13.152 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:32:13.174 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:32:13.175 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:32:13.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:13.196 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:32:13.196 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:32:13.209 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T19:32:13.266 INFO:tasks.rados.rados.0.smithi042.stdout:2045: done (2 left) 2024-09-20T19:32:13.266 INFO:tasks.rados.rados.0.smithi042.stdout:2048: done (1 left) 2024-09-20T19:32:13.266 INFO:tasks.rados.rados.0.smithi042.stdout:2049: done (0 left) 2024-09-20T19:32:13.266 INFO:tasks.rados.rados.0.smithi042.stdout:2050: delete oid 10 current snap is 236 2024-09-20T19:32:13.280 INFO:tasks.rados.rados.0.smithi042.stdout:2050: done (0 left) 2024-09-20T19:32:13.280 INFO:tasks.rados.rados.0.smithi042.stdout:2051: write oid 39 current snap is 236 2024-09-20T19:32:13.280 INFO:tasks.rados.rados.0.smithi042.stdout:2051: seq_num 722 ranges {534654=466960,1766705=367949} 2024-09-20T19:32:13.285 INFO:tasks.rados.rados.0.smithi042.stdout:2051: writing smithi04268048-39 from 534654 to 1001614 tid 1 2024-09-20T19:32:13.292 INFO:tasks.rados.rados.0.smithi042.stdout:2051: writing smithi04268048-39 from 1766705 to 2134654 tid 2 2024-09-20T19:32:13.293 INFO:tasks.rados.rados.0.smithi042.stdout:2052: rollback oid 28 current snap is 236 2024-09-20T19:32:13.293 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 212 2024-09-20T19:32:13.293 INFO:tasks.rados.rados.0.smithi042.stdout:2053: write oid 26 current snap is 236 2024-09-20T19:32:13.293 INFO:tasks.rados.rados.0.smithi042.stdout:2053: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-20T19:32:13.300 INFO:tasks.rados.rados.0.smithi042.stdout:2053: writing smithi04268048-26 from 551461 to 1210023 tid 1 2024-09-20T19:32:13.306 INFO:tasks.rados.rados.0.smithi042.stdout:2053: writing smithi04268048-26 from 1941540 to 2151461 tid 2 2024-09-20T19:32:13.307 INFO:tasks.rados.rados.0.smithi042.stdout:2054: write oid 24 current snap is 236 2024-09-20T19:32:13.307 INFO:tasks.rados.rados.0.smithi042.stdout:2052: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:32:13.307 INFO:tasks.rados.rados.0.smithi042.stdout:2054: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-20T19:32:13.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:13.314 INFO:tasks.rados.rados.0.smithi042.stdout:2054: writing smithi04268048-24 from 568268 to 1302079 tid 1 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:2054: writing smithi04268048-24 from 2000022 to 2168268 tid 2 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:2055: write (excl) oid 3 current snap is 236 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:2052: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:2052: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 722 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:32:13.319 INFO:tasks.rados.rados.0.smithi042.stdout:2055: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-20T19:32:13.324 INFO:tasks.rados.rados.0.smithi042.stdout:2055: writing smithi04268048-3 from 585075 to 994135 tid 1 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2055: writing smithi04268048-3 from 1542151 to 2185075 tid 2 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2052: done (4 left) 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2056: read oid 5 snap 223 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2056: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2057: write oid 12 current snap is 236 2024-09-20T19:32:13.332 INFO:tasks.rados.rados.0.smithi042.stdout:2057: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-20T19:32:13.339 INFO:tasks.rados.rados.0.smithi042.stdout:2057: writing smithi04268048-12 from 601882 to 1086191 tid 1 2024-09-20T19:32:13.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:13.346 INFO:tasks.rados.rados.0.smithi042.stdout:2057: writing smithi04268048-12 from 1600633 to 2125537 tid 2 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2053: finishing write tid 1 to smithi04268048-26 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2053: finishing write tid 2 to smithi04268048-26 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2051: finishing write tid 1 to smithi04268048-39 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2053: finishing write tid 3 to smithi04268048-26 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2058: read oid 10 snap -1 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2058: expect deleted 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2059: read oid 41 snap -1 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2059: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2060: read oid 14 snap -1 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2060: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2061: snap_create 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2051: finishing write tid 2 to smithi04268048-39 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2051: finishing write tid 3 to smithi04268048-39 2024-09-20T19:32:13.347 INFO:tasks.rados.rados.0.smithi042.stdout:2051: finishing write tid 4 to smithi04268048-39 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2051: oid 39 updating version 0 to 701 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2051: oid 39 updating version 701 to 702 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2051: oid 39 version 702 is already newer than 700 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 702 (ObjNum 722 snap 236 seq_num 722) dirty exists 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2051: left oid 39 (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2053: finishing write tid 4 to smithi04268048-26 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2053: oid 26 updating version 0 to 865 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2053: oid 26 version 865 is already newer than 863 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2053: oid 26 version 865 is already newer than 864 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 865 (ObjNum 723 snap 236 seq_num 723) dirty exists 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2053: left oid 26 (ObjNum 723 snap 236 seq_num 723) 2024-09-20T19:32:13.348 INFO:tasks.rados.rados.0.smithi042.stdout:2054: finishing write tid 1 to smithi04268048-24 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: finishing write tid 2 to smithi04268048-24 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: finishing write tid 3 to smithi04268048-24 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: finishing write tid 4 to smithi04268048-24 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: oid 24 updating version 0 to 737 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: oid 24 updating version 737 to 738 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: oid 24 version 738 is already newer than 736 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 738 (ObjNum 724 snap 236 seq_num 724) dirty exists 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2054: left oid 24 (ObjNum 724 snap 236 seq_num 724) 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2055: finishing write tid 1 to smithi04268048-3 2024-09-20T19:32:13.349 INFO:tasks.rados.rados.0.smithi042.stdout:2056: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:13.378 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 216574 2024-09-20T19:32:13.411 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1557362 2024-09-20T19:32:13.422 INFO:tasks.rados.rados.0.smithi042.stdout:2055: finishing write tid 2 to smithi04268048-3 2024-09-20T19:32:13.422 INFO:tasks.rados.rados.0.smithi042.stdout:2055: finishing write tid 3 to smithi04268048-3 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2055: finishing write tid 4 to smithi04268048-3 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2055: oid 3 updating version 0 to 723 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2055: oid 3 version 723 is already newer than 722 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2055: oid 3 updating version 723 to 724 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 724 (ObjNum 725 snap 236 seq_num 725) dirty exists 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2055: left oid 3 (ObjNum 725 snap 236 seq_num 725) 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2057: finishing write tid 1 to smithi04268048-12 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2057: finishing write tid 2 to smithi04268048-12 2024-09-20T19:32:13.423 INFO:tasks.rados.rados.0.smithi042.stdout:2060: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:13 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:13.470 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:13.486 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: finishing write tid 3 to smithi04268048-12 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: finishing write tid 4 to smithi04268048-12 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: oid 12 updating version 0 to 639 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: oid 12 version 639 is already newer than 638 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: oid 12 version 639 is already newer than 637 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 639 (ObjNum 726 snap 236 seq_num 726) dirty exists 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2057: left oid 12 (ObjNum 726 snap 236 seq_num 726) 2024-09-20T19:32:13.519 INFO:tasks.rados.rados.0.smithi042.stdout:2059: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:13.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:13.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.826023+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub starts 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:11.827916+0000 osd.3 (osd.3) 22 : cluster [DBG] 6.b scrub ok 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.306716+0000 mgr.y (mgr.24491) 536 : cluster [DBG] pgmap v933: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 228 MiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.617524+0000 osd.5 (osd.5) 187 : cluster [DBG] 6.a scrub starts 2024-09-20T19:32:13.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.619542+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.a scrub ok 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:12.902537+0000 mon.a (mon.0) 2032 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.013158+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.42:0/1617427852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.204326+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: audit 2024-09-20T19:32:13.262396+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.42:0/3821449911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:32:13.713 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.713 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:13 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.266011+0000 mon.a (mon.0) 2035 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-20T19:32:13.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:13.828 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:13.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:13.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:13.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:13.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:14.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:14.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:14.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:14.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:14.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.353 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[22706]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2051: done (9 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2053: done (8 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2054: done (7 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2055: done (6 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2056: done (5 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2057: done (4 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2058: done (3 left) 2024-09-20T19:32:14.357 INFO:tasks.rados.rados.0.smithi042.stdout:2059: done (2 left) 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2060: done (1 left) 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2061: done (0 left) 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2062: copy_from oid 20 from oid 37 current snap is 237 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2063: rollback oid 25 current snap is 237 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 218 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2064: rollback oid 28 current snap is 237 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 218 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2065: read oid 14 snap -1 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2065: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2066: copy_from oid 2 from oid 29 current snap is 237 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2067: read oid 22 snap -1 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2067: expect deleted 2024-09-20T19:32:14.358 INFO:tasks.rados.rados.0.smithi042.stdout:2068: snap_create 2024-09-20T19:32:14.365 INFO:tasks.rados.rados.0.smithi042.stdout:2064: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:32:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2064: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:32:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2066: finishing copy_from to smithi04268048-2 2024-09-20T19:32:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2066: got expected ENOENT (src dne) 2024-09-20T19:32:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2066: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:32:14.371 INFO:tasks.rados.rados.0.smithi042.stdout:2064: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:32:14.371 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 725 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:32:14.372 INFO:tasks.rados.rados.0.smithi042.stdout:2063: finishing rollback tid 0 to smithi04268048-25 2024-09-20T19:32:14.372 INFO:tasks.rados.rados.0.smithi042.stdout:2063: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:32:14.374 INFO:tasks.rados.rados.0.smithi042.stdout:2063: finishing rollback tid 2 to smithi04268048-25 2024-09-20T19:32:14.374 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 819 (ObjNum 643 snap 206 seq_num 643) dirty exists 2024-09-20T19:32:14.375 INFO:tasks.rados.rados.0.smithi042.stdout:2065: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-20T19:32:14.377 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:14.378 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1260383 2024-09-20T19:32:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:14.406 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 525475 2024-09-20T19:32:14.466 INFO:tasks.rados.rados.0.smithi042.stdout:2062: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:32:14.466 INFO:tasks.rados.rados.0.smithi042.stdout:2062: finishing copy_from to smithi04268048-20 2024-09-20T19:32:14.466 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 757 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:32:14.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.492 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.493 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:14 smithi162 bash[24557]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.508 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:14.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.636 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.872591+0000 osd.3 (osd.3) 23 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:12.873968+0000 osd.3 (osd.3) 24 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.392258+0000 osd.0 (osd.0) 23 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.393742+0000 osd.0 (osd.0) 24 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.570437+0000 osd.5 (osd.5) 189 : cluster [DBG] 5.f scrub starts 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:14 smithi042 bash[32075]: cluster 2024-09-20T19:32:13.572565+0000 osd.5 (osd.5) 190 : cluster [DBG] 5.f scrub ok 2024-09-20T19:32:14.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:14.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:14.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:14.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.898 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:14.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:14.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T19:32:15.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:15.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:15.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:15.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:15.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:15.323 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:15.364 INFO:tasks.rados.rados.0.smithi042.stdout:2062: done (6 left) 2024-09-20T19:32:15.364 INFO:tasks.rados.rados.0.smithi042.stdout:2063: done (5 left) 2024-09-20T19:32:15.364 INFO:tasks.rados.rados.0.smithi042.stdout:2064: done (4 left) 2024-09-20T19:32:15.364 INFO:tasks.rados.rados.0.smithi042.stdout:2065: done (3 left) 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:2066: done (2 left) 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:2067: done (1 left) 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:2068: done (0 left) 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:2069: rollback oid 31 current snap is 238 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 212 2024-09-20T19:32:15.365 INFO:tasks.rados.rados.0.smithi042.stdout:2070: delete oid 19 current snap is 238 2024-09-20T19:32:15.373 INFO:tasks.rados.rados.0.smithi042.stdout:2070: done (1 left) 2024-09-20T19:32:15.373 INFO:tasks.rados.rados.0.smithi042.stdout:2071: snap_remove snap 224 2024-09-20T19:32:15.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:15.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:15.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.568 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:15 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.307450+0000 mgr.y (mgr.24491) 537 : cluster [DBG] pgmap v936: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 232 MiB data, 829 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.354057+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.178370+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:15 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.179827+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:15.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:15.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:15.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:15.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T19:32:15.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:15.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:15.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:15.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:16.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:16.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:16.234 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:16.273 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:16.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:16.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:16.390 INFO:tasks.rados.rados.0.smithi042.stdout:2071: done (1 left) 2024-09-20T19:32:16.390 INFO:tasks.rados.rados.0.smithi042.stdout:2072: delete oid 11 current snap is 238 2024-09-20T19:32:16.390 INFO:tasks.rados.rados.0.smithi042.stdout:2069: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:32:16.390 INFO:tasks.rados.rados.0.smithi042.stdout:2069: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:32:16.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 728 (ObjNum 0 snap 32529 seq_num 304) dirty dne 2024-09-20T19:32:16.398 INFO:tasks.rados.rados.0.smithi042.stdout:2069: done (1 left) 2024-09-20T19:32:16.398 INFO:tasks.rados.rados.0.smithi042.stdout:2072: done (0 left) 2024-09-20T19:32:16.398 INFO:tasks.rados.rados.0.smithi042.stdout:2073: copy_from oid 16 from oid 30 current snap is 238 2024-09-20T19:32:16.398 INFO:tasks.rados.rados.0.smithi042.stdout:2074: delete oid 22 current snap is 238 2024-09-20T19:32:16.406 INFO:tasks.rados.rados.0.smithi042.stdout:2073: finishing copy_from to smithi04268048-16 2024-09-20T19:32:16.406 INFO:tasks.rados.rados.0.smithi042.stdout:2073: got expected ENOENT (src dne) 2024-09-20T19:32:16.407 INFO:tasks.rados.rados.0.smithi042.stdout:2073: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:32:16.408 INFO:tasks.rados.rados.0.smithi042.stdout:2073: done (1 left) 2024-09-20T19:32:16.409 INFO:tasks.rados.rados.0.smithi042.stdout:2074: done (0 left) 2024-09-20T19:32:16.409 INFO:tasks.rados.rados.0.smithi042.stdout:2075: snap_create 2024-09-20T19:32:16.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:16.479 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:16.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.480 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.481 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[32075]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.483 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:16 smithi042 bash[22706]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:16.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:16.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:16.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.824704+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:14.826712+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.094110+0000 osd.1 (osd.1) 215 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.096441+0000 osd.1 (osd.1) 216 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:32:16.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.362878+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.449067+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.451987+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.461215+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.464795+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.775514+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.777732+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:32:16.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.865374+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:32:16.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:16 smithi162 bash[24557]: cluster 2024-09-20T19:32:15.866567+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:32:16.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:16.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:16.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:16.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:17.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:17.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:17.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:17.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:17.381 INFO:tasks.rados.rados.0.smithi042.stdout:2075: done (0 left) 2024-09-20T19:32:17.381 INFO:tasks.rados.rados.0.smithi042.stdout:2076: write oid 41 current snap is 239 2024-09-20T19:32:17.381 INFO:tasks.rados.rados.0.smithi042.stdout:2076: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-20T19:32:17.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:17.399 INFO:tasks.rados.rados.0.smithi042.stdout:2076: writing smithi04268048-41 from 669110 to 1170768 tid 1 2024-09-20T19:32:17.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:17.417 INFO:tasks.rados.rados.0.smithi042.stdout:2076: writing smithi04268048-41 from 1834561 to 2269110 tid 2 2024-09-20T19:32:17.417 INFO:tasks.rados.rados.0.smithi042.stdout:2077: snap_create 2024-09-20T19:32:17.417 INFO:tasks.rados.rados.0.smithi042.stdout:2076: finishing write tid 1 to smithi04268048-41 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: finishing write tid 2 to smithi04268048-41 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: finishing write tid 3 to smithi04268048-41 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: finishing write tid 4 to smithi04268048-41 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: oid 41 updating version 0 to 704 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: oid 41 version 704 is already newer than 703 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: oid 41 updating version 704 to 705 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 705 (ObjNum 730 snap 239 seq_num 730) dirty exists 2024-09-20T19:32:17.422 INFO:tasks.rados.rados.0.smithi042.stdout:2076: left oid 41 (ObjNum 730 snap 239 seq_num 730) 2024-09-20T19:32:17.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:17.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.524 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:17 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:17.644 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.073981+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:32:17.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.075331+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.307961+0000 mgr.y (mgr.24491) 538 : cluster [DBG] pgmap v939: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 235 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.4 MiB/s wr, 3 op/s 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.388706+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.491568+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.493873+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.793483+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:17 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.795114+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:32:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:17.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:17.807 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:17.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:32:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:17.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:17.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:17.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:18.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:18.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.189 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:18.274 INFO:tasks.rados.rados.0.smithi042.stdout:2076: done (1 left) 2024-09-20T19:32:18.274 INFO:tasks.rados.rados.0.smithi042.stdout:2077: done (0 left) 2024-09-20T19:32:18.274 INFO:tasks.rados.rados.0.smithi042.stdout:2078: write (excl) oid 2 current snap is 240 2024-09-20T19:32:18.274 INFO:tasks.rados.rados.0.smithi042.stdout:2078: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-20T19:32:18.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.295 INFO:tasks.rados.rados.0.smithi042.stdout:2078: writing smithi04268048-2 from 685917 to 1262824 tid 1 2024-09-20T19:32:18.311 INFO:tasks.rados.rados.0.smithi042.stdout:2078: writing smithi04268048-2 from 1893043 to 2285917 tid 2 2024-09-20T19:32:18.311 INFO:tasks.rados.rados.0.smithi042.stdout:2079: write oid 10 current snap is 240 2024-09-20T19:32:18.311 INFO:tasks.rados.rados.0.smithi042.stdout:2079: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-20T19:32:18.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.321 INFO:tasks.rados.rados.0.smithi042.stdout:2079: writing smithi04268048-10 from 702724 to 1354880 tid 1 2024-09-20T19:32:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2079: writing smithi04268048-10 from 1951525 to 2302724 tid 2 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: finishing write tid 1 to smithi04268048-2 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: finishing write tid 2 to smithi04268048-2 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: finishing write tid 3 to smithi04268048-2 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: finishing write tid 4 to smithi04268048-2 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: oid 2 updating version 0 to 613 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: oid 2 version 613 is already newer than 611 2024-09-20T19:32:18.332 INFO:tasks.rados.rados.0.smithi042.stdout:2078: oid 2 version 613 is already newer than 612 2024-09-20T19:32:18.333 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 613 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:32:18.333 INFO:tasks.rados.rados.0.smithi042.stdout:2078: left oid 2 (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:32:18.333 INFO:tasks.rados.rados.0.smithi042.stdout:2079: finishing write tid 1 to smithi04268048-10 2024-09-20T19:32:18.333 INFO:tasks.rados.rados.0.smithi042.stdout:2078: done (1 left) 2024-09-20T19:32:18.333 INFO:tasks.rados.rados.0.smithi042.stdout:2080: snap_create 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: finishing write tid 2 to smithi04268048-10 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: finishing write tid 3 to smithi04268048-10 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: finishing write tid 4 to smithi04268048-10 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: oid 10 updating version 0 to 695 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: oid 10 updating version 695 to 696 2024-09-20T19:32:18.338 INFO:tasks.rados.rados.0.smithi042.stdout:2079: oid 10 version 696 is already newer than 694 2024-09-20T19:32:18.339 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 696 (ObjNum 732 snap 240 seq_num 732) dirty exists 2024-09-20T19:32:18.339 INFO:tasks.rados.rados.0.smithi042.stdout:2079: left oid 10 (ObjNum 732 snap 240 seq_num 732) 2024-09-20T19:32:18.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:18.441 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:18.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:18 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:18.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:18.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.782216+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:16.785129+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.379452+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.475949+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:17.478133+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:18 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.272152+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-20T19:32:18.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:18.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:18.837 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:18.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:18.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:18.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:19.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:19.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:19.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:19.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:19.283 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:32:19.288 INFO:tasks.rados.rados.0.smithi042.stdout:2079: done (1 left) 2024-09-20T19:32:19.288 INFO:tasks.rados.rados.0.smithi042.stdout:2080: done (0 left) 2024-09-20T19:32:19.288 INFO:tasks.rados.rados.0.smithi042.stdout:2081: snap_remove snap 204 2024-09-20T19:32:19.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:19.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:19.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:19.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.472 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:19 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:19.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:19.599 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.308405+0000 mgr.y (mgr.24491) 539 : cluster [DBG] pgmap v943: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: audit 2024-09-20T19:32:18.396059+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.396633+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.400298+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.275617+0000 mon.a (mon.0) 2042 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:32:19.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.275639+0000 mon.a (mon.0) 2043 : cluster [INF] Cluster is now healthy 2024-09-20T19:32:19.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: audit 2024-09-20T19:32:19.283072+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.42:0/1222544555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:19.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:19 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.287727+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-20T19:32:19.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:19.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:19.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:19.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:19.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.002 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:20.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:20.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:20.266 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:20.300 INFO:tasks.rados.rados.0.smithi042.stdout:2081: done (0 left) 2024-09-20T19:32:20.300 INFO:tasks.rados.rados.0.smithi042.stdout:2082: snap_create 2024-09-20T19:32:20.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:20.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:20.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:20.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:20.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:20 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:20.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.827977+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:18.830834+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.379858+0000 osd.4 (osd.4) 31 : cluster [DBG] 2.f scrub starts 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.382843+0000 osd.4 (osd.4) 32 : cluster [DBG] 2.f scrub ok 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: audit 2024-09-20T19:32:19.588028+0000 mgr.y (mgr.24491) 540 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.713328+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:19.715163+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:20 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.289775+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-20T19:32:20.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:20.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:20.806 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:20.807 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:20.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:20.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:20.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:21.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:21.041 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:21.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:21.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:21.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:21.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:2082: done (0 left) 2024-09-20T19:32:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:2083: write oid 36 current snap is 242 2024-09-20T19:32:21.303 INFO:tasks.rados.rados.0.smithi042.stdout:2083: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-20T19:32:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:21.324 INFO:tasks.rados.rados.0.smithi042.stdout:2083: writing smithi04268048-36 from 719531 to 1446936 tid 1 2024-09-20T19:32:21.338 INFO:tasks.rados.rados.0.smithi042.stdout:2083: writing smithi04268048-36 from 1893654 to 2319531 tid 2 2024-09-20T19:32:21.338 INFO:tasks.rados.rados.0.smithi042.stdout:2084: snap_create 2024-09-20T19:32:21.338 INFO:tasks.rados.rados.0.smithi042.stdout:2083: finishing write tid 1 to smithi04268048-36 2024-09-20T19:32:21.341 INFO:tasks.rados.rados.0.smithi042.stdout:2083: finishing write tid 2 to smithi04268048-36 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: finishing write tid 3 to smithi04268048-36 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: finishing write tid 4 to smithi04268048-36 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: oid 36 updating version 0 to 572 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: oid 36 updating version 572 to 573 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: oid 36 updating version 573 to 574 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 574 (ObjNum 733 snap 242 seq_num 733) dirty exists 2024-09-20T19:32:21.342 INFO:tasks.rados.rados.0.smithi042.stdout:2083: left oid 36 (ObjNum 733 snap 242 seq_num 733) 2024-09-20T19:32:21.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:21.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:21.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:21 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.564 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:21.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.308949+0000 mgr.y (mgr.24491) 541 : cluster [DBG] pgmap v946: 168 pgs: 6 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 152 active+clean; 245 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-20T19:32:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.367062+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.372281+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.453882+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.455528+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.558266+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.560441+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.800495+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:32:21.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.802047+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:32:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:21 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.301172+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-20T19:32:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:21.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:21.827 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:21.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:21.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:21.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:22.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:22.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:22.308 INFO:tasks.rados.rados.0.smithi042.stdout:2083: done (1 left) 2024-09-20T19:32:22.309 INFO:tasks.rados.rados.0.smithi042.stdout:2084: done (0 left) 2024-09-20T19:32:22.309 INFO:tasks.rados.rados.0.smithi042.stdout:2085: delete oid 29 current snap is 243 2024-09-20T19:32:22.316 INFO:tasks.rados.rados.0.smithi042.stdout:2085: done (0 left) 2024-09-20T19:32:22.316 INFO:tasks.rados.rados.0.smithi042.stdout:2086: snap_remove snap 218 2024-09-20T19:32:22.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:32:22.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:22.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:22.476 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:22 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.605 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.897196+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:20.900880+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.520652+0000 osd.5 (osd.5) 193 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.522941+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.796760+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:21.799243+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:22 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.304447+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-20T19:32:22.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:22.734 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:22.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:22.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:22.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:22.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:22.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:23.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:23.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:23.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:23.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:23.272 INFO:tasks.rados.rados.0.smithi042.stdout:2086: done (0 left) 2024-09-20T19:32:23.272 INFO:tasks.rados.rados.0.smithi042.stdout:2087: rollback oid 49 current snap is 243 2024-09-20T19:32:23.272 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 49 to 238 2024-09-20T19:32:23.272 INFO:tasks.rados.rados.0.smithi042.stdout:2088: rollback oid 48 current snap is 243 2024-09-20T19:32:23.272 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 205 2024-09-20T19:32:23.273 INFO:tasks.rados.rados.0.smithi042.stdout:2089: write oid 5 current snap is 243 2024-09-20T19:32:23.273 INFO:tasks.rados.rados.0.smithi042.stdout:2089: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-20T19:32:23.282 INFO:tasks.rados.rados.0.smithi042.stdout:2089: writing smithi04268048-5 from 753145 to 1231048 tid 1 2024-09-20T19:32:23.289 INFO:tasks.rados.rados.0.smithi042.stdout:2089: writing smithi04268048-5 from 2010618 to 2353145 tid 2 2024-09-20T19:32:23.289 INFO:tasks.rados.rados.0.smithi042.stdout:2090: rollback oid 35 current snap is 243 2024-09-20T19:32:23.289 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 222 2024-09-20T19:32:23.290 INFO:tasks.rados.rados.0.smithi042.stdout:2091: write (excl) oid 1 current snap is 243 2024-09-20T19:32:23.290 INFO:tasks.rados.rados.0.smithi042.stdout:2091: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-20T19:32:23.297 INFO:tasks.rados.rados.0.smithi042.stdout:2091: writing smithi04268048-1 from 769952 to 1323104 tid 1 2024-09-20T19:32:23.304 INFO:tasks.rados.rados.0.smithi042.stdout:2091: writing smithi04268048-1 from 2069100 to 2369952 tid 2 2024-09-20T19:32:23.305 INFO:tasks.rados.rados.0.smithi042.stdout:2092: snap_remove snap 240 2024-09-20T19:32:23.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:23.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:23.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:23.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.503 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:23.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:23.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:23 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:23.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:23.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.309994+0000 mgr.y (mgr.24491) 542 : cluster [DBG] pgmap v949: 168 pgs: 7 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 151 active+clean; 245 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.782288+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:22.784534+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:23 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.271747+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-20T19:32:23.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:23.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:23.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:23.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:24.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:24.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:24.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:24.152 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:24.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:24.263 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:32:24.263 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:32:24.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:24.300 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T19:32:24.304 INFO:tasks.rados.rados.0.smithi042.stdout:2092: done (5 left) 2024-09-20T19:32:24.304 INFO:tasks.rados.rados.0.smithi042.stdout:2093: write oid 38 current snap is 243 2024-09-20T19:32:24.304 INFO:tasks.rados.rados.0.smithi042.stdout:2088: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:32:24.304 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 0 (ObjNum 33 snap 0 seq_num 0) dirty dne 2024-09-20T19:32:24.304 INFO:tasks.rados.rados.0.smithi042.stdout:2087: finishing rollback tid 1 to smithi04268048-49 2024-09-20T19:32:24.305 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1101 (ObjNum 225 snap 0 seq_num 1415333824) dirty dne 2024-09-20T19:32:24.305 INFO:tasks.rados.rados.0.smithi042.stdout:2089: finishing write tid 1 to smithi04268048-5 2024-09-20T19:32:24.305 INFO:tasks.rados.rados.0.smithi042.stdout:2093: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-20T19:32:24.314 INFO:tasks.rados.rados.0.smithi042.stdout:2093: writing smithi04268048-38 from 786759 to 1415160 tid 1 2024-09-20T19:32:24.321 INFO:tasks.rados.rados.0.smithi042.stdout:2093: writing smithi04268048-38 from 2011229 to 2386759 tid 2 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2087: done (5 left) 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2088: done (4 left) 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2094: delete oid 13 current snap is 243 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: finishing write tid 2 to smithi04268048-5 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: finishing write tid 3 to smithi04268048-5 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: finishing write tid 4 to smithi04268048-5 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: oid 5 updating version 0 to 777 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: oid 5 updating version 777 to 778 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: oid 5 version 778 is already newer than 776 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 778 (ObjNum 735 snap 243 seq_num 735) dirty exists 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2089: left oid 5 (ObjNum 735 snap 243 seq_num 735) 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2090: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:32:24.322 INFO:tasks.rados.rados.0.smithi042.stdout:2090: finishing rollback tid 2 to smithi04268048-35 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1045 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: finishing write tid 1 to smithi04268048-1 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: finishing write tid 2 to smithi04268048-1 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: finishing write tid 3 to smithi04268048-1 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: finishing write tid 4 to smithi04268048-1 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: oid 1 updating version 0 to 731 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: oid 1 version 731 is already newer than 729 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: oid 1 version 731 is already newer than 730 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 731 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2091: left oid 1 (ObjNum 736 snap 243 seq_num 736) 2024-09-20T19:32:24.323 INFO:tasks.rados.rados.0.smithi042.stdout:2093: finishing write tid 1 to smithi04268048-38 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2093: finishing write tid 2 to smithi04268048-38 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2089: done (4 left) 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2090: done (3 left) 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2091: done (2 left) 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2094: done (1 left) 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2095: write (excl) oid 13 current snap is 243 2024-09-20T19:32:24.326 INFO:tasks.rados.rados.0.smithi042.stdout:2095: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-20T19:32:24.332 INFO:tasks.rados.rados.0.smithi042.stdout:2095: writing smithi04268048-13 from 420373 to 915625 tid 1 2024-09-20T19:32:24.339 INFO:tasks.rados.rados.0.smithi042.stdout:2095: writing smithi04268048-13 from 1444546 to 2135122 tid 2 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2096: read oid 44 snap -1 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2096: expect deleted 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2097: rollback oid 31 current snap is 243 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 235 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: finishing write tid 3 to smithi04268048-38 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: finishing write tid 4 to smithi04268048-38 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: oid 38 updating version 0 to 705 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: oid 38 updating version 705 to 707 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: oid 38 version 707 is already newer than 706 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 707 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2093: left oid 38 (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:32:24.340 INFO:tasks.rados.rados.0.smithi042.stdout:2095: finishing write tid 1 to smithi04268048-13 2024-09-20T19:32:24.341 INFO:tasks.rados.rados.0.smithi042.stdout:2093: done (3 left) 2024-09-20T19:32:24.341 INFO:tasks.rados.rados.0.smithi042.stdout:2098: snap_create 2024-09-20T19:32:24.342 INFO:tasks.rados.rados.0.smithi042.stdout:2095: finishing write tid 2 to smithi04268048-13 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: finishing write tid 3 to smithi04268048-13 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: finishing write tid 4 to smithi04268048-13 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: oid 13 updating version 0 to 579 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: oid 13 version 579 is already newer than 578 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: oid 13 updating version 579 to 580 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 580 (ObjNum 739 snap 243 seq_num 739) dirty exists 2024-09-20T19:32:24.343 INFO:tasks.rados.rados.0.smithi042.stdout:2095: left oid 13 (ObjNum 739 snap 243 seq_num 739) 2024-09-20T19:32:24.347 INFO:tasks.rados.rados.0.smithi042.stdout:2097: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:32:24.348 INFO:tasks.rados.rados.0.smithi042.stdout:2097: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:32:24.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 733 (ObjNum 718 snap 234 seq_num 718) dirty exists 2024-09-20T19:32:24.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:24.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:24.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:24.539 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:24 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:24.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.322559+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.326711+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.368606+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:32:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.370003+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.503580+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.505463+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: audit 2024-09-20T19:32:23.704146+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.744264+0000 osd.7 (osd.7) 165 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:23.747043+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:32:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: audit 2024-09-20T19:32:24.300424+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? v1:172.21.15.42:0/1123837539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:32:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:24 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.304009+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-20T19:32:24.772 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:24.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:24.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:24.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:25.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:25.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:25.161 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:25.189 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:25.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:25.317 INFO:tasks.rados.rados.0.smithi042.stdout:2095: done (3 left) 2024-09-20T19:32:25.317 INFO:tasks.rados.rados.0.smithi042.stdout:2096: done (2 left) 2024-09-20T19:32:25.317 INFO:tasks.rados.rados.0.smithi042.stdout:2097: done (1 left) 2024-09-20T19:32:25.317 INFO:tasks.rados.rados.0.smithi042.stdout:2098: done (0 left) 2024-09-20T19:32:25.317 INFO:tasks.rados.rados.0.smithi042.stdout:2099: snap_remove snap 221 2024-09-20T19:32:25.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:25.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:25.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[32075]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:25 smithi042 bash[22706]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.451 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:25.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.310617+0000 mgr.y (mgr.24491) 543 : cluster [DBG] pgmap v952: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 244 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 992 KiB/s wr, 6 op/s 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.497284+0000 osd.5 (osd.5) 197 : cluster [DBG] 6.d scrub starts 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:24.498938+0000 osd.5 (osd.5) 198 : cluster [DBG] 6.d scrub ok 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:25 smithi162 bash[24557]: cluster 2024-09-20T19:32:25.316040+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-20T19:32:25.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:25.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:25.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:25.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:25.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:25.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:25.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:25.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:26.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:26.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:26.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:26.229 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:26.321 INFO:tasks.rados.rados.0.smithi042.stdout:2099: done (0 left) 2024-09-20T19:32:26.321 INFO:tasks.rados.rados.0.smithi042.stdout:2100: read oid 25 snap 205 2024-09-20T19:32:26.322 INFO:tasks.rados.rados.0.smithi042.stdout:2100: expect (ObjNum 628 snap 197 seq_num 628) 2024-09-20T19:32:26.322 INFO:tasks.rados.rados.0.smithi042.stdout:2101: read oid 48 snap -1 2024-09-20T19:32:26.322 INFO:tasks.rados.rados.0.smithi042.stdout:2101: expect deleted 2024-09-20T19:32:26.322 INFO:tasks.rados.rados.0.smithi042.stdout:2102: snap_remove snap 236 2024-09-20T19:32:26.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:26.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:26.486 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:26.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:26.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:26.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:26.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:26.747 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:26.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:26.875 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:26.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:26.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:27.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:27.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:27.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:27.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:27.248 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T19:32:27.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:32:27.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:27.279 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T19:32:27.330 INFO:tasks.rados.rados.0.smithi042.stdout:2102: done (2 left) 2024-09-20T19:32:27.330 INFO:tasks.rados.rados.0.smithi042.stdout:2103: rollback oid 2 current snap is 244 2024-09-20T19:32:27.330 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 241 2024-09-20T19:32:27.330 INFO:tasks.rados.rados.0.smithi042.stdout:2104: snap_create 2024-09-20T19:32:27.330 INFO:tasks.rados.rados.0.smithi042.stdout:2100: expect (ObjNum 628 snap 197 seq_num 628) 2024-09-20T19:32:27.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:27.387 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:27.398 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:27.438 INFO:tasks.rados.rados.0.smithi042.stdout:2103: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:32:27.438 INFO:tasks.rados.rados.0.smithi042.stdout:2103: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:32:27.438 INFO:tasks.rados.rados.0.smithi042.stdout:2103: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:32:27.439 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 616 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:32:27.467 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:32:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:32:27.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:27.491 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:32:27.491 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[32075]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:27 smithi042 bash[22706]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.509 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:32:27.509 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:32:27.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:27.527 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:32:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:32:27.552 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:32:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.291430+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.295653+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.311321+0000 mgr.y (mgr.24491) 544 : cluster [DBG] pgmap v954: 168 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 154 active+clean; 241 MiB data, 857 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.320063+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.435036+0000 osd.0 (osd.0) 37 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:27 smithi162 bash[24557]: cluster 2024-09-20T19:32:26.439086+0000 osd.0 (osd.0) 38 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:32:27.573 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:32:27.573 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:32:27.593 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:32:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:32:27.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:27.615 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:32:27.615 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:32:27.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:27.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:27.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:27.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:27.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:28.003 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:28.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:28.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2100: done (3 left) 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2101: done (2 left) 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2103: done (1 left) 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2104: done (0 left) 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2105: write oid 49 current snap is 245 2024-09-20T19:32:28.281 INFO:tasks.rados.rados.0.smithi042.stdout:2105: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-20T19:32:28.287 INFO:tasks.rados.rados.0.smithi042.stdout:2105: writing smithi04268048-49 from 437180 to 1007681 tid 1 2024-09-20T19:32:28.294 INFO:tasks.rados.rados.0.smithi042.stdout:2105: writing smithi04268048-49 from 1503028 to 2053615 tid 2 2024-09-20T19:32:28.294 INFO:tasks.rados.rados.0.smithi042.stdout:2106: write (excl) oid 34 current snap is 245 2024-09-20T19:32:28.294 INFO:tasks.rados.rados.0.smithi042.stdout:2106: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-20T19:32:28.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:28.300 INFO:tasks.rados.rados.0.smithi042.stdout:2106: writing smithi04268048-34 from 453987 to 1099737 tid 1 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2106: writing smithi04268048-34 from 1845157 to 2453987 tid 2 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2107: write oid 47 current snap is 245 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2105: finishing write tid 1 to smithi04268048-49 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2105: finishing write tid 2 to smithi04268048-49 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2105: finishing write tid 3 to smithi04268048-49 2024-09-20T19:32:28.310 INFO:tasks.rados.rados.0.smithi042.stdout:2107: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-20T19:32:28.315 INFO:tasks.rados.rados.0.smithi042.stdout:2107: writing smithi04268048-47 from 470794 to 1191793 tid 1 2024-09-20T19:32:28.323 INFO:tasks.rados.rados.0.smithi042.stdout:2107: writing smithi04268048-47 from 1903639 to 2457895 tid 2 2024-09-20T19:32:28.323 INFO:tasks.rados.rados.0.smithi042.stdout:2108: snap_remove snap 239 2024-09-20T19:32:28.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:28.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:28.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:28.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:28.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:28 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.679 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:28.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.321956+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.446301+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? v1:172.21.15.42:0/1167293030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.827181+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:27.828534+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.919765+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: audit 2024-09-20T19:32:27.921126+0000 mon.a (mon.0) 2058 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:28.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:28 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.277026+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-20T19:32:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:28.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:28.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:28.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:29.040 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:29.170 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:29.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:29.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2108: done (3 left) 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2109: read oid 5 snap -1 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2109: expect (ObjNum 735 snap 243 seq_num 735) 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2110: snap_create 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2105: finishing write tid 4 to smithi04268048-49 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2105: oid 49 updating version 0 to 1135 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2105: oid 49 version 1135 is already newer than 1134 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2105: oid 49 updating version 1135 to 1136 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1136 (ObjNum 740 snap 245 seq_num 740) dirty exists 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2105: left oid 49 (ObjNum 740 snap 245 seq_num 740) 2024-09-20T19:32:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2106: finishing write tid 1 to smithi04268048-34 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: finishing write tid 2 to smithi04268048-34 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: finishing write tid 3 to smithi04268048-34 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: finishing write tid 4 to smithi04268048-34 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: oid 34 updating version 0 to 1046 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: oid 34 updating version 1046 to 1048 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: oid 34 version 1048 is already newer than 1047 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1048 (ObjNum 741 snap 245 seq_num 741) dirty exists 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2106: left oid 34 (ObjNum 741 snap 245 seq_num 741) 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: finishing write tid 1 to smithi04268048-47 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: finishing write tid 2 to smithi04268048-47 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: finishing write tid 3 to smithi04268048-47 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: finishing write tid 4 to smithi04268048-47 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: oid 47 updating version 0 to 897 2024-09-20T19:32:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2107: oid 47 version 897 is already newer than 895 2024-09-20T19:32:29.334 INFO:tasks.rados.rados.0.smithi042.stdout:2107: oid 47 version 897 is already newer than 896 2024-09-20T19:32:29.334 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 897 (ObjNum 742 snap 245 seq_num 742) dirty exists 2024-09-20T19:32:29.334 INFO:tasks.rados.rados.0.smithi042.stdout:2107: left oid 47 (ObjNum 742 snap 245 seq_num 742) 2024-09-20T19:32:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:29.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.336 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[22706]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.350 INFO:tasks.rados.rados.0.smithi042.stdout:2109: expect (ObjNum 735 snap 243 seq_num 735) 2024-09-20T19:32:29.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:29.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:29.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:29 smithi042 bash[32075]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.311954+0000 mgr.y (mgr.24491) 545 : cluster [DBG] pgmap v958: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.434733+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[24557]: cluster 2024-09-20T19:32:28.436474+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:32:29.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:29.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:29.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:29.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:29.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:29.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:29.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:30.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:30.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:30.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:30.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:30.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:30.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:30.383 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2105: done (4 left) 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2106: done (3 left) 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2107: done (2 left) 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2109: done (1 left) 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2110: done (0 left) 2024-09-20T19:32:30.387 INFO:tasks.rados.rados.0.smithi042.stdout:2111: copy_from oid 43 from oid 16 current snap is 246 2024-09-20T19:32:30.388 INFO:tasks.rados.rados.0.smithi042.stdout:2112: write (excl) oid 45 current snap is 246 2024-09-20T19:32:30.388 INFO:tasks.rados.rados.0.smithi042.stdout:2112: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-20T19:32:30.394 INFO:tasks.rados.rados.0.smithi042.stdout:2112: writing smithi04268048-45 from 487601 to 1283849 tid 1 2024-09-20T19:32:30.401 INFO:tasks.rados.rados.0.smithi042.stdout:2112: writing smithi04268048-45 from 1962121 to 2487601 tid 2 2024-09-20T19:32:30.402 INFO:tasks.rados.rados.0.smithi042.stdout:2113: snap_create 2024-09-20T19:32:30.405 INFO:tasks.rados.rados.0.smithi042.stdout:2112: finishing write tid 1 to smithi04268048-45 2024-09-20T19:32:30.409 INFO:tasks.rados.rados.0.smithi042.stdout:2112: finishing write tid 2 to smithi04268048-45 2024-09-20T19:32:30.409 INFO:tasks.rados.rados.0.smithi042.stdout:2112: finishing write tid 3 to smithi04268048-45 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:2112: finishing write tid 4 to smithi04268048-45 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:2112: oid 45 updating version 0 to 1140 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:2112: oid 45 version 1140 is already newer than 1138 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:2112: oid 45 version 1140 is already newer than 1139 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1140 (ObjNum 743 snap 246 seq_num 743) dirty exists 2024-09-20T19:32:30.410 INFO:tasks.rados.rados.0.smithi042.stdout:2112: left oid 45 (ObjNum 743 snap 246 seq_num 743) 2024-09-20T19:32:30.411 INFO:tasks.rados.rados.0.smithi042.stdout:2111: finishing copy_from racing read to smithi04268048-43 2024-09-20T19:32:30.416 INFO:tasks.rados.rados.0.smithi042.stdout:2111: finishing copy_from to smithi04268048-43 2024-09-20T19:32:30.416 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 728 (ObjNum 661 snap 212 seq_num 661) dirty exists 2024-09-20T19:32:30.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:30.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:30.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:30.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:30.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:30.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:30.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:30 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:30.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:30.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:32:30.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:30.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:30.747 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:30.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:30.881 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:30.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:30.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:31.009 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.025657+0000 osd.1 (osd.1) 219 : cluster [DBG] 7.b scrub starts 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.036097+0000 osd.1 (osd.1) 220 : cluster [DBG] 7.b scrub ok 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.221559+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.224160+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.330270+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.431300+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:31.011 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.433991+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: audit 2024-09-20T19:32:29.599486+0000 mgr.y (mgr.24491) 546 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.862358+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:31.012 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:30 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.863939+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:32:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:31.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:31.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2111: done (2 left) 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2112: done (1 left) 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2113: done (0 left) 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2114: read oid 2 snap -1 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2114: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2115: copy_from oid 41 from oid 8 current snap is 247 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2116: read oid 31 snap -1 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2116: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-20T19:32:31.367 INFO:tasks.rados.rados.0.smithi042.stdout:2117: write (excl) oid 13 current snap is 247 2024-09-20T19:32:31.368 INFO:tasks.rados.rados.0.smithi042.stdout:2117: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-20T19:32:31.372 INFO:tasks.rados.rados.0.smithi042.stdout:2117: writing smithi04268048-13 from 504408 to 975905 tid 1 2024-09-20T19:32:31.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:31.383 INFO:tasks.rados.rados.0.smithi042.stdout:2117: writing smithi04268048-13 from 1620603 to 2178528 tid 2 2024-09-20T19:32:31.383 INFO:tasks.rados.rados.0.smithi042.stdout:2118: snap_remove snap 223 2024-09-20T19:32:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:31.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:31.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:31.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:31.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:31 smithi162 bash[24557]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:31.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[32075]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.850279+0000 osd.2 (osd.2) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:29.851892+0000 osd.2 (osd.2) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.312736+0000 mgr.y (mgr.24491) 547 : cluster [DBG] pgmap v960: 168 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 148 active+clean; 243 MiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 9 op/s 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.386796+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.455960+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.464002+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.833129+0000 osd.7 (osd.7) 171 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:31 smithi042 bash[22706]: cluster 2024-09-20T19:32:30.835540+0000 osd.7 (osd.7) 172 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:32:31.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.788 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:31.891 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:31.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:32.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:32.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:32.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:32.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:32.367 INFO:tasks.rados.rados.0.smithi042.stdout:2118: done (4 left) 2024-09-20T19:32:32.368 INFO:tasks.rados.rados.0.smithi042.stdout:2119: copy_from oid 45 from oid 39 current snap is 247 2024-09-20T19:32:32.368 INFO:tasks.rados.rados.0.smithi042.stdout:2116: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-20T19:32:32.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2115: finishing copy_from to smithi04268048-41 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2115: got expected ENOENT (src dne) 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2115: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2117: finishing write tid 1 to smithi04268048-13 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2117: finishing write tid 2 to smithi04268048-13 2024-09-20T19:32:32.431 INFO:tasks.rados.rados.0.smithi042.stdout:2114: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:32:32.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:32.539 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.540 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:32 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.556 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1737067 2024-09-20T19:32:32.576 INFO:tasks.rados.rados.0.smithi042.stdout:2117: finishing write tid 3 to smithi04268048-13 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: finishing write tid 4 to smithi04268048-13 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: oid 13 updating version 0 to 582 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: oid 13 updating version 582 to 583 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: oid 13 version 583 is already newer than 581 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 583 (ObjNum 744 snap 247 seq_num 744) dirty exists 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: left oid 13 (ObjNum 744 snap 247 seq_num 744) 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2119: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2119: finishing copy_from to smithi04268048-45 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 702 (ObjNum 722 snap 236 seq_num 722) dirty exists 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2114: done (4 left) 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2115: done (3 left) 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2116: done (2 left) 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2117: done (1 left) 2024-09-20T19:32:32.577 INFO:tasks.rados.rados.0.smithi042.stdout:2119: done (0 left) 2024-09-20T19:32:32.578 INFO:tasks.rados.rados.0.smithi042.stdout:2120: read oid 23 snap 229 2024-09-20T19:32:32.578 INFO:tasks.rados.rados.0.smithi042.stdout:2120: expect deleted 2024-09-20T19:32:32.578 INFO:tasks.rados.rados.0.smithi042.stdout:2121: write (excl) oid 5 current snap is 247 2024-09-20T19:32:32.578 INFO:tasks.rados.rados.0.smithi042.stdout:2121: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-20T19:32:32.594 INFO:tasks.rados.rados.0.smithi042.stdout:2121: writing smithi04268048-5 from 521215 to 1067961 tid 1 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2121: writing smithi04268048-5 from 1562732 to 2264315 tid 2 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2122: read oid 41 snap -1 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2122: expect (ObjNum 730 snap 239 seq_num 730) 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2123: copy_from oid 21 from oid 37 current snap is 247 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2121: finishing write tid 1 to smithi04268048-5 2024-09-20T19:32:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:2120: done (3 left) 2024-09-20T19:32:32.615 INFO:tasks.rados.rados.0.smithi042.stdout:2124: snap_remove snap 234 2024-09-20T19:32:32.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:32.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.076723+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.080091+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.358445+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.397472+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.401247+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.869472+0000 osd.7 (osd.7) 173 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:32 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.871615+0000 osd.7 (osd.7) 174 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:32:32.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:32.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:32.810 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:32.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:32.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:32.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:33.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:33.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2124: done (3 left) 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2125: rollback oid 28 current snap is 247 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 238 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2126: copy_from oid 25 from oid 14 current snap is 247 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2127: read oid 45 snap -1 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2127: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2128: write (excl) oid 1 current snap is 247 2024-09-20T19:32:33.286 INFO:tasks.rados.rados.0.smithi042.stdout:2128: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-20T19:32:33.287 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:32:33.293 INFO:tasks.rados.rados.0.smithi042.stdout:2128: writing smithi04268048-1 from 538022 to 1276370 tid 1 2024-09-20T19:32:33.301 INFO:tasks.rados.rados.0.smithi042.stdout:2128: writing smithi04268048-1 from 1737567 to 2299161 tid 2 2024-09-20T19:32:33.303 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: finishing write tid 2 to smithi04268048-5 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: finishing write tid 3 to smithi04268048-5 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: finishing write tid 4 to smithi04268048-5 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: oid 5 updating version 0 to 780 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: oid 5 updating version 780 to 781 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: oid 5 version 781 is already newer than 779 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 781 (ObjNum 745 snap 247 seq_num 745) dirty exists 2024-09-20T19:32:33.307 INFO:tasks.rados.rados.0.smithi042.stdout:2121: left oid 5 (ObjNum 745 snap 247 seq_num 745) 2024-09-20T19:32:33.308 INFO:tasks.rados.rados.0.smithi042.stdout:2122: expect (ObjNum 730 snap 239 seq_num 730) 2024-09-20T19:32:33.312 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1379414 2024-09-20T19:32:33.327 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:33.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:33.359 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:33.391 INFO:tasks.rados.rados.0.smithi042.stdout:2123: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:32:33.392 INFO:tasks.rados.rados.0.smithi042.stdout:2123: finishing copy_from to smithi04268048-21 2024-09-20T19:32:33.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 757 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:32:33.392 INFO:tasks.rados.rados.0.smithi042.stdout:2127: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:33.412 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2008813 2024-09-20T19:32:33.413 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 379468 2024-09-20T19:32:33.436 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:33.454 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:2125: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: finishing write tid 1 to smithi04268048-1 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: finishing write tid 2 to smithi04268048-1 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: finishing write tid 3 to smithi04268048-1 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: finishing write tid 4 to smithi04268048-1 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: oid 1 updating version 0 to 741 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: oid 1 updating version 741 to 742 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: oid 1 updating version 742 to 743 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 743 (ObjNum 746 snap 247 seq_num 746) dirty exists 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2128: left oid 1 (ObjNum 746 snap 247 seq_num 746) 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2126: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2125: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:2126: finishing copy_from to smithi04268048-25 2024-09-20T19:32:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 767 (ObjNum 670 snap 216 seq_num 670) dirty exists 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2125: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 729 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2121: done (6 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2122: done (5 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2123: done (4 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2125: done (3 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2126: done (2 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2127: done (1 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2128: done (0 left) 2024-09-20T19:32:33.478 INFO:tasks.rados.rados.0.smithi042.stdout:2129: delete oid 4 current snap is 247 2024-09-20T19:32:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:2129: done (0 left) 2024-09-20T19:32:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:2130: rollback oid 6 current snap is 247 2024-09-20T19:32:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 243 2024-09-20T19:32:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:2131: delete oid 41 current snap is 247 2024-09-20T19:32:33.484 INFO:tasks.rados.rados.0.smithi042.stdout:2131: done (1 left) 2024-09-20T19:32:33.484 INFO:tasks.rados.rados.0.smithi042.stdout:2132: snap_remove snap 238 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:33 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.705431+0000 osd.2 (osd.2) 203 : cluster [DBG] 7.e scrub starts 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.708009+0000 osd.2 (osd.2) 204 : cluster [DBG] 7.e scrub ok 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.813100+0000 osd.1 (osd.1) 223 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:31.814115+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.187211+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.189115+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.313442+0000 mgr.y (mgr.24491) 548 : cluster [DBG] pgmap v963: 168 pgs: 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 148 active+clean; 249 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.367127+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:32.399936+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.42:0/1856302020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:32.400288+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25595 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.470847+0000 osd.0 (osd.0) 43 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:32.473738+0000 osd.0 (osd.0) 44 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.269127+0000 mon.a (mon.0) 2065 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: audit 2024-09-20T19:32:33.271508+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:32:33.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:33 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.280114+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-20T19:32:33.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:33.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:33.815 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:33.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:33.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:34.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:34.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:34.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:34.383 INFO:tasks.rados.rados.0.smithi042.stdout:2132: done (1 left) 2024-09-20T19:32:34.383 INFO:tasks.rados.rados.0.smithi042.stdout:2133: copy_from oid 12 from oid 17 current snap is 247 2024-09-20T19:32:34.383 INFO:tasks.rados.rados.0.smithi042.stdout:2134: copy_from oid 5 from oid 38 current snap is 247 2024-09-20T19:32:34.383 INFO:tasks.rados.rados.0.smithi042.stdout:2135: snap_remove snap 244 2024-09-20T19:32:34.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:34.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:34.591 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:34.602 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[32075]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:34 smithi042 bash[22706]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:34.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:34 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:34 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.195831+0000 osd.4 (osd.4) 43 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:32:34.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:34 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:34 smithi162 bash[24557]: cluster 2024-09-20T19:32:33.197996+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:32:34.748 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:34.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:34.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:34.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:34.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:35.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:35.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:35.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:35.263 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:35.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2135: done (3 left) 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2136: delete oid 11 current snap is 247 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2130: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2130: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2130: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 747 (ObjNum 660 snap 212 seq_num 660) dirty exists 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2134: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2134: finishing copy_from to smithi04268048-5 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 707 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:32:35.388 INFO:tasks.rados.rados.0.smithi042.stdout:2133: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:32:35.389 INFO:tasks.rados.rados.0.smithi042.stdout:2133: finishing copy_from to smithi04268048-12 2024-09-20T19:32:35.389 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 721 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-20T19:32:35.394 INFO:tasks.rados.rados.0.smithi042.stdout:2130: done (3 left) 2024-09-20T19:32:35.394 INFO:tasks.rados.rados.0.smithi042.stdout:2133: done (2 left) 2024-09-20T19:32:35.394 INFO:tasks.rados.rados.0.smithi042.stdout:2134: done (1 left) 2024-09-20T19:32:35.394 INFO:tasks.rados.rados.0.smithi042.stdout:2136: done (0 left) 2024-09-20T19:32:35.394 INFO:tasks.rados.rados.0.smithi042.stdout:2137: snap_create 2024-09-20T19:32:35.396 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:35.499 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:35.537 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[32075]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[32075]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[32075]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[32075]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[22706]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[22706]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[22706]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.538 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:35 smithi042 bash[22706]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.630 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:35.657 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:35.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:35.787 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:35.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:35 smithi162 bash[24557]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:35 smithi162 bash[24557]: cluster 2024-09-20T19:32:34.314122+0000 mgr.y (mgr.24491) 549 : cluster [DBG] pgmap v966: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:32:35.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:35 smithi162 bash[24557]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:35 smithi162 bash[24557]: cluster 2024-09-20T19:32:34.381758+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-20T19:32:35.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:35.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:35.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:36.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:36.116 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2304 4.b 4:d201668e:::obj_delete_at_hint.0000000023:head [call lock.lock in=8b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.37045115399999, 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.05466185, 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:32:36.117 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2304 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743426+0000", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0299999999999997e-07 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743430+0000", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8550000000000001e-06 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743439+0000", 2024-09-20T19:32:36.119 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3930000000000008e-06 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743454+0000", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4882000000000001e-05 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743510+0000", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6486000000000001e-05 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743518+0000", 2024-09-20T19:32:36.120 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4100000000000002e-06 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.763011+0000", 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019493656000000002 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.763015+0000", 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6650000000000001e-06 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.121 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793210+0000", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0301954090000001 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793332+0000", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012176999999999999 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793851+0000", 2024-09-20T19:32:36.122 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00051850500000000005 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795617+0000", 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00176593 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797863+0000", 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002246255 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.123 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797953+0000", 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9913999999999993e-05 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798087+0000", 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000133817 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2381 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:32:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.370408333, 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0519627060000001, 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2381 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.125 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0100000000000004e-07 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743479+0000", 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0648999999999999e-05 2024-09-20T19:32:36.126 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743488+0000", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6479999999999992e-06 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743494+0000", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.798e-06 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743541+0000", 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6360000000000003e-05 2024-09-20T19:32:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743548+0000", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7710000000000001e-06 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762973+0000", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019425462000000001 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762984+0000", 2024-09-20T19:32:36.128 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1403000000000001e-05 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795280+0000", 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0322955819999999 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795430+0000", 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015023200000000001 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2544 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94817798635392 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.370371691, 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.050798683, 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2544 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.130 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743505+0000", 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1200000000000003e-07 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743508+0000", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2389999999999998e-06 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743515+0000", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.5860000000000001e-06 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743521+0000", 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.827e-06 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743576+0000", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4317000000000002e-05 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743583+0000", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4059999999999999e-06 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762986+0000", 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019403238 2024-09-20T19:32:36.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762995+0000", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8000000000000004e-06 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.789018+0000", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.026022295 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.789172+0000", 2024-09-20T19:32:36.134 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000154189 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.794303+0000", 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0051311739999999996 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3928 7.4 7:24603624:::smithi04268048-48:head [write 591745~402925,stat] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.36924355100001, 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0551826959999999, 2024-09-20T19:32:36.135 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3928 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.136 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744634+0000", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9250000000000002e-06 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744981+0000", 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000346468 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.137 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744993+0000", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1878e-05 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745001+0000", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8340000000000005e-06 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746485+0000", 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001483617 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.138 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746497+0000", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2424e-05 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769303+0000", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0228058929999999 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769761+0000", 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00045766200000000001 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.139 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791851+0000", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.022090143 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791978+0000", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012683 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792888+0000", 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00091 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.140 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792995+0000", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107441 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793430+0000", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00043459800000000002 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798331+0000", 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0049014260000000004 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.141 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799616+0000", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012846260000000001 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799686+0000", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9567000000000007e-05 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799815+0000", 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000129364 2024-09-20T19:32:36.142 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3929 7.4 7:24603624:::smithi04268048-48:head [write 1790337~563515] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.36886137299999, 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0564495810000001, 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.143 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3929 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.144 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1500000000000003e-07 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745320+0000", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00030476499999999997 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745330+0000", 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5389999999999996e-06 2024-09-20T19:32:36.145 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745338+0000", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3319999999999992e-06 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746527+0000", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011885789999999999 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746535+0000", 2024-09-20T19:32:36.146 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2160000000000006e-06 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769872+0000", 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0233368110000001 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770035+0000", 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016329300000000001 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.147 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792008+0000", 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021972551999999999 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792074+0000", 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.6872000000000006e-05 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795026+0000", 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0029514490000000001 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.148 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795812+0000", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000786212 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796112+0000", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00030015199999999998 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799835+0000", 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0037226220000000001 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.149 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801207+0000", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013722389999999999 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801282+0000", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4697999999999995e-05 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801464+0000", 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018233499999999999 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.150 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3930 7.4 7:24603624:::smithi04268048-48:head [setxattr _header (58) in=7b,truncate 2991745] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.368528027, 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.05644351, 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3930 2024-09-20T19:32:36.151 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745349+0000", 2024-09-20T19:32:36.152 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0999999999999997e-07 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745354+0000", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8039999999999998e-06 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745360+0000", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8170000000000004e-06 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745366+0000", 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3299999999999998e-06 2024-09-20T19:32:36.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746559+0000", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001192993 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746566+0000", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.1459999999999999e-06 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770610+0000", 2024-09-20T19:32:36.154 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0240446439999999 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770771+0000", 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016099199999999999 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792101+0000", 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021329839999999999 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.155 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792166+0000", 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4751000000000004e-05 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797311+0000", 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0051449699999999996 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797491+0000", 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018041800000000001 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797775+0000", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028395999999999998 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799852+0000", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002076788 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801527+0000", 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016751229999999999 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.157 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801613+0000", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6012000000000003e-05 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801791+0000", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017811199999999999 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3931 7.4 7:24603624:::smithi04268048-48:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.36850412800001, 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0582288870000001, 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3931 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6400000000000003e-07 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745374+0000", 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.232e-06 2024-09-20T19:32:36.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745382+0000", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.807e-06 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745396+0000", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3362e-05 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746595+0000", 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011997240000000001 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.161 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746603+0000", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0139999999999998e-06 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770850+0000", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0242465510000001 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.771097+0000", 2024-09-20T19:32:36.162 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024681600000000002 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792188+0000", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021091675000000001 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792323+0000", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013485099999999999 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798407+0000", 2024-09-20T19:32:36.163 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060842170000000003 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798643+0000", 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023551600000000001 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801873+0000", 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0032300290000000001 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.164 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.802054+0000", 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000180927 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.803601+0000", 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0015467020000000001 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4004 7.9 7:94ee8d8f:::smithi04268048-13:head [stat,delete] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 147.368473096, 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.056561592, 2024-09-20T19:32:36.165 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4004 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5700000000000002e-07 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745406+0000", 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5830000000000001e-06 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.167 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745412+0000", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7049999999999996e-06 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745416+0000", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7190000000000001e-06 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.747326+0000", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001909656 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.168 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.747332+0000", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1090000000000003e-06 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791891+0000", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0445585660000001 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792090+0000", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019927399999999999 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796389+0000", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0042991349999999999 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796569+0000", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000179848 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797353+0000", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00078369699999999997 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797493+0000", 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000140536 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801964+0000", 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044712069999999996 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "MOSDPGPush(5.e 501/500 [PushOp(5:7759931f:::notify.3:head, version: 77'11, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:7759931f:::notify.3:head@77'11, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T19:32:36.171 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:09.773150+0000", 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 146.34072575900001, 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.022187690999999999, 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773150+0000", 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.172 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773150+0000", 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773152+0000", 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.514e-06 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773173+0000", 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1293000000000001e-05 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773197+0000", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3659000000000002e-05 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.773216+0000", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9332000000000001e-05 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795148+0000", 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021932369 2024-09-20T19:32:36.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795163+0000", 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4215e-05 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795338+0000", 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017530900000000001 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.175 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14652.0:21 3.1 3:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e518)", 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 125.81154666499999, 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0601109449999999, 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14652", 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/577469704", 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 21 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.176 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302332+0000", 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6019999999999998e-06 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302342+0000", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7159999999999998e-06 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302360+0000", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8530999999999999e-05 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302375+0000", 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4506e-05 2024-09-20T19:32:36.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.305035+0000", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002660875 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.305076+0000", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0312999999999998e-05 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.362285+0000", 2024-09-20T19:32:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.057209316 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.362440+0000", 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000155086 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4234 7.13 7:cf582c87:::smithi04268048-12:head [copy-from ver 626] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 124.754254567, 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.519326363, 2024-09-20T19:32:36.180 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4234 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359624+0000", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7130000000000001e-06 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359632+0000", 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9400000000000002e-06 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359648+0000", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5883e-05 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359656+0000", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5210000000000002e-06 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359747+0000", 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0464999999999995e-05 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359899+0000", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015204800000000001 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,7", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.868922+0000", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.509023121 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876934+0000", 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0080117520000000005 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.184 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876976+0000", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2599999999999999e-05 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878852+0000", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0018758570000000001 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878890+0000", 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7568999999999999e-05 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878948+0000", 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7893999999999998e-05 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4235 7.13 7:cf582c87:::smithi04268048-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 124.75420281700001, 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.517188041, 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.186 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4235 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.187 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359674+0000", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4099999999999998e-07 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359677+0000", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0400000000000001e-06 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.188 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359683+0000", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8030000000000002e-06 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359687+0000", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6829999999999999e-06 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360140+0000", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000452373 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360199+0000", 2024-09-20T19:32:36.189 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9703000000000001e-05 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876675+0000", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.516475835 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876753+0000", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8146999999999996e-05 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876861+0000", 2024-09-20T19:32:36.190 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107816 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(osd.0.500:1 7.16 7:6c48c6fe:::smithi04268048-7:head [assert-version v626,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 124.753429637, 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5039987930000001, 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360448+0000", 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4020000000000001e-06 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.192 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360453+0000", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1699999999999996e-06 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360463+0000", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0509000000000001e-05 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360470+0000", 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.109e-06 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.193 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360530+0000", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9468000000000001e-05 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360539+0000", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5869999999999999e-06 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.861867+0000", 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5013275930000001 2024-09-20T19:32:36.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.862028+0000", 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016082499999999999 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.864445+0000", 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00241713 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4887 7.f 7:f965d827:::smithi04268048-5:head [write 410269~713794,stat] snapc d8=[d8,d7,d6,d5,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:31:38.270030+0000", 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 57.843845942000002, 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021828606, 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4887 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.196 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.270030+0000", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.270030+0000", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.270038+0000", 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1859999999999992e-06 2024-09-20T19:32:36.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.270948+0000", 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00090967999999999995 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.270991+0000", 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2957000000000002e-05 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.198 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.271012+0000", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1630000000000001e-05 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.271162+0000", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000149333 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.271174+0000", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2500000000000001e-05 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.279268+0000", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0080940720000000008 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.279449+0000", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018052300000000001 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.279747+0000", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000297764 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.291392+0000", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011644985 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.291509+0000", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011738799999999999 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.291701+0000", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000192024 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.291780+0000", 2024-09-20T19:32:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9151000000000002e-05 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:38.291859+0000", 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8412999999999996e-05 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5407 7.4 7:23ed6a02:::smithi04268048-25:head [copy-from ver 767] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 2.8263847169999998, 2024-09-20T19:32:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030478080000000001, 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5407 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287494+0000", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5909999999999998e-06 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287502+0000", 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9279999999999993e-06 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287512+0000", 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0562e-05 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287518+0000", 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0800000000000002e-06 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288171+0000", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00065281200000000001 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288179+0000", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2949999999999994e-06 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295330+0000", 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0071509679999999997 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295627+0000", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029681800000000002 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.304712+0000", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0090846569999999995 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.314385+0000", 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.009672702 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.315214+0000", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00082897899999999998 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317811+0000", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0025975809999999998 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317893+0000", 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1588999999999994e-05 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317969+0000", 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6518000000000002e-05 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5408 7.4 7:23ed6a02:::smithi04268048-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.288103+0000", 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 2.8257731550000003, 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.023518363, 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5408 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288103+0000", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288103+0000", 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288104+0000", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0809999999999999e-06 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288109+0000", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8919999999999999e-06 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288138+0000", 2024-09-20T19:32:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9391999999999999e-05 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288153+0000", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4725999999999999e-05 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288255+0000", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010187300000000001 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288259+0000", 2024-09-20T19:32:36.212 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.724e-06 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.296043+0000", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0077837009999999996 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.296182+0000", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000138639 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.311373+0000", 2024-09-20T19:32:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.015191076 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.311490+0000", 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00011744900000000001 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.311621+0000", 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013081 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5422 7.13 7:cf582c87:::smithi04268048-12:head [copy-from ver 721] snapc f8=[f8,f7,f6,f5,f4,f3,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 1.730374399, 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037795502000000002, 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5422 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.215 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383506+0000", 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0629999999999999e-06 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383520+0000", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4568e-05 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383551+0000", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1041999999999999e-05 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383567+0000", 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5488000000000001e-05 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384047+0000", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000480113 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384060+0000", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3049e-05 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.388415+0000", 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.004354803 2024-09-20T19:32:36.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389079+0000", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00066449499999999997 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,7", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.411175+0000", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.022096352999999999 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.420015+0000", 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0088399080000000005 2024-09-20T19:32:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.420138+0000", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000122879 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421052+0000", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00091352000000000002 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421197+0000", 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014492500000000001 2024-09-20T19:32:36.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421297+0000", 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100296 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5423 7.13 7:cf582c87:::smithi04268048-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 1.7297573320000001, 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.035737366, 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.221 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5423 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.222 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384120+0000", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8220000000000001e-06 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384129+0000", 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8380000000000008e-06 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.223 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384155+0000", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5514999999999999e-05 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384472+0000", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031773400000000002 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384555+0000", 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3004999999999996e-05 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384563+0000", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.328e-06 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389341+0000", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0047780000000000001 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389385+0000", 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4147000000000003e-05 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419500+0000", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030114867 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419666+0000", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016647499999999999 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419856+0000", 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018963500000000001 2024-09-20T19:32:36.226 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5424 7.f 7:f965d827:::smithi04268048-5:head [copy-from ver 707] snapc f8=[f8,f7,f6,f5,f4,f3,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.384501+0000", 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 1.72937454, 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.026567382, 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.227 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5424 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384501+0000", 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384501+0000", 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.228 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384503+0000", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.685e-06 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384512+0000", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4420000000000004e-06 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384526+0000", 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3603e-05 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.229 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384537+0000", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1134e-05 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384608+0000", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0745000000000001e-05 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384616+0000", 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1210000000000007e-06 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.230 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.390219+0000", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0056028800000000002 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.390374+0000", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000155107 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 3,7", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.404197+0000", 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013822674 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.231 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.410205+0000", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060087140000000001 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.410519+0000", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000313443 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.410900+0000", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00038106199999999999 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:32:36.232 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.410990+0000", 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9942e-05 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.411069+0000", 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8830000000000002e-05 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5425 7.f 7:f965d827:::smithi04268048-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.384563+0000", 2024-09-20T19:32:36.233 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 1.729312827, 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.025566019999999998, 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5425 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384563+0000", 2024-09-20T19:32:36.234 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384563+0000", 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384564+0000", 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1179999999999999e-06 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:32:36.235 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384570+0000", 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5269999999999998e-06 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384580+0000", 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8640000000000008e-06 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384589+0000", 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8679999999999994e-06 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.236 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384644+0000", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4286000000000002e-05 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384649+0000", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5400000000000003e-06 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.390671+0000", 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.006021869 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.237 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.390748+0000", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6637999999999994e-05 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.409742+0000", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.018993952000000001 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.409904+0000", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016204199999999999 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:32:36.238 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.410129+0000", 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022531600000000001 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:32:36.239 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:32:36.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:36.259 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:36.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:36.372 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:36.388 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:36.395 INFO:tasks.rados.rados.0.smithi042.stdout:2137: done (0 left) 2024-09-20T19:32:36.395 INFO:tasks.rados.rados.0.smithi042.stdout:2138: rollback oid 6 current snap is 248 2024-09-20T19:32:36.395 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 237 2024-09-20T19:32:36.396 INFO:tasks.rados.rados.0.smithi042.stdout:2139: snap_create 2024-09-20T19:32:36.415 INFO:tasks.rados.rados.0.smithi042.stdout:2138: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:32:36.415 INFO:tasks.rados.rados.0.smithi042.stdout:2138: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:32:36.415 INFO:tasks.rados.rados.0.smithi042.stdout:2138: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:32:36.415 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 751 (ObjNum 660 snap 212 seq_num 660) dirty exists 2024-09-20T19:32:36.433 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:32:36.436 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:32:36.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:36.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:36 smithi042 bash[32075]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:36 smithi042 bash[32075]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:36 smithi042 bash[22706]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.500 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:36 smithi042 bash[22706]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:36.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:36.641 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:36.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:36 smithi162 bash[24557]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:36 smithi162 bash[24557]: cluster 2024-09-20T19:32:35.386826+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-20T19:32:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:36.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:36.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:36.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:36.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:37.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:37.034 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:37.136 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:37.164 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:37.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:37.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:37.410 INFO:tasks.rados.rados.0.smithi042.stdout:2138: done (1 left) 2024-09-20T19:32:37.411 INFO:tasks.rados.rados.0.smithi042.stdout:2139: done (0 left) 2024-09-20T19:32:37.411 INFO:tasks.rados.rados.0.smithi042.stdout:2140: snap_remove snap 216 2024-09-20T19:32:37.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:37.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[32075]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:37 smithi042 bash[22706]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:37.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:37.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:37.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:32:37.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: cluster 2024-09-20T19:32:36.314540+0000 mgr.y (mgr.24491) 550 : cluster [DBG] pgmap v969: 168 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 150 active+clean; 254 MiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-20T19:32:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: cluster 2024-09-20T19:32:36.395146+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-20T19:32:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:37 smithi162 bash[24557]: audit 2024-09-20T19:32:36.420044+0000 mgr.y (mgr.24491) 551 : audit [DBG] from='client.15753 v1:172.21.15.42:0/192192670' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:37.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:32:37.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:37.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":970,"stamp":"2024-09-20T19:32:36.390630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266788832,"num_objects":326,"num_object_clones":80,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":9,"num_read":3652,"num_read_kb":1274922,"num_write":1968,"num_write_kb":461317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":88530899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15725,"ondisk_log_size":15725,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":903256,"kb_used_data":586960,"kb_used_omap":143,"kb_used_meta":315952,"kb_avail":749057960,"statfs":{"total":767960285184,"available":767035351040,"internally_reserved":0,"allocated":601047040,"data_stored":783495279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147312,"internal_metadata":323534992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5639283,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6243,"num_write":9,"num_write_kb":5263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027869"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":1018,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385016+0000","last_change":"2024-09-20T19:31:37.502437+0000","last_active":"2024-09-20T19:32:32.385016+0000","last_peered":"2024-09-20T19:32:32.385016+0000","last_clean":"2024-09-20T19:32:32.385016+0000","last_became_active":"2024-09-20T19:30:59.423955+0000","last_became_peered":"2024-09-20T19:30:59.423955+0000","last_unstale":"2024-09-20T19:32:32.385016+0000","last_undegraded":"2024-09-20T19:32:32.385016+0000","last_fullsized":"2024-09-20T19:32:32.385016+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:42.093962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":995,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.367692+0000","last_change":"2024-09-20T19:32:13.393790+0000","last_active":"2024-09-20T19:32:32.367692+0000","last_peered":"2024-09-20T19:32:32.367692+0000","last_clean":"2024-09-20T19:32:32.367692+0000","last_became_active":"2024-09-20T19:30:31.353214+0000","last_became_peered":"2024-09-20T19:30:31.353214+0000","last_unstale":"2024-09-20T19:32:32.367692+0000","last_undegraded":"2024-09-20T19:32:32.367692+0000","last_fullsized":"2024-09-20T19:32:32.367692+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.908799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1074,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385383+0000","last_change":"2024-09-20T19:30:59.495048+0000","last_active":"2024-09-20T19:32:32.385383+0000","last_peered":"2024-09-20T19:32:32.385383+0000","last_clean":"2024-09-20T19:32:32.385383+0000","last_became_active":"2024-09-20T19:30:59.432572+0000","last_became_peered":"2024-09-20T19:30:59.432572+0000","last_unstale":"2024-09-20T19:32:32.385383+0000","last_undegraded":"2024-09-20T19:32:32.385383+0000","last_fullsized":"2024-09-20T19:32:32.385383+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:25:18.620430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.620430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:55.533243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":967,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.284599+0000","last_change":"2024-09-20T19:30:44.918367+0000","last_active":"2024-09-20T19:32:33.284599+0000","last_peered":"2024-09-20T19:32:33.284599+0000","last_clean":"2024-09-20T19:32:33.284599+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:32:33.284599+0000","last_undegraded":"2024-09-20T19:32:33.284599+0000","last_fullsized":"2024-09-20T19:32:33.284599+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:44.918249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:44.918249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1072,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.388205+0000","last_change":"2024-09-20T19:32:12.874027+0000","last_active":"2024-09-20T19:32:32.388205+0000","last_peered":"2024-09-20T19:32:32.388205+0000","last_clean":"2024-09-20T19:32:32.388205+0000","last_became_active":"2024-09-20T19:30:59.439274+0000","last_became_peered":"2024-09-20T19:30:59.439274+0000","last_unstale":"2024-09-20T19:32:32.388205+0000","last_undegraded":"2024-09-20T19:32:32.388205+0000","last_fullsized":"2024-09-20T19:32:32.388205+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.712357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1071,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.284852+0000","last_change":"2024-09-20T19:30:49.903222+0000","last_active":"2024-09-20T19:32:33.284852+0000","last_peered":"2024-09-20T19:32:33.284852+0000","last_clean":"2024-09-20T19:32:33.284852+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:32:33.284852+0000","last_undegraded":"2024-09-20T19:32:33.284852+0000","last_fullsized":"2024-09-20T19:32:33.284852+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:49.903144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:30:49.903144+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361531+0000","last_change":"2024-09-20T19:29:00.755422+0000","last_active":"2024-09-20T19:32:31.361531+0000","last_peered":"2024-09-20T19:32:31.361531+0000","last_clean":"2024-09-20T19:32:31.361531+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:32:31.361531+0000","last_undegraded":"2024-09-20T19:32:31.361531+0000","last_fullsized":"2024-09-20T19:32:31.361531+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:29:00.755361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.755361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:03.153003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1029,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.285945+0000","last_change":"2024-09-20T19:31:20.821748+0000","last_active":"2024-09-20T19:32:33.285945+0000","last_peered":"2024-09-20T19:32:33.285945+0000","last_clean":"2024-09-20T19:32:33.285945+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:32:33.285945+0000","last_undegraded":"2024-09-20T19:32:33.285945+0000","last_fullsized":"2024-09-20T19:32:33.285945+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:20.821598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.821598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:36.498203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"632'1040","reported_seq":3532,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.473837+0000","last_change":"2024-09-20T19:32:32.473837+0000","last_active":"2024-09-20T19:32:32.473837+0000","last_peered":"2024-09-20T19:32:32.473837+0000","last_clean":"2024-09-20T19:32:32.473837+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:32:32.473837+0000","last_undegraded":"2024-09-20T19:32:32.473837+0000","last_fullsized":"2024-09-20T19:32:32.473837+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.284333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0026592299999999998,"stat_sum":{"num_bytes":6269242,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":17260,"num_write":25,"num_write_kb":6122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1209,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.382236+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:32:31.382236+0000","last_peered":"2024-09-20T19:32:31.382236+0000","last_clean":"2024-09-20T19:32:31.382236+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:32:31.382236+0000","last_undegraded":"2024-09-20T19:32:31.382236+0000","last_fullsized":"2024-09-20T19:32:31.382236+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.958849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1035,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.288197+0000","last_change":"2024-09-20T19:32:29.224300+0000","last_active":"2024-09-20T19:32:33.288197+0000","last_peered":"2024-09-20T19:32:33.288197+0000","last_clean":"2024-09-20T19:32:33.288197+0000","last_became_active":"2024-09-20T19:30:59.395790+0000","last_became_peered":"2024-09-20T19:30:59.395790+0000","last_unstale":"2024-09-20T19:32:33.288197+0000","last_undegraded":"2024-09-20T19:32:33.288197+0000","last_fullsized":"2024-09-20T19:32:33.288197+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:32:29.224169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.224169+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:53.488936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1193,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.378048+0000","last_change":"2024-09-20T19:30:31.352779+0000","last_active":"2024-09-20T19:32:31.378048+0000","last_peered":"2024-09-20T19:32:31.378048+0000","last_clean":"2024-09-20T19:32:31.378048+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:32:31.378048+0000","last_undegraded":"2024-09-20T19:32:31.378048+0000","last_fullsized":"2024-09-20T19:32:31.378048+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:11.324315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.324315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1205,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.397457+0000","last_change":"2024-09-20T19:32:28.436574+0000","last_active":"2024-09-20T19:32:31.397457+0000","last_peered":"2024-09-20T19:32:31.397457+0000","last_clean":"2024-09-20T19:32:31.397457+0000","last_became_active":"2024-09-20T19:30:59.422108+0000","last_became_peered":"2024-09-20T19:30:59.422108+0000","last_unstale":"2024-09-20T19:32:31.397457+0000","last_undegraded":"2024-09-20T19:32:31.397457+0000","last_fullsized":"2024-09-20T19:32:31.397457+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.196618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"641'347","reported_seq":2867,"reported_epoch":643,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:31.367712+0000","last_change":"2024-09-20T19:32:30.440338+0000","last_active":"2024-09-20T19:32:31.367712+0000","last_peered":"2024-09-20T19:32:31.367712+0000","last_clean":"2024-09-20T19:32:31.367712+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:32:31.367712+0000","last_undegraded":"2024-09-20T19:32:31.367712+0000","last_fullsized":"2024-09-20T19:32:31.367712+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'290","last_scrub_stamp":"2024-09-20T19:29:01.442771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:01.442771+0000","objects_scrubbed":4,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.0926730469999999,"stat_sum":{"num_bytes":6696501,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":11058,"num_write":15,"num_write_kb":3458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1198,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.381932+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:32:31.381932+0000","last_peered":"2024-09-20T19:32:31.381932+0000","last_clean":"2024-09-20T19:32:31.381932+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:32:31.381932+0000","last_undegraded":"2024-09-20T19:32:31.381932+0000","last_fullsized":"2024-09-20T19:32:31.381932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.698584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1081,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.285396+0000","last_change":"2024-09-20T19:32:15.452097+0000","last_active":"2024-09-20T19:32:33.285396+0000","last_peered":"2024-09-20T19:32:33.285396+0000","last_clean":"2024-09-20T19:32:33.285396+0000","last_became_active":"2024-09-20T19:30:59.403992+0000","last_became_peered":"2024-09-20T19:30:59.403992+0000","last_unstale":"2024-09-20T19:32:33.285396+0000","last_undegraded":"2024-09-20T19:32:33.285396+0000","last_fullsized":"2024-09-20T19:32:33.285396+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.451996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.451996+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:23.255186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1193,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.397895+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:32:31.397895+0000","last_peered":"2024-09-20T19:32:31.397895+0000","last_clean":"2024-09-20T19:32:31.397895+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:32:31.397895+0000","last_undegraded":"2024-09-20T19:32:31.397895+0000","last_fullsized":"2024-09-20T19:32:31.397895+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.435769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":171,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.364090+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:32:32.364090+0000","last_peered":"2024-09-20T19:32:32.364090+0000","last_clean":"2024-09-20T19:32:32.364090+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:32:32.364090+0000","last_undegraded":"2024-09-20T19:32:32.364090+0000","last_fullsized":"2024-09-20T19:32:32.364090+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.693370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":809,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.395932+0000","last_change":"2024-09-20T19:30:59.413612+0000","last_active":"2024-09-20T19:32:31.395932+0000","last_peered":"2024-09-20T19:32:31.395932+0000","last_clean":"2024-09-20T19:32:31.395932+0000","last_became_active":"2024-09-20T19:30:59.413239+0000","last_became_peered":"2024-09-20T19:30:59.413239+0000","last_unstale":"2024-09-20T19:32:31.395932+0000","last_undegraded":"2024-09-20T19:32:31.395932+0000","last_fullsized":"2024-09-20T19:32:31.395932+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:04.483789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:04.483789+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"639'200","reported_seq":2747,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.708119+0000","last_change":"2024-09-20T19:32:31.708119+0000","last_active":"2024-09-20T19:32:31.708119+0000","last_peered":"2024-09-20T19:32:31.708119+0000","last_clean":"2024-09-20T19:32:31.708119+0000","last_became_active":"2024-09-20T19:30:09.790218+0000","last_became_peered":"2024-09-20T19:30:09.790218+0000","last_unstale":"2024-09-20T19:32:31.708119+0000","last_undegraded":"2024-09-20T19:32:31.708119+0000","last_fullsized":"2024-09-20T19:32:31.708119+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'200","last_scrub_stamp":"2024-09-20T19:32:31.708023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.708023+0000","objects_scrubbed":2,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.952101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00136775,"stat_sum":{"num_bytes":4975136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":37797,"num_write":81,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2317410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.d","version":"523'31","reported_seq":1057,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.289354+0000","last_change":"2024-09-20T19:32:23.326867+0000","last_active":"2024-09-20T19:32:33.289354+0000","last_peered":"2024-09-20T19:32:33.289354+0000","last_clean":"2024-09-20T19:32:33.289354+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:32:33.289354+0000","last_undegraded":"2024-09-20T19:32:33.289354+0000","last_fullsized":"2024-09-20T19:32:33.289354+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:32:23.326721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.326721+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:26.897588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.378065+0000","last_change":"2024-09-20T19:30:31.374792+0000","last_active":"2024-09-20T19:32:31.378065+0000","last_peered":"2024-09-20T19:32:31.378065+0000","last_clean":"2024-09-20T19:32:31.378065+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:32:31.378065+0000","last_undegraded":"2024-09-20T19:32:31.378065+0000","last_fullsized":"2024-09-20T19:32:31.378065+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:10.363587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:10.363587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1213,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366292+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:32:31.366292+0000","last_peered":"2024-09-20T19:32:31.366292+0000","last_clean":"2024-09-20T19:32:31.366292+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:32:31.366292+0000","last_undegraded":"2024-09-20T19:32:31.366292+0000","last_fullsized":"2024-09-20T19:32:31.366292+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:51.922142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1208,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.363028+0000","last_change":"2024-09-20T19:31:31.525823+0000","last_active":"2024-09-20T19:32:31.363028+0000","last_peered":"2024-09-20T19:32:31.363028+0000","last_clean":"2024-09-20T19:32:31.363028+0000","last_became_active":"2024-09-20T19:30:59.407192+0000","last_became_peered":"2024-09-20T19:30:59.407192+0000","last_unstale":"2024-09-20T19:32:31.363028+0000","last_undegraded":"2024-09-20T19:32:31.363028+0000","last_fullsized":"2024-09-20T19:32:31.363028+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:31.525745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.525745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.839611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1265,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.378270+0000","last_change":"2024-09-20T19:30:59.437810+0000","last_active":"2024-09-20T19:32:31.378270+0000","last_peered":"2024-09-20T19:32:31.378270+0000","last_clean":"2024-09-20T19:32:31.378270+0000","last_became_active":"2024-09-20T19:30:59.437496+0000","last_became_peered":"2024-09-20T19:30:59.437496+0000","last_unstale":"2024-09-20T19:32:31.378270+0000","last_undegraded":"2024-09-20T19:32:31.378270+0000","last_fullsized":"2024-09-20T19:32:31.378270+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:55.361511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:55.361511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"644'463","reported_seq":2724,"reported_epoch":644,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:32.622353+0000","last_change":"2024-09-20T19:32:32.454268+0000","last_active":"2024-09-20T19:32:32.622353+0000","last_peered":"2024-09-20T19:32:32.622353+0000","last_clean":"2024-09-20T19:32:32.622353+0000","last_became_active":"2024-09-20T19:30:59.433690+0000","last_became_peered":"2024-09-20T19:30:59.433690+0000","last_unstale":"2024-09-20T19:32:32.622353+0000","last_undegraded":"2024-09-20T19:32:32.622353+0000","last_fullsized":"2024-09-20T19:32:32.622353+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'449","last_scrub_stamp":"2024-09-20T19:32:09.344681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.344681+0000","objects_scrubbed":6,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:29.832173+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.086739464000000002,"stat_sum":{"num_bytes":17676816,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":235,"num_read_kb":207145,"num_write":152,"num_write_kb":63605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17328954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.c","version":"523'20","reported_seq":1089,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.289125+0000","last_change":"2024-09-20T19:32:10.444211+0000","last_active":"2024-09-20T19:32:33.289125+0000","last_peered":"2024-09-20T19:32:33.289125+0000","last_clean":"2024-09-20T19:32:33.289125+0000","last_became_active":"2024-09-20T19:30:59.412354+0000","last_became_peered":"2024-09-20T19:30:59.412354+0000","last_unstale":"2024-09-20T19:32:33.289125+0000","last_undegraded":"2024-09-20T19:32:33.289125+0000","last_fullsized":"2024-09-20T19:32:33.289125+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:32:10.444013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.444013+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:11.431206+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1214,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366326+0000","last_change":"2024-09-20T19:31:39.214509+0000","last_active":"2024-09-20T19:32:31.366326+0000","last_peered":"2024-09-20T19:32:31.366326+0000","last_clean":"2024-09-20T19:32:31.366326+0000","last_became_active":"2024-09-20T19:30:59.435459+0000","last_became_peered":"2024-09-20T19:30:59.435459+0000","last_unstale":"2024-09-20T19:32:31.366326+0000","last_undegraded":"2024-09-20T19:32:31.366326+0000","last_fullsized":"2024-09-20T19:32:31.366326+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:39.214431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.214431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:39.611002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1388,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361700+0000","last_change":"2024-09-20T19:29:54.085338+0000","last_active":"2024-09-20T19:32:31.361700+0000","last_peered":"2024-09-20T19:32:31.361700+0000","last_clean":"2024-09-20T19:32:31.361700+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:32:31.361700+0000","last_undegraded":"2024-09-20T19:32:31.361700+0000","last_fullsized":"2024-09-20T19:32:31.361700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:29:54.085246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:54.085246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.534025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1017,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.290610+0000","last_change":"2024-09-20T19:30:31.380645+0000","last_active":"2024-09-20T19:32:33.290610+0000","last_peered":"2024-09-20T19:32:33.290610+0000","last_clean":"2024-09-20T19:32:33.290610+0000","last_became_active":"2024-09-20T19:30:31.379668+0000","last_became_peered":"2024-09-20T19:30:31.379668+0000","last_unstale":"2024-09-20T19:32:33.290610+0000","last_undegraded":"2024-09-20T19:32:33.290610+0000","last_fullsized":"2024-09-20T19:32:33.290610+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:20.948186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:20.948186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":791,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.396953+0000","last_change":"2024-09-20T19:28:03.366431+0000","last_active":"2024-09-20T19:32:31.396953+0000","last_peered":"2024-09-20T19:32:31.396953+0000","last_clean":"2024-09-20T19:32:31.396953+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:32:31.396953+0000","last_undegraded":"2024-09-20T19:32:31.396953+0000","last_fullsized":"2024-09-20T19:32:31.396953+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:03.366356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.366356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"640'737","reported_seq":3294,"reported_epoch":643,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:31.377190+0000","last_change":"2024-09-20T19:32:30.440136+0000","last_active":"2024-09-20T19:32:31.377190+0000","last_peered":"2024-09-20T19:32:31.377190+0000","last_clean":"2024-09-20T19:32:31.377190+0000","last_became_active":"2024-09-20T19:30:59.437942+0000","last_became_peered":"2024-09-20T19:30:59.437942+0000","last_unstale":"2024-09-20T19:32:31.377190+0000","last_undegraded":"2024-09-20T19:32:31.377190+0000","last_fullsized":"2024-09-20T19:32:31.377190+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'237","last_scrub_stamp":"2024-09-20T19:24:35.542766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.542766+0000","objects_scrubbed":12,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":1.093595976,"stat_sum":{"num_bytes":17896953,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":395,"num_read_kb":287352,"num_write":262,"num_write_kb":84816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":11179590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"}]},{"pgid":"4.f","version":"523'27","reported_seq":1265,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366948+0000","last_change":"2024-09-20T19:30:31.397678+0000","last_active":"2024-09-20T19:32:31.366948+0000","last_peered":"2024-09-20T19:32:31.366948+0000","last_clean":"2024-09-20T19:32:31.366948+0000","last_became_active":"2024-09-20T19:30:31.365973+0000","last_became_peered":"2024-09-20T19:30:31.365973+0000","last_unstale":"2024-09-20T19:32:31.366948+0000","last_undegraded":"2024-09-20T19:32:31.366948+0000","last_fullsized":"2024-09-20T19:32:31.366948+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.137611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1214,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361750+0000","last_change":"2024-09-20T19:30:59.435287+0000","last_active":"2024-09-20T19:32:31.361750+0000","last_peered":"2024-09-20T19:32:31.361750+0000","last_clean":"2024-09-20T19:32:31.361750+0000","last_became_active":"2024-09-20T19:30:59.434928+0000","last_became_peered":"2024-09-20T19:30:59.434928+0000","last_unstale":"2024-09-20T19:32:31.361750+0000","last_undegraded":"2024-09-20T19:32:31.361750+0000","last_fullsized":"2024-09-20T19:32:31.361750+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.895903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.895903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.157414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1178,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.290861+0000","last_change":"2024-09-20T19:30:38.015879+0000","last_active":"2024-09-20T19:32:33.290861+0000","last_peered":"2024-09-20T19:32:33.290861+0000","last_clean":"2024-09-20T19:32:33.290861+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:32:33.290861+0000","last_undegraded":"2024-09-20T19:32:33.290861+0000","last_fullsized":"2024-09-20T19:32:33.290861+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:38.015815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:38.015815+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:42.732365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1198,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.398629+0000","last_change":"2024-09-20T19:32:24.498998+0000","last_active":"2024-09-20T19:32:31.398629+0000","last_peered":"2024-09-20T19:32:31.398629+0000","last_clean":"2024-09-20T19:32:31.398629+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:32:31.398629+0000","last_undegraded":"2024-09-20T19:32:31.398629+0000","last_fullsized":"2024-09-20T19:32:31.398629+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:24.498942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:24.498942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.513248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"638'476","reported_seq":3021,"reported_epoch":643,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:31.371794+0000","last_change":"2024-09-20T19:32:30.439346+0000","last_active":"2024-09-20T19:32:31.371794+0000","last_peered":"2024-09-20T19:32:31.371794+0000","last_clean":"2024-09-20T19:32:31.371794+0000","last_became_active":"2024-09-20T19:30:09.785274+0000","last_became_peered":"2024-09-20T19:30:09.785274+0000","last_unstale":"2024-09-20T19:32:31.371794+0000","last_undegraded":"2024-09-20T19:32:31.371794+0000","last_fullsized":"2024-09-20T19:32:31.371794+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'149","last_scrub_stamp":"2024-09-20T19:24:44.564761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.564761+0000","objects_scrubbed":9,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0926344889999999,"stat_sum":{"num_bytes":16861690,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":270,"num_read_kb":230214,"num_write":172,"num_write_kb":65414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":15038144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"}]},{"pgid":"4.e","version":"523'19","reported_seq":1089,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.291225+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:32:33.291225+0000","last_peered":"2024-09-20T19:32:33.291225+0000","last_clean":"2024-09-20T19:32:33.291225+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:32:33.291225+0000","last_undegraded":"2024-09-20T19:32:33.291225+0000","last_fullsized":"2024-09-20T19:32:33.291225+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:01.570152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1089,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.358924+0000","last_change":"2024-09-20T19:31:26.549536+0000","last_active":"2024-09-20T19:32:31.358924+0000","last_peered":"2024-09-20T19:32:31.358924+0000","last_clean":"2024-09-20T19:32:31.358924+0000","last_became_active":"2024-09-20T19:30:59.407964+0000","last_became_peered":"2024-09-20T19:30:59.407964+0000","last_unstale":"2024-09-20T19:32:31.358924+0000","last_undegraded":"2024-09-20T19:32:31.358924+0000","last_fullsized":"2024-09-20T19:32:31.358924+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.395778+0000","last_change":"2024-09-20T19:32:13.572694+0000","last_active":"2024-09-20T19:32:31.395778+0000","last_peered":"2024-09-20T19:32:31.395778+0000","last_clean":"2024-09-20T19:32:31.395778+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:32:31.395778+0000","last_undegraded":"2024-09-20T19:32:31.395778+0000","last_fullsized":"2024-09-20T19:32:31.395778+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.572577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.572577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:15.678184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1005,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385573+0000","last_change":"2024-09-20T19:30:59.413132+0000","last_active":"2024-09-20T19:32:32.385573+0000","last_peered":"2024-09-20T19:32:32.385573+0000","last_clean":"2024-09-20T19:32:32.385573+0000","last_became_active":"2024-09-20T19:30:59.412722+0000","last_became_peered":"2024-09-20T19:30:59.412722+0000","last_unstale":"2024-09-20T19:32:32.385573+0000","last_undegraded":"2024-09-20T19:32:32.385573+0000","last_fullsized":"2024-09-20T19:32:32.385573+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":629,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.387733+0000","last_change":"2024-09-20T19:30:59.428471+0000","last_active":"2024-09-20T19:32:32.387733+0000","last_peered":"2024-09-20T19:32:32.387733+0000","last_clean":"2024-09-20T19:32:32.387733+0000","last_became_active":"2024-09-20T19:30:59.426646+0000","last_became_peered":"2024-09-20T19:30:59.426646+0000","last_unstale":"2024-09-20T19:32:32.387733+0000","last_undegraded":"2024-09-20T19:32:32.387733+0000","last_fullsized":"2024-09-20T19:32:32.387733+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:07.247985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:07.247985+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:33:04.665624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"634'190","reported_seq":2516,"reported_epoch":644,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:32.387835+0000","last_change":"2024-09-20T19:32:32.387812+0000","last_active":"2024-09-20T19:32:32.387835+0000","last_peered":"2024-09-20T19:32:32.387835+0000","last_clean":"2024-09-20T19:32:32.387835+0000","last_became_active":"2024-09-20T19:30:59.420070+0000","last_became_peered":"2024-09-20T19:30:59.420070+0000","last_unstale":"2024-09-20T19:32:32.387835+0000","last_undegraded":"2024-09-20T19:32:32.387835+0000","last_fullsized":"2024-09-20T19:32:32.387835+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'186","last_scrub_stamp":"2024-09-20T19:31:31.606808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.606808+0000","objects_scrubbed":3,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:42.141968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.059160239000000003,"stat_sum":{"num_bytes":2773287,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.1","version":"523'28","reported_seq":1059,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.285509+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:32:33.285509+0000","last_peered":"2024-09-20T19:32:33.285509+0000","last_clean":"2024-09-20T19:32:33.285509+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:32:33.285509+0000","last_undegraded":"2024-09-20T19:32:33.285509+0000","last_fullsized":"2024-09-20T19:32:33.285509+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.002901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1199,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.374008+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:32:31.374008+0000","last_peered":"2024-09-20T19:32:31.374008+0000","last_clean":"2024-09-20T19:32:31.374008+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:32:31.374008+0000","last_undegraded":"2024-09-20T19:32:31.374008+0000","last_fullsized":"2024-09-20T19:32:31.374008+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"554'15","reported_seq":1120,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.384908+0000","last_change":"2024-09-20T19:30:59.419924+0000","last_active":"2024-09-20T19:32:32.384908+0000","last_peered":"2024-09-20T19:32:32.384908+0000","last_clean":"2024-09-20T19:32:32.384908+0000","last_became_active":"2024-09-20T19:30:59.418231+0000","last_became_peered":"2024-09-20T19:30:59.418231+0000","last_unstale":"2024-09-20T19:32:32.384908+0000","last_undegraded":"2024-09-20T19:32:32.384908+0000","last_fullsized":"2024-09-20T19:32:32.384908+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:32.089810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.089810+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.508745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1179,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.380202+0000","last_change":"2024-09-20T19:30:03.184615+0000","last_active":"2024-09-20T19:32:31.380202+0000","last_peered":"2024-09-20T19:32:31.380202+0000","last_clean":"2024-09-20T19:32:31.380202+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:32:31.380202+0000","last_undegraded":"2024-09-20T19:32:31.380202+0000","last_fullsized":"2024-09-20T19:32:31.380202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:03.184552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:30:03.184552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":591,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.289894+0000","last_change":"2024-09-20T19:30:34.035838+0000","last_active":"2024-09-20T19:32:33.289894+0000","last_peered":"2024-09-20T19:32:33.289894+0000","last_clean":"2024-09-20T19:32:33.289894+0000","last_became_active":"2024-09-20T19:30:31.377908+0000","last_became_peered":"2024-09-20T19:30:31.377908+0000","last_unstale":"2024-09-20T19:32:33.289894+0000","last_undegraded":"2024-09-20T19:32:33.289894+0000","last_fullsized":"2024-09-20T19:32:33.289894+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:30:34.035776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:30:34.035776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:47.612744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"644'371","reported_seq":2710,"reported_epoch":644,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:32.386355+0000","last_change":"2024-09-20T19:32:32.386355+0000","last_active":"2024-09-20T19:32:32.386355+0000","last_peered":"2024-09-20T19:32:32.386355+0000","last_clean":"2024-09-20T19:32:32.386355+0000","last_became_active":"2024-09-20T19:30:59.407988+0000","last_became_peered":"2024-09-20T19:30:59.407988+0000","last_unstale":"2024-09-20T19:32:32.386355+0000","last_undegraded":"2024-09-20T19:32:32.386355+0000","last_fullsized":"2024-09-20T19:32:32.386355+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'350","last_scrub_stamp":"2024-09-20T19:31:19.546439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.546439+0000","objects_scrubbed":2,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.057581227999999998,"stat_sum":{"num_bytes":11076143,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":7193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.0","version":"523'30","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.384969+0000","last_change":"2024-09-20T19:32:20.900983+0000","last_active":"2024-09-20T19:32:32.384969+0000","last_peered":"2024-09-20T19:32:32.384969+0000","last_clean":"2024-09-20T19:32:32.384969+0000","last_became_active":"2024-09-20T19:30:59.431399+0000","last_became_peered":"2024-09-20T19:30:59.431399+0000","last_unstale":"2024-09-20T19:32:32.384969+0000","last_undegraded":"2024-09-20T19:32:32.384969+0000","last_fullsized":"2024-09-20T19:32:32.384969+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.542941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1207,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366495+0000","last_change":"2024-09-20T19:32:15.096541+0000","last_active":"2024-09-20T19:32:31.366495+0000","last_peered":"2024-09-20T19:32:31.366495+0000","last_clean":"2024-09-20T19:32:31.366495+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:32:31.366495+0000","last_undegraded":"2024-09-20T19:32:31.366495+0000","last_fullsized":"2024-09-20T19:32:31.366495+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:32:15.096450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.096450+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:25.106545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1011,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.284724+0000","last_change":"2024-09-20T19:32:18.400368+0000","last_active":"2024-09-20T19:32:33.284724+0000","last_peered":"2024-09-20T19:32:33.284724+0000","last_clean":"2024-09-20T19:32:33.284724+0000","last_became_active":"2024-09-20T19:30:59.403694+0000","last_became_peered":"2024-09-20T19:30:59.403694+0000","last_unstale":"2024-09-20T19:32:33.284724+0000","last_undegraded":"2024-09-20T19:32:33.284724+0000","last_fullsized":"2024-09-20T19:32:33.284724+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:18.400304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.400304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.263499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":957,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.285710+0000","last_change":"2024-09-20T19:30:59.403484+0000","last_active":"2024-09-20T19:32:33.285710+0000","last_peered":"2024-09-20T19:32:33.285710+0000","last_clean":"2024-09-20T19:32:33.285710+0000","last_became_active":"2024-09-20T19:30:59.403080+0000","last_became_peered":"2024-09-20T19:30:59.403080+0000","last_unstale":"2024-09-20T19:32:33.285710+0000","last_undegraded":"2024-09-20T19:32:33.285710+0000","last_fullsized":"2024-09-20T19:32:33.285710+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.522323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":904,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.372087+0000","last_change":"2024-09-20T19:30:59.410472+0000","last_active":"2024-09-20T19:32:31.372087+0000","last_peered":"2024-09-20T19:32:31.372087+0000","last_clean":"2024-09-20T19:32:31.372087+0000","last_became_active":"2024-09-20T19:30:59.410103+0000","last_became_peered":"2024-09-20T19:30:59.410103+0000","last_unstale":"2024-09-20T19:32:31.372087+0000","last_undegraded":"2024-09-20T19:32:31.372087+0000","last_fullsized":"2024-09-20T19:32:31.372087+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.382463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.382463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"641'719","reported_seq":3287,"reported_epoch":643,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:31.396115+0000","last_change":"2024-09-20T19:32:30.431547+0000","last_active":"2024-09-20T19:32:31.396115+0000","last_peered":"2024-09-20T19:32:31.396115+0000","last_clean":"2024-09-20T19:32:31.396115+0000","last_became_active":"2024-09-20T19:30:59.413518+0000","last_became_peered":"2024-09-20T19:30:59.413518+0000","last_unstale":"2024-09-20T19:32:31.396115+0000","last_undegraded":"2024-09-20T19:32:31.396115+0000","last_fullsized":"2024-09-20T19:32:31.396115+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'685","last_scrub_stamp":"2024-09-20T19:32:09.614901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.614901+0000","objects_scrubbed":9,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:16.902346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.086900655,"stat_sum":{"num_bytes":18499692,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":1961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.3","version":"523'29","reported_seq":1020,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369185+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:32:32.369185+0000","last_peered":"2024-09-20T19:32:32.369185+0000","last_clean":"2024-09-20T19:32:32.369185+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:32:32.369185+0000","last_undegraded":"2024-09-20T19:32:32.369185+0000","last_fullsized":"2024-09-20T19:32:32.369185+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.517393+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1158,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361546+0000","last_change":"2024-09-20T19:30:31.348665+0000","last_active":"2024-09-20T19:32:31.361546+0000","last_peered":"2024-09-20T19:32:31.361546+0000","last_clean":"2024-09-20T19:32:31.361546+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:32:31.361546+0000","last_undegraded":"2024-09-20T19:32:31.361546+0000","last_fullsized":"2024-09-20T19:32:31.361546+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:22.169579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:22.169579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.508877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.369339+0000","last_change":"2024-09-20T19:30:09.775734+0000","last_active":"2024-09-20T19:32:31.369339+0000","last_peered":"2024-09-20T19:32:31.369339+0000","last_clean":"2024-09-20T19:32:31.369339+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:32:31.369339+0000","last_undegraded":"2024-09-20T19:32:31.369339+0000","last_fullsized":"2024-09-20T19:32:31.369339+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:59.649128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:59.649128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:04.937740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1175,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361130+0000","last_change":"2024-09-20T19:27:53.899785+0000","last_active":"2024-09-20T19:32:31.361130+0000","last_peered":"2024-09-20T19:32:31.361130+0000","last_clean":"2024-09-20T19:32:31.361130+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:32:31.361130+0000","last_undegraded":"2024-09-20T19:32:31.361130+0000","last_fullsized":"2024-09-20T19:32:31.361130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:53.899739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:53.899739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.125297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":905,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.371918+0000","last_change":"2024-09-20T19:30:59.409489+0000","last_active":"2024-09-20T19:32:31.371918+0000","last_peered":"2024-09-20T19:32:31.371918+0000","last_clean":"2024-09-20T19:32:31.371918+0000","last_became_active":"2024-09-20T19:30:59.409151+0000","last_became_peered":"2024-09-20T19:30:59.409151+0000","last_unstale":"2024-09-20T19:32:31.371918+0000","last_undegraded":"2024-09-20T19:32:31.371918+0000","last_fullsized":"2024-09-20T19:32:31.371918+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"644'545","reported_seq":3007,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.286250+0000","last_change":"2024-09-20T19:32:33.286250+0000","last_active":"2024-09-20T19:32:33.286250+0000","last_peered":"2024-09-20T19:32:33.286250+0000","last_clean":"2024-09-20T19:32:33.286250+0000","last_became_active":"2024-09-20T19:31:48.294687+0000","last_became_peered":"2024-09-20T19:31:48.294687+0000","last_unstale":"2024-09-20T19:32:33.286250+0000","last_undegraded":"2024-09-20T19:32:33.286250+0000","last_fullsized":"2024-09-20T19:32:33.286250+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'452","last_scrub_stamp":"2024-09-20T19:30:36.040764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:36.040764+0000","objects_scrubbed":6,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001175932,"stat_sum":{"num_bytes":10706587,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":36455,"num_write":23,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6670312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.2","version":"523'20","reported_seq":1251,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366019+0000","last_change":"2024-09-20T19:31:19.351200+0000","last_active":"2024-09-20T19:32:31.366019+0000","last_peered":"2024-09-20T19:32:31.366019+0000","last_clean":"2024-09-20T19:32:31.366019+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:32:31.366019+0000","last_undegraded":"2024-09-20T19:32:31.366019+0000","last_fullsized":"2024-09-20T19:32:31.366019+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:19.351134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.351134+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:32.835514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1198,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.367274+0000","last_change":"2024-09-20T19:30:09.760759+0000","last_active":"2024-09-20T19:32:31.367274+0000","last_peered":"2024-09-20T19:32:31.367274+0000","last_clean":"2024-09-20T19:32:31.367274+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:32:31.367274+0000","last_undegraded":"2024-09-20T19:32:31.367274+0000","last_fullsized":"2024-09-20T19:32:31.367274+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.151053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1110,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369448+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:32:32.369448+0000","last_peered":"2024-09-20T19:32:32.369448+0000","last_clean":"2024-09-20T19:32:32.369448+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:32:32.369448+0000","last_undegraded":"2024-09-20T19:32:32.369448+0000","last_fullsized":"2024-09-20T19:32:32.369448+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:32.990270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":984,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.367192+0000","last_change":"2024-09-20T19:31:33.455642+0000","last_active":"2024-09-20T19:32:32.367192+0000","last_peered":"2024-09-20T19:32:32.367192+0000","last_clean":"2024-09-20T19:32:32.367192+0000","last_became_active":"2024-09-20T19:30:59.433962+0000","last_became_peered":"2024-09-20T19:30:59.433962+0000","last_unstale":"2024-09-20T19:32:32.367192+0000","last_undegraded":"2024-09-20T19:32:32.367192+0000","last_fullsized":"2024-09-20T19:32:32.367192+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.455584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.455584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.875559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":803,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.396933+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:32:31.396933+0000","last_peered":"2024-09-20T19:32:31.396933+0000","last_clean":"2024-09-20T19:32:31.396933+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:32:31.396933+0000","last_undegraded":"2024-09-20T19:32:31.396933+0000","last_fullsized":"2024-09-20T19:32:31.396933+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.285306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"641'414","reported_seq":2991,"reported_epoch":643,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:31.373069+0000","last_change":"2024-09-20T19:32:30.439298+0000","last_active":"2024-09-20T19:32:31.373069+0000","last_peered":"2024-09-20T19:32:31.373069+0000","last_clean":"2024-09-20T19:32:31.373069+0000","last_became_active":"2024-09-20T19:30:59.409330+0000","last_became_peered":"2024-09-20T19:30:59.409330+0000","last_unstale":"2024-09-20T19:32:31.373069+0000","last_undegraded":"2024-09-20T19:32:31.373069+0000","last_fullsized":"2024-09-20T19:32:31.373069+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'400","last_scrub_stamp":"2024-09-20T19:31:36.808095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.808095+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:56.521701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.11706116,"stat_sum":{"num_bytes":6460566,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":4866,"num_write":13,"num_write_kb":5272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"}]},{"pgid":"4.4","version":"554'44","reported_seq":1471,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.367113+0000","last_change":"2024-09-20T19:31:31.390088+0000","last_active":"2024-09-20T19:32:31.367113+0000","last_peered":"2024-09-20T19:32:31.367113+0000","last_clean":"2024-09-20T19:32:31.367113+0000","last_became_active":"2024-09-20T19:30:59.436599+0000","last_became_peered":"2024-09-20T19:30:59.436599+0000","last_unstale":"2024-09-20T19:32:31.367113+0000","last_undegraded":"2024-09-20T19:32:31.367113+0000","last_fullsized":"2024-09-20T19:32:31.367113+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'44","last_scrub_stamp":"2024-09-20T19:31:31.390009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.390009+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.692796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1207,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.397887+0000","last_change":"2024-09-20T19:30:02.530981+0000","last_active":"2024-09-20T19:32:31.397887+0000","last_peered":"2024-09-20T19:32:31.397887+0000","last_clean":"2024-09-20T19:32:31.397887+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:32:31.397887+0000","last_undegraded":"2024-09-20T19:32:31.397887+0000","last_fullsized":"2024-09-20T19:32:31.397887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:02.530860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:30:02.530860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":995,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.365189+0000","last_change":"2024-09-20T19:32:17.478298+0000","last_active":"2024-09-20T19:32:32.365189+0000","last_peered":"2024-09-20T19:32:32.365189+0000","last_clean":"2024-09-20T19:32:32.365189+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:32:32.365189+0000","last_undegraded":"2024-09-20T19:32:32.365189+0000","last_fullsized":"2024-09-20T19:32:32.365189+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:17.478143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:17.478143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":994,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.388031+0000","last_change":"2024-09-20T19:32:14.826848+0000","last_active":"2024-09-20T19:32:32.388031+0000","last_peered":"2024-09-20T19:32:32.388031+0000","last_clean":"2024-09-20T19:32:32.388031+0000","last_became_active":"2024-09-20T19:30:59.424738+0000","last_became_peered":"2024-09-20T19:30:59.424738+0000","last_unstale":"2024-09-20T19:32:32.388031+0000","last_undegraded":"2024-09-20T19:32:32.388031+0000","last_fullsized":"2024-09-20T19:32:32.388031+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.259591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1273,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.871893+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:32:31.871893+0000","last_peered":"2024-09-20T19:32:31.871893+0000","last_clean":"2024-09-20T19:32:31.871893+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:32:31.871893+0000","last_undegraded":"2024-09-20T19:32:31.871893+0000","last_fullsized":"2024-09-20T19:32:31.871893+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.269206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"644'832","reported_seq":3057,"reported_epoch":644,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:32.454310+0000","last_change":"2024-09-20T19:32:32.454310+0000","last_active":"2024-09-20T19:32:32.454310+0000","last_peered":"2024-09-20T19:32:32.454310+0000","last_clean":"2024-09-20T19:32:32.454310+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:32:32.454310+0000","last_undegraded":"2024-09-20T19:32:32.454310+0000","last_fullsized":"2024-09-20T19:32:32.454310+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'779","last_scrub_stamp":"2024-09-20T19:31:22.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.595437+0000","objects_scrubbed":4,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0084521419999999993,"stat_sum":{"num_bytes":10687802,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":44308,"num_write":16,"num_write_kb":3428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.7","version":"523'17","reported_seq":1168,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.362588+0000","last_change":"2024-09-20T19:31:39.546516+0000","last_active":"2024-09-20T19:32:31.362588+0000","last_peered":"2024-09-20T19:32:31.362588+0000","last_clean":"2024-09-20T19:32:31.362588+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:32:31.362588+0000","last_undegraded":"2024-09-20T19:32:31.362588+0000","last_fullsized":"2024-09-20T19:32:31.362588+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:31:39.546441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.546441+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.667130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2051,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.360875+0000","last_change":"2024-09-20T19:28:48.600562+0000","last_active":"2024-09-20T19:32:31.360875+0000","last_peered":"2024-09-20T19:32:31.360875+0000","last_clean":"2024-09-20T19:32:31.360875+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:32:31.360875+0000","last_undegraded":"2024-09-20T19:32:31.360875+0000","last_fullsized":"2024-09-20T19:32:31.360875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:28:48.600384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.600384+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.135772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":847,"num_read_kb":842,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1212,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.359637+0000","last_change":"2024-09-20T19:31:23.521703+0000","last_active":"2024-09-20T19:32:31.359637+0000","last_peered":"2024-09-20T19:32:31.359637+0000","last_clean":"2024-09-20T19:32:31.359637+0000","last_became_active":"2024-09-20T19:30:59.416753+0000","last_became_peered":"2024-09-20T19:30:59.416753+0000","last_unstale":"2024-09-20T19:32:31.359637+0000","last_undegraded":"2024-09-20T19:32:31.359637+0000","last_fullsized":"2024-09-20T19:32:31.359637+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:23.521555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:23.521555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.929656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1202,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.362894+0000","last_change":"2024-09-20T19:29:52.092899+0000","last_active":"2024-09-20T19:32:31.362894+0000","last_peered":"2024-09-20T19:32:31.362894+0000","last_clean":"2024-09-20T19:32:31.362894+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:32:31.362894+0000","last_undegraded":"2024-09-20T19:32:31.362894+0000","last_fullsized":"2024-09-20T19:32:31.362894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:29:52.092846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:52.092846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:47.539955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1198,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.380696+0000","last_change":"2024-09-20T19:32:21.799367+0000","last_active":"2024-09-20T19:32:31.380696+0000","last_peered":"2024-09-20T19:32:31.380696+0000","last_clean":"2024-09-20T19:32:31.380696+0000","last_became_active":"2024-09-20T19:30:59.437740+0000","last_became_peered":"2024-09-20T19:30:59.437740+0000","last_unstale":"2024-09-20T19:32:31.380696+0000","last_undegraded":"2024-09-20T19:32:31.380696+0000","last_fullsized":"2024-09-20T19:32:31.380696+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.069051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":805,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.366175+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:32:31.366175+0000","last_peered":"2024-09-20T19:32:31.366175+0000","last_clean":"2024-09-20T19:32:31.366175+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:32:31.366175+0000","last_undegraded":"2024-09-20T19:32:31.366175+0000","last_fullsized":"2024-09-20T19:32:31.366175+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.171547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"640'834","reported_seq":3433,"reported_epoch":643,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:31.380671+0000","last_change":"2024-09-20T19:32:30.445103+0000","last_active":"2024-09-20T19:32:31.380671+0000","last_peered":"2024-09-20T19:32:31.380671+0000","last_clean":"2024-09-20T19:32:31.380671+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:32:31.380671+0000","last_undegraded":"2024-09-20T19:32:31.380671+0000","last_fullsized":"2024-09-20T19:32:31.380671+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:47.486671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.088583756999999999,"stat_sum":{"num_bytes":7260987,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":3330,"num_write":5,"num_write_kb":783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f2","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":985,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369430+0000","last_change":"2024-09-20T19:30:09.829566+0000","last_active":"2024-09-20T19:32:32.369430+0000","last_peered":"2024-09-20T19:32:32.369430+0000","last_clean":"2024-09-20T19:32:32.369430+0000","last_became_active":"2024-09-20T19:30:09.783033+0000","last_became_peered":"2024-09-20T19:30:09.783033+0000","last_unstale":"2024-09-20T19:32:32.369430+0000","last_undegraded":"2024-09-20T19:32:32.369430+0000","last_fullsized":"2024-09-20T19:32:32.369430+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:21.127630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.127630+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.428713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":930,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.365226+0000","last_change":"2024-09-20T19:30:59.438831+0000","last_active":"2024-09-20T19:32:32.365226+0000","last_peered":"2024-09-20T19:32:32.365226+0000","last_clean":"2024-09-20T19:32:32.365226+0000","last_became_active":"2024-09-20T19:30:59.438639+0000","last_became_peered":"2024-09-20T19:30:59.438639+0000","last_unstale":"2024-09-20T19:32:32.365226+0000","last_undegraded":"2024-09-20T19:32:32.365226+0000","last_fullsized":"2024-09-20T19:32:32.365226+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:24:39.000765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.000765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.485035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1211,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.376740+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:32:31.376740+0000","last_peered":"2024-09-20T19:32:31.376740+0000","last_clean":"2024-09-20T19:32:31.376740+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:32:31.376740+0000","last_undegraded":"2024-09-20T19:32:31.376740+0000","last_fullsized":"2024-09-20T19:32:31.376740+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:41.273146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1202,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.399360+0000","last_change":"2024-09-20T19:32:23.505615+0000","last_active":"2024-09-20T19:32:31.399360+0000","last_peered":"2024-09-20T19:32:31.399360+0000","last_clean":"2024-09-20T19:32:31.399360+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:32:31.399360+0000","last_undegraded":"2024-09-20T19:32:31.399360+0000","last_fullsized":"2024-09-20T19:32:31.399360+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.505475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.505475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.486718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1210,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.365945+0000","last_change":"2024-09-20T19:30:59.436100+0000","last_active":"2024-09-20T19:32:31.365945+0000","last_peered":"2024-09-20T19:32:31.365945+0000","last_clean":"2024-09-20T19:32:31.365945+0000","last_became_active":"2024-09-20T19:30:59.435560+0000","last_became_peered":"2024-09-20T19:30:59.435560+0000","last_unstale":"2024-09-20T19:32:31.365945+0000","last_undegraded":"2024-09-20T19:32:31.365945+0000","last_fullsized":"2024-09-20T19:32:31.365945+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:14.942057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:14.942057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.153381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":2496,"reported_epoch":644,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:32.369725+0000","last_change":"2024-09-20T19:32:32.369703+0000","last_active":"2024-09-20T19:32:32.369725+0000","last_peered":"2024-09-20T19:32:32.369725+0000","last_clean":"2024-09-20T19:32:32.369725+0000","last_became_active":"2024-09-20T19:30:59.449561+0000","last_became_peered":"2024-09-20T19:30:59.449561+0000","last_unstale":"2024-09-20T19:32:32.369725+0000","last_undegraded":"2024-09-20T19:32:32.369725+0000","last_fullsized":"2024-09-20T19:32:32.369725+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:31:29.529433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.529433+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:40.634209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067332039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.5","version":"523'30","reported_seq":1412,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.371805+0000","last_change":"2024-09-20T19:30:31.397491+0000","last_active":"2024-09-20T19:32:31.371805+0000","last_peered":"2024-09-20T19:32:31.371805+0000","last_clean":"2024-09-20T19:32:31.371805+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:32:31.371805+0000","last_undegraded":"2024-09-20T19:32:31.371805+0000","last_fullsized":"2024-09-20T19:32:31.371805+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:24:43.584143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.584143+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":1016,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.386280+0000","last_change":"2024-09-20T19:30:59.412249+0000","last_active":"2024-09-20T19:32:32.386280+0000","last_peered":"2024-09-20T19:32:32.386280+0000","last_clean":"2024-09-20T19:32:32.386280+0000","last_became_active":"2024-09-20T19:30:59.411724+0000","last_became_peered":"2024-09-20T19:30:59.411724+0000","last_unstale":"2024-09-20T19:32:32.386280+0000","last_undegraded":"2024-09-20T19:32:32.386280+0000","last_fullsized":"2024-09-20T19:32:32.386280+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:24:46.018399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.018399+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.503059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1191,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.398353+0000","last_change":"2024-09-20T19:28:48.425075+0000","last_active":"2024-09-20T19:32:31.398353+0000","last_peered":"2024-09-20T19:32:31.398353+0000","last_clean":"2024-09-20T19:32:31.398353+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:32:31.398353+0000","last_undegraded":"2024-09-20T19:32:31.398353+0000","last_fullsized":"2024-09-20T19:32:31.398353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:48.424987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.424987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1252,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.377364+0000","last_change":"2024-09-20T19:30:09.775314+0000","last_active":"2024-09-20T19:32:31.377364+0000","last_peered":"2024-09-20T19:32:31.377364+0000","last_clean":"2024-09-20T19:32:31.377364+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:32:31.377364+0000","last_undegraded":"2024-09-20T19:32:31.377364+0000","last_fullsized":"2024-09-20T19:32:31.377364+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:26:13.424049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.424049+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1183,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.380912+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:32:31.380912+0000","last_peered":"2024-09-20T19:32:31.380912+0000","last_clean":"2024-09-20T19:32:31.380912+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:32:31.380912+0000","last_undegraded":"2024-09-20T19:32:31.380912+0000","last_fullsized":"2024-09-20T19:32:31.380912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.790580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1197,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.399984+0000","last_change":"2024-09-20T19:32:10.638734+0000","last_active":"2024-09-20T19:32:31.399984+0000","last_peered":"2024-09-20T19:32:31.399984+0000","last_clean":"2024-09-20T19:32:31.399984+0000","last_became_active":"2024-09-20T19:30:59.410315+0000","last_became_peered":"2024-09-20T19:30:59.410315+0000","last_unstale":"2024-09-20T19:32:31.399984+0000","last_undegraded":"2024-09-20T19:32:31.399984+0000","last_fullsized":"2024-09-20T19:32:31.399984+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:10.638671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.638671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.768982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"644'335","reported_seq":2607,"reported_epoch":644,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:32.367133+0000","last_change":"2024-09-20T19:32:32.367133+0000","last_active":"2024-09-20T19:32:32.367133+0000","last_peered":"2024-09-20T19:32:32.367133+0000","last_clean":"2024-09-20T19:32:32.367133+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:32:32.367133+0000","last_undegraded":"2024-09-20T19:32:32.367133+0000","last_fullsized":"2024-09-20T19:32:32.367133+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'300","last_scrub_stamp":"2024-09-20T19:31:19.575078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.575078+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011231682999999999,"stat_sum":{"num_bytes":7013011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":20,"num_write_kb":5041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.a","version":"523'35","reported_seq":1242,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.370114+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:32:31.370114+0000","last_peered":"2024-09-20T19:32:31.370114+0000","last_clean":"2024-09-20T19:32:31.370114+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:32:31.370114+0000","last_undegraded":"2024-09-20T19:32:31.370114+0000","last_fullsized":"2024-09-20T19:32:31.370114+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1213,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361748+0000","last_change":"2024-09-20T19:31:40.539723+0000","last_active":"2024-09-20T19:32:31.361748+0000","last_peered":"2024-09-20T19:32:31.361748+0000","last_clean":"2024-09-20T19:32:31.361748+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:32:31.361748+0000","last_undegraded":"2024-09-20T19:32:31.361748+0000","last_fullsized":"2024-09-20T19:32:31.361748+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:40.539596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.539596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1205,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.360147+0000","last_change":"2024-09-20T19:31:29.553212+0000","last_active":"2024-09-20T19:32:31.360147+0000","last_peered":"2024-09-20T19:32:31.360147+0000","last_clean":"2024-09-20T19:32:31.360147+0000","last_became_active":"2024-09-20T19:30:09.789268+0000","last_became_peered":"2024-09-20T19:30:09.789268+0000","last_unstale":"2024-09-20T19:32:31.360147+0000","last_undegraded":"2024-09-20T19:32:31.360147+0000","last_fullsized":"2024-09-20T19:32:31.360147+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.075543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1182,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.377076+0000","last_change":"2024-09-20T19:32:15.777865+0000","last_active":"2024-09-20T19:32:31.377076+0000","last_peered":"2024-09-20T19:32:31.377076+0000","last_clean":"2024-09-20T19:32:31.377076+0000","last_became_active":"2024-09-20T19:30:59.440497+0000","last_became_peered":"2024-09-20T19:30:59.440497+0000","last_unstale":"2024-09-20T19:32:31.377076+0000","last_undegraded":"2024-09-20T19:32:31.377076+0000","last_fullsized":"2024-09-20T19:32:31.377076+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.419161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"642'1140","reported_seq":3647,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.358973+0000","last_change":"2024-09-20T19:32:29.418807+0000","last_active":"2024-09-20T19:32:31.358973+0000","last_peered":"2024-09-20T19:32:31.358973+0000","last_clean":"2024-09-20T19:32:31.358973+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:32:31.358973+0000","last_undegraded":"2024-09-20T19:32:31.358973+0000","last_fullsized":"2024-09-20T19:32:31.358973+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'1120","last_scrub_stamp":"2024-09-20T19:32:08.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:08.554870+0000","objects_scrubbed":7,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:25.828918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001605015,"stat_sum":{"num_bytes":14649106,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":18684,"num_write":49,"num_write_kb":21068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.b","version":"523'15","reported_seq":993,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369349+0000","last_change":"2024-09-20T19:32:15.464944+0000","last_active":"2024-09-20T19:32:32.369349+0000","last_peered":"2024-09-20T19:32:32.369349+0000","last_clean":"2024-09-20T19:32:32.369349+0000","last_became_active":"2024-09-20T19:30:31.349098+0000","last_became_peered":"2024-09-20T19:30:31.349098+0000","last_unstale":"2024-09-20T19:32:32.369349+0000","last_undegraded":"2024-09-20T19:32:32.369349+0000","last_fullsized":"2024-09-20T19:32:32.369349+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.464807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.464807+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.851627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1221,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.368448+0000","last_change":"2024-09-20T19:31:29.374819+0000","last_active":"2024-09-20T19:32:31.368448+0000","last_peered":"2024-09-20T19:32:31.368448+0000","last_clean":"2024-09-20T19:32:31.368448+0000","last_became_active":"2024-09-20T19:30:59.434929+0000","last_became_peered":"2024-09-20T19:30:59.434929+0000","last_unstale":"2024-09-20T19:32:31.368448+0000","last_undegraded":"2024-09-20T19:32:31.368448+0000","last_fullsized":"2024-09-20T19:32:31.368448+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.983029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1212,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.360024+0000","last_change":"2024-09-20T19:31:36.575855+0000","last_active":"2024-09-20T19:32:31.360024+0000","last_peered":"2024-09-20T19:32:31.360024+0000","last_clean":"2024-09-20T19:32:31.360024+0000","last_became_active":"2024-09-20T19:30:59.407432+0000","last_became_peered":"2024-09-20T19:30:59.407432+0000","last_unstale":"2024-09-20T19:32:31.360024+0000","last_undegraded":"2024-09-20T19:32:31.360024+0000","last_fullsized":"2024-09-20T19:32:31.360024+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.575803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.575803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.311917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":978,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.365339+0000","last_change":"2024-09-20T19:32:10.330837+0000","last_active":"2024-09-20T19:32:32.365339+0000","last_peered":"2024-09-20T19:32:32.365339+0000","last_clean":"2024-09-20T19:32:32.365339+0000","last_became_active":"2024-09-20T19:30:09.787067+0000","last_became_peered":"2024-09-20T19:30:09.787067+0000","last_unstale":"2024-09-20T19:32:32.365339+0000","last_undegraded":"2024-09-20T19:32:32.365339+0000","last_fullsized":"2024-09-20T19:32:32.365339+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:32:10.330783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.330783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.559325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"641'407","reported_seq":2931,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.365773+0000","last_change":"2024-09-20T19:32:31.340735+0000","last_active":"2024-09-20T19:32:31.365773+0000","last_peered":"2024-09-20T19:32:31.365773+0000","last_clean":"2024-09-20T19:32:31.365773+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:32:31.365773+0000","last_undegraded":"2024-09-20T19:32:31.365773+0000","last_fullsized":"2024-09-20T19:32:31.365773+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'405","last_scrub_stamp":"2024-09-20T19:32:29.036109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.036109+0000","objects_scrubbed":7,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:55.102731+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.95816396299999995,"stat_sum":{"num_bytes":18473081,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":168,"num_read_kb":125466,"num_write":106,"num_write_kb":50551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.8","version":"523'27","reported_seq":1238,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.396366+0000","last_change":"2024-09-20T19:29:49.731520+0000","last_active":"2024-09-20T19:32:31.396366+0000","last_peered":"2024-09-20T19:32:31.396366+0000","last_clean":"2024-09-20T19:32:31.396366+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:32:31.396366+0000","last_undegraded":"2024-09-20T19:32:31.396366+0000","last_fullsized":"2024-09-20T19:32:31.396366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'17","last_scrub_stamp":"2024-09-20T19:29:49.731377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:29:49.731377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":1229,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.363586+0000","last_change":"2024-09-20T19:31:35.568682+0000","last_active":"2024-09-20T19:32:31.363586+0000","last_peered":"2024-09-20T19:32:31.363586+0000","last_clean":"2024-09-20T19:32:31.363586+0000","last_became_active":"2024-09-20T19:30:59.407885+0000","last_became_peered":"2024-09-20T19:30:59.407885+0000","last_unstale":"2024-09-20T19:32:31.363586+0000","last_undegraded":"2024-09-20T19:32:31.363586+0000","last_fullsized":"2024-09-20T19:32:31.363586+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:35.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.568627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:00.653341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1388,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.377635+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:32:31.377635+0000","last_peered":"2024-09-20T19:32:31.377635+0000","last_clean":"2024-09-20T19:32:31.377635+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:32:31.377635+0000","last_undegraded":"2024-09-20T19:32:31.377635+0000","last_fullsized":"2024-09-20T19:32:31.377635+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.376352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1190,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.397181+0000","last_change":"2024-09-20T19:32:12.619664+0000","last_active":"2024-09-20T19:32:31.397181+0000","last_peered":"2024-09-20T19:32:31.397181+0000","last_clean":"2024-09-20T19:32:31.397181+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:32:31.397181+0000","last_undegraded":"2024-09-20T19:32:31.397181+0000","last_fullsized":"2024-09-20T19:32:31.397181+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.619550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.619550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.382228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"644'411","reported_seq":2686,"reported_epoch":644,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:32.386221+0000","last_change":"2024-09-20T19:32:32.386221+0000","last_active":"2024-09-20T19:32:32.386221+0000","last_peered":"2024-09-20T19:32:32.386221+0000","last_clean":"2024-09-20T19:32:32.386221+0000","last_became_active":"2024-09-20T19:30:59.413000+0000","last_became_peered":"2024-09-20T19:30:59.413000+0000","last_unstale":"2024-09-20T19:32:32.386221+0000","last_undegraded":"2024-09-20T19:32:32.386221+0000","last_fullsized":"2024-09-20T19:32:32.386221+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'344","last_scrub_stamp":"2024-09-20T19:27:23.499201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:23.499201+0000","objects_scrubbed":6,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.061928311,"stat_sum":{"num_bytes":13832590,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":182,"num_read_kb":149074,"num_write":131,"num_write_kb":56239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4084895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.9","version":"523'18","reported_seq":1066,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.285399+0000","last_change":"2024-09-20T19:32:26.295823+0000","last_active":"2024-09-20T19:32:33.285399+0000","last_peered":"2024-09-20T19:32:33.285399+0000","last_clean":"2024-09-20T19:32:33.285399+0000","last_became_active":"2024-09-20T19:30:59.404224+0000","last_became_peered":"2024-09-20T19:30:59.404224+0000","last_unstale":"2024-09-20T19:32:33.285399+0000","last_undegraded":"2024-09-20T19:32:33.285399+0000","last_fullsized":"2024-09-20T19:32:33.285399+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.872180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1065,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.290969+0000","last_change":"2024-09-20T19:32:19.382969+0000","last_active":"2024-09-20T19:32:33.290969+0000","last_peered":"2024-09-20T19:32:33.290969+0000","last_clean":"2024-09-20T19:32:33.290969+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:32:33.290969+0000","last_undegraded":"2024-09-20T19:32:33.290969+0000","last_fullsized":"2024-09-20T19:32:33.290969+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:32:19.382850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:19.382850+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:24.971441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1205,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.359255+0000","last_change":"2024-09-20T19:31:32.549379+0000","last_active":"2024-09-20T19:32:31.359255+0000","last_peered":"2024-09-20T19:32:31.359255+0000","last_clean":"2024-09-20T19:32:31.359255+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:32:31.359255+0000","last_undegraded":"2024-09-20T19:32:31.359255+0000","last_fullsized":"2024-09-20T19:32:31.359255+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:32.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:32.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.516267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1031,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385362+0000","last_change":"2024-09-20T19:32:11.827975+0000","last_active":"2024-09-20T19:32:32.385362+0000","last_peered":"2024-09-20T19:32:32.385362+0000","last_clean":"2024-09-20T19:32:32.385362+0000","last_became_active":"2024-09-20T19:30:59.425091+0000","last_became_peered":"2024-09-20T19:30:59.425091+0000","last_unstale":"2024-09-20T19:32:32.385362+0000","last_undegraded":"2024-09-20T19:32:32.385362+0000","last_fullsized":"2024-09-20T19:32:32.385362+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.305742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"642'899","reported_seq":3559,"reported_epoch":643,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:31.382355+0000","last_change":"2024-09-20T19:32:30.443086+0000","last_active":"2024-09-20T19:32:31.382355+0000","last_peered":"2024-09-20T19:32:31.382355+0000","last_clean":"2024-09-20T19:32:31.382355+0000","last_became_active":"2024-09-20T19:28:48.211105+0000","last_became_peered":"2024-09-20T19:28:48.211105+0000","last_unstale":"2024-09-20T19:32:31.382355+0000","last_undegraded":"2024-09-20T19:32:31.382355+0000","last_fullsized":"2024-09-20T19:32:31.382355+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"483'743","last_scrub_stamp":"2024-09-20T19:29:15.367390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:15.367390+0000","objects_scrubbed":10,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.12673780700000001,"stat_sum":{"num_bytes":24124394,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":43687,"num_write":53,"num_write_kb":16705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f2","length":"1"}]},{"pgid":"4.16","version":"523'15","reported_seq":995,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369213+0000","last_change":"2024-09-20T19:32:16.493984+0000","last_active":"2024-09-20T19:32:32.369213+0000","last_peered":"2024-09-20T19:32:32.369213+0000","last_clean":"2024-09-20T19:32:32.369213+0000","last_became_active":"2024-09-20T19:30:59.447014+0000","last_became_peered":"2024-09-20T19:30:59.447014+0000","last_unstale":"2024-09-20T19:32:32.369213+0000","last_undegraded":"2024-09-20T19:32:32.369213+0000","last_fullsized":"2024-09-20T19:32:32.369213+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:18.681371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1229,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.359362+0000","last_change":"2024-09-20T19:31:41.498323+0000","last_active":"2024-09-20T19:32:31.359362+0000","last_peered":"2024-09-20T19:32:31.359362+0000","last_clean":"2024-09-20T19:32:31.359362+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:32:31.359362+0000","last_undegraded":"2024-09-20T19:32:31.359362+0000","last_fullsized":"2024-09-20T19:32:31.359362+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:41.498237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.498237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.336115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1009,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.388030+0000","last_change":"2024-09-20T19:30:59.429090+0000","last_active":"2024-09-20T19:32:32.388030+0000","last_peered":"2024-09-20T19:32:32.388030+0000","last_clean":"2024-09-20T19:32:32.388030+0000","last_became_active":"2024-09-20T19:30:59.426634+0000","last_became_peered":"2024-09-20T19:30:59.426634+0000","last_unstale":"2024-09-20T19:32:32.388030+0000","last_undegraded":"2024-09-20T19:32:32.388030+0000","last_fullsized":"2024-09-20T19:32:32.388030+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:01.260486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:01.260486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1193,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.360220+0000","last_change":"2024-09-20T19:30:31.368696+0000","last_active":"2024-09-20T19:32:31.360220+0000","last_peered":"2024-09-20T19:32:31.360220+0000","last_clean":"2024-09-20T19:32:31.360220+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:32:31.360220+0000","last_undegraded":"2024-09-20T19:32:31.360220+0000","last_fullsized":"2024-09-20T19:32:31.360220+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:31.849481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:31.849481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:01.544107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"640'940","reported_seq":3416,"reported_epoch":644,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:32.388102+0000","last_change":"2024-09-20T19:32:32.388075+0000","last_active":"2024-09-20T19:32:32.388102+0000","last_peered":"2024-09-20T19:32:32.388102+0000","last_clean":"2024-09-20T19:32:32.388102+0000","last_became_active":"2024-09-20T19:31:02.399794+0000","last_became_peered":"2024-09-20T19:31:02.399794+0000","last_unstale":"2024-09-20T19:32:32.388102+0000","last_undegraded":"2024-09-20T19:32:32.388102+0000","last_fullsized":"2024-09-20T19:32:32.388102+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'769","last_scrub_stamp":"2024-09-20T19:28:51.165603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:51.165603+0000","objects_scrubbed":9,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.061569690000000003,"stat_sum":{"num_bytes":15149270,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":57,"num_read_kb":32313,"num_write":40,"num_write_kb":15381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":13361098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"2.11","version":"0'0","reported_seq":1202,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.372455+0000","last_change":"2024-09-20T19:30:31.386421+0000","last_active":"2024-09-20T19:32:31.372455+0000","last_peered":"2024-09-20T19:32:31.372455+0000","last_clean":"2024-09-20T19:32:31.372455+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:32:31.372455+0000","last_undegraded":"2024-09-20T19:32:31.372455+0000","last_fullsized":"2024-09-20T19:32:31.372455+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.575807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.575807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1048,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.388535+0000","last_change":"2024-09-20T19:31:41.454838+0000","last_active":"2024-09-20T19:32:32.388535+0000","last_peered":"2024-09-20T19:32:32.388535+0000","last_clean":"2024-09-20T19:32:32.388535+0000","last_became_active":"2024-09-20T19:30:59.429434+0000","last_became_peered":"2024-09-20T19:30:59.429434+0000","last_unstale":"2024-09-20T19:32:32.388535+0000","last_undegraded":"2024-09-20T19:32:32.388535+0000","last_fullsized":"2024-09-20T19:32:32.388535+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:54.613627+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1201,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.398921+0000","last_change":"2024-09-20T19:30:59.412406+0000","last_active":"2024-09-20T19:32:31.398921+0000","last_peered":"2024-09-20T19:32:31.398921+0000","last_clean":"2024-09-20T19:32:31.398921+0000","last_became_active":"2024-09-20T19:30:59.411374+0000","last_became_peered":"2024-09-20T19:30:59.411374+0000","last_unstale":"2024-09-20T19:32:31.398921+0000","last_undegraded":"2024-09-20T19:32:31.398921+0000","last_fullsized":"2024-09-20T19:32:31.398921+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:59.345399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:27:59.345399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1181,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.379159+0000","last_change":"2024-09-20T19:30:31.376526+0000","last_active":"2024-09-20T19:32:31.379159+0000","last_peered":"2024-09-20T19:32:31.379159+0000","last_clean":"2024-09-20T19:32:31.379159+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:32:31.379159+0000","last_undegraded":"2024-09-20T19:32:31.379159+0000","last_fullsized":"2024-09-20T19:32:31.379159+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.296694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.296694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"642'526","reported_seq":2992,"reported_epoch":643,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:31.392141+0000","last_change":"2024-09-20T19:32:30.445173+0000","last_active":"2024-09-20T19:32:31.392141+0000","last_peered":"2024-09-20T19:32:31.392141+0000","last_clean":"2024-09-20T19:32:31.392141+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:32:31.392141+0000","last_undegraded":"2024-09-20T19:32:31.392141+0000","last_fullsized":"2024-09-20T19:32:31.392141+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:29.920633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.083670761999999996,"stat_sum":{"num_bytes":11311781,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":11,"num_read_kb":4546,"num_write":6,"num_write_kb":948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f2","length":"1"}]},{"pgid":"4.14","version":"523'20","reported_seq":1058,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.387854+0000","last_change":"2024-09-20T19:31:40.419678+0000","last_active":"2024-09-20T19:32:32.387854+0000","last_peered":"2024-09-20T19:32:32.387854+0000","last_clean":"2024-09-20T19:32:32.387854+0000","last_became_active":"2024-09-20T19:30:59.428000+0000","last_became_peered":"2024-09-20T19:30:59.428000+0000","last_unstale":"2024-09-20T19:32:32.387854+0000","last_undegraded":"2024-09-20T19:32:32.387854+0000","last_fullsized":"2024-09-20T19:32:32.387854+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:55.130318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1210,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.397600+0000","last_change":"2024-09-20T19:30:59.412019+0000","last_active":"2024-09-20T19:32:31.397600+0000","last_peered":"2024-09-20T19:32:31.397600+0000","last_clean":"2024-09-20T19:32:31.397600+0000","last_became_active":"2024-09-20T19:30:59.411617+0000","last_became_peered":"2024-09-20T19:30:59.411617+0000","last_unstale":"2024-09-20T19:32:31.397600+0000","last_undegraded":"2024-09-20T19:32:31.397600+0000","last_fullsized":"2024-09-20T19:32:31.397600+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.424475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.424475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1387,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.398859+0000","last_change":"2024-09-20T19:32:11.594974+0000","last_active":"2024-09-20T19:32:31.398859+0000","last_peered":"2024-09-20T19:32:31.398859+0000","last_clean":"2024-09-20T19:32:31.398859+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:32:31.398859+0000","last_undegraded":"2024-09-20T19:32:31.398859+0000","last_fullsized":"2024-09-20T19:32:31.398859+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:11.594902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.594902+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:28.976985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":977,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.370224+0000","last_change":"2024-09-20T19:32:11.370896+0000","last_active":"2024-09-20T19:32:32.370224+0000","last_peered":"2024-09-20T19:32:32.370224+0000","last_clean":"2024-09-20T19:32:32.370224+0000","last_became_active":"2024-09-20T19:30:59.447481+0000","last_became_peered":"2024-09-20T19:30:59.447481+0000","last_unstale":"2024-09-20T19:32:32.370224+0000","last_undegraded":"2024-09-20T19:32:32.370224+0000","last_fullsized":"2024-09-20T19:32:32.370224+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.301214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"645'356","reported_seq":2632,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.294384+0000","last_change":"2024-09-20T19:32:33.294865+0000","last_active":"2024-09-20T19:32:33.294384+0000","last_peered":"2024-09-20T19:32:33.294384+0000","last_clean":"2024-09-20T19:32:33.294384+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:32:33.294384+0000","last_undegraded":"2024-09-20T19:32:33.294384+0000","last_fullsized":"2024-09-20T19:32:33.294384+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"527'291","last_scrub_stamp":"2024-09-20T19:30:37.025255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:37.025255+0000","objects_scrubbed":5,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003885271,"stat_sum":{"num_bytes":12496367,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":14373,"num_write":23,"num_write_kb":8912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.15","version":"523'15","reported_seq":1237,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.399097+0000","last_change":"2024-09-20T19:30:59.427602+0000","last_active":"2024-09-20T19:32:31.399097+0000","last_peered":"2024-09-20T19:32:31.399097+0000","last_clean":"2024-09-20T19:32:31.399097+0000","last_became_active":"2024-09-20T19:30:59.404598+0000","last_became_peered":"2024-09-20T19:30:59.404598+0000","last_unstale":"2024-09-20T19:32:31.399097+0000","last_undegraded":"2024-09-20T19:32:31.399097+0000","last_fullsized":"2024-09-20T19:32:31.399097+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:27:22.506346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:22.506346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":950,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369544+0000","last_change":"2024-09-20T19:32:26.439201+0000","last_active":"2024-09-20T19:32:32.369544+0000","last_peered":"2024-09-20T19:32:32.369544+0000","last_clean":"2024-09-20T19:32:32.369544+0000","last_became_active":"2024-09-20T19:30:59.448308+0000","last_became_peered":"2024-09-20T19:30:59.448308+0000","last_unstale":"2024-09-20T19:32:32.369544+0000","last_undegraded":"2024-09-20T19:32:32.369544+0000","last_fullsized":"2024-09-20T19:32:32.369544+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.836552+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"554'13","reported_seq":1176,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.387621+0000","last_change":"2024-09-20T19:30:59.427899+0000","last_active":"2024-09-20T19:32:32.387621+0000","last_peered":"2024-09-20T19:32:32.387621+0000","last_clean":"2024-09-20T19:32:32.387621+0000","last_became_active":"2024-09-20T19:30:59.425446+0000","last_became_peered":"2024-09-20T19:30:59.425446+0000","last_unstale":"2024-09-20T19:32:32.387621+0000","last_undegraded":"2024-09-20T19:32:32.387621+0000","last_fullsized":"2024-09-20T19:32:32.387621+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:28:18.243973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:18.243973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1027,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.290852+0000","last_change":"2024-09-20T19:31:19.806432+0000","last_active":"2024-09-20T19:32:33.290852+0000","last_peered":"2024-09-20T19:32:33.290852+0000","last_clean":"2024-09-20T19:32:33.290852+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:32:33.290852+0000","last_undegraded":"2024-09-20T19:32:33.290852+0000","last_fullsized":"2024-09-20T19:32:33.290852+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:19.806369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.806369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"599'455","reported_seq":2984,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.376457+0000","last_change":"2024-09-20T19:32:30.442859+0000","last_active":"2024-09-20T19:32:31.376457+0000","last_peered":"2024-09-20T19:32:31.376457+0000","last_clean":"2024-09-20T19:32:31.376457+0000","last_became_active":"2024-09-20T19:31:00.440451+0000","last_became_peered":"2024-09-20T19:31:00.440451+0000","last_unstale":"2024-09-20T19:32:31.376457+0000","last_undegraded":"2024-09-20T19:32:31.376457+0000","last_fullsized":"2024-09-20T19:32:31.376457+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.088434788,"stat_sum":{"num_bytes":3428727,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.12","version":"523'15","reported_seq":1232,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.365819+0000","last_change":"2024-09-20T19:32:31.080255+0000","last_active":"2024-09-20T19:32:31.365819+0000","last_peered":"2024-09-20T19:32:31.365819+0000","last_clean":"2024-09-20T19:32:31.365819+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:32:31.365819+0000","last_undegraded":"2024-09-20T19:32:31.365819+0000","last_fullsized":"2024-09-20T19:32:31.365819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:31.080102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.080102+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.008745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.371528+0000","last_change":"2024-09-20T19:30:59.412393+0000","last_active":"2024-09-20T19:32:31.371528+0000","last_peered":"2024-09-20T19:32:31.371528+0000","last_clean":"2024-09-20T19:32:31.371528+0000","last_became_active":"2024-09-20T19:30:59.412120+0000","last_became_peered":"2024-09-20T19:30:59.412120+0000","last_unstale":"2024-09-20T19:32:31.371528+0000","last_undegraded":"2024-09-20T19:32:31.371528+0000","last_fullsized":"2024-09-20T19:32:31.371528+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.625450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.625450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1047,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385800+0000","last_change":"2024-09-20T19:31:35.565401+0000","last_active":"2024-09-20T19:32:32.385800+0000","last_peered":"2024-09-20T19:32:32.385800+0000","last_clean":"2024-09-20T19:32:32.385800+0000","last_became_active":"2024-09-20T19:30:59.432219+0000","last_became_peered":"2024-09-20T19:30:59.432219+0000","last_unstale":"2024-09-20T19:32:32.385800+0000","last_undegraded":"2024-09-20T19:32:32.385800+0000","last_fullsized":"2024-09-20T19:32:32.385800+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.758533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":959,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.364657+0000","last_change":"2024-09-20T19:30:09.763911+0000","last_active":"2024-09-20T19:32:32.364657+0000","last_peered":"2024-09-20T19:32:32.364657+0000","last_clean":"2024-09-20T19:32:32.364657+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:32:32.364657+0000","last_undegraded":"2024-09-20T19:32:32.364657+0000","last_fullsized":"2024-09-20T19:32:32.364657+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:31.041093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.041093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:59.430814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"632'591","reported_seq":3063,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385535+0000","last_change":"2024-09-20T19:32:32.386214+0000","last_active":"2024-09-20T19:32:32.385535+0000","last_peered":"2024-09-20T19:32:32.385535+0000","last_clean":"2024-09-20T19:32:32.385535+0000","last_became_active":"2024-09-20T19:31:48.284326+0000","last_became_peered":"2024-09-20T19:31:48.284326+0000","last_unstale":"2024-09-20T19:32:32.385535+0000","last_undegraded":"2024-09-20T19:32:32.385535+0000","last_fullsized":"2024-09-20T19:32:32.385535+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'561","last_scrub_stamp":"2024-09-20T19:28:49.166999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:49.166999+0000","objects_scrubbed":1,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00097392799999999999,"stat_sum":{"num_bytes":3473963,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":5242,"num_write":10,"num_write_kb":2213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3019792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.13","version":"523'19","reported_seq":1042,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.287779+0000","last_change":"2024-09-20T19:32:32.189200+0000","last_active":"2024-09-20T19:32:33.287779+0000","last_peered":"2024-09-20T19:32:33.287779+0000","last_clean":"2024-09-20T19:32:33.287779+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:32:33.287779+0000","last_undegraded":"2024-09-20T19:32:33.287779+0000","last_fullsized":"2024-09-20T19:32:33.287779+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:32.189122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.189122+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.082120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1221,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.368307+0000","last_change":"2024-09-20T19:31:20.343942+0000","last_active":"2024-09-20T19:32:31.368307+0000","last_peered":"2024-09-20T19:32:31.368307+0000","last_clean":"2024-09-20T19:32:31.368307+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:32:31.368307+0000","last_undegraded":"2024-09-20T19:32:31.368307+0000","last_fullsized":"2024-09-20T19:32:31.368307+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:20.343882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.343882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.923955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1201,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.365692+0000","last_change":"2024-09-20T19:31:45.587402+0000","last_active":"2024-09-20T19:32:31.365692+0000","last_peered":"2024-09-20T19:32:31.365692+0000","last_clean":"2024-09-20T19:32:31.365692+0000","last_became_active":"2024-09-20T19:31:45.586887+0000","last_became_peered":"2024-09-20T19:31:45.586887+0000","last_unstale":"2024-09-20T19:32:31.365692+0000","last_undegraded":"2024-09-20T19:32:31.365692+0000","last_fullsized":"2024-09-20T19:32:31.365692+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.571921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.571921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:54.116440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1043,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.384875+0000","last_change":"2024-09-20T19:31:36.528455+0000","last_active":"2024-09-20T19:32:32.384875+0000","last_peered":"2024-09-20T19:32:32.384875+0000","last_clean":"2024-09-20T19:32:32.384875+0000","last_became_active":"2024-09-20T19:30:59.431451+0000","last_became_peered":"2024-09-20T19:30:59.431451+0000","last_unstale":"2024-09-20T19:32:32.384875+0000","last_undegraded":"2024-09-20T19:32:32.384875+0000","last_fullsized":"2024-09-20T19:32:32.384875+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.528327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.528327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.129198+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"638'418","reported_seq":2855,"reported_epoch":644,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:32.369111+0000","last_change":"2024-09-20T19:32:32.368968+0000","last_active":"2024-09-20T19:32:32.369111+0000","last_peered":"2024-09-20T19:32:32.369111+0000","last_clean":"2024-09-20T19:32:32.369111+0000","last_became_active":"2024-09-20T19:30:12.800137+0000","last_became_peered":"2024-09-20T19:30:12.800137+0000","last_unstale":"2024-09-20T19:32:32.369111+0000","last_undegraded":"2024-09-20T19:32:32.369111+0000","last_fullsized":"2024-09-20T19:32:32.369111+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"524'363","last_scrub_stamp":"2024-09-20T19:30:33.692750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:33.692750+0000","objects_scrubbed":3,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0093286670000000006,"stat_sum":{"num_bytes":4672643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":15,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"4.10","version":"523'8","reported_seq":1051,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385103+0000","last_change":"2024-09-20T19:32:18.830939+0000","last_active":"2024-09-20T19:32:32.385103+0000","last_peered":"2024-09-20T19:32:32.385103+0000","last_clean":"2024-09-20T19:32:32.385103+0000","last_became_active":"2024-09-20T19:30:59.418597+0000","last_became_peered":"2024-09-20T19:30:59.418597+0000","last_unstale":"2024-09-20T19:32:32.385103+0000","last_undegraded":"2024-09-20T19:32:32.385103+0000","last_fullsized":"2024-09-20T19:32:32.385103+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:32:18.830843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.830843+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.107094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1182,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.399082+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:32:31.399082+0000","last_peered":"2024-09-20T19:32:31.399082+0000","last_clean":"2024-09-20T19:32:31.399082+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:32:31.399082+0000","last_undegraded":"2024-09-20T19:32:31.399082+0000","last_fullsized":"2024-09-20T19:32:31.399082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:57.565392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1194,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.371592+0000","last_change":"2024-09-20T19:30:31.387063+0000","last_active":"2024-09-20T19:32:31.371592+0000","last_peered":"2024-09-20T19:32:31.371592+0000","last_clean":"2024-09-20T19:32:31.371592+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:32:31.371592+0000","last_undegraded":"2024-09-20T19:32:31.371592+0000","last_fullsized":"2024-09-20T19:32:31.371592+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.544996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.544996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1182,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.377806+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:32:31.377806+0000","last_peered":"2024-09-20T19:32:31.377806+0000","last_clean":"2024-09-20T19:32:31.377806+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:32:31.377806+0000","last_undegraded":"2024-09-20T19:32:31.377806+0000","last_fullsized":"2024-09-20T19:32:31.377806+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:00.258729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":2592,"reported_epoch":643,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:31.368156+0000","last_change":"2024-09-20T19:32:30.439239+0000","last_active":"2024-09-20T19:32:31.368156+0000","last_peered":"2024-09-20T19:32:31.368156+0000","last_clean":"2024-09-20T19:32:31.368156+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:32:31.368156+0000","last_undegraded":"2024-09-20T19:32:31.368156+0000","last_fullsized":"2024-09-20T19:32:31.368156+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:29:00.472909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.472909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0934241149999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ce","length":"1"},{"start":"dc","length":"1"},{"start":"df","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.11","version":"523'19","reported_seq":1066,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385139+0000","last_change":"2024-09-20T19:32:16.785202+0000","last_active":"2024-09-20T19:32:32.385139+0000","last_peered":"2024-09-20T19:32:32.385139+0000","last_clean":"2024-09-20T19:32:32.385139+0000","last_became_active":"2024-09-20T19:30:59.426703+0000","last_became_peered":"2024-09-20T19:30:59.426703+0000","last_unstale":"2024-09-20T19:32:32.385139+0000","last_undegraded":"2024-09-20T19:32:32.385139+0000","last_fullsized":"2024-09-20T19:32:32.385139+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.001001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1198,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.373777+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:32:31.373777+0000","last_peered":"2024-09-20T19:32:31.373777+0000","last_clean":"2024-09-20T19:32:31.373777+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:32:31.373777+0000","last_undegraded":"2024-09-20T19:32:31.373777+0000","last_fullsized":"2024-09-20T19:32:31.373777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1202,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.376213+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:32:31.376213+0000","last_peered":"2024-09-20T19:32:31.376213+0000","last_clean":"2024-09-20T19:32:31.376213+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:32:31.376213+0000","last_undegraded":"2024-09-20T19:32:31.376213+0000","last_fullsized":"2024-09-20T19:32:31.376213+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.725052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1051,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385608+0000","last_change":"2024-09-20T19:30:59.432385+0000","last_active":"2024-09-20T19:32:32.385608+0000","last_peered":"2024-09-20T19:32:32.385608+0000","last_clean":"2024-09-20T19:32:32.385608+0000","last_became_active":"2024-09-20T19:30:59.431821+0000","last_became_peered":"2024-09-20T19:30:59.431821+0000","last_unstale":"2024-09-20T19:32:32.385608+0000","last_undegraded":"2024-09-20T19:32:32.385608+0000","last_fullsized":"2024-09-20T19:32:32.385608+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.682353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.682353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:57.593210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":1180,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.380760+0000","last_change":"2024-09-20T19:28:13.311223+0000","last_active":"2024-09-20T19:32:31.380760+0000","last_peered":"2024-09-20T19:32:31.380760+0000","last_clean":"2024-09-20T19:32:31.380760+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:32:31.380760+0000","last_undegraded":"2024-09-20T19:32:31.380760+0000","last_fullsized":"2024-09-20T19:32:31.380760+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:28:13.311036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:13.311036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1217,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.401371+0000","last_change":"2024-09-20T19:32:31.401371+0000","last_active":"2024-09-20T19:32:31.401371+0000","last_peered":"2024-09-20T19:32:31.401371+0000","last_clean":"2024-09-20T19:32:31.401371+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:32:31.401371+0000","last_undegraded":"2024-09-20T19:32:31.401371+0000","last_fullsized":"2024-09-20T19:32:31.401371+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:31.401257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.401257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.241223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":984,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369334+0000","last_change":"2024-09-20T19:31:36.551849+0000","last_active":"2024-09-20T19:32:32.369334+0000","last_peered":"2024-09-20T19:32:32.369334+0000","last_clean":"2024-09-20T19:32:32.369334+0000","last_became_active":"2024-09-20T19:30:59.447209+0000","last_became_peered":"2024-09-20T19:30:59.447209+0000","last_unstale":"2024-09-20T19:32:32.369334+0000","last_undegraded":"2024-09-20T19:32:32.369334+0000","last_fullsized":"2024-09-20T19:32:32.369334+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:36.551731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.551731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.578982+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1391,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.373259+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:32:31.373259+0000","last_peered":"2024-09-20T19:32:31.373259+0000","last_clean":"2024-09-20T19:32:31.373259+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:32:31.373259+0000","last_undegraded":"2024-09-20T19:32:31.373259+0000","last_fullsized":"2024-09-20T19:32:31.373259+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1201,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.367594+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:32:31.367594+0000","last_peered":"2024-09-20T19:32:31.367594+0000","last_clean":"2024-09-20T19:32:31.367594+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:32:31.367594+0000","last_undegraded":"2024-09-20T19:32:31.367594+0000","last_fullsized":"2024-09-20T19:32:31.367594+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:37.323926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":1064,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385214+0000","last_change":"2024-09-20T19:30:59.433439+0000","last_active":"2024-09-20T19:32:32.385214+0000","last_peered":"2024-09-20T19:32:32.385214+0000","last_clean":"2024-09-20T19:32:32.385214+0000","last_became_active":"2024-09-20T19:30:59.432846+0000","last_became_peered":"2024-09-20T19:30:59.432846+0000","last_unstale":"2024-09-20T19:32:32.385214+0000","last_undegraded":"2024-09-20T19:32:32.385214+0000","last_fullsized":"2024-09-20T19:32:32.385214+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:24:30.106245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.106245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:42.482721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1248,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.373525+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:32:31.373525+0000","last_peered":"2024-09-20T19:32:31.373525+0000","last_clean":"2024-09-20T19:32:31.373525+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:32:31.373525+0000","last_undegraded":"2024-09-20T19:32:31.373525+0000","last_fullsized":"2024-09-20T19:32:31.373525+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":982,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.369906+0000","last_change":"2024-09-20T19:32:20.455596+0000","last_active":"2024-09-20T19:32:32.369906+0000","last_peered":"2024-09-20T19:32:32.369906+0000","last_clean":"2024-09-20T19:32:32.369906+0000","last_became_active":"2024-09-20T19:30:09.786731+0000","last_became_peered":"2024-09-20T19:30:09.786731+0000","last_unstale":"2024-09-20T19:32:32.369906+0000","last_undegraded":"2024-09-20T19:32:32.369906+0000","last_fullsized":"2024-09-20T19:32:32.369906+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.455533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.455533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.459393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1025,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.290727+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:32:33.290727+0000","last_peered":"2024-09-20T19:32:33.290727+0000","last_clean":"2024-09-20T19:32:33.290727+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:32:33.290727+0000","last_undegraded":"2024-09-20T19:32:33.290727+0000","last_fullsized":"2024-09-20T19:32:33.290727+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:00.429934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1209,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.370916+0000","last_change":"2024-09-20T19:32:15.866641+0000","last_active":"2024-09-20T19:32:31.370916+0000","last_peered":"2024-09-20T19:32:31.370916+0000","last_clean":"2024-09-20T19:32:31.370916+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:32:31.370916+0000","last_undegraded":"2024-09-20T19:32:31.370916+0000","last_fullsized":"2024-09-20T19:32:31.370916+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.866572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.866572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.064376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":1260,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.361061+0000","last_change":"2024-09-20T19:30:59.448664+0000","last_active":"2024-09-20T19:32:31.361061+0000","last_peered":"2024-09-20T19:32:31.361061+0000","last_clean":"2024-09-20T19:32:31.361061+0000","last_became_active":"2024-09-20T19:30:59.414544+0000","last_became_peered":"2024-09-20T19:30:59.414544+0000","last_unstale":"2024-09-20T19:32:31.361061+0000","last_undegraded":"2024-09-20T19:32:31.361061+0000","last_fullsized":"2024-09-20T19:32:31.361061+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:28:02.451854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.451854+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:46.533831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1212,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-20T19:32:31.367955+0000","last_change":"2024-09-20T19:32:16.075391+0000","last_active":"2024-09-20T19:32:31.367955+0000","last_peered":"2024-09-20T19:32:31.367955+0000","last_clean":"2024-09-20T19:32:31.367955+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:32:31.367955+0000","last_undegraded":"2024-09-20T19:32:31.367955+0000","last_fullsized":"2024-09-20T19:32:31.367955+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.075336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.075336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.476752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1005,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.388412+0000","last_change":"2024-09-20T19:30:59.414333+0000","last_active":"2024-09-20T19:32:32.388412+0000","last_peered":"2024-09-20T19:32:32.388412+0000","last_clean":"2024-09-20T19:32:32.388412+0000","last_became_active":"2024-09-20T19:30:59.413650+0000","last_became_peered":"2024-09-20T19:30:59.413650+0000","last_unstale":"2024-09-20T19:32:32.388412+0000","last_undegraded":"2024-09-20T19:32:32.388412+0000","last_fullsized":"2024-09-20T19:32:32.388412+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:54.253853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.253853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1016,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.385715+0000","last_change":"2024-09-20T19:30:59.425869+0000","last_active":"2024-09-20T19:32:32.385715+0000","last_peered":"2024-09-20T19:32:32.385715+0000","last_clean":"2024-09-20T19:32:32.385715+0000","last_became_active":"2024-09-20T19:30:59.424644+0000","last_became_peered":"2024-09-20T19:30:59.424644+0000","last_unstale":"2024-09-20T19:32:32.385715+0000","last_undegraded":"2024-09-20T19:32:32.385715+0000","last_fullsized":"2024-09-20T19:32:32.385715+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:58.281416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:58.281416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":1024,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-20T19:32:32.387699+0000","last_change":"2024-09-20T19:32:10.800441+0000","last_active":"2024-09-20T19:32:32.387699+0000","last_peered":"2024-09-20T19:32:32.387699+0000","last_clean":"2024-09-20T19:32:32.387699+0000","last_became_active":"2024-09-20T19:30:59.418441+0000","last_became_peered":"2024-09-20T19:30:59.418441+0000","last_unstale":"2024-09-20T19:32:32.387699+0000","last_undegraded":"2024-09-20T19:32:32.387699+0000","last_fullsized":"2024-09-20T19:32:32.387699+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:10.800347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.800347+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.402765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1023,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-20T19:32:33.289014+0000","last_change":"2024-09-20T19:32:33.198084+0000","last_active":"2024-09-20T19:32:33.289014+0000","last_peered":"2024-09-20T19:32:33.289014+0000","last_clean":"2024-09-20T19:32:33.289014+0000","last_became_active":"2024-09-20T19:30:59.409071+0000","last_became_peered":"2024-09-20T19:30:59.409071+0000","last_unstale":"2024-09-20T19:32:33.289014+0000","last_undegraded":"2024-09-20T19:32:33.289014+0000","last_fullsized":"2024-09-20T19:32:33.289014+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:33.198003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:33.198003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.209339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":266466335,"num_objects":128,"num_object_clones":80,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":9,"num_read":1672,"num_read_kb":1273180,"num_write":1236,"num_write_kb":461124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":88476679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564367360,"data_stored":763040877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":14049,"ondisk_log_size":14049,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":847,"num_read_kb":842,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907126,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144092,"kb_used_data":90692,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93601060,"statfs":{"total":95995035648,"available":95847485440,"internally_reserved":0,"allocated":92868608,"data_stored":114498738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234177,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95636,"kb_used_data":64448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93649516,"statfs":{"total":95995035648,"available":95897104384,"internally_reserved":0,"allocated":65994752,"data_stored":84701565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561227,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91372,"kb_used_data":64336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93653780,"statfs":{"total":95995035648,"available":95901470720,"internally_reserved":0,"allocated":65880064,"data_stored":92621132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928538,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84752,"kb_used_data":52728,"kb_used_omap":35,"kb_used_meta":31964,"kb_avail":93660400,"statfs":{"total":95995035648,"available":95908249600,"internally_reserved":0,"allocated":53993472,"data_stored":72047812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":32732081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":550,"seq":2362232012820,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124460,"kb_used_data":87568,"kb_used_omap":51,"kb_used_meta":36812,"kb_avail":93620692,"statfs":{"total":95995035648,"available":95867588608,"internally_reserved":0,"allocated":89669632,"data_stored":113988084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52777,"internal_metadata":37695959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378863,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129724,"kb_used_data":84644,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93615428,"statfs":{"total":95995035648,"available":95862198272,"internally_reserved":0,"allocated":86675456,"data_stored":114441195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126920,"kb_used_data":73520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93618232,"statfs":{"total":95995035648,"available":95865069568,"internally_reserved":0,"allocated":75284480,"data_stored":99398853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648030,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106300,"kb_used_data":69024,"kb_used_omap":49,"kb_used_meta":37198,"kb_avail":93638852,"statfs":{"total":95995035648,"available":95886184448,"internally_reserved":0,"allocated":70680576,"data_stored":91797900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":38091305},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67272704,"data_stored":91353576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71675904,"data_stored":97905194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81760256,"data_stored":111388443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84414464,"data_stored":110607952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49324032,"data_stored":69243699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":60661760,"data_stored":89248072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61362176,"data_stored":81912556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":87896064,"data_stored":111381385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:32:37.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:32:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:37.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:37.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:38.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:38.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:38.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:38.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:38.288 INFO:tasks.rados.rados.0.smithi042.stdout:2140: done (0 left) 2024-09-20T19:32:38.288 INFO:tasks.rados.rados.0.smithi042.stdout:2141: read oid 7 snap -1 2024-09-20T19:32:38.288 INFO:tasks.rados.rados.0.smithi042.stdout:2141: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:38.294 INFO:tasks.rados.rados.0.smithi042.stdout:2142: write (excl) oid 30 current snap is 249 2024-09-20T19:32:38.294 INFO:tasks.rados.rados.0.smithi042.stdout:2142: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-20T19:32:38.312 INFO:tasks.rados.rados.0.smithi042.stdout:2142: writing smithi04268048-30 from 605250 to 1244594 tid 1 2024-09-20T19:32:38.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:38.348 INFO:tasks.rados.rados.0.smithi042.stdout:2142: writing smithi04268048-30 from 1855142 to 2307721 tid 2 2024-09-20T19:32:38.348 INFO:tasks.rados.rados.0.smithi042.stdout:2143: copy_from oid 10 from oid 36 current snap is 249 2024-09-20T19:32:38.349 INFO:tasks.rados.rados.0.smithi042.stdout:2141: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:38.367 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:38.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:38.413 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 621478 2024-09-20T19:32:38.435 INFO:tasks.rados.rados.0.smithi042.stdout:2142: finishing write tid 1 to smithi04268048-30 2024-09-20T19:32:38.435 INFO:tasks.rados.rados.0.smithi042.stdout:2141: done (2 left) 2024-09-20T19:32:38.435 INFO:tasks.rados.rados.0.smithi042.stdout:2144: read oid 46 snap -1 2024-09-20T19:32:38.435 INFO:tasks.rados.rados.0.smithi042.stdout:2144: expect (ObjNum 675 snap 216 seq_num 675) 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2145: read oid 38 snap -1 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2145: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2146: delete oid 42 current snap is 249 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: finishing write tid 2 to smithi04268048-30 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: finishing write tid 3 to smithi04268048-30 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: finishing write tid 4 to smithi04268048-30 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: oid 30 updating version 0 to 949 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: oid 30 version 949 is already newer than 948 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: oid 30 version 949 is already newer than 947 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 949 (ObjNum 750 snap 249 seq_num 750) dirty exists 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2142: left oid 30 (ObjNum 750 snap 249 seq_num 750) 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2143: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:32:38.436 INFO:tasks.rados.rados.0.smithi042.stdout:2143: finishing copy_from to smithi04268048-10 2024-09-20T19:32:38.437 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 574 (ObjNum 733 snap 242 seq_num 733) dirty exists 2024-09-20T19:32:38.439 INFO:tasks.rados.rados.0.smithi042.stdout:2142: done (4 left) 2024-09-20T19:32:38.439 INFO:tasks.rados.rados.0.smithi042.stdout:2143: done (3 left) 2024-09-20T19:32:38.439 INFO:tasks.rados.rados.0.smithi042.stdout:2146: done (2 left) 2024-09-20T19:32:38.439 INFO:tasks.rados.rados.0.smithi042.stdout:2147: delete oid 35 current snap is 249 2024-09-20T19:32:38.441 INFO:tasks.rados.rados.0.smithi042.stdout:2145: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:32:38.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:38.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:38.591 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 204313 2024-09-20T19:32:38.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:38.695 INFO:tasks.rados.rados.0.smithi042.stdout:2144: expect (ObjNum 675 snap 216 seq_num 675) 2024-09-20T19:32:38.696 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[32075]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[32075]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[32075]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[32075]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.696 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[22706]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[22706]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[22706]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:38 smithi042 bash[22706]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.705 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:38.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:38.741 INFO:tasks.rados.rados.0.smithi042.stdout:2144: done (2 left) 2024-09-20T19:32:38.741 INFO:tasks.rados.rados.0.smithi042.stdout:2145: done (1 left) 2024-09-20T19:32:38.741 INFO:tasks.rados.rados.0.smithi042.stdout:2147: done (0 left) 2024-09-20T19:32:38.741 INFO:tasks.rados.rados.0.smithi042.stdout:2148: snap_create 2024-09-20T19:32:38.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:38 smithi162 bash[24557]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:38 smithi162 bash[24557]: cluster 2024-09-20T19:32:37.408541+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-20T19:32:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:38 smithi162 bash[24557]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:38 smithi162 bash[24557]: cluster 2024-09-20T19:32:38.286185+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-20T19:32:38.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:38.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:38.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:39.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:39.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:39.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:39.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:39.410 INFO:tasks.rados.rados.0.smithi042.stdout:2148: done (0 left) 2024-09-20T19:32:39.410 INFO:tasks.rados.rados.0.smithi042.stdout:2149: snap_create 2024-09-20T19:32:39.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:39.482 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:39.489 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:39 smithi162 bash[24557]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.489 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:39 smithi162 bash[24557]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:39.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:39 smithi042 bash[32075]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:39 smithi042 bash[32075]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:39 smithi042 bash[22706]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:39 smithi042 bash[22706]: cluster 2024-09-20T19:32:38.315423+0000 mgr.y (mgr.24491) 552 : cluster [DBG] pgmap v973: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2024-09-20T19:32:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:39.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:39.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:39.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:39.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:39.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:39.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:40.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:40.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:40.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:40.156 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:40.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:40.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:40.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2149: done (0 left) 2024-09-20T19:32:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2150: write (excl) oid 36 current snap is 251 2024-09-20T19:32:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2150: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-20T19:32:40.423 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:40.423 INFO:tasks.rados.rados.0.smithi042.stdout:2150: writing smithi04268048-36 from 655671 to 1237115 tid 1 2024-09-20T19:32:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:40.432 INFO:tasks.rados.rados.0.smithi042.stdout:2150: writing smithi04268048-36 from 2030588 to 2655671 tid 2 2024-09-20T19:32:40.433 INFO:tasks.rados.rados.0.smithi042.stdout:2151: delete oid 30 current snap is 251 2024-09-20T19:32:40.438 INFO:tasks.rados.rados.0.smithi042.stdout:2151: done (1 left) 2024-09-20T19:32:40.438 INFO:tasks.rados.rados.0.smithi042.stdout:2152: read oid 39 snap -1 2024-09-20T19:32:40.438 INFO:tasks.rados.rados.0.smithi042.stdout:2152: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:40.438 INFO:tasks.rados.rados.0.smithi042.stdout:2153: snap_create 2024-09-20T19:32:40.442 INFO:tasks.rados.rados.0.smithi042.stdout:2150: finishing write tid 1 to smithi04268048-36 2024-09-20T19:32:40.447 INFO:tasks.rados.rados.0.smithi042.stdout:2152: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:40.477 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 381385 2024-09-20T19:32:40.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:40.540 INFO:tasks.rados.rados.0.smithi042.stdout:2150: finishing write tid 2 to smithi04268048-36 2024-09-20T19:32:40.540 INFO:tasks.rados.rados.0.smithi042.stdout:2150: finishing write tid 3 to smithi04268048-36 2024-09-20T19:32:40.540 INFO:tasks.rados.rados.0.smithi042.stdout:2150: finishing write tid 4 to smithi04268048-36 2024-09-20T19:32:40.541 INFO:tasks.rados.rados.0.smithi042.stdout:2150: oid 36 updating version 0 to 576 2024-09-20T19:32:40.541 INFO:tasks.rados.rados.0.smithi042.stdout:2150: oid 36 updating version 576 to 577 2024-09-20T19:32:40.541 INFO:tasks.rados.rados.0.smithi042.stdout:2150: oid 36 version 577 is already newer than 575 2024-09-20T19:32:40.541 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 577 (ObjNum 753 snap 251 seq_num 753) dirty exists 2024-09-20T19:32:40.541 INFO:tasks.rados.rados.0.smithi042.stdout:2150: left oid 36 (ObjNum 753 snap 251 seq_num 753) 2024-09-20T19:32:40.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:40 smithi162 bash[24557]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:40 smithi162 bash[24557]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:40 smithi162 bash[24557]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.542 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:40 smithi162 bash[24557]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:40.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:40.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[22706]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[22706]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[22706]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[22706]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[32075]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[32075]: cluster 2024-09-20T19:32:39.408436+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[32075]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:40 smithi042 bash[32075]: audit 2024-09-20T19:32:39.605580+0000 mgr.y (mgr.24491) 553 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:40.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:40.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:40.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:40.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:40.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:41.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:41.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:41.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:41.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:41.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:2150: done (2 left) 2024-09-20T19:32:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:2152: done (1 left) 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2153: done (0 left) 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2154: read oid 30 snap -1 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2154: expect deleted 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2155: read oid 26 snap -1 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2155: expect (ObjNum 723 snap 236 seq_num 723) 2024-09-20T19:32:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2156: snap_remove snap 245 2024-09-20T19:32:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:41.505 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:32:41.505 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:32:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:41.523 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:32:41.524 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:32:41.543 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:32:41.543 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:32:41.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:41.561 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:32:41.561 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:32:41.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:41.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[22706]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[22706]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[22706]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[22706]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[32075]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[32075]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[32075]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:41 smithi042 bash[32075]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.583 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:32:41.583 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:32:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:41.603 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:32:41.603 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:32:41.624 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:32:41.624 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:32:41.651 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:32:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:32:41.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:41 smithi162 bash[24557]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:41 smithi162 bash[24557]: cluster 2024-09-20T19:32:40.316190+0000 mgr.y (mgr.24491) 554 : cluster [DBG] pgmap v975: 168 pgs: 12 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 147 active+clean; 265 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2024-09-20T19:32:41.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:41 smithi162 bash[24557]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:41 smithi162 bash[24557]: cluster 2024-09-20T19:32:40.413524+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-20T19:32:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:41.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.841 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:41.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:41.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:41.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:42.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:42.104 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:42.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:42.338 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.367 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2156: done (2 left) 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2157: read oid 19 snap -1 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2157: expect deleted 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2158: read oid 18 snap -1 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2158: expect (ObjNum 681 snap 221 seq_num 681) 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2159: copy_from oid 38 from oid 12 current snap is 252 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2160: rollback oid 8 current snap is 252 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 249 2024-09-20T19:32:42.440 INFO:tasks.rados.rados.0.smithi042.stdout:2161: write oid 32 current snap is 252 2024-09-20T19:32:42.441 INFO:tasks.rados.rados.0.smithi042.stdout:2161: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-20T19:32:42.448 INFO:tasks.rados.rados.0.smithi042.stdout:2161: writing smithi04268048-32 from 689285 to 1421227 tid 1 2024-09-20T19:32:42.456 INFO:tasks.rados.rados.0.smithi042.stdout:2161: writing smithi04268048-32 from 2147552 to 2689285 tid 2 2024-09-20T19:32:42.456 INFO:tasks.rados.rados.0.smithi042.stdout:2155: expect (ObjNum 723 snap 236 seq_num 723) 2024-09-20T19:32:42.456 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:42.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:42.502 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 909843 2024-09-20T19:32:42.553 INFO:tasks.rados.rados.0.smithi042.stdout:2158: expect (ObjNum 681 snap 221 seq_num 681) 2024-09-20T19:32:42.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[32075]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.607 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.608 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:42 smithi042 bash[22706]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:42.720 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:32:42.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2161: finishing write tid 1 to smithi04268048-32 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2160: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 722 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2161: finishing write tid 2 to smithi04268048-32 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2154: done (6 left) 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2155: done (5 left) 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2157: done (4 left) 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2158: done (3 left) 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2160: done (2 left) 2024-09-20T19:32:42.733 INFO:tasks.rados.rados.0.smithi042.stdout:2162: rollback oid 11 current snap is 252 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 237 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: finishing write tid 3 to smithi04268048-32 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: finishing write tid 4 to smithi04268048-32 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: oid 32 updating version 0 to 1157 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: oid 32 version 1157 is already newer than 1155 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: oid 32 version 1157 is already newer than 1156 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1157 (ObjNum 755 snap 252 seq_num 755) dirty exists 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: left oid 32 (ObjNum 755 snap 252 seq_num 755) 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2159: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2159: finishing copy_from to smithi04268048-38 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 721 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2159: done (2 left) 2024-09-20T19:32:42.734 INFO:tasks.rados.rados.0.smithi042.stdout:2161: done (1 left) 2024-09-20T19:32:42.735 INFO:tasks.rados.rados.0.smithi042.stdout:2163: snap_remove snap 235 2024-09-20T19:32:42.735 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.736 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: cluster 2024-09-20T19:32:41.435439+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-20T19:32:42.736 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.736 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: audit 2024-09-20T19:32:41.486006+0000 mon.a (mon.0) 2076 : audit [DBG] from='client.? v1:172.21.15.42:0/836992619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:42.736 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.736 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:42 smithi162 bash[24557]: cluster 2024-09-20T19:32:42.438480+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-20T19:32:42.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:42.852 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.878 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:42.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:42.980 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:43.003 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:43.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:43.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:43.236 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:2163: done (1 left) 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:2164: write (excl) oid 34 current snap is 252 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:2162: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:2162: finishing rollback tid 2 to smithi04268048-11 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 839 (ObjNum 716 snap 234 seq_num 716) dirty exists 2024-09-20T19:32:43.283 INFO:tasks.rados.rados.0.smithi042.stdout:2164: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-20T19:32:43.290 INFO:tasks.rados.rados.0.smithi042.stdout:2164: writing smithi04268048-34 from 706092 to 1113283 tid 1 2024-09-20T19:32:43.297 INFO:tasks.rados.rados.0.smithi042.stdout:2164: writing smithi04268048-34 from 1806034 to 2269620 tid 2 2024-09-20T19:32:43.298 INFO:tasks.rados.rados.0.smithi042.stdout:2162: done (1 left) 2024-09-20T19:32:43.298 INFO:tasks.rados.rados.0.smithi042.stdout:2165: read oid 50 snap -1 2024-09-20T19:32:43.298 INFO:tasks.rados.rados.0.smithi042.stdout:2165: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:32:43.298 INFO:tasks.rados.rados.0.smithi042.stdout:2166: snap_create 2024-09-20T19:32:43.319 INFO:tasks.rados.rados.0.smithi042.stdout:2164: finishing write tid 1 to smithi04268048-34 2024-09-20T19:32:43.320 INFO:tasks.rados.rados.0.smithi042.stdout:2164: finishing write tid 2 to smithi04268048-34 2024-09-20T19:32:43.320 INFO:tasks.rados.rados.0.smithi042.stdout:2164: finishing write tid 3 to smithi04268048-34 2024-09-20T19:32:43.320 INFO:tasks.rados.rados.0.smithi042.stdout:2165: expect (ObjNum 429 snap 133 seq_num 429) 2024-09-20T19:32:43.320 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:43.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:43.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:2164: finishing write tid 4 to smithi04268048-34 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:2164: oid 34 updating version 0 to 1052 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:2164: oid 34 version 1052 is already newer than 1050 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:2164: oid 34 version 1052 is already newer than 1051 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1052 (ObjNum 756 snap 252 seq_num 756) dirty exists 2024-09-20T19:32:43.421 INFO:tasks.rados.rados.0.smithi042.stdout:2164: left oid 34 (ObjNum 756 snap 252 seq_num 756) 2024-09-20T19:32:43.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:43.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:43.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.631 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:43 smithi162 bash[24557]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:43.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[22706]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: cluster 2024-09-20T19:32:42.317330+0000 mgr.y (mgr.24491) 555 : cluster [DBG] pgmap v978: 168 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 151 active+clean; 267 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 855 KiB/s wr, 10 op/s 2024-09-20T19:32:43.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: audit 2024-09-20T19:32:42.939708+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:32:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: audit 2024-09-20T19:32:42.940703+0000 mon.a (mon.0) 2079 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:43 smithi042 bash[32075]: cluster 2024-09-20T19:32:43.280124+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-20T19:32:43.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:43.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.902 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:43.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:44.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:44.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:44.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:32:44.299 INFO:tasks.rados.rados.0.smithi042.stdout:2164: done (2 left) 2024-09-20T19:32:44.300 INFO:tasks.rados.rados.0.smithi042.stdout:2165: done (1 left) 2024-09-20T19:32:44.300 INFO:tasks.rados.rados.0.smithi042.stdout:2166: done (0 left) 2024-09-20T19:32:44.300 INFO:tasks.rados.rados.0.smithi042.stdout:2167: delete oid 40 current snap is 253 2024-09-20T19:32:44.308 INFO:tasks.rados.rados.0.smithi042.stdout:2167: done (0 left) 2024-09-20T19:32:44.308 INFO:tasks.rados.rados.0.smithi042.stdout:2168: read oid 22 snap 205 2024-09-20T19:32:44.308 INFO:tasks.rados.rados.0.smithi042.stdout:2168: expect (ObjNum 641 snap 205 seq_num 641) 2024-09-20T19:32:44.308 INFO:tasks.rados.rados.0.smithi042.stdout:2169: snap_create 2024-09-20T19:32:44.312 INFO:tasks.rados.rados.0.smithi042.stdout:2168: expect (ObjNum 641 snap 205 seq_num 641) 2024-09-20T19:32:44.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:44.444 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:32:44.444 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:32:44.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:44.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.533 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:44 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:44.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:43.642431+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.42:0/442013167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:43.642830+0000 mon.a (mon.0) 2081 : audit [INF] from='client.25613 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.287888+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:44.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.648 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: audit 2024-09-20T19:32:44.290050+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25613 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:32:44.648 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.648 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:44 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.299022+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-20T19:32:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:44.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:44.897 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:44.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:45.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:45.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:32:45.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:45.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:45.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T19:32:45.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:45.310 INFO:tasks.rados.rados.0.smithi042.stdout:2168: done (1 left) 2024-09-20T19:32:45.310 INFO:tasks.rados.rados.0.smithi042.stdout:2169: done (0 left) 2024-09-20T19:32:45.310 INFO:tasks.rados.rados.0.smithi042.stdout:2170: write oid 49 current snap is 254 2024-09-20T19:32:45.310 INFO:tasks.rados.rados.0.smithi042.stdout:2170: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-20T19:32:45.316 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:45.317 INFO:tasks.rados.rados.0.smithi042.stdout:2170: writing smithi04268048-49 from 739706 to 1297395 tid 1 2024-09-20T19:32:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:2170: writing smithi04268048-49 from 1806645 to 2273900 tid 2 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:2171: rollback oid 42 current snap is 254 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 205 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:2172: read oid 38 snap -1 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:2172: expect (ObjNum 656 snap 210 seq_num 656) 2024-09-20T19:32:45.323 INFO:tasks.rados.rados.0.smithi042.stdout:2173: copy_from oid 48 from oid 22 current snap is 254 2024-09-20T19:32:45.324 INFO:tasks.rados.rados.0.smithi042.stdout:2174: snap_remove snap 242 2024-09-20T19:32:45.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:45.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:45.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[32075]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[32075]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[32075]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[22706]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[22706]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:45 smithi042 bash[22706]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:45.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:45.705 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:45 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:45 smithi162 bash[24557]: cluster 2024-09-20T19:32:44.317903+0000 mgr.y (mgr.24491) 556 : cluster [DBG] pgmap v982: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 929 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-20T19:32:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:45 smithi162 bash[24557]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:45 smithi162 bash[24557]: cluster 2024-09-20T19:32:45.304048+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-20T19:32:45.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:45.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:45.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:45.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:46.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:46.094 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:46.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:46.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:46.315 INFO:tasks.rados.rados.0.smithi042.stdout:2174: done (4 left) 2024-09-20T19:32:46.315 INFO:tasks.rados.rados.0.smithi042.stdout:2175: write oid 37 current snap is 254 2024-09-20T19:32:46.315 INFO:tasks.rados.rados.0.smithi042.stdout:2170: finishing write tid 1 to smithi04268048-49 2024-09-20T19:32:46.315 INFO:tasks.rados.rados.0.smithi042.stdout:2175: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-20T19:32:46.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:46.334 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:46.338 INFO:tasks.rados.rados.0.smithi042.stdout:2175: writing smithi04268048-37 from 756513 to 1389451 tid 1 2024-09-20T19:32:46.347 INFO:tasks.rados.rados.0.smithi042.stdout:2175: writing smithi04268048-37 from 1865127 to 2592393 tid 2 2024-09-20T19:32:46.347 INFO:tasks.rados.rados.0.smithi042.stdout:2176: snap_create 2024-09-20T19:32:46.347 INFO:tasks.rados.rados.0.smithi042.stdout:2171: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2171: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 927 (ObjNum 642 snap 205 seq_num 642) dirty exists 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2173: finishing copy_from to smithi04268048-48 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2173: got expected ENOENT (src dne) 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2173: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2170: finishing write tid 2 to smithi04268048-49 2024-09-20T19:32:46.348 INFO:tasks.rados.rados.0.smithi042.stdout:2170: finishing write tid 3 to smithi04268048-49 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2170: finishing write tid 4 to smithi04268048-49 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2170: oid 49 updating version 0 to 1163 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2170: oid 49 version 1163 is already newer than 1162 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2170: oid 49 version 1163 is already newer than 1161 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1163 (ObjNum 758 snap 254 seq_num 758) dirty exists 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2170: left oid 49 (ObjNum 758 snap 254 seq_num 758) 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:2172: expect (ObjNum 656 snap 210 seq_num 656) 2024-09-20T19:32:46.349 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:46.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:46.411 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:46.437 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:32:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:32:46.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: finishing write tid 1 to smithi04268048-37 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: finishing write tid 2 to smithi04268048-37 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: finishing write tid 3 to smithi04268048-37 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: finishing write tid 4 to smithi04268048-37 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: oid 37 updating version 0 to 783 2024-09-20T19:32:46.539 INFO:tasks.rados.rados.0.smithi042.stdout:2175: oid 37 version 783 is already newer than 782 2024-09-20T19:32:46.540 INFO:tasks.rados.rados.0.smithi042.stdout:2175: oid 37 updating version 783 to 784 2024-09-20T19:32:46.540 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 784 (ObjNum 759 snap 254 seq_num 759) dirty exists 2024-09-20T19:32:46.540 INFO:tasks.rados.rados.0.smithi042.stdout:2175: left oid 37 (ObjNum 759 snap 254 seq_num 759) 2024-09-20T19:32:46.577 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:46.603 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:46.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:46.733 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:46.836 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:46.964 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:46.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:47.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:47.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:47.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:47.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2170: done (5 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2171: done (4 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2172: done (3 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2173: done (2 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2175: done (1 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2176: done (0 left) 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2177: read oid 20 snap -1 2024-09-20T19:32:47.322 INFO:tasks.rados.rados.0.smithi042.stdout:2177: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:47.323 INFO:tasks.rados.rados.0.smithi042.stdout:2178: delete oid 18 current snap is 255 2024-09-20T19:32:47.333 INFO:tasks.rados.rados.0.smithi042.stdout:2177: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:47.333 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:47.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.365 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:47 smithi162 bash[24557]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.380 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:47.419 INFO:tasks.rados.rados.0.smithi042.stdout:2177: done (1 left) 2024-09-20T19:32:47.419 INFO:tasks.rados.rados.0.smithi042.stdout:2178: done (0 left) 2024-09-20T19:32:47.419 INFO:tasks.rados.rados.0.smithi042.stdout:2179: copy_from oid 23 from oid 28 current snap is 255 2024-09-20T19:32:47.419 INFO:tasks.rados.rados.0.smithi042.stdout:2180: write (excl) oid 25 current snap is 255 2024-09-20T19:32:47.419 INFO:tasks.rados.rados.0.smithi042.stdout:2180: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-20T19:32:47.427 INFO:tasks.rados.rados.0.smithi042.stdout:2180: writing smithi04268048-25 from 790127 to 1289916 tid 1 2024-09-20T19:32:47.435 INFO:tasks.rados.rados.0.smithi042.stdout:2180: writing smithi04268048-25 from 1698444 to 2429379 tid 2 2024-09-20T19:32:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:2181: copy_from oid 46 from oid 47 current snap is 255 2024-09-20T19:32:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:2182: write oid 24 current snap is 255 2024-09-20T19:32:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:2182: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-20T19:32:47.442 INFO:tasks.rados.rados.0.smithi042.stdout:2182: writing smithi04268048-24 from 406934 to 981972 tid 1 2024-09-20T19:32:47.449 INFO:tasks.rados.rados.0.smithi042.stdout:2182: writing smithi04268048-24 from 1640573 to 2115166 tid 2 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2182: writing smithi04268048-24 from 2642791 to 2806934 tid 3 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2183: read oid 8 snap -1 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2183: expect deleted 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2184: rollback oid 21 current snap is 255 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 251 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2179: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2180: finishing write tid 1 to smithi04268048-25 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2180: finishing write tid 2 to smithi04268048-25 2024-09-20T19:32:47.454 INFO:tasks.rados.rados.0.smithi042.stdout:2179: finishing copy_from to smithi04268048-23 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 729 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: finishing write tid 3 to smithi04268048-25 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2182: finishing write tid 1 to smithi04268048-24 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: finishing write tid 4 to smithi04268048-25 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: oid 25 updating version 0 to 847 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: oid 25 version 847 is already newer than 846 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: oid 25 updating version 847 to 848 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 848 (ObjNum 761 snap 255 seq_num 761) dirty exists 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: left oid 25 (ObjNum 761 snap 255 seq_num 761) 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2182: finishing write tid 2 to smithi04268048-24 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2179: done (5 left) 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2180: done (4 left) 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2185: write oid 43 current snap is 255 2024-09-20T19:32:47.455 INFO:tasks.rados.rados.0.smithi042.stdout:2185: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-20T19:32:47.460 INFO:tasks.rados.rados.0.smithi042.stdout:2185: writing smithi04268048-43 from 423741 to 1074028 tid 1 2024-09-20T19:32:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:2185: writing smithi04268048-43 from 1699055 to 2317306 tid 2 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2186: delete oid 33 current snap is 255 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: finishing write tid 3 to smithi04268048-24 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: finishing write tid 4 to smithi04268048-24 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2184: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2184: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2184: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 965 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: finishing write tid 5 to smithi04268048-24 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: oid 24 updating version 0 to 739 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: oid 24 updating version 739 to 740 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: oid 24 updating version 740 to 741 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:2182: oid 24 updating version 741 to 742 2024-09-20T19:32:47.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 742 (ObjNum 762 snap 255 seq_num 762) dirty exists 2024-09-20T19:32:47.469 INFO:tasks.rados.rados.0.smithi042.stdout:2182: left oid 24 (ObjNum 762 snap 255 seq_num 762) 2024-09-20T19:32:47.469 INFO:tasks.rados.rados.0.smithi042.stdout:2181: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:32:47.470 INFO:tasks.rados.rados.0.smithi042.stdout:2185: finishing write tid 1 to smithi04268048-43 2024-09-20T19:32:47.472 INFO:tasks.rados.rados.0.smithi042.stdout:2182: done (5 left) 2024-09-20T19:32:47.472 INFO:tasks.rados.rados.0.smithi042.stdout:2183: done (4 left) 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2184: done (3 left) 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2186: done (2 left) 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2187: copy_from oid 10 from oid 13 current snap is 255 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2188: read oid 45 snap -1 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2188: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2189: read oid 44 snap 252 2024-09-20T19:32:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2189: expect deleted 2024-09-20T19:32:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:2190: write oid 23 current snap is 255 2024-09-20T19:32:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:2190: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-20T19:32:47.477 INFO:tasks.rados.rados.0.smithi042.stdout:2190: writing smithi04268048-23 from 457355 to 858140 tid 1 2024-09-20T19:32:47.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:47.485 INFO:tasks.rados.rados.0.smithi042.stdout:2190: writing smithi04268048-23 from 1416019 to 2037939 tid 2 2024-09-20T19:32:47.489 INFO:tasks.rados.rados.0.smithi042.stdout:2190: writing smithi04268048-23 from 2725060 to 2857355 tid 3 2024-09-20T19:32:47.489 INFO:tasks.rados.rados.0.smithi042.stdout:2191: write (excl) oid 50 current snap is 255 2024-09-20T19:32:47.490 INFO:tasks.rados.rados.0.smithi042.stdout:2191: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-20T19:32:47.494 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":985,"stamp":"2024-09-20T19:32:46.318134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281885910,"num_objects":337,"num_object_clones":91,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":13,"num_read":3741,"num_read_kb":1337098,"num_write":1997,"num_write_kb":479626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":88530899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15910,"ondisk_log_size":15910,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":956240,"kb_used_data":631628,"kb_used_omap":143,"kb_used_meta":324272,"kb_avail":749004976,"statfs":{"total":767960285184,"available":766981095424,"internally_reserved":0,"allocated":646787072,"data_stored":836036235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147312,"internal_metadata":332054672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1704589,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":21,"num_read_kb":22410,"num_write":5,"num_write_kb":1067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.884069"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":1038,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.438449+0000","last_change":"2024-09-20T19:31:37.502437+0000","last_active":"2024-09-20T19:32:42.438449+0000","last_peered":"2024-09-20T19:32:42.438449+0000","last_clean":"2024-09-20T19:32:42.438449+0000","last_became_active":"2024-09-20T19:30:59.423955+0000","last_became_peered":"2024-09-20T19:30:59.423955+0000","last_unstale":"2024-09-20T19:32:42.438449+0000","last_undegraded":"2024-09-20T19:32:42.438449+0000","last_fullsized":"2024-09-20T19:32:42.438449+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1015,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457754+0000","last_change":"2024-09-20T19:32:13.393790+0000","last_active":"2024-09-20T19:32:42.457754+0000","last_peered":"2024-09-20T19:32:42.457754+0000","last_clean":"2024-09-20T19:32:42.457754+0000","last_became_active":"2024-09-20T19:30:31.353214+0000","last_became_peered":"2024-09-20T19:30:31.353214+0000","last_unstale":"2024-09-20T19:32:42.457754+0000","last_undegraded":"2024-09-20T19:32:42.457754+0000","last_fullsized":"2024-09-20T19:32:42.457754+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.908799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1094,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.436515+0000","last_change":"2024-09-20T19:30:59.495048+0000","last_active":"2024-09-20T19:32:42.436515+0000","last_peered":"2024-09-20T19:32:42.436515+0000","last_clean":"2024-09-20T19:32:42.436515+0000","last_became_active":"2024-09-20T19:30:59.432572+0000","last_became_peered":"2024-09-20T19:30:59.432572+0000","last_unstale":"2024-09-20T19:32:42.436515+0000","last_undegraded":"2024-09-20T19:32:42.436515+0000","last_fullsized":"2024-09-20T19:32:42.436515+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:25:18.620430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:18.620430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:55.533243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":987,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.301932+0000","last_change":"2024-09-20T19:30:44.918367+0000","last_active":"2024-09-20T19:32:43.301932+0000","last_peered":"2024-09-20T19:32:43.301932+0000","last_clean":"2024-09-20T19:32:43.301932+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:32:43.301932+0000","last_undegraded":"2024-09-20T19:32:43.301932+0000","last_fullsized":"2024-09-20T19:32:43.301932+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:44.918249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:44.918249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1090,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436854+0000","last_change":"2024-09-20T19:32:12.874027+0000","last_active":"2024-09-20T19:32:41.436854+0000","last_peered":"2024-09-20T19:32:41.436854+0000","last_clean":"2024-09-20T19:32:41.436854+0000","last_became_active":"2024-09-20T19:30:59.439274+0000","last_became_peered":"2024-09-20T19:30:59.439274+0000","last_unstale":"2024-09-20T19:32:41.436854+0000","last_undegraded":"2024-09-20T19:32:41.436854+0000","last_fullsized":"2024-09-20T19:32:41.436854+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.712357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1091,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.302483+0000","last_change":"2024-09-20T19:30:49.903222+0000","last_active":"2024-09-20T19:32:43.302483+0000","last_peered":"2024-09-20T19:32:43.302483+0000","last_clean":"2024-09-20T19:32:43.302483+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:32:43.302483+0000","last_undegraded":"2024-09-20T19:32:43.302483+0000","last_fullsized":"2024-09-20T19:32:43.302483+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:49.903144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:30:49.903144+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1215,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.433848+0000","last_change":"2024-09-20T19:29:00.755422+0000","last_active":"2024-09-20T19:32:41.433848+0000","last_peered":"2024-09-20T19:32:41.433848+0000","last_clean":"2024-09-20T19:32:41.433848+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:32:41.433848+0000","last_undegraded":"2024-09-20T19:32:41.433848+0000","last_fullsized":"2024-09-20T19:32:41.433848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:29:00.755361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.755361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:03.153003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1049,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303244+0000","last_change":"2024-09-20T19:31:20.821748+0000","last_active":"2024-09-20T19:32:43.303244+0000","last_peered":"2024-09-20T19:32:43.303244+0000","last_clean":"2024-09-20T19:32:43.303244+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:32:43.303244+0000","last_undegraded":"2024-09-20T19:32:43.303244+0000","last_fullsized":"2024-09-20T19:32:43.303244+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:20.821598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.821598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"632'1040","reported_seq":3586,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.478229+0000","last_change":"2024-09-20T19:32:42.479335+0000","last_active":"2024-09-20T19:32:42.478229+0000","last_peered":"2024-09-20T19:32:42.478229+0000","last_clean":"2024-09-20T19:32:42.478229+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:32:42.478229+0000","last_undegraded":"2024-09-20T19:32:42.478229+0000","last_fullsized":"2024-09-20T19:32:42.478229+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.314986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022930806000000001,"stat_sum":{"num_bytes":6269242,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31343,"num_write":25,"num_write_kb":6122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"2"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1229,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658241+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:32:41.658241+0000","last_peered":"2024-09-20T19:32:41.658241+0000","last_clean":"2024-09-20T19:32:41.658241+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:32:41.658241+0000","last_undegraded":"2024-09-20T19:32:41.658241+0000","last_fullsized":"2024-09-20T19:32:41.658241+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.958849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1055,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303426+0000","last_change":"2024-09-20T19:32:29.224300+0000","last_active":"2024-09-20T19:32:43.303426+0000","last_peered":"2024-09-20T19:32:43.303426+0000","last_clean":"2024-09-20T19:32:43.303426+0000","last_became_active":"2024-09-20T19:30:59.395790+0000","last_became_peered":"2024-09-20T19:30:59.395790+0000","last_unstale":"2024-09-20T19:32:43.303426+0000","last_undegraded":"2024-09-20T19:32:43.303426+0000","last_fullsized":"2024-09-20T19:32:43.303426+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:32:29.224169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.224169+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:53.488936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1213,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653968+0000","last_change":"2024-09-20T19:30:31.352779+0000","last_active":"2024-09-20T19:32:41.653968+0000","last_peered":"2024-09-20T19:32:41.653968+0000","last_clean":"2024-09-20T19:32:41.653968+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:32:41.653968+0000","last_undegraded":"2024-09-20T19:32:41.653968+0000","last_fullsized":"2024-09-20T19:32:41.653968+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:11.324315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.324315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1225,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.651906+0000","last_change":"2024-09-20T19:32:28.436574+0000","last_active":"2024-09-20T19:32:41.651906+0000","last_peered":"2024-09-20T19:32:41.651906+0000","last_clean":"2024-09-20T19:32:41.651906+0000","last_became_active":"2024-09-20T19:30:59.422108+0000","last_became_peered":"2024-09-20T19:30:59.422108+0000","last_unstale":"2024-09-20T19:32:41.651906+0000","last_undegraded":"2024-09-20T19:32:41.651906+0000","last_fullsized":"2024-09-20T19:32:41.651906+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.196618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"652'357","reported_seq":2920,"reported_epoch":653,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:41.651090+0000","last_change":"2024-09-20T19:32:40.462065+0000","last_active":"2024-09-20T19:32:41.651090+0000","last_peered":"2024-09-20T19:32:41.651090+0000","last_clean":"2024-09-20T19:32:41.651090+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:32:41.651090+0000","last_undegraded":"2024-09-20T19:32:41.651090+0000","last_fullsized":"2024-09-20T19:32:41.651090+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'290","last_scrub_stamp":"2024-09-20T19:29:01.442771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:01.442771+0000","objects_scrubbed":4,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.17422858999999999,"stat_sum":{"num_bytes":7903028,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":11059,"num_write":19,"num_write_kb":4637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1218,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.655924+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:32:41.655924+0000","last_peered":"2024-09-20T19:32:41.655924+0000","last_clean":"2024-09-20T19:32:41.655924+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:32:41.655924+0000","last_undegraded":"2024-09-20T19:32:41.655924+0000","last_fullsized":"2024-09-20T19:32:41.655924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.698584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1101,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.302130+0000","last_change":"2024-09-20T19:32:15.452097+0000","last_active":"2024-09-20T19:32:43.302130+0000","last_peered":"2024-09-20T19:32:43.302130+0000","last_clean":"2024-09-20T19:32:43.302130+0000","last_became_active":"2024-09-20T19:30:59.403992+0000","last_became_peered":"2024-09-20T19:30:59.403992+0000","last_unstale":"2024-09-20T19:32:43.302130+0000","last_undegraded":"2024-09-20T19:32:43.302130+0000","last_fullsized":"2024-09-20T19:32:43.302130+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.451996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.451996+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:23.255186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1213,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653141+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:32:41.653141+0000","last_peered":"2024-09-20T19:32:41.653141+0000","last_clean":"2024-09-20T19:32:41.653141+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:32:41.653141+0000","last_undegraded":"2024-09-20T19:32:41.653141+0000","last_fullsized":"2024-09-20T19:32:41.653141+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.435769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":191,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.455987+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:32:42.455987+0000","last_peered":"2024-09-20T19:32:42.455987+0000","last_clean":"2024-09-20T19:32:42.455987+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:32:42.455987+0000","last_undegraded":"2024-09-20T19:32:42.455987+0000","last_fullsized":"2024-09-20T19:32:42.455987+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.693370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":829,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.651233+0000","last_change":"2024-09-20T19:30:59.413612+0000","last_active":"2024-09-20T19:32:41.651233+0000","last_peered":"2024-09-20T19:32:41.651233+0000","last_clean":"2024-09-20T19:32:41.651233+0000","last_became_active":"2024-09-20T19:30:59.413239+0000","last_became_peered":"2024-09-20T19:30:59.413239+0000","last_unstale":"2024-09-20T19:32:41.651233+0000","last_undegraded":"2024-09-20T19:32:41.651233+0000","last_fullsized":"2024-09-20T19:32:41.651233+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:04.483789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:04.483789+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.360692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"650'206","reported_seq":2803,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657006+0000","last_change":"2024-09-20T19:32:38.332072+0000","last_active":"2024-09-20T19:32:41.657006+0000","last_peered":"2024-09-20T19:32:41.657006+0000","last_clean":"2024-09-20T19:32:41.657006+0000","last_became_active":"2024-09-20T19:30:09.790218+0000","last_became_peered":"2024-09-20T19:30:09.790218+0000","last_unstale":"2024-09-20T19:32:41.657006+0000","last_undegraded":"2024-09-20T19:32:41.657006+0000","last_fullsized":"2024-09-20T19:32:41.657006+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'200","last_scrub_stamp":"2024-09-20T19:32:31.708023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.708023+0000","objects_scrubbed":2,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.485229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040184820000000003,"stat_sum":{"num_bytes":4975136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":37797,"num_write":81,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2317410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.d","version":"523'31","reported_seq":1078,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304162+0000","last_change":"2024-09-20T19:32:23.326867+0000","last_active":"2024-09-20T19:32:43.304162+0000","last_peered":"2024-09-20T19:32:43.304162+0000","last_clean":"2024-09-20T19:32:43.304162+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:32:43.304162+0000","last_undegraded":"2024-09-20T19:32:43.304162+0000","last_fullsized":"2024-09-20T19:32:43.304162+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:32:23.326721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.326721+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:26.897588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1214,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.655668+0000","last_change":"2024-09-20T19:30:31.374792+0000","last_active":"2024-09-20T19:32:41.655668+0000","last_peered":"2024-09-20T19:32:41.655668+0000","last_clean":"2024-09-20T19:32:41.655668+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:32:41.655668+0000","last_undegraded":"2024-09-20T19:32:41.655668+0000","last_fullsized":"2024-09-20T19:32:41.655668+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:10.363587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:10.363587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1233,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435015+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:32:41.435015+0000","last_peered":"2024-09-20T19:32:41.435015+0000","last_clean":"2024-09-20T19:32:41.435015+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:32:41.435015+0000","last_undegraded":"2024-09-20T19:32:41.435015+0000","last_fullsized":"2024-09-20T19:32:41.435015+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:51.922142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1228,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657103+0000","last_change":"2024-09-20T19:31:31.525823+0000","last_active":"2024-09-20T19:32:41.657103+0000","last_peered":"2024-09-20T19:32:41.657103+0000","last_clean":"2024-09-20T19:32:41.657103+0000","last_became_active":"2024-09-20T19:30:59.407192+0000","last_became_peered":"2024-09-20T19:30:59.407192+0000","last_unstale":"2024-09-20T19:32:41.657103+0000","last_undegraded":"2024-09-20T19:32:41.657103+0000","last_fullsized":"2024-09-20T19:32:41.657103+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:31.525745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.525745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.839611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1285,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653242+0000","last_change":"2024-09-20T19:30:59.437810+0000","last_active":"2024-09-20T19:32:41.653242+0000","last_peered":"2024-09-20T19:32:41.653242+0000","last_clean":"2024-09-20T19:32:41.653242+0000","last_became_active":"2024-09-20T19:30:59.437496+0000","last_became_peered":"2024-09-20T19:30:59.437496+0000","last_unstale":"2024-09-20T19:32:41.653242+0000","last_undegraded":"2024-09-20T19:32:41.653242+0000","last_fullsized":"2024-09-20T19:32:41.653242+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:55.361511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:55.361511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"647'468","reported_seq":2781,"reported_epoch":654,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:42.478139+0000","last_change":"2024-09-20T19:32:42.478191+0000","last_active":"2024-09-20T19:32:42.478139+0000","last_peered":"2024-09-20T19:32:42.478139+0000","last_clean":"2024-09-20T19:32:42.478139+0000","last_became_active":"2024-09-20T19:30:59.433690+0000","last_became_peered":"2024-09-20T19:30:59.433690+0000","last_unstale":"2024-09-20T19:32:42.478139+0000","last_undegraded":"2024-09-20T19:32:42.478139+0000","last_fullsized":"2024-09-20T19:32:42.478139+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'449","last_scrub_stamp":"2024-09-20T19:32:09.344681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.344681+0000","objects_scrubbed":6,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:22.784637+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.020661011,"stat_sum":{"num_bytes":18647176,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":236,"num_read_kb":207145,"num_write":153,"num_write_kb":65936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17328954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"2"}]},{"pgid":"4.c","version":"523'20","reported_seq":1110,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303933+0000","last_change":"2024-09-20T19:32:10.444211+0000","last_active":"2024-09-20T19:32:43.303933+0000","last_peered":"2024-09-20T19:32:43.303933+0000","last_clean":"2024-09-20T19:32:43.303933+0000","last_became_active":"2024-09-20T19:30:59.412354+0000","last_became_peered":"2024-09-20T19:30:59.412354+0000","last_unstale":"2024-09-20T19:32:43.303933+0000","last_undegraded":"2024-09-20T19:32:43.303933+0000","last_fullsized":"2024-09-20T19:32:43.303933+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:32:10.444013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.444013+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:11.431206+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1234,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.434898+0000","last_change":"2024-09-20T19:31:39.214509+0000","last_active":"2024-09-20T19:32:41.434898+0000","last_peered":"2024-09-20T19:32:41.434898+0000","last_clean":"2024-09-20T19:32:41.434898+0000","last_became_active":"2024-09-20T19:30:59.435459+0000","last_became_peered":"2024-09-20T19:30:59.435459+0000","last_unstale":"2024-09-20T19:32:41.434898+0000","last_undegraded":"2024-09-20T19:32:41.434898+0000","last_fullsized":"2024-09-20T19:32:41.434898+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:39.214431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.214431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1410,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656829+0000","last_change":"2024-09-20T19:29:54.085338+0000","last_active":"2024-09-20T19:32:41.656829+0000","last_peered":"2024-09-20T19:32:41.656829+0000","last_clean":"2024-09-20T19:32:41.656829+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:32:41.656829+0000","last_undegraded":"2024-09-20T19:32:41.656829+0000","last_fullsized":"2024-09-20T19:32:41.656829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:29:54.085246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:54.085246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.534025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1038,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304320+0000","last_change":"2024-09-20T19:30:31.380645+0000","last_active":"2024-09-20T19:32:43.304320+0000","last_peered":"2024-09-20T19:32:43.304320+0000","last_clean":"2024-09-20T19:32:43.304320+0000","last_became_active":"2024-09-20T19:30:31.379668+0000","last_became_peered":"2024-09-20T19:30:31.379668+0000","last_unstale":"2024-09-20T19:32:43.304320+0000","last_undegraded":"2024-09-20T19:32:43.304320+0000","last_fullsized":"2024-09-20T19:32:43.304320+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:20.948186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:26:20.948186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.134190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":811,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653390+0000","last_change":"2024-09-20T19:28:03.366431+0000","last_active":"2024-09-20T19:32:41.653390+0000","last_peered":"2024-09-20T19:32:41.653390+0000","last_clean":"2024-09-20T19:32:41.653390+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:32:41.653390+0000","last_undegraded":"2024-09-20T19:32:41.653390+0000","last_fullsized":"2024-09-20T19:32:41.653390+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:03.366356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.366356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"652'759","reported_seq":3352,"reported_epoch":653,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:41.657002+0000","last_change":"2024-09-20T19:32:38.462526+0000","last_active":"2024-09-20T19:32:41.657002+0000","last_peered":"2024-09-20T19:32:41.657002+0000","last_clean":"2024-09-20T19:32:41.657002+0000","last_became_active":"2024-09-20T19:30:59.437942+0000","last_became_peered":"2024-09-20T19:30:59.437942+0000","last_unstale":"2024-09-20T19:32:41.657002+0000","last_undegraded":"2024-09-20T19:32:41.657002+0000","last_fullsized":"2024-09-20T19:32:41.657002+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'237","last_scrub_stamp":"2024-09-20T19:24:35.542766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.542766+0000","objects_scrubbed":12,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.169086231,"stat_sum":{"num_bytes":19364965,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":397,"num_read_kb":287353,"num_write":270,"num_write_kb":86087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":11179590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e1","length":"1"},{"start":"ec","length":"1"}]},{"pgid":"4.f","version":"523'27","reported_seq":1285,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437774+0000","last_change":"2024-09-20T19:30:31.397678+0000","last_active":"2024-09-20T19:32:41.437774+0000","last_peered":"2024-09-20T19:32:41.437774+0000","last_clean":"2024-09-20T19:32:41.437774+0000","last_became_active":"2024-09-20T19:30:31.365973+0000","last_became_peered":"2024-09-20T19:30:31.365973+0000","last_unstale":"2024-09-20T19:32:41.437774+0000","last_undegraded":"2024-09-20T19:32:41.437774+0000","last_fullsized":"2024-09-20T19:32:41.437774+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:23:04.313226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:23:04.313226+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.137611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1234,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.433953+0000","last_change":"2024-09-20T19:30:59.435287+0000","last_active":"2024-09-20T19:32:41.433953+0000","last_peered":"2024-09-20T19:32:41.433953+0000","last_clean":"2024-09-20T19:32:41.433953+0000","last_became_active":"2024-09-20T19:30:59.434928+0000","last_became_peered":"2024-09-20T19:30:59.434928+0000","last_unstale":"2024-09-20T19:32:41.433953+0000","last_undegraded":"2024-09-20T19:32:41.433953+0000","last_fullsized":"2024-09-20T19:32:41.433953+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.895903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.895903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.157414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1201,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304925+0000","last_change":"2024-09-20T19:30:38.015879+0000","last_active":"2024-09-20T19:32:43.304925+0000","last_peered":"2024-09-20T19:32:43.304925+0000","last_clean":"2024-09-20T19:32:43.304925+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:32:43.304925+0000","last_undegraded":"2024-09-20T19:32:43.304925+0000","last_fullsized":"2024-09-20T19:32:43.304925+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:38.015815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:38.015815+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1218,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657184+0000","last_change":"2024-09-20T19:32:24.498998+0000","last_active":"2024-09-20T19:32:41.657184+0000","last_peered":"2024-09-20T19:32:41.657184+0000","last_clean":"2024-09-20T19:32:41.657184+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:32:41.657184+0000","last_undegraded":"2024-09-20T19:32:41.657184+0000","last_fullsized":"2024-09-20T19:32:41.657184+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:24.498942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:24.498942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.513248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"650'484","reported_seq":3071,"reported_epoch":653,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:41.657671+0000","last_change":"2024-09-20T19:32:40.461930+0000","last_active":"2024-09-20T19:32:41.657671+0000","last_peered":"2024-09-20T19:32:41.657671+0000","last_clean":"2024-09-20T19:32:41.657671+0000","last_became_active":"2024-09-20T19:30:09.785274+0000","last_became_peered":"2024-09-20T19:30:09.785274+0000","last_unstale":"2024-09-20T19:32:41.657671+0000","last_undegraded":"2024-09-20T19:32:41.657671+0000","last_fullsized":"2024-09-20T19:32:41.657671+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'149","last_scrub_stamp":"2024-09-20T19:24:44.564761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.564761+0000","objects_scrubbed":9,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.043277940000000001,"stat_sum":{"num_bytes":16861690,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":270,"num_read_kb":230214,"num_write":172,"num_write_kb":65414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":15038144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"}]},{"pgid":"4.e","version":"523'19","reported_seq":1110,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.306630+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:32:43.306630+0000","last_peered":"2024-09-20T19:32:43.306630+0000","last_clean":"2024-09-20T19:32:43.306630+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:32:43.306630+0000","last_undegraded":"2024-09-20T19:32:43.306630+0000","last_fullsized":"2024-09-20T19:32:43.306630+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:01.570152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1109,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.643942+0000","last_change":"2024-09-20T19:31:26.549536+0000","last_active":"2024-09-20T19:32:41.643942+0000","last_peered":"2024-09-20T19:32:41.643942+0000","last_clean":"2024-09-20T19:32:41.643942+0000","last_became_active":"2024-09-20T19:30:59.407964+0000","last_became_peered":"2024-09-20T19:30:59.407964+0000","last_unstale":"2024-09-20T19:32:41.643942+0000","last_undegraded":"2024-09-20T19:32:41.643942+0000","last_fullsized":"2024-09-20T19:32:41.643942+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1214,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659161+0000","last_change":"2024-09-20T19:32:13.572694+0000","last_active":"2024-09-20T19:32:41.659161+0000","last_peered":"2024-09-20T19:32:41.659161+0000","last_clean":"2024-09-20T19:32:41.659161+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:32:41.659161+0000","last_undegraded":"2024-09-20T19:32:41.659161+0000","last_fullsized":"2024-09-20T19:32:41.659161+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.572577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.572577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:15.678184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1023,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435525+0000","last_change":"2024-09-20T19:30:59.413132+0000","last_active":"2024-09-20T19:32:41.435525+0000","last_peered":"2024-09-20T19:32:41.435525+0000","last_clean":"2024-09-20T19:32:41.435525+0000","last_became_active":"2024-09-20T19:30:59.412722+0000","last_became_peered":"2024-09-20T19:30:59.412722+0000","last_unstale":"2024-09-20T19:32:41.435525+0000","last_undegraded":"2024-09-20T19:32:41.435525+0000","last_fullsized":"2024-09-20T19:32:41.435525+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":647,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435953+0000","last_change":"2024-09-20T19:30:59.428471+0000","last_active":"2024-09-20T19:32:41.435953+0000","last_peered":"2024-09-20T19:32:41.435953+0000","last_clean":"2024-09-20T19:32:41.435953+0000","last_became_active":"2024-09-20T19:30:59.426646+0000","last_became_peered":"2024-09-20T19:30:59.426646+0000","last_unstale":"2024-09-20T19:32:41.435953+0000","last_undegraded":"2024-09-20T19:32:41.435953+0000","last_fullsized":"2024-09-20T19:32:41.435953+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:07.247985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:07.247985+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-20T19:33:04.665624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"634'190","reported_seq":2566,"reported_epoch":654,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:42.437141+0000","last_change":"2024-09-20T19:32:42.437109+0000","last_active":"2024-09-20T19:32:42.437141+0000","last_peered":"2024-09-20T19:32:42.437141+0000","last_clean":"2024-09-20T19:32:42.437141+0000","last_became_active":"2024-09-20T19:30:59.420070+0000","last_became_peered":"2024-09-20T19:30:59.420070+0000","last_unstale":"2024-09-20T19:32:42.437141+0000","last_undegraded":"2024-09-20T19:32:42.437141+0000","last_fullsized":"2024-09-20T19:32:42.437141+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'186","last_scrub_stamp":"2024-09-20T19:31:31.606808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.606808+0000","objects_scrubbed":3,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:46.135515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.98939534799999995,"stat_sum":{"num_bytes":2773287,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.1","version":"523'28","reported_seq":1079,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304894+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:32:43.304894+0000","last_peered":"2024-09-20T19:32:43.304894+0000","last_clean":"2024-09-20T19:32:43.304894+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:32:43.304894+0000","last_undegraded":"2024-09-20T19:32:43.304894+0000","last_fullsized":"2024-09-20T19:32:43.304894+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.002901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1219,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659750+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:32:41.659750+0000","last_peered":"2024-09-20T19:32:41.659750+0000","last_clean":"2024-09-20T19:32:41.659750+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:32:41.659750+0000","last_undegraded":"2024-09-20T19:32:41.659750+0000","last_fullsized":"2024-09-20T19:32:41.659750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"554'15","reported_seq":1142,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.435533+0000","last_change":"2024-09-20T19:30:59.419924+0000","last_active":"2024-09-20T19:32:42.435533+0000","last_peered":"2024-09-20T19:32:42.435533+0000","last_clean":"2024-09-20T19:32:42.435533+0000","last_became_active":"2024-09-20T19:30:59.418231+0000","last_became_peered":"2024-09-20T19:30:59.418231+0000","last_unstale":"2024-09-20T19:32:42.435533+0000","last_undegraded":"2024-09-20T19:32:42.435533+0000","last_fullsized":"2024-09-20T19:32:42.435533+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:32.089810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.089810+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.508745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1199,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.654487+0000","last_change":"2024-09-20T19:30:03.184615+0000","last_active":"2024-09-20T19:32:41.654487+0000","last_peered":"2024-09-20T19:32:41.654487+0000","last_clean":"2024-09-20T19:32:41.654487+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:32:41.654487+0000","last_undegraded":"2024-09-20T19:32:41.654487+0000","last_fullsized":"2024-09-20T19:32:41.654487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:03.184552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:30:03.184552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":611,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304496+0000","last_change":"2024-09-20T19:30:34.035838+0000","last_active":"2024-09-20T19:32:43.304496+0000","last_peered":"2024-09-20T19:32:43.304496+0000","last_clean":"2024-09-20T19:32:43.304496+0000","last_became_active":"2024-09-20T19:30:31.377908+0000","last_became_peered":"2024-09-20T19:30:31.377908+0000","last_unstale":"2024-09-20T19:32:43.304496+0000","last_undegraded":"2024-09-20T19:32:43.304496+0000","last_fullsized":"2024-09-20T19:32:43.304496+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:30:34.035776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:30:34.035776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:47.612744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"650'375","reported_seq":2759,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.425113+0000","last_change":"2024-09-20T19:32:42.425871+0000","last_active":"2024-09-20T19:32:42.425113+0000","last_peered":"2024-09-20T19:32:42.425113+0000","last_clean":"2024-09-20T19:32:42.425113+0000","last_became_active":"2024-09-20T19:30:59.407988+0000","last_became_peered":"2024-09-20T19:30:59.407988+0000","last_unstale":"2024-09-20T19:32:42.425113+0000","last_undegraded":"2024-09-20T19:32:42.425113+0000","last_fullsized":"2024-09-20T19:32:42.425113+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'350","last_scrub_stamp":"2024-09-20T19:31:19.546439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.546439+0000","objects_scrubbed":2,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.99032699300000004,"stat_sum":{"num_bytes":11076143,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":7193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.0","version":"523'30","reported_seq":1110,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.435670+0000","last_change":"2024-09-20T19:32:20.900983+0000","last_active":"2024-09-20T19:32:42.435670+0000","last_peered":"2024-09-20T19:32:42.435670+0000","last_clean":"2024-09-20T19:32:42.435670+0000","last_became_active":"2024-09-20T19:30:59.431399+0000","last_became_peered":"2024-09-20T19:30:59.431399+0000","last_unstale":"2024-09-20T19:32:42.435670+0000","last_undegraded":"2024-09-20T19:32:42.435670+0000","last_fullsized":"2024-09-20T19:32:42.435670+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.542941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1227,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437570+0000","last_change":"2024-09-20T19:32:15.096541+0000","last_active":"2024-09-20T19:32:41.437570+0000","last_peered":"2024-09-20T19:32:41.437570+0000","last_clean":"2024-09-20T19:32:41.437570+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:32:41.437570+0000","last_undegraded":"2024-09-20T19:32:41.437570+0000","last_fullsized":"2024-09-20T19:32:41.437570+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:32:15.096450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.096450+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:25.106545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1031,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.301791+0000","last_change":"2024-09-20T19:32:18.400368+0000","last_active":"2024-09-20T19:32:43.301791+0000","last_peered":"2024-09-20T19:32:43.301791+0000","last_clean":"2024-09-20T19:32:43.301791+0000","last_became_active":"2024-09-20T19:30:59.403694+0000","last_became_peered":"2024-09-20T19:30:59.403694+0000","last_unstale":"2024-09-20T19:32:43.301791+0000","last_undegraded":"2024-09-20T19:32:43.301791+0000","last_fullsized":"2024-09-20T19:32:43.301791+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:18.400304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.400304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.263499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":977,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.302519+0000","last_change":"2024-09-20T19:30:59.403484+0000","last_active":"2024-09-20T19:32:43.302519+0000","last_peered":"2024-09-20T19:32:43.302519+0000","last_clean":"2024-09-20T19:32:43.302519+0000","last_became_active":"2024-09-20T19:30:59.403080+0000","last_became_peered":"2024-09-20T19:30:59.403080+0000","last_unstale":"2024-09-20T19:32:43.302519+0000","last_undegraded":"2024-09-20T19:32:43.302519+0000","last_fullsized":"2024-09-20T19:32:43.302519+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:48.185791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:23:48.185791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.522323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":924,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658693+0000","last_change":"2024-09-20T19:30:59.410472+0000","last_active":"2024-09-20T19:32:41.658693+0000","last_peered":"2024-09-20T19:32:41.658693+0000","last_clean":"2024-09-20T19:32:41.658693+0000","last_became_active":"2024-09-20T19:30:59.410103+0000","last_became_peered":"2024-09-20T19:30:59.410103+0000","last_unstale":"2024-09-20T19:32:41.658693+0000","last_undegraded":"2024-09-20T19:32:41.658693+0000","last_fullsized":"2024-09-20T19:32:41.658693+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.382463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.382463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"650'733","reported_seq":3358,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.650957+0000","last_change":"2024-09-20T19:32:40.330385+0000","last_active":"2024-09-20T19:32:41.650957+0000","last_peered":"2024-09-20T19:32:41.650957+0000","last_clean":"2024-09-20T19:32:41.650957+0000","last_became_active":"2024-09-20T19:30:59.413518+0000","last_became_peered":"2024-09-20T19:30:59.413518+0000","last_unstale":"2024-09-20T19:32:41.650957+0000","last_undegraded":"2024-09-20T19:32:41.650957+0000","last_fullsized":"2024-09-20T19:32:41.650957+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'685","last_scrub_stamp":"2024-09-20T19:32:09.614901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.614901+0000","objects_scrubbed":9,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:24.072662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.91705127600000003,"stat_sum":{"num_bytes":20819223,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":12,"num_read_kb":6452,"num_write":10,"num_write_kb":4227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.3","version":"523'29","reported_seq":1040,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.458458+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:32:42.458458+0000","last_peered":"2024-09-20T19:32:42.458458+0000","last_clean":"2024-09-20T19:32:42.458458+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:32:42.458458+0000","last_undegraded":"2024-09-20T19:32:42.458458+0000","last_fullsized":"2024-09-20T19:32:42.458458+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.517393+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1178,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656805+0000","last_change":"2024-09-20T19:30:31.348665+0000","last_active":"2024-09-20T19:32:41.656805+0000","last_peered":"2024-09-20T19:32:41.656805+0000","last_clean":"2024-09-20T19:32:41.656805+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:32:41.656805+0000","last_undegraded":"2024-09-20T19:32:41.656805+0000","last_fullsized":"2024-09-20T19:32:41.656805+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:22.169579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:22.169579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.508877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1214,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.652603+0000","last_change":"2024-09-20T19:30:09.775734+0000","last_active":"2024-09-20T19:32:41.652603+0000","last_peered":"2024-09-20T19:32:41.652603+0000","last_clean":"2024-09-20T19:32:41.652603+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:32:41.652603+0000","last_undegraded":"2024-09-20T19:32:41.652603+0000","last_fullsized":"2024-09-20T19:32:41.652603+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:59.649128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:23:59.649128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:04.937740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1195,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.433656+0000","last_change":"2024-09-20T19:27:53.899785+0000","last_active":"2024-09-20T19:32:41.433656+0000","last_peered":"2024-09-20T19:32:41.433656+0000","last_clean":"2024-09-20T19:32:41.433656+0000","last_became_active":"2024-09-20T19:18:13.489851+0000","last_became_peered":"2024-09-20T19:18:13.489851+0000","last_unstale":"2024-09-20T19:32:41.433656+0000","last_undegraded":"2024-09-20T19:32:41.433656+0000","last_fullsized":"2024-09-20T19:32:41.433656+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:53.899739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:53.899739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.125297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":925,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658792+0000","last_change":"2024-09-20T19:30:59.409489+0000","last_active":"2024-09-20T19:32:41.658792+0000","last_peered":"2024-09-20T19:32:41.658792+0000","last_clean":"2024-09-20T19:32:41.658792+0000","last_became_active":"2024-09-20T19:30:59.409151+0000","last_became_peered":"2024-09-20T19:30:59.409151+0000","last_unstale":"2024-09-20T19:32:41.658792+0000","last_undegraded":"2024-09-20T19:32:41.658792+0000","last_fullsized":"2024-09-20T19:32:41.658792+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"650'547","reported_seq":3059,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303488+0000","last_change":"2024-09-20T19:32:43.306934+0000","last_active":"2024-09-20T19:32:43.303488+0000","last_peered":"2024-09-20T19:32:43.303488+0000","last_clean":"2024-09-20T19:32:43.303488+0000","last_became_active":"2024-09-20T19:31:48.294687+0000","last_became_peered":"2024-09-20T19:31:48.294687+0000","last_unstale":"2024-09-20T19:32:43.303488+0000","last_undegraded":"2024-09-20T19:32:43.303488+0000","last_fullsized":"2024-09-20T19:32:43.303488+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'452","last_scrub_stamp":"2024-09-20T19:30:36.040764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:36.040764+0000","objects_scrubbed":6,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041168400000000001,"stat_sum":{"num_bytes":10706587,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":36455,"num_write":23,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6670312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"da","length":"1"},{"start":"ec","length":"3"},{"start":"f0","length":"3"},{"start":"f6","length":"2"}]},{"pgid":"4.2","version":"523'20","reported_seq":1271,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.434579+0000","last_change":"2024-09-20T19:31:19.351200+0000","last_active":"2024-09-20T19:32:41.434579+0000","last_peered":"2024-09-20T19:32:41.434579+0000","last_clean":"2024-09-20T19:32:41.434579+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:32:41.434579+0000","last_undegraded":"2024-09-20T19:32:41.434579+0000","last_fullsized":"2024-09-20T19:32:41.434579+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:19.351134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.351134+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:36.023354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1218,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436265+0000","last_change":"2024-09-20T19:30:09.760759+0000","last_active":"2024-09-20T19:32:41.436265+0000","last_peered":"2024-09-20T19:32:41.436265+0000","last_clean":"2024-09-20T19:32:41.436265+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:32:41.436265+0000","last_undegraded":"2024-09-20T19:32:41.436265+0000","last_fullsized":"2024-09-20T19:32:41.436265+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:22:21.888700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:22:21.888700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.151053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1132,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.458772+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:32:42.458772+0000","last_peered":"2024-09-20T19:32:42.458772+0000","last_clean":"2024-09-20T19:32:42.458772+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:32:42.458772+0000","last_undegraded":"2024-09-20T19:32:42.458772+0000","last_fullsized":"2024-09-20T19:32:42.458772+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:32.990270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1004,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457018+0000","last_change":"2024-09-20T19:31:33.455642+0000","last_active":"2024-09-20T19:32:42.457018+0000","last_peered":"2024-09-20T19:32:42.457018+0000","last_clean":"2024-09-20T19:32:42.457018+0000","last_became_active":"2024-09-20T19:30:59.433962+0000","last_became_peered":"2024-09-20T19:30:59.433962+0000","last_unstale":"2024-09-20T19:32:42.457018+0000","last_undegraded":"2024-09-20T19:32:42.457018+0000","last_fullsized":"2024-09-20T19:32:42.457018+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.455584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.455584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.875559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":823,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.650823+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:32:41.650823+0000","last_peered":"2024-09-20T19:32:41.650823+0000","last_clean":"2024-09-20T19:32:41.650823+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:32:41.650823+0000","last_undegraded":"2024-09-20T19:32:41.650823+0000","last_fullsized":"2024-09-20T19:32:41.650823+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.285306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"645'416","reported_seq":3044,"reported_epoch":653,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:41.659177+0000","last_change":"2024-09-20T19:32:38.461914+0000","last_active":"2024-09-20T19:32:41.659177+0000","last_peered":"2024-09-20T19:32:41.659177+0000","last_clean":"2024-09-20T19:32:41.659177+0000","last_became_active":"2024-09-20T19:30:59.409330+0000","last_became_peered":"2024-09-20T19:30:59.409330+0000","last_unstale":"2024-09-20T19:32:41.659177+0000","last_undegraded":"2024-09-20T19:32:41.659177+0000","last_fullsized":"2024-09-20T19:32:41.659177+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'400","last_scrub_stamp":"2024-09-20T19:31:36.808095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.808095+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:00.316811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.17180720199999999,"stat_sum":{"num_bytes":6460566,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":31,"num_read_kb":20880,"num_write":14,"num_write_kb":5272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.4","version":"554'44","reported_seq":1493,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436193+0000","last_change":"2024-09-20T19:31:31.390088+0000","last_active":"2024-09-20T19:32:41.436193+0000","last_peered":"2024-09-20T19:32:41.436193+0000","last_clean":"2024-09-20T19:32:41.436193+0000","last_became_active":"2024-09-20T19:30:59.436599+0000","last_became_peered":"2024-09-20T19:30:59.436599+0000","last_unstale":"2024-09-20T19:32:41.436193+0000","last_undegraded":"2024-09-20T19:32:41.436193+0000","last_fullsized":"2024-09-20T19:32:41.436193+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'44","last_scrub_stamp":"2024-09-20T19:31:31.390009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.390009+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.692796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1227,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.655134+0000","last_change":"2024-09-20T19:30:02.530981+0000","last_active":"2024-09-20T19:32:41.655134+0000","last_peered":"2024-09-20T19:32:41.655134+0000","last_clean":"2024-09-20T19:32:41.655134+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:32:41.655134+0000","last_undegraded":"2024-09-20T19:32:41.655134+0000","last_fullsized":"2024-09-20T19:32:41.655134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:02.530860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:30:02.530860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1015,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.456819+0000","last_change":"2024-09-20T19:32:17.478298+0000","last_active":"2024-09-20T19:32:42.456819+0000","last_peered":"2024-09-20T19:32:42.456819+0000","last_clean":"2024-09-20T19:32:42.456819+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:32:42.456819+0000","last_undegraded":"2024-09-20T19:32:42.456819+0000","last_fullsized":"2024-09-20T19:32:42.456819+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:17.478143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:17.478143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1012,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436761+0000","last_change":"2024-09-20T19:32:14.826848+0000","last_active":"2024-09-20T19:32:41.436761+0000","last_peered":"2024-09-20T19:32:41.436761+0000","last_clean":"2024-09-20T19:32:41.436761+0000","last_became_active":"2024-09-20T19:30:59.424738+0000","last_became_peered":"2024-09-20T19:30:59.424738+0000","last_unstale":"2024-09-20T19:32:41.436761+0000","last_undegraded":"2024-09-20T19:32:41.436761+0000","last_fullsized":"2024-09-20T19:32:41.436761+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.259591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1293,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.654436+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:32:41.654436+0000","last_peered":"2024-09-20T19:32:41.654436+0000","last_clean":"2024-09-20T19:32:41.654436+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:32:41.654436+0000","last_undegraded":"2024-09-20T19:32:41.654436+0000","last_fullsized":"2024-09-20T19:32:41.654436+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.269206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"652'840","reported_seq":3110,"reported_epoch":654,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:42.477703+0000","last_change":"2024-09-20T19:32:42.477754+0000","last_active":"2024-09-20T19:32:42.477703+0000","last_peered":"2024-09-20T19:32:42.477703+0000","last_clean":"2024-09-20T19:32:42.477703+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:32:42.477703+0000","last_undegraded":"2024-09-20T19:32:42.477703+0000","last_fullsized":"2024-09-20T19:32:42.477703+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'779","last_scrub_stamp":"2024-09-20T19:31:22.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.595437+0000","objects_scrubbed":4,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.021139304000000001,"stat_sum":{"num_bytes":13948492,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":44308,"num_write":17,"num_write_kb":6613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"}]},{"pgid":"4.7","version":"523'17","reported_seq":1188,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656978+0000","last_change":"2024-09-20T19:31:39.546516+0000","last_active":"2024-09-20T19:32:41.656978+0000","last_peered":"2024-09-20T19:32:41.656978+0000","last_clean":"2024-09-20T19:32:41.656978+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:32:41.656978+0000","last_undegraded":"2024-09-20T19:32:41.656978+0000","last_fullsized":"2024-09-20T19:32:41.656978+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:31:39.546441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.546441+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.667130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2081,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.434153+0000","last_change":"2024-09-20T19:28:48.600562+0000","last_active":"2024-09-20T19:32:41.434153+0000","last_peered":"2024-09-20T19:32:41.434153+0000","last_clean":"2024-09-20T19:32:41.434153+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:32:41.434153+0000","last_undegraded":"2024-09-20T19:32:41.434153+0000","last_fullsized":"2024-09-20T19:32:41.434153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:28:48.600384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.600384+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.135772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":857,"num_read_kb":852,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1233,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644265+0000","last_change":"2024-09-20T19:31:23.521703+0000","last_active":"2024-09-20T19:32:41.644265+0000","last_peered":"2024-09-20T19:32:41.644265+0000","last_clean":"2024-09-20T19:32:41.644265+0000","last_became_active":"2024-09-20T19:30:59.416753+0000","last_became_peered":"2024-09-20T19:30:59.416753+0000","last_unstale":"2024-09-20T19:32:41.644265+0000","last_undegraded":"2024-09-20T19:32:41.644265+0000","last_fullsized":"2024-09-20T19:32:41.644265+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:23.521555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:23.521555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.929656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1222,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656967+0000","last_change":"2024-09-20T19:29:52.092899+0000","last_active":"2024-09-20T19:32:41.656967+0000","last_peered":"2024-09-20T19:32:41.656967+0000","last_clean":"2024-09-20T19:32:41.656967+0000","last_became_active":"2024-09-20T19:18:11.328805+0000","last_became_peered":"2024-09-20T19:18:11.328805+0000","last_unstale":"2024-09-20T19:32:41.656967+0000","last_undegraded":"2024-09-20T19:32:41.656967+0000","last_fullsized":"2024-09-20T19:32:41.656967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:29:52.092846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:52.092846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:47.539955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1218,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657015+0000","last_change":"2024-09-20T19:32:21.799367+0000","last_active":"2024-09-20T19:32:41.657015+0000","last_peered":"2024-09-20T19:32:41.657015+0000","last_clean":"2024-09-20T19:32:41.657015+0000","last_became_active":"2024-09-20T19:30:59.437740+0000","last_became_peered":"2024-09-20T19:30:59.437740+0000","last_unstale":"2024-09-20T19:32:41.657015+0000","last_undegraded":"2024-09-20T19:32:41.657015+0000","last_fullsized":"2024-09-20T19:32:41.657015+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.069051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":825,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436383+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:32:41.436383+0000","last_peered":"2024-09-20T19:32:41.436383+0000","last_clean":"2024-09-20T19:32:41.436383+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:32:41.436383+0000","last_undegraded":"2024-09-20T19:32:41.436383+0000","last_fullsized":"2024-09-20T19:32:41.436383+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:52.171547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"647'837","reported_seq":3489,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656612+0000","last_change":"2024-09-20T19:32:40.470872+0000","last_active":"2024-09-20T19:32:41.656612+0000","last_peered":"2024-09-20T19:32:41.656612+0000","last_clean":"2024-09-20T19:32:41.656612+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:32:41.656612+0000","last_undegraded":"2024-09-20T19:32:41.656612+0000","last_fullsized":"2024-09-20T19:32:41.656612+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:46.687067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.033876781000000002,"stat_sum":{"num_bytes":7260987,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":3330,"num_write":5,"num_write_kb":783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":1005,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457176+0000","last_change":"2024-09-20T19:30:09.829566+0000","last_active":"2024-09-20T19:32:42.457176+0000","last_peered":"2024-09-20T19:32:42.457176+0000","last_clean":"2024-09-20T19:32:42.457176+0000","last_became_active":"2024-09-20T19:30:09.783033+0000","last_became_peered":"2024-09-20T19:30:09.783033+0000","last_unstale":"2024-09-20T19:32:42.457176+0000","last_undegraded":"2024-09-20T19:32:42.457176+0000","last_fullsized":"2024-09-20T19:32:42.457176+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-20T19:25:21.127630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.127630+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.428713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":950,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.459884+0000","last_change":"2024-09-20T19:30:59.438831+0000","last_active":"2024-09-20T19:32:42.459884+0000","last_peered":"2024-09-20T19:32:42.459884+0000","last_clean":"2024-09-20T19:32:42.459884+0000","last_became_active":"2024-09-20T19:30:59.438639+0000","last_became_peered":"2024-09-20T19:30:59.438639+0000","last_unstale":"2024-09-20T19:32:42.459884+0000","last_undegraded":"2024-09-20T19:32:42.459884+0000","last_fullsized":"2024-09-20T19:32:42.459884+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:24:39.000765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.000765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.485035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1231,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653433+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:32:41.653433+0000","last_peered":"2024-09-20T19:32:41.653433+0000","last_clean":"2024-09-20T19:32:41.653433+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:32:41.653433+0000","last_undegraded":"2024-09-20T19:32:41.653433+0000","last_fullsized":"2024-09-20T19:32:41.653433+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:41.273146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1222,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659257+0000","last_change":"2024-09-20T19:32:23.505615+0000","last_active":"2024-09-20T19:32:41.659257+0000","last_peered":"2024-09-20T19:32:41.659257+0000","last_clean":"2024-09-20T19:32:41.659257+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:32:41.659257+0000","last_undegraded":"2024-09-20T19:32:41.659257+0000","last_fullsized":"2024-09-20T19:32:41.659257+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.505475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.505475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.486718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1230,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435004+0000","last_change":"2024-09-20T19:30:59.436100+0000","last_active":"2024-09-20T19:32:41.435004+0000","last_peered":"2024-09-20T19:32:41.435004+0000","last_clean":"2024-09-20T19:32:41.435004+0000","last_became_active":"2024-09-20T19:30:59.435560+0000","last_became_peered":"2024-09-20T19:30:59.435560+0000","last_unstale":"2024-09-20T19:32:41.435004+0000","last_undegraded":"2024-09-20T19:32:41.435004+0000","last_fullsized":"2024-09-20T19:32:41.435004+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:14.942057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:14.942057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.153381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":2549,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.479396+0000","last_change":"2024-09-20T19:32:42.480371+0000","last_active":"2024-09-20T19:32:42.479396+0000","last_peered":"2024-09-20T19:32:42.479396+0000","last_clean":"2024-09-20T19:32:42.479396+0000","last_became_active":"2024-09-20T19:30:59.449561+0000","last_became_peered":"2024-09-20T19:30:59.449561+0000","last_unstale":"2024-09-20T19:32:42.479396+0000","last_undegraded":"2024-09-20T19:32:42.479396+0000","last_fullsized":"2024-09-20T19:32:42.479396+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:31:29.529433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.529433+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022871589000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"2"}]},{"pgid":"4.5","version":"523'30","reported_seq":1434,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657759+0000","last_change":"2024-09-20T19:30:31.397491+0000","last_active":"2024-09-20T19:32:41.657759+0000","last_peered":"2024-09-20T19:32:41.657759+0000","last_clean":"2024-09-20T19:32:41.657759+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:32:41.657759+0000","last_undegraded":"2024-09-20T19:32:41.657759+0000","last_fullsized":"2024-09-20T19:32:41.657759+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:24:43.584143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.584143+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":1036,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.436796+0000","last_change":"2024-09-20T19:30:59.412249+0000","last_active":"2024-09-20T19:32:42.436796+0000","last_peered":"2024-09-20T19:32:42.436796+0000","last_clean":"2024-09-20T19:32:42.436796+0000","last_became_active":"2024-09-20T19:30:59.411724+0000","last_became_peered":"2024-09-20T19:30:59.411724+0000","last_unstale":"2024-09-20T19:32:42.436796+0000","last_undegraded":"2024-09-20T19:32:42.436796+0000","last_fullsized":"2024-09-20T19:32:42.436796+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:24:46.018399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:46.018399+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.503059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1211,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653952+0000","last_change":"2024-09-20T19:28:48.425075+0000","last_active":"2024-09-20T19:32:41.653952+0000","last_peered":"2024-09-20T19:32:41.653952+0000","last_clean":"2024-09-20T19:32:41.653952+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:32:41.653952+0000","last_undegraded":"2024-09-20T19:32:41.653952+0000","last_fullsized":"2024-09-20T19:32:41.653952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:48.424987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.424987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1272,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653948+0000","last_change":"2024-09-20T19:30:09.775314+0000","last_active":"2024-09-20T19:32:41.653948+0000","last_peered":"2024-09-20T19:32:41.653948+0000","last_clean":"2024-09-20T19:32:41.653948+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:32:41.653948+0000","last_undegraded":"2024-09-20T19:32:41.653948+0000","last_fullsized":"2024-09-20T19:32:41.653948+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:26:13.424049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.424049+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:39.829392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1203,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.655420+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:32:41.655420+0000","last_peered":"2024-09-20T19:32:41.655420+0000","last_clean":"2024-09-20T19:32:41.655420+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:32:41.655420+0000","last_undegraded":"2024-09-20T19:32:41.655420+0000","last_fullsized":"2024-09-20T19:32:41.655420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.790580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1217,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659703+0000","last_change":"2024-09-20T19:32:10.638734+0000","last_active":"2024-09-20T19:32:41.659703+0000","last_peered":"2024-09-20T19:32:41.659703+0000","last_clean":"2024-09-20T19:32:41.659703+0000","last_became_active":"2024-09-20T19:30:59.410315+0000","last_became_peered":"2024-09-20T19:30:59.410315+0000","last_unstale":"2024-09-20T19:32:41.659703+0000","last_undegraded":"2024-09-20T19:32:41.659703+0000","last_fullsized":"2024-09-20T19:32:41.659703+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:10.638671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.638671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.768982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"654'339","reported_seq":2661,"reported_epoch":654,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:42.480461+0000","last_change":"2024-09-20T19:32:42.480461+0000","last_active":"2024-09-20T19:32:42.480461+0000","last_peered":"2024-09-20T19:32:42.480461+0000","last_clean":"2024-09-20T19:32:42.480461+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:32:42.480461+0000","last_undegraded":"2024-09-20T19:32:42.480461+0000","last_fullsized":"2024-09-20T19:32:42.480461+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'300","last_scrub_stamp":"2024-09-20T19:31:19.575078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.575078+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.93531378499999995,"stat_sum":{"num_bytes":7013011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":20,"num_write_kb":5041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.a","version":"523'35","reported_seq":1262,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653219+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:32:41.653219+0000","last_peered":"2024-09-20T19:32:41.653219+0000","last_clean":"2024-09-20T19:32:41.653219+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:32:41.653219+0000","last_undegraded":"2024-09-20T19:32:41.653219+0000","last_fullsized":"2024-09-20T19:32:41.653219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1233,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656742+0000","last_change":"2024-09-20T19:31:40.539723+0000","last_active":"2024-09-20T19:32:41.656742+0000","last_peered":"2024-09-20T19:32:41.656742+0000","last_clean":"2024-09-20T19:32:41.656742+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:32:41.656742+0000","last_undegraded":"2024-09-20T19:32:41.656742+0000","last_fullsized":"2024-09-20T19:32:41.656742+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:40.539596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.539596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:45.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1225,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644313+0000","last_change":"2024-09-20T19:31:29.553212+0000","last_active":"2024-09-20T19:32:41.644313+0000","last_peered":"2024-09-20T19:32:41.644313+0000","last_clean":"2024-09-20T19:32:41.644313+0000","last_became_active":"2024-09-20T19:30:09.789268+0000","last_became_peered":"2024-09-20T19:30:09.789268+0000","last_unstale":"2024-09-20T19:32:41.644313+0000","last_undegraded":"2024-09-20T19:32:41.644313+0000","last_fullsized":"2024-09-20T19:32:41.644313+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.075543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1202,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653689+0000","last_change":"2024-09-20T19:32:15.777865+0000","last_active":"2024-09-20T19:32:41.653689+0000","last_peered":"2024-09-20T19:32:41.653689+0000","last_clean":"2024-09-20T19:32:41.653689+0000","last_became_active":"2024-09-20T19:30:59.440497+0000","last_became_peered":"2024-09-20T19:30:59.440497+0000","last_unstale":"2024-09-20T19:32:41.653689+0000","last_undegraded":"2024-09-20T19:32:41.653689+0000","last_fullsized":"2024-09-20T19:32:41.653689+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.419161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"650'1152","reported_seq":3712,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644121+0000","last_change":"2024-09-20T19:32:38.328172+0000","last_active":"2024-09-20T19:32:41.644121+0000","last_peered":"2024-09-20T19:32:41.644121+0000","last_clean":"2024-09-20T19:32:41.644121+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:32:41.644121+0000","last_undegraded":"2024-09-20T19:32:41.644121+0000","last_fullsized":"2024-09-20T19:32:41.644121+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'1120","last_scrub_stamp":"2024-09-20T19:32:08.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:08.554870+0000","objects_scrubbed":7,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.779947+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0069432679999999998,"stat_sum":{"num_bytes":15419141,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":44,"num_read_kb":26534,"num_write":50,"num_write_kb":23153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.b","version":"523'15","reported_seq":1013,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.458664+0000","last_change":"2024-09-20T19:32:15.464944+0000","last_active":"2024-09-20T19:32:42.458664+0000","last_peered":"2024-09-20T19:32:42.458664+0000","last_clean":"2024-09-20T19:32:42.458664+0000","last_became_active":"2024-09-20T19:30:31.349098+0000","last_became_peered":"2024-09-20T19:30:31.349098+0000","last_unstale":"2024-09-20T19:32:42.458664+0000","last_undegraded":"2024-09-20T19:32:42.458664+0000","last_fullsized":"2024-09-20T19:32:42.458664+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.464807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.464807+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.851627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1241,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437155+0000","last_change":"2024-09-20T19:31:29.374819+0000","last_active":"2024-09-20T19:32:41.437155+0000","last_peered":"2024-09-20T19:32:41.437155+0000","last_clean":"2024-09-20T19:32:41.437155+0000","last_became_active":"2024-09-20T19:30:59.434929+0000","last_became_peered":"2024-09-20T19:30:59.434929+0000","last_unstale":"2024-09-20T19:32:41.437155+0000","last_undegraded":"2024-09-20T19:32:41.437155+0000","last_fullsized":"2024-09-20T19:32:41.437155+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.983029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1232,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644382+0000","last_change":"2024-09-20T19:31:36.575855+0000","last_active":"2024-09-20T19:32:41.644382+0000","last_peered":"2024-09-20T19:32:41.644382+0000","last_clean":"2024-09-20T19:32:41.644382+0000","last_became_active":"2024-09-20T19:30:59.407432+0000","last_became_peered":"2024-09-20T19:30:59.407432+0000","last_unstale":"2024-09-20T19:32:41.644382+0000","last_undegraded":"2024-09-20T19:32:41.644382+0000","last_fullsized":"2024-09-20T19:32:41.644382+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.575803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.575803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.311917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":998,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.460000+0000","last_change":"2024-09-20T19:32:10.330837+0000","last_active":"2024-09-20T19:32:42.460000+0000","last_peered":"2024-09-20T19:32:42.460000+0000","last_clean":"2024-09-20T19:32:42.460000+0000","last_became_active":"2024-09-20T19:30:09.787067+0000","last_became_peered":"2024-09-20T19:30:09.787067+0000","last_unstale":"2024-09-20T19:32:42.460000+0000","last_undegraded":"2024-09-20T19:32:42.460000+0000","last_fullsized":"2024-09-20T19:32:42.460000+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:32:10.330783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.330783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.559325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"650'423","reported_seq":2993,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435096+0000","last_change":"2024-09-20T19:32:40.410203+0000","last_active":"2024-09-20T19:32:41.435096+0000","last_peered":"2024-09-20T19:32:41.435096+0000","last_clean":"2024-09-20T19:32:41.435096+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:32:41.435096+0000","last_undegraded":"2024-09-20T19:32:41.435096+0000","last_fullsized":"2024-09-20T19:32:41.435096+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'405","last_scrub_stamp":"2024-09-20T19:32:29.036109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.036109+0000","objects_scrubbed":7,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:32.887220+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":1.002304761,"stat_sum":{"num_bytes":16828051,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":169,"num_read_kb":125466,"num_write":107,"num_write_kb":50551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.8","version":"523'27","reported_seq":1258,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.651553+0000","last_change":"2024-09-20T19:29:49.731520+0000","last_active":"2024-09-20T19:32:41.651553+0000","last_peered":"2024-09-20T19:32:41.651553+0000","last_clean":"2024-09-20T19:32:41.651553+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:32:41.651553+0000","last_undegraded":"2024-09-20T19:32:41.651553+0000","last_fullsized":"2024-09-20T19:32:41.651553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'17","last_scrub_stamp":"2024-09-20T19:29:49.731377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:29:49.731377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":1249,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657211+0000","last_change":"2024-09-20T19:31:35.568682+0000","last_active":"2024-09-20T19:32:41.657211+0000","last_peered":"2024-09-20T19:32:41.657211+0000","last_clean":"2024-09-20T19:32:41.657211+0000","last_became_active":"2024-09-20T19:30:59.407885+0000","last_became_peered":"2024-09-20T19:30:59.407885+0000","last_unstale":"2024-09-20T19:32:41.657211+0000","last_undegraded":"2024-09-20T19:32:41.657211+0000","last_fullsized":"2024-09-20T19:32:41.657211+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:35.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.568627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:00.653341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1410,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.652864+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:32:41.652864+0000","last_peered":"2024-09-20T19:32:41.652864+0000","last_clean":"2024-09-20T19:32:41.652864+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:32:41.652864+0000","last_undegraded":"2024-09-20T19:32:41.652864+0000","last_fullsized":"2024-09-20T19:32:41.652864+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.376352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1210,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653724+0000","last_change":"2024-09-20T19:32:12.619664+0000","last_active":"2024-09-20T19:32:41.653724+0000","last_peered":"2024-09-20T19:32:41.653724+0000","last_clean":"2024-09-20T19:32:41.653724+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:32:41.653724+0000","last_undegraded":"2024-09-20T19:32:41.653724+0000","last_fullsized":"2024-09-20T19:32:41.653724+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.619550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.619550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.382228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"651'423","reported_seq":2747,"reported_epoch":654,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:42.436686+0000","last_change":"2024-09-20T19:32:42.436724+0000","last_active":"2024-09-20T19:32:42.436686+0000","last_peered":"2024-09-20T19:32:42.436686+0000","last_clean":"2024-09-20T19:32:42.436686+0000","last_became_active":"2024-09-20T19:30:59.413000+0000","last_became_peered":"2024-09-20T19:30:59.413000+0000","last_unstale":"2024-09-20T19:32:42.436686+0000","last_undegraded":"2024-09-20T19:32:42.436686+0000","last_fullsized":"2024-09-20T19:32:42.436686+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'344","last_scrub_stamp":"2024-09-20T19:27:23.499201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:23.499201+0000","objects_scrubbed":6,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99035099100000001,"stat_sum":{"num_bytes":13832590,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":182,"num_read_kb":149074,"num_write":133,"num_write_kb":56239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4084895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.9","version":"523'18","reported_seq":1087,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304564+0000","last_change":"2024-09-20T19:32:26.295823+0000","last_active":"2024-09-20T19:32:43.304564+0000","last_peered":"2024-09-20T19:32:43.304564+0000","last_clean":"2024-09-20T19:32:43.304564+0000","last_became_active":"2024-09-20T19:30:59.404224+0000","last_became_peered":"2024-09-20T19:30:59.404224+0000","last_unstale":"2024-09-20T19:32:43.304564+0000","last_undegraded":"2024-09-20T19:32:43.304564+0000","last_fullsized":"2024-09-20T19:32:43.304564+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.872180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1086,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.306181+0000","last_change":"2024-09-20T19:32:19.382969+0000","last_active":"2024-09-20T19:32:43.306181+0000","last_peered":"2024-09-20T19:32:43.306181+0000","last_clean":"2024-09-20T19:32:43.306181+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:32:43.306181+0000","last_undegraded":"2024-09-20T19:32:43.306181+0000","last_fullsized":"2024-09-20T19:32:43.306181+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:32:19.382850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:19.382850+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:24.971441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1225,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644087+0000","last_change":"2024-09-20T19:31:32.549379+0000","last_active":"2024-09-20T19:32:41.644087+0000","last_peered":"2024-09-20T19:32:41.644087+0000","last_clean":"2024-09-20T19:32:41.644087+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:32:41.644087+0000","last_undegraded":"2024-09-20T19:32:41.644087+0000","last_fullsized":"2024-09-20T19:32:41.644087+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:32.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:32.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:44.516267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1051,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.438644+0000","last_change":"2024-09-20T19:32:11.827975+0000","last_active":"2024-09-20T19:32:42.438644+0000","last_peered":"2024-09-20T19:32:42.438644+0000","last_clean":"2024-09-20T19:32:42.438644+0000","last_became_active":"2024-09-20T19:30:59.425091+0000","last_became_peered":"2024-09-20T19:30:59.425091+0000","last_unstale":"2024-09-20T19:32:42.438644+0000","last_undegraded":"2024-09-20T19:32:42.438644+0000","last_fullsized":"2024-09-20T19:32:42.438644+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.305742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"652'917","reported_seq":3623,"reported_epoch":653,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:41.657751+0000","last_change":"2024-09-20T19:32:40.474166+0000","last_active":"2024-09-20T19:32:41.657751+0000","last_peered":"2024-09-20T19:32:41.657751+0000","last_clean":"2024-09-20T19:32:41.657751+0000","last_became_active":"2024-09-20T19:28:48.211105+0000","last_became_peered":"2024-09-20T19:28:48.211105+0000","last_unstale":"2024-09-20T19:32:41.657751+0000","last_undegraded":"2024-09-20T19:32:41.657751+0000","last_fullsized":"2024-09-20T19:32:41.657751+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"483'743","last_scrub_stamp":"2024-09-20T19:29:15.367390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:15.367390+0000","objects_scrubbed":10,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.034746612000000003,"stat_sum":{"num_bytes":24124394,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":66,"num_read_kb":43687,"num_write":55,"num_write_kb":16705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"}]},{"pgid":"4.16","version":"523'15","reported_seq":1015,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.456822+0000","last_change":"2024-09-20T19:32:16.493984+0000","last_active":"2024-09-20T19:32:42.456822+0000","last_peered":"2024-09-20T19:32:42.456822+0000","last_clean":"2024-09-20T19:32:42.456822+0000","last_became_active":"2024-09-20T19:30:59.447014+0000","last_became_peered":"2024-09-20T19:30:59.447014+0000","last_unstale":"2024-09-20T19:32:42.456822+0000","last_undegraded":"2024-09-20T19:32:42.456822+0000","last_fullsized":"2024-09-20T19:32:42.456822+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:18.681371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1249,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644345+0000","last_change":"2024-09-20T19:31:41.498323+0000","last_active":"2024-09-20T19:32:41.644345+0000","last_peered":"2024-09-20T19:32:41.644345+0000","last_clean":"2024-09-20T19:32:41.644345+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:32:41.644345+0000","last_undegraded":"2024-09-20T19:32:41.644345+0000","last_fullsized":"2024-09-20T19:32:41.644345+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:41.498237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.498237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:06.336115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1027,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436805+0000","last_change":"2024-09-20T19:30:59.429090+0000","last_active":"2024-09-20T19:32:41.436805+0000","last_peered":"2024-09-20T19:32:41.436805+0000","last_clean":"2024-09-20T19:32:41.436805+0000","last_became_active":"2024-09-20T19:30:59.426634+0000","last_became_peered":"2024-09-20T19:30:59.426634+0000","last_unstale":"2024-09-20T19:32:41.436805+0000","last_undegraded":"2024-09-20T19:32:41.436805+0000","last_fullsized":"2024-09-20T19:32:41.436805+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:01.260486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:01.260486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1213,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.644381+0000","last_change":"2024-09-20T19:30:31.368696+0000","last_active":"2024-09-20T19:32:41.644381+0000","last_peered":"2024-09-20T19:32:41.644381+0000","last_clean":"2024-09-20T19:32:41.644381+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:32:41.644381+0000","last_undegraded":"2024-09-20T19:32:41.644381+0000","last_fullsized":"2024-09-20T19:32:41.644381+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:31.849481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:24:31.849481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:01.544107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"653'957","reported_seq":3468,"reported_epoch":653,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:42.422628+0000","last_change":"2024-09-20T19:32:40.412904+0000","last_active":"2024-09-20T19:32:42.422628+0000","last_peered":"2024-09-20T19:32:42.422628+0000","last_clean":"2024-09-20T19:32:42.422628+0000","last_became_active":"2024-09-20T19:31:02.399794+0000","last_became_peered":"2024-09-20T19:31:02.399794+0000","last_unstale":"2024-09-20T19:32:42.422628+0000","last_undegraded":"2024-09-20T19:32:42.422628+0000","last_fullsized":"2024-09-20T19:32:42.422628+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'769","last_scrub_stamp":"2024-09-20T19:28:51.165603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:51.165603+0000","objects_scrubbed":9,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.102027935,"stat_sum":{"num_bytes":19369783,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":61,"num_read_kb":32314,"num_write":46,"num_write_kb":18418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":13361098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"2"}]},{"pgid":"2.11","version":"0'0","reported_seq":1222,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653091+0000","last_change":"2024-09-20T19:30:31.386421+0000","last_active":"2024-09-20T19:32:41.653091+0000","last_peered":"2024-09-20T19:32:41.653091+0000","last_clean":"2024-09-20T19:32:41.653091+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:32:41.653091+0000","last_undegraded":"2024-09-20T19:32:41.653091+0000","last_fullsized":"2024-09-20T19:32:41.653091+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.575807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.575807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1066,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437231+0000","last_change":"2024-09-20T19:31:41.454838+0000","last_active":"2024-09-20T19:32:41.437231+0000","last_peered":"2024-09-20T19:32:41.437231+0000","last_clean":"2024-09-20T19:32:41.437231+0000","last_became_active":"2024-09-20T19:30:59.429434+0000","last_became_peered":"2024-09-20T19:30:59.429434+0000","last_unstale":"2024-09-20T19:32:41.437231+0000","last_undegraded":"2024-09-20T19:32:41.437231+0000","last_fullsized":"2024-09-20T19:32:41.437231+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:54.613627+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1221,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658435+0000","last_change":"2024-09-20T19:30:59.412406+0000","last_active":"2024-09-20T19:32:41.658435+0000","last_peered":"2024-09-20T19:32:41.658435+0000","last_clean":"2024-09-20T19:32:41.658435+0000","last_became_active":"2024-09-20T19:30:59.411374+0000","last_became_peered":"2024-09-20T19:30:59.411374+0000","last_unstale":"2024-09-20T19:32:41.658435+0000","last_undegraded":"2024-09-20T19:32:41.658435+0000","last_fullsized":"2024-09-20T19:32:41.658435+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:59.345399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:27:59.345399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1202,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656880+0000","last_change":"2024-09-20T19:30:31.376526+0000","last_active":"2024-09-20T19:32:41.656880+0000","last_peered":"2024-09-20T19:32:41.656880+0000","last_clean":"2024-09-20T19:32:41.656880+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:32:41.656880+0000","last_undegraded":"2024-09-20T19:32:41.656880+0000","last_fullsized":"2024-09-20T19:32:41.656880+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.296694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.296694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.879308+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"650'536","reported_seq":3052,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656285+0000","last_change":"2024-09-20T19:32:40.472950+0000","last_active":"2024-09-20T19:32:41.656285+0000","last_peered":"2024-09-20T19:32:41.656285+0000","last_clean":"2024-09-20T19:32:41.656285+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:32:41.656285+0000","last_undegraded":"2024-09-20T19:32:41.656285+0000","last_fullsized":"2024-09-20T19:32:41.656285+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:18.486613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.033386315,"stat_sum":{"num_bytes":11311781,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":11,"num_read_kb":4546,"num_write":6,"num_write_kb":948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.14","version":"523'20","reported_seq":1076,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436060+0000","last_change":"2024-09-20T19:31:40.419678+0000","last_active":"2024-09-20T19:32:41.436060+0000","last_peered":"2024-09-20T19:32:41.436060+0000","last_clean":"2024-09-20T19:32:41.436060+0000","last_became_active":"2024-09-20T19:30:59.428000+0000","last_became_peered":"2024-09-20T19:30:59.428000+0000","last_unstale":"2024-09-20T19:32:41.436060+0000","last_undegraded":"2024-09-20T19:32:41.436060+0000","last_fullsized":"2024-09-20T19:32:41.436060+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:55.130318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1230,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.654645+0000","last_change":"2024-09-20T19:30:59.412019+0000","last_active":"2024-09-20T19:32:41.654645+0000","last_peered":"2024-09-20T19:32:41.654645+0000","last_clean":"2024-09-20T19:32:41.654645+0000","last_became_active":"2024-09-20T19:30:59.411617+0000","last_became_peered":"2024-09-20T19:30:59.411617+0000","last_unstale":"2024-09-20T19:32:41.654645+0000","last_undegraded":"2024-09-20T19:32:41.654645+0000","last_fullsized":"2024-09-20T19:32:41.654645+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.424475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.424475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1410,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658070+0000","last_change":"2024-09-20T19:32:11.594974+0000","last_active":"2024-09-20T19:32:41.658070+0000","last_peered":"2024-09-20T19:32:41.658070+0000","last_clean":"2024-09-20T19:32:41.658070+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:32:41.658070+0000","last_undegraded":"2024-09-20T19:32:41.658070+0000","last_fullsized":"2024-09-20T19:32:41.658070+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:11.594902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.594902+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:28.976985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":997,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457859+0000","last_change":"2024-09-20T19:32:11.370896+0000","last_active":"2024-09-20T19:32:42.457859+0000","last_peered":"2024-09-20T19:32:42.457859+0000","last_clean":"2024-09-20T19:32:42.457859+0000","last_became_active":"2024-09-20T19:30:59.447481+0000","last_became_peered":"2024-09-20T19:30:59.447481+0000","last_unstale":"2024-09-20T19:32:42.457859+0000","last_undegraded":"2024-09-20T19:32:42.457859+0000","last_fullsized":"2024-09-20T19:32:42.457859+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.301214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"655'360","reported_seq":2692,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.314381+0000","last_change":"2024-09-20T19:32:43.318094+0000","last_active":"2024-09-20T19:32:43.314381+0000","last_peered":"2024-09-20T19:32:43.314381+0000","last_clean":"2024-09-20T19:32:43.314381+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:32:43.314381+0000","last_undegraded":"2024-09-20T19:32:43.314381+0000","last_fullsized":"2024-09-20T19:32:43.314381+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"527'291","last_scrub_stamp":"2024-09-20T19:30:37.025255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:37.025255+0000","objects_scrubbed":5,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.012498458,"stat_sum":{"num_bytes":11997415,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":51,"num_read_kb":32139,"num_write":23,"num_write_kb":8912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"da","length":"1"},{"start":"ec","length":"3"},{"start":"f0","length":"3"},{"start":"f6","length":"2"}]},{"pgid":"4.15","version":"523'15","reported_seq":1258,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657537+0000","last_change":"2024-09-20T19:30:59.427602+0000","last_active":"2024-09-20T19:32:41.657537+0000","last_peered":"2024-09-20T19:32:41.657537+0000","last_clean":"2024-09-20T19:32:41.657537+0000","last_became_active":"2024-09-20T19:30:59.404598+0000","last_became_peered":"2024-09-20T19:30:59.404598+0000","last_unstale":"2024-09-20T19:32:41.657537+0000","last_undegraded":"2024-09-20T19:32:41.657537+0000","last_fullsized":"2024-09-20T19:32:41.657537+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:27:22.506346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:22.506346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:40.357492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":970,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.458884+0000","last_change":"2024-09-20T19:32:26.439201+0000","last_active":"2024-09-20T19:32:42.458884+0000","last_peered":"2024-09-20T19:32:42.458884+0000","last_clean":"2024-09-20T19:32:42.458884+0000","last_became_active":"2024-09-20T19:30:59.448308+0000","last_became_peered":"2024-09-20T19:30:59.448308+0000","last_unstale":"2024-09-20T19:32:42.458884+0000","last_undegraded":"2024-09-20T19:32:42.458884+0000","last_fullsized":"2024-09-20T19:32:42.458884+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.836552+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"554'13","reported_seq":1196,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435836+0000","last_change":"2024-09-20T19:30:59.427899+0000","last_active":"2024-09-20T19:32:41.435836+0000","last_peered":"2024-09-20T19:32:41.435836+0000","last_clean":"2024-09-20T19:32:41.435836+0000","last_became_active":"2024-09-20T19:30:59.425446+0000","last_became_peered":"2024-09-20T19:30:59.425446+0000","last_unstale":"2024-09-20T19:32:41.435836+0000","last_undegraded":"2024-09-20T19:32:41.435836+0000","last_fullsized":"2024-09-20T19:32:41.435836+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:28:18.243973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:18.243973+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1047,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.305563+0000","last_change":"2024-09-20T19:31:19.806432+0000","last_active":"2024-09-20T19:32:43.305563+0000","last_peered":"2024-09-20T19:32:43.305563+0000","last_clean":"2024-09-20T19:32:43.305563+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:32:43.305563+0000","last_undegraded":"2024-09-20T19:32:43.305563+0000","last_fullsized":"2024-09-20T19:32:43.305563+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:19.806369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.806369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"599'455","reported_seq":3034,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.652464+0000","last_change":"2024-09-20T19:32:38.469273+0000","last_active":"2024-09-20T19:32:41.652464+0000","last_peered":"2024-09-20T19:32:41.652464+0000","last_clean":"2024-09-20T19:32:41.652464+0000","last_became_active":"2024-09-20T19:31:00.440451+0000","last_became_peered":"2024-09-20T19:31:00.440451+0000","last_unstale":"2024-09-20T19:32:41.652464+0000","last_undegraded":"2024-09-20T19:32:41.652464+0000","last_fullsized":"2024-09-20T19:32:41.652464+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.18167428399999999,"stat_sum":{"num_bytes":3428727,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.12","version":"523'15","reported_seq":1252,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.434575+0000","last_change":"2024-09-20T19:32:31.080255+0000","last_active":"2024-09-20T19:32:41.434575+0000","last_peered":"2024-09-20T19:32:41.434575+0000","last_clean":"2024-09-20T19:32:41.434575+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:32:41.434575+0000","last_undegraded":"2024-09-20T19:32:41.434575+0000","last_fullsized":"2024-09-20T19:32:41.434575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:31.080102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.080102+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.008745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1214,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.657102+0000","last_change":"2024-09-20T19:30:59.412393+0000","last_active":"2024-09-20T19:32:41.657102+0000","last_peered":"2024-09-20T19:32:41.657102+0000","last_clean":"2024-09-20T19:32:41.657102+0000","last_became_active":"2024-09-20T19:30:59.412120+0000","last_became_peered":"2024-09-20T19:30:59.412120+0000","last_unstale":"2024-09-20T19:32:41.657102+0000","last_undegraded":"2024-09-20T19:32:41.657102+0000","last_fullsized":"2024-09-20T19:32:41.657102+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.625450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.625450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1067,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.438963+0000","last_change":"2024-09-20T19:31:35.565401+0000","last_active":"2024-09-20T19:32:42.438963+0000","last_peered":"2024-09-20T19:32:42.438963+0000","last_clean":"2024-09-20T19:32:42.438963+0000","last_became_active":"2024-09-20T19:30:59.432219+0000","last_became_peered":"2024-09-20T19:30:59.432219+0000","last_unstale":"2024-09-20T19:32:42.438963+0000","last_undegraded":"2024-09-20T19:32:42.438963+0000","last_fullsized":"2024-09-20T19:32:42.438963+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:53.758533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":979,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457161+0000","last_change":"2024-09-20T19:30:09.763911+0000","last_active":"2024-09-20T19:32:42.457161+0000","last_peered":"2024-09-20T19:32:42.457161+0000","last_clean":"2024-09-20T19:32:42.457161+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:32:42.457161+0000","last_undegraded":"2024-09-20T19:32:42.457161+0000","last_fullsized":"2024-09-20T19:32:42.457161+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:31.041093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:31.041093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:59.430814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"632'591","reported_seq":3110,"reported_epoch":654,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:42.436508+0000","last_change":"2024-09-20T19:32:42.436471+0000","last_active":"2024-09-20T19:32:42.436508+0000","last_peered":"2024-09-20T19:32:42.436508+0000","last_clean":"2024-09-20T19:32:42.436508+0000","last_became_active":"2024-09-20T19:31:48.284326+0000","last_became_peered":"2024-09-20T19:31:48.284326+0000","last_unstale":"2024-09-20T19:32:42.436508+0000","last_undegraded":"2024-09-20T19:32:42.436508+0000","last_fullsized":"2024-09-20T19:32:42.436508+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'561","last_scrub_stamp":"2024-09-20T19:28:49.166999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:49.166999+0000","objects_scrubbed":1,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.97232803000000001,"stat_sum":{"num_bytes":3473963,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":5242,"num_write":10,"num_write_kb":2213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3019792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"},{"start":"f6","length":"1"}]},{"pgid":"4.13","version":"523'19","reported_seq":1062,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303135+0000","last_change":"2024-09-20T19:32:32.189200+0000","last_active":"2024-09-20T19:32:43.303135+0000","last_peered":"2024-09-20T19:32:43.303135+0000","last_clean":"2024-09-20T19:32:43.303135+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:32:43.303135+0000","last_undegraded":"2024-09-20T19:32:43.303135+0000","last_fullsized":"2024-09-20T19:32:43.303135+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:32.189122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.189122+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.082120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1242,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437056+0000","last_change":"2024-09-20T19:31:20.343942+0000","last_active":"2024-09-20T19:32:41.437056+0000","last_peered":"2024-09-20T19:32:41.437056+0000","last_clean":"2024-09-20T19:32:41.437056+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:32:41.437056+0000","last_undegraded":"2024-09-20T19:32:41.437056+0000","last_fullsized":"2024-09-20T19:32:41.437056+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:20.343882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.343882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:43.923955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1221,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.434325+0000","last_change":"2024-09-20T19:31:45.587402+0000","last_active":"2024-09-20T19:32:41.434325+0000","last_peered":"2024-09-20T19:32:41.434325+0000","last_clean":"2024-09-20T19:32:41.434325+0000","last_became_active":"2024-09-20T19:31:45.586887+0000","last_became_peered":"2024-09-20T19:31:45.586887+0000","last_unstale":"2024-09-20T19:32:41.434325+0000","last_undegraded":"2024-09-20T19:32:41.434325+0000","last_fullsized":"2024-09-20T19:32:41.434325+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.571921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.571921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:54.116440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1063,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.435491+0000","last_change":"2024-09-20T19:31:36.528455+0000","last_active":"2024-09-20T19:32:42.435491+0000","last_peered":"2024-09-20T19:32:42.435491+0000","last_clean":"2024-09-20T19:32:42.435491+0000","last_became_active":"2024-09-20T19:30:59.431451+0000","last_became_peered":"2024-09-20T19:30:59.431451+0000","last_unstale":"2024-09-20T19:32:42.435491+0000","last_undegraded":"2024-09-20T19:32:42.435491+0000","last_fullsized":"2024-09-20T19:32:42.435491+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.528327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.528327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:50.129198+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"653'426","reported_seq":2911,"reported_epoch":654,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:32:42.458376+0000","last_change":"2024-09-20T19:32:42.406504+0000","last_active":"2024-09-20T19:32:42.458376+0000","last_peered":"2024-09-20T19:32:42.458376+0000","last_clean":"2024-09-20T19:32:42.458376+0000","last_became_active":"2024-09-20T19:30:12.800137+0000","last_became_peered":"2024-09-20T19:30:12.800137+0000","last_unstale":"2024-09-20T19:32:42.458376+0000","last_undegraded":"2024-09-20T19:32:42.458376+0000","last_fullsized":"2024-09-20T19:32:42.458376+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"524'363","last_scrub_stamp":"2024-09-20T19:30:33.692750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:33.692750+0000","objects_scrubbed":3,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032316286,"stat_sum":{"num_bytes":7698035,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":6307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"3"}]},{"pgid":"4.10","version":"523'8","reported_seq":1071,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.435889+0000","last_change":"2024-09-20T19:32:18.830939+0000","last_active":"2024-09-20T19:32:42.435889+0000","last_peered":"2024-09-20T19:32:42.435889+0000","last_clean":"2024-09-20T19:32:42.435889+0000","last_became_active":"2024-09-20T19:30:59.418597+0000","last_became_peered":"2024-09-20T19:30:59.418597+0000","last_unstale":"2024-09-20T19:32:42.435889+0000","last_undegraded":"2024-09-20T19:32:42.435889+0000","last_fullsized":"2024-09-20T19:32:42.435889+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:32:18.830843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.830843+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.107094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1203,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.658975+0000","last_change":"2024-09-20T19:23:42.758089+0000","last_active":"2024-09-20T19:32:41.658975+0000","last_peered":"2024-09-20T19:32:41.658975+0000","last_clean":"2024-09-20T19:32:41.658975+0000","last_became_active":"2024-09-20T19:18:07.261392+0000","last_became_peered":"2024-09-20T19:18:07.261392+0000","last_unstale":"2024-09-20T19:32:41.658975+0000","last_undegraded":"2024-09-20T19:32:41.658975+0000","last_fullsized":"2024-09-20T19:32:41.658975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:23:42.757941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:23:42.757941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:57.565392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1214,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.652086+0000","last_change":"2024-09-20T19:30:31.387063+0000","last_active":"2024-09-20T19:32:41.652086+0000","last_peered":"2024-09-20T19:32:41.652086+0000","last_clean":"2024-09-20T19:32:41.652086+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:32:41.652086+0000","last_undegraded":"2024-09-20T19:32:41.652086+0000","last_fullsized":"2024-09-20T19:32:41.652086+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.544996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.544996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1202,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653767+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:32:41.653767+0000","last_peered":"2024-09-20T19:32:41.653767+0000","last_clean":"2024-09-20T19:32:41.653767+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:32:41.653767+0000","last_undegraded":"2024-09-20T19:32:41.653767+0000","last_fullsized":"2024-09-20T19:32:41.653767+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:00.258729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":2638,"reported_epoch":653,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:32:41.652037+0000","last_change":"2024-09-20T19:32:38.462487+0000","last_active":"2024-09-20T19:32:41.652037+0000","last_peered":"2024-09-20T19:32:41.652037+0000","last_clean":"2024-09-20T19:32:41.652037+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:32:41.652037+0000","last_undegraded":"2024-09-20T19:32:41.652037+0000","last_fullsized":"2024-09-20T19:32:41.652037+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:29:00.472909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.472909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.17521742700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f0","length":"1"}]},{"pgid":"4.11","version":"523'19","reported_seq":1086,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.436214+0000","last_change":"2024-09-20T19:32:16.785202+0000","last_active":"2024-09-20T19:32:42.436214+0000","last_peered":"2024-09-20T19:32:42.436214+0000","last_clean":"2024-09-20T19:32:42.436214+0000","last_became_active":"2024-09-20T19:30:59.426703+0000","last_became_peered":"2024-09-20T19:30:59.426703+0000","last_unstale":"2024-09-20T19:32:42.436214+0000","last_undegraded":"2024-09-20T19:32:42.436214+0000","last_fullsized":"2024-09-20T19:32:42.436214+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.001001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1219,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659651+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:32:41.659651+0000","last_peered":"2024-09-20T19:32:41.659651+0000","last_clean":"2024-09-20T19:32:41.659651+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:32:41.659651+0000","last_undegraded":"2024-09-20T19:32:41.659651+0000","last_fullsized":"2024-09-20T19:32:41.659651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1222,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.652224+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:32:41.652224+0000","last_peered":"2024-09-20T19:32:41.652224+0000","last_clean":"2024-09-20T19:32:41.652224+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:32:41.652224+0000","last_undegraded":"2024-09-20T19:32:41.652224+0000","last_fullsized":"2024-09-20T19:32:41.652224+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.725052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1071,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.438764+0000","last_change":"2024-09-20T19:30:59.432385+0000","last_active":"2024-09-20T19:32:42.438764+0000","last_peered":"2024-09-20T19:32:42.438764+0000","last_clean":"2024-09-20T19:32:42.438764+0000","last_became_active":"2024-09-20T19:30:59.431821+0000","last_became_peered":"2024-09-20T19:30:59.431821+0000","last_unstale":"2024-09-20T19:32:42.438764+0000","last_undegraded":"2024-09-20T19:32:42.438764+0000","last_fullsized":"2024-09-20T19:32:42.438764+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:20.682353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:20.682353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:57.593210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":1200,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.655201+0000","last_change":"2024-09-20T19:28:13.311223+0000","last_active":"2024-09-20T19:32:41.655201+0000","last_peered":"2024-09-20T19:32:41.655201+0000","last_clean":"2024-09-20T19:32:41.655201+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:32:41.655201+0000","last_undegraded":"2024-09-20T19:32:41.655201+0000","last_fullsized":"2024-09-20T19:32:41.655201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:28:13.311036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:13.311036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1238,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.651724+0000","last_change":"2024-09-20T19:32:31.401371+0000","last_active":"2024-09-20T19:32:41.651724+0000","last_peered":"2024-09-20T19:32:41.651724+0000","last_clean":"2024-09-20T19:32:41.651724+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:32:41.651724+0000","last_undegraded":"2024-09-20T19:32:41.651724+0000","last_fullsized":"2024-09-20T19:32:41.651724+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:31.401257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.401257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.241223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":1004,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457052+0000","last_change":"2024-09-20T19:31:36.551849+0000","last_active":"2024-09-20T19:32:42.457052+0000","last_peered":"2024-09-20T19:32:42.457052+0000","last_clean":"2024-09-20T19:32:42.457052+0000","last_became_active":"2024-09-20T19:30:59.447209+0000","last_became_peered":"2024-09-20T19:30:59.447209+0000","last_unstale":"2024-09-20T19:32:42.457052+0000","last_undegraded":"2024-09-20T19:32:42.457052+0000","last_fullsized":"2024-09-20T19:32:42.457052+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:36.551731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.551731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:02.578982+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1413,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659081+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:32:41.659081+0000","last_peered":"2024-09-20T19:32:41.659081+0000","last_clean":"2024-09-20T19:32:41.659081+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:32:41.659081+0000","last_undegraded":"2024-09-20T19:32:41.659081+0000","last_fullsized":"2024-09-20T19:32:41.659081+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1221,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.438196+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:32:41.438196+0000","last_peered":"2024-09-20T19:32:41.438196+0000","last_clean":"2024-09-20T19:32:41.438196+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:32:41.438196+0000","last_undegraded":"2024-09-20T19:32:41.438196+0000","last_fullsized":"2024-09-20T19:32:41.438196+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:41.117115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":1084,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.436314+0000","last_change":"2024-09-20T19:30:59.433439+0000","last_active":"2024-09-20T19:32:42.436314+0000","last_peered":"2024-09-20T19:32:42.436314+0000","last_clean":"2024-09-20T19:32:42.436314+0000","last_became_active":"2024-09-20T19:30:59.432846+0000","last_became_peered":"2024-09-20T19:30:59.432846+0000","last_unstale":"2024-09-20T19:32:42.436314+0000","last_undegraded":"2024-09-20T19:32:42.436314+0000","last_fullsized":"2024-09-20T19:32:42.436314+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:24:30.106245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.106245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:42.482721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1268,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.659349+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:32:41.659349+0000","last_peered":"2024-09-20T19:32:41.659349+0000","last_clean":"2024-09-20T19:32:41.659349+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:32:41.659349+0000","last_undegraded":"2024-09-20T19:32:41.659349+0000","last_fullsized":"2024-09-20T19:32:41.659349+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1002,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.457735+0000","last_change":"2024-09-20T19:32:20.455596+0000","last_active":"2024-09-20T19:32:42.457735+0000","last_peered":"2024-09-20T19:32:42.457735+0000","last_clean":"2024-09-20T19:32:42.457735+0000","last_became_active":"2024-09-20T19:30:09.786731+0000","last_became_peered":"2024-09-20T19:30:09.786731+0000","last_unstale":"2024-09-20T19:32:42.457735+0000","last_undegraded":"2024-09-20T19:32:42.457735+0000","last_fullsized":"2024-09-20T19:32:42.457735+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.455533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.455533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.459393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1045,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.304462+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:32:43.304462+0000","last_peered":"2024-09-20T19:32:43.304462+0000","last_clean":"2024-09-20T19:32:43.304462+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:32:43.304462+0000","last_undegraded":"2024-09-20T19:32:43.304462+0000","last_fullsized":"2024-09-20T19:32:43.304462+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:00.429934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1229,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.653676+0000","last_change":"2024-09-20T19:32:15.866641+0000","last_active":"2024-09-20T19:32:41.653676+0000","last_peered":"2024-09-20T19:32:41.653676+0000","last_clean":"2024-09-20T19:32:41.653676+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:32:41.653676+0000","last_undegraded":"2024-09-20T19:32:41.653676+0000","last_fullsized":"2024-09-20T19:32:41.653676+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.866572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.866572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.064376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":1280,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.656509+0000","last_change":"2024-09-20T19:30:59.448664+0000","last_active":"2024-09-20T19:32:41.656509+0000","last_peered":"2024-09-20T19:32:41.656509+0000","last_clean":"2024-09-20T19:32:41.656509+0000","last_became_active":"2024-09-20T19:30:59.414544+0000","last_became_peered":"2024-09-20T19:30:59.414544+0000","last_unstale":"2024-09-20T19:32:41.656509+0000","last_undegraded":"2024-09-20T19:32:41.656509+0000","last_fullsized":"2024-09-20T19:32:41.656509+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-20T19:28:02.451854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.451854+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:32:46.533831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1232,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.436754+0000","last_change":"2024-09-20T19:32:16.075391+0000","last_active":"2024-09-20T19:32:41.436754+0000","last_peered":"2024-09-20T19:32:41.436754+0000","last_clean":"2024-09-20T19:32:41.436754+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:32:41.436754+0000","last_undegraded":"2024-09-20T19:32:41.436754+0000","last_fullsized":"2024-09-20T19:32:41.436754+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.075336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.075336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.476752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1023,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.437075+0000","last_change":"2024-09-20T19:30:59.414333+0000","last_active":"2024-09-20T19:32:41.437075+0000","last_peered":"2024-09-20T19:32:41.437075+0000","last_clean":"2024-09-20T19:32:41.437075+0000","last_became_active":"2024-09-20T19:30:59.413650+0000","last_became_peered":"2024-09-20T19:30:59.413650+0000","last_unstale":"2024-09-20T19:32:41.437075+0000","last_undegraded":"2024-09-20T19:32:41.437075+0000","last_fullsized":"2024-09-20T19:32:41.437075+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:54.253853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.253853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1036,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-20T19:32:42.438873+0000","last_change":"2024-09-20T19:30:59.425869+0000","last_active":"2024-09-20T19:32:42.438873+0000","last_peered":"2024-09-20T19:32:42.438873+0000","last_clean":"2024-09-20T19:32:42.438873+0000","last_became_active":"2024-09-20T19:30:59.424644+0000","last_became_peered":"2024-09-20T19:30:59.424644+0000","last_unstale":"2024-09-20T19:32:42.438873+0000","last_undegraded":"2024-09-20T19:32:42.438873+0000","last_fullsized":"2024-09-20T19:32:42.438873+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:58.281416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:58.281416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":1042,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-09-20T19:32:41.435509+0000","last_change":"2024-09-20T19:32:10.800441+0000","last_active":"2024-09-20T19:32:41.435509+0000","last_peered":"2024-09-20T19:32:41.435509+0000","last_clean":"2024-09-20T19:32:41.435509+0000","last_became_active":"2024-09-20T19:30:59.418441+0000","last_became_peered":"2024-09-20T19:30:59.418441+0000","last_unstale":"2024-09-20T19:32:41.435509+0000","last_undegraded":"2024-09-20T19:32:41.435509+0000","last_fullsized":"2024-09-20T19:32:41.435509+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:10.800347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.800347+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.402765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1043,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-20T19:32:43.303545+0000","last_change":"2024-09-20T19:32:33.198084+0000","last_active":"2024-09-20T19:32:43.303545+0000","last_peered":"2024-09-20T19:32:43.303545+0000","last_clean":"2024-09-20T19:32:43.303545+0000","last_became_active":"2024-09-20T19:30:59.409071+0000","last_became_peered":"2024-09-20T19:30:59.409071+0000","last_unstale":"2024-09-20T19:32:43.303545+0000","last_undegraded":"2024-09-20T19:32:43.303545+0000","last_fullsized":"2024-09-20T19:32:43.303545+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:33.198003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:33.198003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.209339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":281563413,"num_objects":139,"num_object_clones":91,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":13,"num_read":1751,"num_read_kb":1335346,"num_write":1265,"num_write_kb":479433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":88476679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606834688,"data_stored":812141513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":14234,"ondisk_log_size":14234,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":857,"num_read_kb":852,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907128,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":157592,"kb_used_data":100028,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93587560,"statfs":{"total":95995035648,"available":95833661440,"internally_reserved":0,"allocated":102428672,"data_stored":127228929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234179,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96628,"kb_used_data":65436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93648524,"statfs":{"total":95995035648,"available":95896088576,"internally_reserved":0,"allocated":67006464,"data_stored":88244865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561229,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94816,"kb_used_data":67788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93650336,"statfs":{"total":95995035648,"available":95897944064,"internally_reserved":0,"allocated":69414912,"data_stored":95721641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928540,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89176,"kb_used_data":52988,"kb_used_omap":35,"kb_used_meta":36124,"kb_avail":93655976,"statfs":{"total":95995035648,"available":95903719424,"internally_reserved":0,"allocated":54259712,"data_stored":69641005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":36991921},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":550,"seq":2362232012822,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":133108,"kb_used_data":96224,"kb_used_omap":51,"kb_used_meta":36812,"kb_avail":93612044,"statfs":{"total":95995035648,"available":95858733056,"internally_reserved":0,"allocated":98533376,"data_stored":127576872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52777,"internal_metadata":37695959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378865,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138116,"kb_used_data":93032,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93607036,"statfs":{"total":95995035648,"available":95853604864,"internally_reserved":0,"allocated":95264768,"data_stored":124078749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575101,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131916,"kb_used_data":78512,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93613236,"statfs":{"total":95995035648,"available":95859953664,"internally_reserved":0,"allocated":80396288,"data_stored":102998665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648032,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114888,"kb_used_data":77620,"kb_used_omap":49,"kb_used_meta":37198,"kb_avail":93630264,"statfs":{"total":95995035648,"available":95877390336,"internally_reserved":0,"allocated":79482880,"data_stored":100545509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":38091305},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74698752,"data_stored":98474031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75628544,"data_stored":100140690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90226688,"data_stored":120951179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":93155328,"data_stored":124121922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49467392,"data_stored":66762116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":64073728,"data_stored":92273779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":62251008,"data_stored":85381038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":97333248,"data_stored":124036758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:32:47.502 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T19:32:47.502 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:32:47.503 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.6177893687443328 2024-09-20T19:32:47.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 0.6177893687443328 2024-09-20T19:32:47.504 INFO:tasks.rados.rados.0.smithi042.stdout:2191: writing smithi04268048-50 from 474162 to 950196 tid 1 2024-09-20T19:32:47.504 INFO:tasks.rados.rados.0.smithi042.stdout:2191: writing smithi04268048-50 from 1358148 to 1840079 tid 2 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2191: writing smithi04268048-50 from 2352483 to 2817786 tid 3 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2192: delete oid 9 current snap is 255 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2181: finishing copy_from to smithi04268048-46 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 897 (ObjNum 742 snap 245 seq_num 742) dirty exists 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2185: finishing write tid 2 to smithi04268048-43 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2185: finishing write tid 3 to smithi04268048-43 2024-09-20T19:32:47.507 INFO:tasks.rados.rados.0.smithi042.stdout:2185: finishing write tid 4 to smithi04268048-43 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:2185: oid 43 updating version 0 to 731 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:2185: oid 43 version 731 is already newer than 730 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:2185: oid 43 version 731 is already newer than 729 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 731 (ObjNum 763 snap 255 seq_num 763) dirty exists 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:2185: left oid 43 (ObjNum 763 snap 255 seq_num 763) 2024-09-20T19:32:47.508 INFO:tasks.rados.rados.0.smithi042.stdout:2188: expect (ObjNum 722 snap 236 seq_num 722) 2024-09-20T19:32:47.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:47.513 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 590295 2024-09-20T19:32:47.567 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.599 INFO:tasks.rados.rados.0.smithi042.stdout:2190: finishing write tid 1 to smithi04268048-23 2024-09-20T19:32:47.599 INFO:tasks.rados.rados.0.smithi042.stdout:2181: done (7 left) 2024-09-20T19:32:47.599 INFO:tasks.rados.rados.0.smithi042.stdout:2185: done (6 left) 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2188: done (5 left) 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2189: done (4 left) 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2192: done (3 left) 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2193: write (excl) oid 26 current snap is 255 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2187: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2187: finishing copy_from to smithi04268048-10 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 583 (ObjNum 744 snap 247 seq_num 744) dirty exists 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: finishing write tid 2 to smithi04268048-23 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: finishing write tid 3 to smithi04268048-23 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: finishing write tid 4 to smithi04268048-23 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: finishing write tid 5 to smithi04268048-23 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: oid 23 updating version 0 to 731 2024-09-20T19:32:47.600 INFO:tasks.rados.rados.0.smithi042.stdout:2190: oid 23 updating version 731 to 733 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2190: oid 23 version 733 is already newer than 732 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2190: oid 23 version 733 is already newer than 730 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 733 (ObjNum 765 snap 255 seq_num 765) dirty exists 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2190: left oid 23 (ObjNum 765 snap 255 seq_num 765) 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2191: finishing write tid 1 to smithi04268048-50 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2191: finishing write tid 2 to smithi04268048-50 2024-09-20T19:32:47.601 INFO:tasks.rados.rados.0.smithi042.stdout:2193: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-20T19:32:47.607 INFO:tasks.rados.rados.0.smithi042.stdout:2193: writing smithi04268048-26 from 507776 to 1250661 tid 1 2024-09-20T19:32:47.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2193: writing smithi04268048-26 from 1991465 to 2477065 tid 2 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2187: done (3 left) 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2190: done (2 left) 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2194: write (excl) oid 37 current snap is 255 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: finishing write tid 3 to smithi04268048-50 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: finishing write tid 4 to smithi04268048-50 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: finishing write tid 5 to smithi04268048-50 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: oid 50 updating version 0 to 843 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: oid 50 updating version 843 to 844 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: oid 50 updating version 844 to 845 2024-09-20T19:32:47.616 INFO:tasks.rados.rados.0.smithi042.stdout:2191: oid 50 updating version 845 to 846 2024-09-20T19:32:47.617 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 846 (ObjNum 766 snap 255 seq_num 766) dirty exists 2024-09-20T19:32:47.617 INFO:tasks.rados.rados.0.smithi042.stdout:2191: left oid 50 (ObjNum 766 snap 255 seq_num 766) 2024-09-20T19:32:47.617 INFO:tasks.rados.rados.0.smithi042.stdout:2194: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-20T19:32:47.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.621 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[32075]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.622 INFO:tasks.rados.rados.0.smithi042.stdout:2194: writing smithi04268048-37 from 524583 to 942717 tid 1 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: cluster 2024-09-20T19:32:46.308736+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: cluster 2024-09-20T19:32:46.318355+0000 mgr.y (mgr.24491) 557 : cluster [DBG] pgmap v985: 168 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 154 active+clean; 269 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 275 KiB/s wr, 6 op/s 2024-09-20T19:32:47.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:47 smithi042 bash[22706]: audit 2024-09-20T19:32:46.421364+0000 mgr.y (mgr.24491) 558 : audit [DBG] from='client.15771 v1:172.21.15.42:0/3884507273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:32:47.630 INFO:tasks.rados.rados.0.smithi042.stdout:2194: writing smithi04268048-37 from 1649947 to 2279205 tid 2 2024-09-20T19:32:47.634 INFO:tasks.rados.rados.0.smithi042.stdout:2194: writing smithi04268048-37 from 2834752 to 2924583 tid 3 2024-09-20T19:32:47.634 INFO:tasks.rados.rados.0.smithi042.stdout:2191: done (2 left) 2024-09-20T19:32:47.634 INFO:tasks.rados.rados.0.smithi042.stdout:2195: write (excl) oid 46 current snap is 255 2024-09-20T19:32:47.635 INFO:tasks.rados.rados.0.smithi042.stdout:2193: finishing write tid 1 to smithi04268048-26 2024-09-20T19:32:47.635 INFO:tasks.rados.rados.0.smithi042.stdout:2195: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-09-20T19:32:47.637 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:47.640 INFO:tasks.rados.rados.0.smithi042.stdout:2195: writing smithi04268048-46 from 541390 to 1034773 tid 1 2024-09-20T19:32:47.646 INFO:tasks.rados.rados.0.smithi042.stdout:2195: writing smithi04268048-46 from 1592076 to 2081345 tid 2 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2195: writing smithi04268048-46 from 2862175 to 2941390 tid 3 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2196: read oid 31 snap -1 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2196: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2197: read oid 21 snap -1 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2197: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2198: write oid 38 current snap is 255 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2193: finishing write tid 2 to smithi04268048-26 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2193: finishing write tid 3 to smithi04268048-26 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2193: finishing write tid 4 to smithi04268048-26 2024-09-20T19:32:47.652 INFO:tasks.rados.rados.0.smithi042.stdout:2193: oid 26 updating version 0 to 868 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:2193: oid 26 version 868 is already newer than 867 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:2193: oid 26 version 868 is already newer than 866 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 868 (ObjNum 768 snap 255 seq_num 768) dirty exists 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:2193: left oid 26 (ObjNum 768 snap 255 seq_num 768) 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:2194: finishing write tid 1 to smithi04268048-37 2024-09-20T19:32:47.653 INFO:tasks.rados.rados.0.smithi042.stdout:2198: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-20T19:32:47.658 INFO:tasks.rados.rados.0.smithi042.stdout:2198: writing smithi04268048-38 from 558197 to 1126829 tid 1 2024-09-20T19:32:47.665 INFO:tasks.rados.rados.0.smithi042.stdout:2198: writing smithi04268048-38 from 1650558 to 2283485 tid 2 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2198: writing smithi04268048-38 from 2773245 to 2958197 tid 3 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2193: done (5 left) 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2199: rollback oid 14 current snap is 255 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 246 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2194: finishing write tid 2 to smithi04268048-37 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2194: finishing write tid 3 to smithi04268048-37 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2194: finishing write tid 4 to smithi04268048-37 2024-09-20T19:32:47.670 INFO:tasks.rados.rados.0.smithi042.stdout:2194: finishing write tid 5 to smithi04268048-37 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: oid 37 updating version 0 to 787 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: oid 37 version 787 is already newer than 786 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: oid 37 updating version 787 to 788 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: oid 37 version 788 is already newer than 785 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 788 (ObjNum 769 snap 255 seq_num 769) dirty exists 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: left oid 37 (ObjNum 769 snap 255 seq_num 769) 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2194: done (5 left) 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2200: read oid 34 snap -1 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2200: expect (ObjNum 756 snap 252 seq_num 756) 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2201: write (excl) oid 6 current snap is 255 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2195: finishing write tid 1 to smithi04268048-46 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2195: finishing write tid 2 to smithi04268048-46 2024-09-20T19:32:47.671 INFO:tasks.rados.rados.0.smithi042.stdout:2201: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-20T19:32:47.676 INFO:tasks.rados.rados.0.smithi042.stdout:2201: writing smithi04268048-6 from 575004 to 1218885 tid 1 2024-09-20T19:32:47.682 INFO:tasks.rados.rados.0.smithi042.stdout:2201: writing smithi04268048-6 from 1709040 to 2201978 tid 2 2024-09-20T19:32:47.687 INFO:tasks.rados.rados.0.smithi042.stdout:2201: writing smithi04268048-6 from 2917021 to 2975004 tid 3 2024-09-20T19:32:47.688 INFO:tasks.rados.rados.0.smithi042.stdout:2202: snap_create 2024-09-20T19:32:47.688 INFO:tasks.rados.rados.0.smithi042.stdout:2195: finishing write tid 3 to smithi04268048-46 2024-09-20T19:32:47.688 INFO:tasks.rados.rados.0.smithi042.stdout:2195: finishing write tid 4 to smithi04268048-46 2024-09-20T19:32:47.688 INFO:tasks.rados.rados.0.smithi042.stdout:2197: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:47.698 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1300526 2024-09-20T19:32:47.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:47.741 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:47.768 INFO:tasks.rados.rados.0.smithi042.stdout:2195: finishing write tid 5 to smithi04268048-46 2024-09-20T19:32:47.768 INFO:tasks.rados.rados.0.smithi042.stdout:2195: oid 46 updating version 0 to 1046 2024-09-20T19:32:47.768 INFO:tasks.rados.rados.0.smithi042.stdout:2195: oid 46 version 1046 is already newer than 1044 2024-09-20T19:32:47.768 INFO:tasks.rados.rados.0.smithi042.stdout:2195: oid 46 version 1046 is already newer than 1043 2024-09-20T19:32:47.769 INFO:tasks.rados.rados.0.smithi042.stdout:2195: oid 46 version 1046 is already newer than 1045 2024-09-20T19:32:47.769 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1046 (ObjNum 770 snap 255 seq_num 770) dirty exists 2024-09-20T19:32:47.769 INFO:tasks.rados.rados.0.smithi042.stdout:2195: left oid 46 (ObjNum 770 snap 255 seq_num 770) 2024-09-20T19:32:47.769 INFO:tasks.rados.rados.0.smithi042.stdout:2196: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-20T19:32:47.769 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:47.790 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.829 INFO:tasks.rados.rados.0.smithi042.stdout:2198: finishing write tid 1 to smithi04268048-38 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2199: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: finishing write tid 2 to smithi04268048-38 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: finishing write tid 3 to smithi04268048-38 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2199: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: finishing write tid 4 to smithi04268048-38 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2199: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 791 (ObjNum 670 snap 216 seq_num 670) dirty exists 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: finishing write tid 5 to smithi04268048-38 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: oid 38 updating version 0 to 749 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: oid 38 updating version 749 to 750 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: oid 38 updating version 750 to 751 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: oid 38 updating version 751 to 752 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 752 (ObjNum 771 snap 255 seq_num 771) dirty exists 2024-09-20T19:32:47.830 INFO:tasks.rados.rados.0.smithi042.stdout:2198: left oid 38 (ObjNum 771 snap 255 seq_num 771) 2024-09-20T19:32:47.831 INFO:tasks.rados.rados.0.smithi042.stdout:2200: expect (ObjNum 756 snap 252 seq_num 756) 2024-09-20T19:32:47.856 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:32:47.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:47.891 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:47.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:47.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: finishing write tid 1 to smithi04268048-6 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: finishing write tid 2 to smithi04268048-6 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: finishing write tid 3 to smithi04268048-6 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: finishing write tid 4 to smithi04268048-6 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: finishing write tid 5 to smithi04268048-6 2024-09-20T19:32:48.007 INFO:tasks.rados.rados.0.smithi042.stdout:2201: oid 6 updating version 0 to 767 2024-09-20T19:32:48.008 INFO:tasks.rados.rados.0.smithi042.stdout:2201: oid 6 version 767 is already newer than 766 2024-09-20T19:32:48.008 INFO:tasks.rados.rados.0.smithi042.stdout:2201: oid 6 updating version 767 to 769 2024-09-20T19:32:48.008 INFO:tasks.rados.rados.0.smithi042.stdout:2201: oid 6 version 769 is already newer than 768 2024-09-20T19:32:48.008 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 769 (ObjNum 772 snap 255 seq_num 772) dirty exists 2024-09-20T19:32:48.008 INFO:tasks.rados.rados.0.smithi042.stdout:2201: left oid 6 (ObjNum 772 snap 255 seq_num 772) 2024-09-20T19:32:48.013 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:48.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:48.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:48.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:2195: done (7 left) 2024-09-20T19:32:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:2196: done (6 left) 2024-09-20T19:32:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:2197: done (5 left) 2024-09-20T19:32:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:2198: done (4 left) 2024-09-20T19:32:48.290 INFO:tasks.rados.rados.0.smithi042.stdout:2199: done (3 left) 2024-09-20T19:32:48.291 INFO:tasks.rados.rados.0.smithi042.stdout:2200: done (2 left) 2024-09-20T19:32:48.291 INFO:tasks.rados.rados.0.smithi042.stdout:2201: done (1 left) 2024-09-20T19:32:48.291 INFO:tasks.rados.rados.0.smithi042.stdout:2202: done (0 left) 2024-09-20T19:32:48.291 INFO:tasks.rados.rados.0.smithi042.stdout:2203: delete oid 30 current snap is 256 2024-09-20T19:32:48.293 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:48.297 INFO:tasks.rados.rados.0.smithi042.stdout:2203: done (0 left) 2024-09-20T19:32:48.297 INFO:tasks.rados.rados.0.smithi042.stdout:2204: copy_from oid 50 from oid 48 current snap is 256 2024-09-20T19:32:48.297 INFO:tasks.rados.rados.0.smithi042.stdout:2205: snap_create 2024-09-20T19:32:48.313 INFO:tasks.rados.rados.0.smithi042.stdout:2204: finishing copy_from to smithi04268048-50 2024-09-20T19:32:48.313 INFO:tasks.rados.rados.0.smithi042.stdout:2204: got expected ENOENT (src dne) 2024-09-20T19:32:48.313 INFO:tasks.rados.rados.0.smithi042.stdout:2204: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:32:48.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:48.407 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:48 smithi162 bash[24557]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.407 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:48 smithi162 bash[24557]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.408 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:48 smithi162 bash[24557]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.408 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:48 smithi162 bash[24557]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:48.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:48.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:48.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[32075]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[32075]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[32075]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[32075]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[22706]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[22706]: cluster 2024-09-20T19:32:47.319833+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[22706]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:48 smithi042 bash[22706]: cluster 2024-09-20T19:32:48.285026+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-20T19:32:48.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:48.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:48.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:48.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:48.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:48.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:49.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:49.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:49.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:2204: done (1 left) 2024-09-20T19:32:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:2205: done (0 left) 2024-09-20T19:32:49.291 INFO:tasks.rados.rados.0.smithi042.stdout:2206: snap_remove snap 256 2024-09-20T19:32:49.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:49.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:49 smithi162 bash[24557]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:49 smithi162 bash[24557]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:49 smithi162 bash[24557]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:49 smithi162 bash[24557]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:49.564 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[32075]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[32075]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[32075]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[32075]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[22706]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[22706]: cluster 2024-09-20T19:32:48.318790+0000 mgr.y (mgr.24491) 559 : cluster [DBG] pgmap v988: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 283 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[22706]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:49 smithi042 bash[22706]: cluster 2024-09-20T19:32:49.289146+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-20T19:32:49.593 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:49.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.707 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:49.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:49.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:49.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:49.987 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:49.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:50.005 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:50.064 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:32:50.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:50.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:50.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:50.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:50.295 INFO:tasks.rados.rados.0.smithi042.stdout:2206: done (0 left) 2024-09-20T19:32:50.295 INFO:tasks.rados.rados.0.smithi042.stdout:2207: read oid 9 snap 247 2024-09-20T19:32:50.295 INFO:tasks.rados.rados.0.smithi042.stdout:2207: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:50.295 INFO:tasks.rados.rados.0.smithi042.stdout:2208: snap_remove snap 229 2024-09-20T19:32:50.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[32075]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[32075]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[32075]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[32075]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[22706]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[22706]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.354 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[22706]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.355 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:50 smithi042 bash[22706]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:50.471 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:50.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:50.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:50 smithi162 bash[24557]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:50 smithi162 bash[24557]: audit 2024-09-20T19:32:49.616967+0000 mgr.y (mgr.24491) 560 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:32:50.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:50 smithi162 bash[24557]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:50 smithi162 bash[24557]: cluster 2024-09-20T19:32:50.292240+0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-20T19:32:50.601 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:50.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:50.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:50.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:50.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:50.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:50.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:50.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:51.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:51.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:51.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:51.284 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:51.296 INFO:tasks.rados.rados.0.smithi042.stdout:2208: done (1 left) 2024-09-20T19:32:51.296 INFO:tasks.rados.rados.0.smithi042.stdout:2209: write (excl) oid 10 current snap is 257 2024-09-20T19:32:51.296 INFO:tasks.rados.rados.0.smithi042.stdout:2209: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-20T19:32:51.305 INFO:tasks.rados.rados.0.smithi042.stdout:2209: writing smithi04268048-10 from 608618 to 1402997 tid 1 2024-09-20T19:32:51.314 INFO:tasks.rados.rados.0.smithi042.stdout:2209: writing smithi04268048-10 from 2109651 to 2889905 tid 2 2024-09-20T19:32:51.314 INFO:tasks.rados.rados.0.smithi042.stdout:2210: read oid 50 snap -1 2024-09-20T19:32:51.314 INFO:tasks.rados.rados.0.smithi042.stdout:2210: expect (ObjNum 766 snap 255 seq_num 766) 2024-09-20T19:32:51.314 INFO:tasks.rados.rados.0.smithi042.stdout:2211: snap_create 2024-09-20T19:32:51.314 INFO:tasks.rados.rados.0.smithi042.stdout:2207: expect (ObjNum 596 snap 189 seq_num 596) 2024-09-20T19:32:51.355 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1232528 2024-09-20T19:32:51.386 INFO:tasks.rados.rados.0.smithi042.stdout:2209: finishing write tid 1 to smithi04268048-10 2024-09-20T19:32:51.387 INFO:tasks.rados.rados.0.smithi042.stdout:2210: expect (ObjNum 766 snap 255 seq_num 766) 2024-09-20T19:32:51.387 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:51.468 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:51.512 INFO:tasks.rados.rados.0.smithi042.stdout:2209: finishing write tid 2 to smithi04268048-10 2024-09-20T19:32:51.512 INFO:tasks.rados.rados.0.smithi042.stdout:2209: finishing write tid 3 to smithi04268048-10 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:2209: finishing write tid 4 to smithi04268048-10 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:2209: oid 10 updating version 0 to 760 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:2209: oid 10 version 760 is already newer than 759 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:2209: oid 10 version 760 is already newer than 758 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 760 (ObjNum 774 snap 257 seq_num 774) dirty exists 2024-09-20T19:32:51.513 INFO:tasks.rados.rados.0.smithi042.stdout:2209: left oid 10 (ObjNum 774 snap 257 seq_num 774) 2024-09-20T19:32:51.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:51.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:51 smithi162 bash[24557]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:51 smithi162 bash[24557]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:51 smithi162 bash[24557]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:51 smithi162 bash[24557]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:51.644 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:51.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[22706]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[22706]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[22706]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[22706]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[32075]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[32075]: cluster 2024-09-20T19:32:50.319485+0000 mgr.y (mgr.24491) 561 : cluster [DBG] pgmap v991: 168 pgs: 8 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 150 active+clean; 286 MiB data, 967 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-20T19:32:51.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[32075]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:51 smithi042 bash[32075]: cluster 2024-09-20T19:32:51.295781+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-20T19:32:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:51.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:51.799 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:51.799 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:32:51.799 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:32:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:51.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:51.926 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:51.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:51.950 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:32:51.950 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:32:51.967 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:32:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:32:51.985 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:32:51.985 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:32:52.002 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:32:52.002 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:32:52.023 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:32:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:32:52.028 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:52.028 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:32:52.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:52.052 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:32:52.053 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:32:52.073 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:32:52.074 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:32:52.094 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:32:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:32:52.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:52.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:52.307 INFO:tasks.rados.rados.0.smithi042.stdout:2207: done (3 left) 2024-09-20T19:32:52.307 INFO:tasks.rados.rados.0.smithi042.stdout:2209: done (2 left) 2024-09-20T19:32:52.307 INFO:tasks.rados.rados.0.smithi042.stdout:2210: done (1 left) 2024-09-20T19:32:52.307 INFO:tasks.rados.rados.0.smithi042.stdout:2211: done (0 left) 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2212: copy_from oid 38 from oid 23 current snap is 258 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2213: read oid 15 snap -1 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2213: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2214: rollback oid 31 current snap is 258 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 257 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2215: write oid 20 current snap is 258 2024-09-20T19:32:52.308 INFO:tasks.rados.rados.0.smithi042.stdout:2215: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-20T19:32:52.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:52.315 INFO:tasks.rados.rados.0.smithi042.stdout:2215: writing smithi04268048-20 from 625425 to 1095053 tid 1 2024-09-20T19:32:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:52.328 INFO:tasks.rados.rados.0.smithi042.stdout:2215: writing smithi04268048-20 from 1768133 to 2408398 tid 2 2024-09-20T19:32:52.328 INFO:tasks.rados.rados.0.smithi042.stdout:2216: write oid 18 current snap is 258 2024-09-20T19:32:52.328 INFO:tasks.rados.rados.0.smithi042.stdout:2216: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-20T19:32:52.342 INFO:tasks.rados.rados.0.smithi042.stdout:2216: writing smithi04268048-18 from 642232 to 1303462 tid 1 2024-09-20T19:32:52.355 INFO:tasks.rados.rados.0.smithi042.stdout:2216: writing smithi04268048-18 from 1942968 to 2726891 tid 2 2024-09-20T19:32:52.355 INFO:tasks.rados.rados.0.smithi042.stdout:2217: read oid 25 snap -1 2024-09-20T19:32:52.355 INFO:tasks.rados.rados.0.smithi042.stdout:2217: expect (ObjNum 761 snap 255 seq_num 761) 2024-09-20T19:32:52.355 INFO:tasks.rados.rados.0.smithi042.stdout:2218: write (excl) oid 46 current snap is 258 2024-09-20T19:32:52.355 INFO:tasks.rados.rados.0.smithi042.stdout:2214: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:32:52.356 INFO:tasks.rados.rados.0.smithi042.stdout:2218: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-20T19:32:52.368 INFO:tasks.rados.rados.0.smithi042.stdout:2218: writing smithi04268048-46 from 659039 to 1395518 tid 1 2024-09-20T19:32:52.379 INFO:tasks.rados.rados.0.smithi042.stdout:2218: writing smithi04268048-46 from 2001450 to 2645384 tid 2 2024-09-20T19:32:52.379 INFO:tasks.rados.rados.0.smithi042.stdout:2219: snap_remove snap 237 2024-09-20T19:32:52.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:52 smithi162 bash[24557]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:52 smithi162 bash[24557]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:52 smithi162 bash[24557]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:52 smithi162 bash[24557]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[22706]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[22706]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[22706]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[22706]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[32075]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[32075]: audit 2024-09-20T19:32:51.930943+0000 mon.a (mon.0) 2092 : audit [DBG] from='client.? v1:172.21.15.42:0/4221461142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:32:52.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[32075]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:52 smithi042 bash[32075]: cluster 2024-09-20T19:32:52.302231+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-20T19:32:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:52.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:52.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:52.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T19:32:52.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:52.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:52.936 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:52.965 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:52.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:53.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:53.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:53.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:53.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:53.291 INFO:tasks.rados.rados.0.smithi042.stdout:2219: done (7 left) 2024-09-20T19:32:53.292 INFO:tasks.rados.rados.0.smithi042.stdout:2220: snap_remove snap 205 2024-09-20T19:32:53.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:53.353 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:53.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:53.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:53.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[32075]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[32075]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[32075]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[32075]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[22706]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[22706]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[22706]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:53 smithi042 bash[22706]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:53.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:53.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:53 smithi162 bash[24557]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:53 smithi162 bash[24557]: cluster 2024-09-20T19:32:52.320507+0000 mgr.y (mgr.24491) 562 : cluster [DBG] pgmap v994: 168 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 149 active+clean; 297 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:32:53.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:53 smithi162 bash[24557]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:53 smithi162 bash[24557]: cluster 2024-09-20T19:32:53.289214+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-20T19:32:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:53.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:53.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:53.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:53.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:54.104 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:54.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:54.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:54.296 INFO:tasks.rados.rados.0.smithi042.stdout:2220: done (7 left) 2024-09-20T19:32:54.296 INFO:tasks.rados.rados.0.smithi042.stdout:2221: snap_remove snap 253 2024-09-20T19:32:54.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.390 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:54.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:54.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:54.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:54.753 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:54.777 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:54.880 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:54.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:55.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:55.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:55.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:55.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:55.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:55.297 INFO:tasks.rados.rados.0.smithi042.stdout:2221: done (7 left) 2024-09-20T19:32:55.298 INFO:tasks.rados.rados.0.smithi042.stdout:2222: read oid 7 snap -1 2024-09-20T19:32:55.298 INFO:tasks.rados.rados.0.smithi042.stdout:2222: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:55.298 INFO:tasks.rados.rados.0.smithi042.stdout:2223: write oid 22 current snap is 258 2024-09-20T19:32:55.298 INFO:tasks.rados.rados.0.smithi042.stdout:2223: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-20T19:32:55.302 INFO:tasks.rados.rados.0.smithi042.stdout:2223: writing smithi04268048-22 from 675846 to 1087574 tid 1 2024-09-20T19:32:55.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:55.310 INFO:tasks.rados.rados.0.smithi042.stdout:2223: writing smithi04268048-22 from 1543579 to 2331171 tid 2 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:2223: writing smithi04268048-22 from 2848853 to 3075846 tid 3 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:2224: snap_create 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:2214: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:2214: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 777 (ObjNum 718 snap 234 seq_num 718) dirty exists 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:2213: expect (ObjNum 561 snap 178 seq_num 561) 2024-09-20T19:32:55.315 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:55.374 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3349765 2024-09-20T19:32:55.378 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 686784 2024-09-20T19:32:55.407 INFO:tasks.rados.rados.0.smithi042.stdout:2212: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:32:55.407 INFO:tasks.rados.rados.0.smithi042.stdout:2212: finishing copy_from to smithi04268048-38 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 733 (ObjNum 765 snap 255 seq_num 765) dirty exists 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: finishing write tid 1 to smithi04268048-20 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: finishing write tid 2 to smithi04268048-20 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: finishing write tid 3 to smithi04268048-20 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: finishing write tid 4 to smithi04268048-20 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: oid 20 updating version 0 to 984 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: oid 20 version 984 is already newer than 983 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: oid 20 updating version 984 to 985 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 985 (ObjNum 775 snap 258 seq_num 775) dirty exists 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2215: left oid 20 (ObjNum 775 snap 258 seq_num 775) 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2216: finishing write tid 1 to smithi04268048-18 2024-09-20T19:32:55.408 INFO:tasks.rados.rados.0.smithi042.stdout:2216: finishing write tid 2 to smithi04268048-18 2024-09-20T19:32:55.409 INFO:tasks.rados.rados.0.smithi042.stdout:2216: finishing write tid 3 to smithi04268048-18 2024-09-20T19:32:55.409 INFO:tasks.rados.rados.0.smithi042.stdout:2217: expect (ObjNum 761 snap 255 seq_num 761) 2024-09-20T19:32:55.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:55.410 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 549083 2024-09-20T19:32:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:55.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2216: finishing write tid 4 to smithi04268048-18 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2216: oid 18 updating version 0 to 971 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2216: oid 18 version 971 is already newer than 969 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2216: oid 18 version 971 is already newer than 970 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 971 (ObjNum 776 snap 258 seq_num 776) dirty exists 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2216: left oid 18 (ObjNum 776 snap 258 seq_num 776) 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2218: finishing write tid 1 to smithi04268048-46 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2218: finishing write tid 2 to smithi04268048-46 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2218: finishing write tid 3 to smithi04268048-46 2024-09-20T19:32:55.533 INFO:tasks.rados.rados.0.smithi042.stdout:2218: finishing write tid 4 to smithi04268048-46 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:2218: oid 46 updating version 0 to 1050 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:2218: oid 46 updating version 1050 to 1052 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:2218: oid 46 version 1052 is already newer than 1051 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1052 (ObjNum 777 snap 258 seq_num 777) dirty exists 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:2218: left oid 46 (ObjNum 777 snap 258 seq_num 777) 2024-09-20T19:32:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:2222: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:32:55.536 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1623405 2024-09-20T19:32:55.542 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[32075]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.542 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[32075]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.542 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[32075]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.542 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[32075]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[22706]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[22706]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[22706]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:55 smithi042 bash[22706]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:55 smithi162 bash[24557]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:55 smithi162 bash[24557]: cluster 2024-09-20T19:32:54.291488+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-20T19:32:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:55 smithi162 bash[24557]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:55 smithi162 bash[24557]: cluster 2024-09-20T19:32:54.321110+0000 mgr.y (mgr.24491) 563 : cluster [DBG] pgmap v997: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 302 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-20T19:32:55.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:55.583 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1523059 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: finishing write tid 1 to smithi04268048-22 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: finishing write tid 2 to smithi04268048-22 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: finishing write tid 3 to smithi04268048-22 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: finishing write tid 4 to smithi04268048-22 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: finishing write tid 5 to smithi04268048-22 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: oid 22 updating version 0 to 1013 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: oid 22 updating version 1013 to 1014 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: oid 22 version 1014 is already newer than 1011 2024-09-20T19:32:55.600 INFO:tasks.rados.rados.0.smithi042.stdout:2223: oid 22 version 1014 is already newer than 1012 2024-09-20T19:32:55.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1014 (ObjNum 778 snap 258 seq_num 778) dirty exists 2024-09-20T19:32:55.601 INFO:tasks.rados.rados.0.smithi042.stdout:2223: left oid 22 (ObjNum 778 snap 258 seq_num 778) 2024-09-20T19:32:55.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:55.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:55.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:55.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:55.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:56.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:56.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:56.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:56.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:56.301 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:32:56.303 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:56.313 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 0.617789 (9e27) 2024-09-20T19:32:56.313 INFO:tasks.rados.rados.0.smithi042.stdout:2212: done (9 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2213: done (8 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2214: done (7 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2215: done (6 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2216: done (5 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2217: done (4 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2218: done (3 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2222: done (2 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2223: done (1 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2224: done (0 left) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2225: read oid 16 snap -1 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2225: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:32:56.314 INFO:tasks.rados.rados.0.smithi042.stdout:2226: read oid 43 snap -1 2024-09-20T19:32:56.315 INFO:tasks.rados.rados.0.smithi042.stdout:2226: expect (ObjNum 763 snap 255 seq_num 763) 2024-09-20T19:32:56.315 INFO:tasks.rados.rados.0.smithi042.stdout:2227: delete oid 29 current snap is 259 2024-09-20T19:32:56.332 INFO:tasks.rados.rados.0.smithi042.stdout:2225: expect (ObjNum 661 snap 212 seq_num 661) 2024-09-20T19:32:56.332 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:32:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:56.353 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:32:56.405 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2605091 2024-09-20T19:32:56.439 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:56.455 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3062953 2024-09-20T19:32:56.455 INFO:tasks.rados.rados.0.smithi042.stdout:2226: expect (ObjNum 763 snap 255 seq_num 763) 2024-09-20T19:32:56.456 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 146558 2024-09-20T19:32:56.460 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:56.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1104917 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:56 smithi162 bash[24557]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:56.570 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:32:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[32075]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.294389+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-20T19:32:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.687240+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.42:0/3566443169' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.687688+0000 mon.a (mon.0) 2097 : audit [INF] from='client.25631 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328} : dispatch 2024-09-20T19:32:56.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.572 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:56 smithi042 bash[22706]: audit 2024-09-20T19:32:55.695594+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:32:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:56.602 INFO:tasks.rados.rados.0.smithi042.stdout:2225: done (2 left) 2024-09-20T19:32:56.602 INFO:tasks.rados.rados.0.smithi042.stdout:2226: done (1 left) 2024-09-20T19:32:56.602 INFO:tasks.rados.rados.0.smithi042.stdout:2227: done (0 left) 2024-09-20T19:32:56.602 INFO:tasks.rados.rados.0.smithi042.stdout:2228: copy_from oid 25 from oid 5 current snap is 259 2024-09-20T19:32:56.602 INFO:tasks.rados.rados.0.smithi042.stdout:2229: snap_create 2024-09-20T19:32:56.608 INFO:tasks.rados.rados.0.smithi042.stdout:2228: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:32:56.613 INFO:tasks.rados.rados.0.smithi042.stdout:2228: finishing copy_from to smithi04268048-25 2024-09-20T19:32:56.613 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 707 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:32:56.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:56.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:56.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:56.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:56.975 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:56.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:57.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:57.210 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:57.322 INFO:tasks.rados.rados.0.smithi042.stdout:2228: done (1 left) 2024-09-20T19:32:57.322 INFO:tasks.rados.rados.0.smithi042.stdout:2229: done (0 left) 2024-09-20T19:32:57.322 INFO:tasks.rados.rados.0.smithi042.stdout:2230: copy_from oid 38 from oid 2 current snap is 260 2024-09-20T19:32:57.322 INFO:tasks.rados.rados.0.smithi042.stdout:2231: copy_from oid 14 from oid 44 current snap is 260 2024-09-20T19:32:57.323 INFO:tasks.rados.rados.0.smithi042.stdout:2232: snap_remove snap 212 2024-09-20T19:32:57.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:57.369 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[32075]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:57 smithi042 bash[22706]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.446 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:32:57.472 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:57.500 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:57.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.835461+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:32:57.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:55.836442+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.291318+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: audit 2024-09-20T19:32:56.300765+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6177893687443328}]': finished 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: audit 2024-09-20T19:32:56.300888+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.42:0/3386577947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.309641+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.321652+0000 mgr.y (mgr.24491) 564 : cluster [DBG] pgmap v1000: 168 pgs: 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 147 active+clean; 298 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 1000 KiB/s wr, 8 op/s 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.915355+0000 osd.4 (osd.4) 45 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:32:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:57 smithi162 bash[24557]: cluster 2024-09-20T19:32:56.921126+0000 osd.4 (osd.4) 46 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:32:57.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:57.628 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:57.731 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:57.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:57.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:57.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:57.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:57.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:57.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:58.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:32:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:32:58.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:58.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:58.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:58.281 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:32:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:32:58.291 INFO:tasks.rados.rados.0.smithi042.stdout:2232: done (2 left) 2024-09-20T19:32:58.291 INFO:tasks.rados.rados.0.smithi042.stdout:2233: read oid 28 snap -1 2024-09-20T19:32:58.291 INFO:tasks.rados.rados.0.smithi042.stdout:2233: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:32:58.291 INFO:tasks.rados.rados.0.smithi042.stdout:2234: read oid 34 snap -1 2024-09-20T19:32:58.291 INFO:tasks.rados.rados.0.smithi042.stdout:2234: expect (ObjNum 756 snap 252 seq_num 756) 2024-09-20T19:32:58.292 INFO:tasks.rados.rados.0.smithi042.stdout:2235: rollback oid 20 current snap is 260 2024-09-20T19:32:58.292 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 20 to 257 2024-09-20T19:32:58.297 INFO:tasks.rados.rados.0.smithi042.stdout:2236: write (excl) oid 49 current snap is 260 2024-09-20T19:32:58.297 INFO:tasks.rados.rados.0.smithi042.stdout:2236: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-20T19:32:58.299 INFO:tasks.rados.rados.0.smithi042.stdout:2236: writing smithi04268048-49 from 709460 to 1271686 tid 1 2024-09-20T19:32:58.309 INFO:tasks.rados.rados.0.smithi042.stdout:2236: writing smithi04268048-49 from 2060543 to 2851804 tid 2 2024-09-20T19:32:58.309 INFO:tasks.rados.rados.0.smithi042.stdout:2237: snap_remove snap 246 2024-09-20T19:32:58.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:58 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:58.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:58.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.527 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.247351+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.249031+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.318778+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-20T19:32:58.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.494546+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.496952+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: audit 2024-09-20T19:32:57.953841+0000 mon.a (mon.0) 2104 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.954667+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.e scrub starts 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.956328+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.e scrub ok 2024-09-20T19:32:58.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:58 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.287261+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-20T19:32:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:58.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:58.673 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:32:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:32:58.776 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:58.905 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:58.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:58.994 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:32:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2234 4.1a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.lock in=8b] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.64468615800001, 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0273693370000001, 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:32:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2234 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352499+0000", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4540000000000001e-06 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352507+0000", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0539999999999995e-06 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352518+0000", 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0154e-05 2024-09-20T19:32:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352527+0000", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0250000000000008e-06 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352886+0000", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035973599999999999 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352896+0000", 2024-09-20T19:32:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4369999999999996e-06 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.371744+0000", 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0188486000000001 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.371982+0000", 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023746300000000001 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-20T19:32:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.372358+0000", 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037604499999999998 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.378302+0000", 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059445499999999998 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379623+0000", 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001320821 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379724+0000", 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010050700000000001 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379866+0000", 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014249099999999999 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2592 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94817798633088 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:32:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.64463809399999, 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0268885000000001, 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2592 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352546+0000", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8299999999999996e-07 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352549+0000", 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4929999999999998e-06 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352555+0000", 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2230000000000004e-06 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352562+0000", 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1920000000000002e-06 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.354341+0000", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017789629999999999 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.354350+0000", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2469999999999997e-06 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.351955+0000", 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99760481000000001 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.352331+0000", 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000375922 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379433+0000", 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027102866999999999 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "pg_update_log_missing_reply(5.e epoch 523/522 rep_tid 0 lcod 523'15)", 2024-09-20T19:32:59.007 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:31.367196+0000", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 147.62998727199999, 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012244405999999999, 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367196+0000", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367196+0000", 2024-09-20T19:32:59.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367198+0000", 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1849999999999998e-06 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367206+0000", 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7249999999999997e-06 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367223+0000", 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7062e-05 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.367236+0000", 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379338+0000", 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012101608 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379440+0000", 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102421 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4457 7.16 7:6c48c6fe:::smithi04268048-7:head [delete] snapc c2=[c2,c0,bb,b3,b2,aa] ondisk+write+known_if_redirected+full_force+supports_pool_eio e546)", 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:54.337839+0000", 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.659343968, 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014846421, 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4457 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337839+0000", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337839+0000", 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337844+0000", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6829999999999999e-06 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337860+0000", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6422999999999998e-05 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337890+0000", 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0066e-05 2024-09-20T19:32:59.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.337912+0000", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1625999999999999e-05 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.338093+0000", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000180812 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.338119+0000", 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6539000000000001e-05 2024-09-20T19:32:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.344396+0000", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00627678 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.344571+0000", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017497199999999999 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.345022+0000", 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045141700000000002 2024-09-20T19:32:59.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.348613+0000", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003590293 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.351281+0000", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026687920000000001 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.352491+0000", 2024-09-20T19:32:59.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012093360000000001 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.352608+0000", 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117222 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:54.352685+0000", 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7459999999999994e-05 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14904.0:4471 7.8 e547/522 7:17ea136f:::smithi04268048-32:head v 547'1046, mlcod=546'1042)", 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.647796861, 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018403108000000001, 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349387+0000", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4799999999999998e-07 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360853+0000", 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011466551 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360875+0000", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1763000000000002e-05 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360886+0000", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360959+0000", 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2787000000000006e-05 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360986+0000", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7191000000000001e-05 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.367755+0000", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067691649999999997 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.367789+0000", 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3827e-05 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.021 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4692 7.1 7:861e105e:::smithi04268048-24:head [copy-from ver 586] snapc ce=[ce,cc,cb,c8,c7,aa] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:18.691394+0000", 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.30578898100001, 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015865918999999999, 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4692 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691394+0000", 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691394+0000", 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691399+0000", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8629999999999999e-06 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691425+0000", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5593e-05 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691462+0000", 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7416000000000002e-05 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691487+0000", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5272000000000001e-05 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691727+0000", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000239748 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691741+0000", 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4090999999999999e-05 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.694795+0000", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003053582 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.694897+0000", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010192 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.699786+0000", 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048896620000000003 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.705192+0000", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054054369999999999 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.706869+0000", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016777980000000001 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.707101+0000", 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023176300000000001 2024-09-20T19:32:59.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.707182+0000", 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1131000000000006e-05 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.707260+0000", 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7643000000000002e-05 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.028 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4693 7.1 7:861e105e:::smithi04268048-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e570)", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:18.691555+0000", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.305628082, 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012660975999999999, 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4693 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691555+0000", 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691555+0000", 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691557+0000", 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0770000000000002e-06 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691570+0000", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3020000000000001e-05 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691619+0000", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9128000000000001e-05 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691636+0000", 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7082999999999999e-05 2024-09-20T19:32:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691848+0000", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211308 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.691858+0000", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0298e-05 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.695113+0000", 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032552309999999999 2024-09-20T19:32:59.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.695168+0000", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5226999999999998e-05 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.704002+0000", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088337740000000008 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.704091+0000", 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8978000000000001e-05 2024-09-20T19:32:59.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:18.704216+0000", 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000124852 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14904.0:4739 7.3 e576/550 7:c0f7e14f:::smithi04268048-43:head v 576'352, mlcod=562'348)", 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:24.286227+0000", 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.710956222999997, 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013536703000000001, 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.034 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.286227+0000", 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.286227+0000", 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.286227+0000", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3300000000000001e-07 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.293907+0000", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076797109999999997 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.293923+0000", 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5820000000000001e-05 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.293932+0000", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2420000000000006e-06 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294018+0000", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5697000000000003e-05 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.294051+0000", 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3969e-05 2024-09-20T19:32:59.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.299716+0000", 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056641920000000002 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:24.299764+0000", 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7839000000000001e-05 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4928 7.1 7:861e105e:::smithi04268048-24:head [append 0~0] snapc da=[da,d8,d7,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:42.394739+0000", 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.602443985999997, 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010519382000000001, 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4928 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.039 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394739+0000", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394739+0000", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394742+0000", 2024-09-20T19:32:59.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.791e-06 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394749+0000", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2880000000000004e-06 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394764+0000", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4606e-05 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394772+0000", 2024-09-20T19:32:59.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.738e-06 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394942+0000", 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000169989 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395067+0000", 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012405399999999999 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-20T19:32:59.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395226+0000", 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159751 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.404896+0000", 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096699379999999995 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405058+0000", 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016144099999999999 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405122+0000", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3905000000000006e-05 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405189+0000", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7633999999999997e-05 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405258+0000", 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9246999999999996e-05 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4929 7.1 7:861e105e:::smithi04268048-24:head [rollback d7] snapc da=[da,d8,d7,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:42.394795+0000", 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.602388106000006, 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010640933, 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4929 2024-09-20T19:32:59.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394795+0000", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394795+0000", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394796+0000", 2024-09-20T19:32:59.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4499999999999996e-07 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394799+0000", 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.529e-06 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394807+0000", 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.055e-06 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394813+0000", 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8170000000000004e-06 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395891+0000", 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010777499999999999 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395979+0000", 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8570000000000001e-05 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.396165+0000", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018524599999999999 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.404963+0000", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087981580000000004 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405065+0000", 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102552 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405347+0000", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028116799999999999 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405397+0000", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9997999999999998e-05 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405436+0000", 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9245000000000001e-05 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4930 7.1 7:861e105e:::smithi04268048-24:head [append 0~0] snapc da=[da,d8,d7,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:42.394864+0000", 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.602319012999999, 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011965907, 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.051 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4930 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394864+0000", 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394864+0000", 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394865+0000", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.61e-07 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394870+0000", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2390000000000002e-06 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394877+0000", 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4789999999999999e-06 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394898+0000", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0293e-05 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.396679+0000", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017811669999999999 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.396761+0000", 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2501000000000002e-05 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.396925+0000", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016325500000000001 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405042+0000", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0081169880000000003 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405075+0000", 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3590000000000002e-05 2024-09-20T19:32:59.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.406761+0000", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016854929999999999 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.406799+0000", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8702999999999999e-05 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.406830+0000", 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0537999999999998e-05 2024-09-20T19:32:59.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:4940 7.1 7:8715d6d2:::smithi04268048-16:d6 [read 0~3109427,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e594)", 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:31:42.394999+0000", 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.602183769999996, 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011723552, 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4940 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394999+0000", 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.394999+0000", 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395000+0000", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6599999999999996e-07 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395004+0000", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9369999999999997e-06 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395010+0000", 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1530000000000002e-06 2024-09-20T19:32:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.395015+0000", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4639999999999997e-06 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405485+0000", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010469579999999999 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.405625+0000", 2024-09-20T19:32:59.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013952499999999999 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:31:42.406723+0000", 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001098227 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5157 7.1 7:8715d6d2:::smithi04268048-16:head [read 0~3109427,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e614)", 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:02.403167+0000", 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.594016236999998, 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012594098999999999, 2024-09-20T19:32:59.061 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5157 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403167+0000", 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403167+0000", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403168+0000", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6169999999999999e-06 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403178+0000", 2024-09-20T19:32:59.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2730000000000008e-06 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403198+0000", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0720000000000002e-05 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403212+0000", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3129000000000001e-05 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403500+0000", 2024-09-20T19:32:59.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028808000000000002 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403521+0000", 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1206e-05 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.413060+0000", 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095393969999999998 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.413363+0000", 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030258900000000003 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.415761+0000", 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002398088 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5158 7.1 7:8715d6d2:::smithi04268048-16:head [read 0~3109427,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e614)", 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:02.403266+0000", 2024-09-20T19:32:59.066 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.593917439000002, 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014264829999999999, 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5158 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403266+0000", 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403266+0000", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403267+0000", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7400000000000001e-06 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403282+0000", 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4749999999999999e-05 2024-09-20T19:32:59.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403299+0000", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7258999999999999e-05 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403310+0000", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0477e-05 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403600+0000", 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029063299999999999 2024-09-20T19:32:59.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.403611+0000", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0798999999999999e-05 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.415831+0000", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012219495 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.415966+0000", 2024-09-20T19:32:59.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013484099999999999 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:02.417530+0000", 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001564836 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5203 7.16 7:6dcf8a02:::smithi04268048-26:head [write 450619~490715] snapc eb=[ea,e9,e7,e5,e2,e1,e0,df,dc,da,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:09.334417+0000", 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.662766349999998, 2024-09-20T19:32:59.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016532237000000002, 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5203 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.334417+0000", 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.334417+0000", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.334419+0000", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4880000000000001e-06 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335222+0000", 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080285700000000003 2024-09-20T19:32:59.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335250+0000", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335261+0000", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0284e-05 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335347+0000", 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6644000000000006e-05 2024-09-20T19:32:59.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335435+0000", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8138000000000002e-05 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.335615+0000", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000179431 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.337544+0000", 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001929004 2024-09-20T19:32:59.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.337936+0000", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000392438 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.350800+0000", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012864161000000001 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.350881+0000", 2024-09-20T19:32:59.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0606999999999996e-05 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:09.350949+0000", 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7877000000000001e-05 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5220 7.16 7:6dcf8a02:::smithi04268048-26:head [copy-from ver 0] snapc ec=[ec,e7,e5,e2,e1,e0,df,dc,da,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e624)", 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:12.335139+0000", 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.662044494, 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014032958, 2024-09-20T19:32:59.077 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5220 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335139+0000", 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335139+0000", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335144+0000", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3759999999999999e-06 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335162+0000", 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7785999999999999e-05 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335193+0000", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.133e-05 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335215+0000", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2382e-05 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335395+0000", 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017913 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335667+0000", 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000272623 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.349171+0000", 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013504331 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.081 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5221 7.16 7:6dcf8a02:::smithi04268048-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e624)", 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:12.335264+0000", 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.661918534000002, 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014787583999999999, 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5221 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335264+0000", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335264+0000", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335267+0000", 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3800000000000001e-06 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335276+0000", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0159999999999997e-06 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335295+0000", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9380000000000001e-05 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.335312+0000", 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7173e-05 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.336075+0000", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076301400000000001 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.336212+0000", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136377 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.349456+0000", 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013244402000000001 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.349766+0000", 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030950200000000002 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:12.350052+0000", 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028634 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.086 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5651 7.1 7:8715d6d2:::smithi04268048-16:head [read 0~3109427,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e668)", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:56.318763+0000", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.6784204909999998, 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013661236, 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5651 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318763+0000", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318763+0000", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318768+0000", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8599999999999998e-06 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318789+0000", 2024-09-20T19:32:59.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0285999999999999e-05 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318839+0000", 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0206e-05 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318864+0000", 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4926000000000002e-05 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319002+0000", 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013841299999999999 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319015+0000", 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3266e-05 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.327964+0000", 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089485720000000001 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.328224+0000", 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000259774 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.332424+0000", 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041999330000000003 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.091 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5652 7.1 7:8715d6d2:::smithi04268048-16:head [read 2605091~504336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e668)", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:56.318920+0000", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.678263281, 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015344287999999999, 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5652 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318920+0000", 2024-09-20T19:32:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318920+0000", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318922+0000", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1969999999999999e-06 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318934+0000", 2024-09-20T19:32:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.208e-05 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318953+0000", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9468999999999999e-05 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318970+0000", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6915999999999999e-05 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319113+0000", 2024-09-20T19:32:59.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014221700000000001 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319122+0000", 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7580000000000004e-06 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.332519+0000", 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013396119999999999 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.332851+0000", 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033275600000000002 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.334264+0000", 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001412775 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:5653 7.1 7:8715d6d2:::smithi04268048-16:head [read 3062953~46474,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e668)", 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:32:56.318993+0000", 2024-09-20T19:32:59.096 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.6781895790000001, 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016005700000000001, 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5653 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318993+0000", 2024-09-20T19:32:59.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318993+0000", 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.318995+0000", 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5060000000000001e-06 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:32:59.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319004+0000", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.67e-06 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319023+0000", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9789999999999999e-05 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319039+0000", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5583e-05 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319159+0000", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012008800000000001 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.319168+0000", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0690000000000008e-06 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.334347+0000", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015178952000000001 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:32:59.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.334579+0000", 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023207100000000001 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:32:56.334999+0000", 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041997099999999999 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:32:59.101 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:32:59.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:59.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:32:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:32:59.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:59.256 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:59.312 INFO:tasks.rados.rados.0.smithi042.stdout:2237: done (6 left) 2024-09-20T19:32:59.312 INFO:tasks.rados.rados.0.smithi042.stdout:2238: read oid 11 snap 241 2024-09-20T19:32:59.312 INFO:tasks.rados.rados.0.smithi042.stdout:2238: expect deleted 2024-09-20T19:32:59.312 INFO:tasks.rados.rados.0.smithi042.stdout:2239: delete oid 37 current snap is 260 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2231: finishing copy_from to smithi04268048-14 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2231: got expected ENOENT (src dne) 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2231: finishing copy_from racing read to smithi04268048-14 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2230: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2230: finishing copy_from to smithi04268048-38 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 616 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:32:59.314 INFO:tasks.rados.rados.0.smithi042.stdout:2233: expect (ObjNum 538 snap 173 seq_num 538) 2024-09-20T19:32:59.359 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:32:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:32:59.447 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1928860 2024-09-20T19:32:59.487 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:59.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:59.520 INFO:tasks.rados.rados.0.smithi042.stdout:2234: expect (ObjNum 756 snap 252 seq_num 756) 2024-09-20T19:32:59.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[24557]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[32075]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.872727+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:57.875777+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.322276+0000 mgr.y (mgr.24491) 565 : cluster [DBG] pgmap v1003: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.9 MiB/s wr, 7 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.915475+0000 osd.4 (osd.4) 49 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.919235+0000 osd.4 (osd.4) 50 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:32:59 smithi042 bash[22706]: cluster 2024-09-20T19:32:59.305025+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-20T19:32:59.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:32:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: finishing write tid 1 to smithi04268048-49 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2235: finishing rollback tid 0 to smithi04268048-20 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: finishing write tid 2 to smithi04268048-49 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2235: finishing rollback tid 1 to smithi04268048-20 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2235: finishing rollback tid 2 to smithi04268048-20 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 988 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: finishing write tid 3 to smithi04268048-49 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: finishing write tid 4 to smithi04268048-49 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: oid 49 updating version 0 to 1175 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: oid 49 updating version 1175 to 1176 2024-09-20T19:32:59.710 INFO:tasks.rados.rados.0.smithi042.stdout:2236: oid 49 updating version 1176 to 1177 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1177 (ObjNum 780 snap 260 seq_num 780) dirty exists 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2236: left oid 49 (ObjNum 780 snap 260 seq_num 780) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2230: done (7 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2231: done (6 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2233: done (5 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2234: done (4 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2235: done (3 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2236: done (2 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2238: done (1 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2239: done (0 left) 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2240: rollback oid 43 current snap is 260 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 250 2024-09-20T19:32:59.711 INFO:tasks.rados.rados.0.smithi042.stdout:2241: snap_remove snap 259 2024-09-20T19:32:59.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:59.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:32:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:32:59.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:59.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:32:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:32:59.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:32:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:32:59.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:32:59.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:32:59.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:00.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:00.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:00.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:2241: done (1 left) 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:2242: delete oid 6 current snap is 260 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:2240: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:2240: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:2240: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:33:00.313 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 734 (ObjNum 661 snap 212 seq_num 661) dirty exists 2024-09-20T19:33:00.320 INFO:tasks.rados.rados.0.smithi042.stdout:2240: done (1 left) 2024-09-20T19:33:00.320 INFO:tasks.rados.rados.0.smithi042.stdout:2242: done (0 left) 2024-09-20T19:33:00.320 INFO:tasks.rados.rados.0.smithi042.stdout:2243: write oid 11 current snap is 260 2024-09-20T19:33:00.321 INFO:tasks.rados.rados.0.smithi042.stdout:2243: seq_num 783 ranges {759881=787973,2119636=654941} 2024-09-20T19:33:00.347 INFO:tasks.rados.rados.0.smithi042.stdout:2243: writing smithi04268048-11 from 759881 to 1547854 tid 1 2024-09-20T19:33:00.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:2243: writing smithi04268048-11 from 2119636 to 2774577 tid 2 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:2244: rollback oid 16 current snap is 260 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 241 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:2243: finishing write tid 1 to smithi04268048-11 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:2245: copy_from oid 1 from oid 33 current snap is 260 2024-09-20T19:33:00.361 INFO:tasks.rados.rados.0.smithi042.stdout:2246: snap_remove snap 222 2024-09-20T19:33:00.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.398 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.399 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:00.515 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:00.642 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.832341+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:58.834579+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:32:59.321853+0000 mon.a (mon.0) 2107 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:00.653 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: audit 2024-09-20T19:32:59.624565+0000 mgr.y (mgr.24491) 566 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:00.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.308873+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-20T19:33:00.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:00.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:00.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:00.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:01.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:01.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:01.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:01.153 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:01.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:01.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:01.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:01.370 INFO:tasks.rados.rados.0.smithi042.stdout:2246: done (3 left) 2024-09-20T19:33:01.370 INFO:tasks.rados.rados.0.smithi042.stdout:2247: write (excl) oid 41 current snap is 260 2024-09-20T19:33:01.370 INFO:tasks.rados.rados.0.smithi042.stdout:2247: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-20T19:33:01.375 INFO:tasks.rados.rados.0.smithi042.stdout:2247: writing smithi04268048-41 from 776688 to 1356263 tid 1 2024-09-20T19:33:01.384 INFO:tasks.rados.rados.0.smithi042.stdout:2247: writing smithi04268048-41 from 1894471 to 2693070 tid 2 2024-09-20T19:33:01.384 INFO:tasks.rados.rados.0.smithi042.stdout:2248: read oid 35 snap -1 2024-09-20T19:33:01.384 INFO:tasks.rados.rados.0.smithi042.stdout:2248: expect deleted 2024-09-20T19:33:01.384 INFO:tasks.rados.rados.0.smithi042.stdout:2249: write (excl) oid 40 current snap is 260 2024-09-20T19:33:01.384 INFO:tasks.rados.rados.0.smithi042.stdout:2249: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-20T19:33:01.393 INFO:tasks.rados.rados.0.smithi042.stdout:2249: writing smithi04268048-40 from 793495 to 1448319 tid 1 2024-09-20T19:33:01.401 INFO:tasks.rados.rados.0.smithi042.stdout:2249: writing smithi04268048-40 from 1952953 to 2611563 tid 2 2024-09-20T19:33:01.405 INFO:tasks.rados.rados.0.smithi042.stdout:2249: writing smithi04268048-40 from 3157167 to 3193495 tid 3 2024-09-20T19:33:01.406 INFO:tasks.rados.rados.0.smithi042.stdout:2250: write (excl) oid 31 current snap is 260 2024-09-20T19:33:01.406 INFO:tasks.rados.rados.0.smithi042.stdout:2250: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-20T19:33:01.413 INFO:tasks.rados.rados.0.smithi042.stdout:2250: writing smithi04268048-31 from 410302 to 1140375 tid 1 2024-09-20T19:33:01.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:01.420 INFO:tasks.rados.rados.0.smithi042.stdout:2250: writing smithi04268048-31 from 1895082 to 2297350 tid 2 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.423 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[22706]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.424 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:01 smithi042 bash[32075]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.427 INFO:tasks.rados.rados.0.smithi042.stdout:2250: writing smithi04268048-31 from 2951884 to 3210302 tid 3 2024-09-20T19:33:01.427 INFO:tasks.rados.rados.0.smithi042.stdout:2251: write (excl) oid 47 current snap is 260 2024-09-20T19:33:01.427 INFO:tasks.rados.rados.0.smithi042.stdout:2245: finishing copy_from to smithi04268048-1 2024-09-20T19:33:01.427 INFO:tasks.rados.rados.0.smithi042.stdout:2245: got expected ENOENT (src dne) 2024-09-20T19:33:01.427 INFO:tasks.rados.rados.0.smithi042.stdout:2251: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-20T19:33:01.432 INFO:tasks.rados.rados.0.smithi042.stdout:2251: writing smithi04268048-47 from 427109 to 832431 tid 1 2024-09-20T19:33:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:01.439 INFO:tasks.rados.rados.0.smithi042.stdout:2251: writing smithi04268048-47 from 1553564 to 2099490 tid 2 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2251: writing smithi04268048-47 from 2579307 to 3227109 tid 3 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2252: rollback oid 42 current snap is 260 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 227 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2245: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: finishing write tid 2 to smithi04268048-11 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: finishing write tid 3 to smithi04268048-11 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: finishing write tid 4 to smithi04268048-11 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: oid 11 updating version 0 to 867 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: oid 11 updating version 867 to 868 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: oid 11 version 868 is already newer than 866 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 868 (ObjNum 783 snap 260 seq_num 783) dirty exists 2024-09-20T19:33:01.447 INFO:tasks.rados.rados.0.smithi042.stdout:2243: left oid 11 (ObjNum 783 snap 260 seq_num 783) 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2244: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2244: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2243: done (8 left) 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2245: done (7 left) 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2253: rollback oid 5 current snap is 260 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 227 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2244: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 745 (ObjNum 661 snap 212 seq_num 661) dirty exists 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: finishing write tid 1 to smithi04268048-41 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: finishing write tid 2 to smithi04268048-41 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: finishing write tid 3 to smithi04268048-41 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: finishing write tid 4 to smithi04268048-41 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: oid 41 updating version 0 to 707 2024-09-20T19:33:01.448 INFO:tasks.rados.rados.0.smithi042.stdout:2247: oid 41 updating version 707 to 709 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2247: oid 41 version 709 is already newer than 708 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 709 (ObjNum 784 snap 260 seq_num 784) dirty exists 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2247: left oid 41 (ObjNum 784 snap 260 seq_num 784) 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: finishing write tid 1 to smithi04268048-40 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: finishing write tid 2 to smithi04268048-40 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: finishing write tid 3 to smithi04268048-40 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: finishing write tid 4 to smithi04268048-40 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: finishing write tid 5 to smithi04268048-40 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: oid 40 updating version 0 to 603 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: oid 40 updating version 603 to 605 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: oid 40 version 605 is already newer than 602 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: oid 40 version 605 is already newer than 604 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 605 (ObjNum 785 snap 260 seq_num 785) dirty exists 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2249: left oid 40 (ObjNum 785 snap 260 seq_num 785) 2024-09-20T19:33:01.449 INFO:tasks.rados.rados.0.smithi042.stdout:2250: finishing write tid 1 to smithi04268048-31 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2250: finishing write tid 2 to smithi04268048-31 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2244: done (7 left) 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2247: done (6 left) 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2248: done (5 left) 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2249: done (4 left) 2024-09-20T19:33:01.450 INFO:tasks.rados.rados.0.smithi042.stdout:2254: snap_remove snap 258 2024-09-20T19:33:01.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.324702+0000 mgr.y (mgr.24491) 567 : cluster [DBG] pgmap v1006: 168 pgs: 1 remapped+peering, 6 peering, 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 141 active+clean; 291 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 112 B/s, 1 objects/s recovering 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.331766+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:01 smithi162 bash[24557]: cluster 2024-09-20T19:33:00.335277+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:33:01.565 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:01.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:01.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:01.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:01.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:01.928 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:01.957 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:01.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:02.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:02.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:02.216 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:02.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:02.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:02.371 INFO:tasks.rados.rados.0.smithi042.stdout:2254: done (4 left) 2024-09-20T19:33:02.371 INFO:tasks.rados.rados.0.smithi042.stdout:2255: write oid 41 current snap is 260 2024-09-20T19:33:02.371 INFO:tasks.rados.rados.0.smithi042.stdout:2255: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-20T19:33:02.377 INFO:tasks.rados.rados.0.smithi042.stdout:2255: writing smithi04268048-41 from 443916 to 924487 tid 1 2024-09-20T19:33:02.384 INFO:tasks.rados.rados.0.smithi042.stdout:2255: writing smithi04268048-41 from 1612046 to 2017983 tid 2 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2255: writing smithi04268048-41 from 2606730 to 3101782 tid 3 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2256: delete oid 50 current snap is 260 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: finishing write tid 3 to smithi04268048-31 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: finishing write tid 4 to smithi04268048-31 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: finishing write tid 5 to smithi04268048-31 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: oid 31 updating version 0 to 795 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: oid 31 updating version 795 to 796 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: oid 31 version 796 is already newer than 794 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: oid 31 version 796 is already newer than 793 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 796 (ObjNum 786 snap 260 seq_num 786) dirty exists 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2250: left oid 31 (ObjNum 786 snap 260 seq_num 786) 2024-09-20T19:33:02.392 INFO:tasks.rados.rados.0.smithi042.stdout:2251: finishing write tid 1 to smithi04268048-47 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: finishing write tid 2 to smithi04268048-47 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2253: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2253: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2253: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 795 (ObjNum 596 snap 189 seq_num 596) dirty exists 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: finishing write tid 3 to smithi04268048-47 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: finishing write tid 4 to smithi04268048-47 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: finishing write tid 5 to smithi04268048-47 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: oid 47 updating version 0 to 947 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: oid 47 updating version 947 to 948 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: oid 47 version 948 is already newer than 945 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: oid 47 version 948 is already newer than 946 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 948 (ObjNum 787 snap 260 seq_num 787) dirty exists 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2251: left oid 47 (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:33:02.393 INFO:tasks.rados.rados.0.smithi042.stdout:2252: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:33:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2252: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:33:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2252: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:33:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 952 (ObjNum 682 snap 221 seq_num 682) dirty exists 2024-09-20T19:33:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2255: finishing write tid 1 to smithi04268048-41 2024-09-20T19:33:02.394 INFO:tasks.rados.rados.0.smithi042.stdout:2255: finishing write tid 2 to smithi04268048-41 2024-09-20T19:33:02.395 INFO:tasks.rados.rados.0.smithi042.stdout:2250: done (5 left) 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2251: done (4 left) 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2252: done (3 left) 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2253: done (2 left) 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2256: done (1 left) 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2257: read oid 50 snap -1 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2257: expect deleted 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2258: write (excl) oid 4 current snap is 260 2024-09-20T19:33:02.396 INFO:tasks.rados.rados.0.smithi042.stdout:2258: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-20T19:33:02.404 INFO:tasks.rados.rados.0.smithi042.stdout:2258: writing smithi04268048-4 from 477530 to 1108599 tid 1 2024-09-20T19:33:02.410 INFO:tasks.rados.rados.0.smithi042.stdout:2258: writing smithi04268048-4 from 1612657 to 2022263 tid 2 2024-09-20T19:33:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2258: writing smithi04268048-4 from 2545223 to 3062819 tid 3 2024-09-20T19:33:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2259: read oid 24 snap -1 2024-09-20T19:33:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2259: expect (ObjNum 762 snap 255 seq_num 762) 2024-09-20T19:33:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2260: read oid 10 snap -1 2024-09-20T19:33:02.418 INFO:tasks.rados.rados.0.smithi042.stdout:2260: expect (ObjNum 774 snap 257 seq_num 774) 2024-09-20T19:33:02.419 INFO:tasks.rados.rados.0.smithi042.stdout:2261: snap_remove snap 227 2024-09-20T19:33:02.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:02.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:02 smithi162 bash[24557]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:02 smithi162 bash[24557]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:02.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:02.597 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:02 smithi042 bash[32075]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:02 smithi042 bash[32075]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:02 smithi042 bash[22706]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:02 smithi042 bash[22706]: cluster 2024-09-20T19:33:01.347071+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-20T19:33:02.699 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:02.726 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:02.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:02.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:02.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:03.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:03.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:03.133 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:33:03.133 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:33:03.151 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:33:03.152 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:33:03.168 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:33:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:33:03.186 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:33:03.186 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:33:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:33:03.208 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:33:03.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:03.228 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:33:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:33:03.236 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:03.252 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:33:03.252 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:33:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:33:03.274 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:33:03.294 INFO:tasks.rados.rados.0.smithi042.stdout:2261: done (5 left) 2024-09-20T19:33:03.294 INFO:tasks.rados.rados.0.smithi042.stdout:2262: write (excl) oid 14 current snap is 260 2024-09-20T19:33:03.294 INFO:tasks.rados.rados.0.smithi042.stdout:2262: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-20T19:33:03.295 INFO:tasks.rados.rados.0.smithi042.stdout:2262: writing smithi04268048-14 from 494337 to 917008 tid 1 2024-09-20T19:33:03.302 INFO:tasks.rados.rados.0.smithi042.stdout:2262: writing smithi04268048-14 from 1387492 to 1940756 tid 2 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2262: writing smithi04268048-14 from 2688999 to 3134220 tid 3 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2263: delete oid 46 current snap is 260 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2255: finishing write tid 3 to smithi04268048-41 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2255: finishing write tid 4 to smithi04268048-41 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2255: finishing write tid 5 to smithi04268048-41 2024-09-20T19:33:03.309 INFO:tasks.rados.rados.0.smithi042.stdout:2255: oid 41 updating version 0 to 710 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2255: oid 41 updating version 710 to 711 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2255: oid 41 updating version 711 to 712 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2255: oid 41 updating version 712 to 713 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 713 (ObjNum 788 snap 260 seq_num 788) dirty exists 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2255: left oid 41 (ObjNum 788 snap 260 seq_num 788) 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: finishing write tid 1 to smithi04268048-4 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: finishing write tid 2 to smithi04268048-4 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: finishing write tid 3 to smithi04268048-4 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: finishing write tid 4 to smithi04268048-4 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: finishing write tid 5 to smithi04268048-4 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: oid 4 updating version 0 to 958 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: oid 4 version 958 is already newer than 957 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: oid 4 version 958 is already newer than 955 2024-09-20T19:33:03.310 INFO:tasks.rados.rados.0.smithi042.stdout:2258: oid 4 version 958 is already newer than 956 2024-09-20T19:33:03.311 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 958 (ObjNum 790 snap 260 seq_num 790) dirty exists 2024-09-20T19:33:03.311 INFO:tasks.rados.rados.0.smithi042.stdout:2258: left oid 4 (ObjNum 790 snap 260 seq_num 790) 2024-09-20T19:33:03.311 INFO:tasks.rados.rados.0.smithi042.stdout:2260: expect (ObjNum 774 snap 257 seq_num 774) 2024-09-20T19:33:03.322 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:33:03.322 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:33:03.322 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-20T19:33:03.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:33:03.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:03.369 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:03.375 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 975595 2024-09-20T19:33:03.416 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:03.471 INFO:tasks.rados.rados.0.smithi042.stdout:2259: expect (ObjNum 762 snap 255 seq_num 762) 2024-09-20T19:33:03.471 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:03.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.500 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:03.563 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1280737 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: finishing write tid 1 to smithi04268048-14 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: finishing write tid 2 to smithi04268048-14 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: finishing write tid 3 to smithi04268048-14 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: finishing write tid 4 to smithi04268048-14 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: finishing write tid 5 to smithi04268048-14 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: oid 14 updating version 0 to 796 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: oid 14 updating version 796 to 799 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: oid 14 version 799 is already newer than 798 2024-09-20T19:33:03.589 INFO:tasks.rados.rados.0.smithi042.stdout:2262: oid 14 version 799 is already newer than 797 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 799 (ObjNum 791 snap 260 seq_num 791) dirty exists 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2262: left oid 14 (ObjNum 791 snap 260 seq_num 791) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2255: done (6 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2257: done (5 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2258: done (4 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2259: done (3 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2260: done (2 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2262: done (1 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2263: done (0 left) 2024-09-20T19:33:03.590 INFO:tasks.rados.rados.0.smithi042.stdout:2264: snap_create 2024-09-20T19:33:03.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:03 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.629 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.127379+0000 osd.2 (osd.2) 205 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.129700+0000 osd.2 (osd.2) 206 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.326234+0000 mgr.y (mgr.24491) 568 : cluster [DBG] pgmap v1008: 168 pgs: 1 remapped+peering, 11 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 282 MiB data, 977 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 4.1 MiB/s, 14 objects/s recovering 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.356859+0000 mon.a (mon.0) 2110 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.367512+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.556757+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.558927+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.902194+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.904208+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.953120+0000 osd.7 (osd.7) 175 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:33:03.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.955357+0000 osd.7 (osd.7) 176 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:33:03.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: audit 2024-09-20T19:33:03.114212+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.42:0/3162743338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:03.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:03 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.289524+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-20T19:33:03.732 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:03.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:03.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:03.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:04.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:04.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:04.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:04.253 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:04.319 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:2264: done (0 left) 2024-09-20T19:33:04.394 INFO:tasks.rados.rados.0.smithi042.stdout:2265: delete oid 28 current snap is 261 2024-09-20T19:33:04.397 INFO:tasks.rados.rados.0.smithi042.stdout:2265: done (0 left) 2024-09-20T19:33:04.397 INFO:tasks.rados.rados.0.smithi042.stdout:2266: snap_create 2024-09-20T19:33:04.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:04.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:04.562 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[32075]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:04 smithi042 bash[22706]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:04.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:04.709 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:04.736 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:04.737 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:04.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.766273+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:33:04.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:02.767871+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:33:04.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.323365+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:33:04.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:04 smithi162 bash[24557]: cluster 2024-09-20T19:33:03.325417+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:33:04.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:04.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:04.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:04.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:05.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:05.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:05.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:05.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:05.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:05.410 INFO:tasks.rados.rados.0.smithi042.stdout:2266: done (0 left) 2024-09-20T19:33:05.410 INFO:tasks.rados.rados.0.smithi042.stdout:2267: rollback oid 15 current snap is 262 2024-09-20T19:33:05.410 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 260 2024-09-20T19:33:05.410 INFO:tasks.rados.rados.0.smithi042.stdout:2268: write (excl) oid 12 current snap is 262 2024-09-20T19:33:05.410 INFO:tasks.rados.rados.0.smithi042.stdout:2268: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-20T19:33:05.431 INFO:tasks.rados.rados.0.smithi042.stdout:2268: writing smithi04268048-12 from 544758 to 1193176 tid 1 2024-09-20T19:33:05.446 INFO:tasks.rados.rados.0.smithi042.stdout:2268: writing smithi04268048-12 from 1846585 to 2263529 tid 2 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2268: writing smithi04268048-12 from 2771268 to 3344758 tid 3 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2269: write oid 45 current snap is 262 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2267: finishing rollback tid 0 to smithi04268048-15 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2267: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2267: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 719 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2268: finishing write tid 1 to smithi04268048-12 2024-09-20T19:33:05.457 INFO:tasks.rados.rados.0.smithi042.stdout:2268: finishing write tid 2 to smithi04268048-12 2024-09-20T19:33:05.458 INFO:tasks.rados.rados.0.smithi042.stdout:2269: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-20T19:33:05.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:05.469 INFO:tasks.rados.rados.0.smithi042.stdout:2269: writing smithi04268048-45 from 561565 to 1285232 tid 1 2024-09-20T19:33:05.479 INFO:tasks.rados.rados.0.smithi042.stdout:2269: writing smithi04268048-45 from 1905067 to 2465669 tid 2 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2269: writing smithi04268048-45 from 3082338 to 3361565 tid 3 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2267: done (2 left) 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2270: read oid 26 snap -1 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2270: expect (ObjNum 768 snap 255 seq_num 768) 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2271: rollback oid 2 current snap is 262 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 251 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: finishing write tid 3 to smithi04268048-12 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: finishing write tid 4 to smithi04268048-12 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: finishing write tid 5 to smithi04268048-12 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: oid 12 updating version 0 to 724 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: oid 12 version 724 is already newer than 723 2024-09-20T19:33:05.488 INFO:tasks.rados.rados.0.smithi042.stdout:2268: oid 12 updating version 724 to 725 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2268: oid 12 version 725 is already newer than 722 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 725 (ObjNum 794 snap 262 seq_num 794) dirty exists 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2268: left oid 12 (ObjNum 794 snap 262 seq_num 794) 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2269: finishing write tid 1 to smithi04268048-45 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2269: finishing write tid 2 to smithi04268048-45 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2268: done (3 left) 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2272: read oid 29 snap 243 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2272: expect deleted 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2273: read oid 14 snap -1 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2273: expect (ObjNum 791 snap 260 seq_num 791) 2024-09-20T19:33:05.489 INFO:tasks.rados.rados.0.smithi042.stdout:2274: delete oid 43 current snap is 262 2024-09-20T19:33:05.492 INFO:tasks.rados.rados.0.smithi042.stdout:2271: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:33:05.492 INFO:tasks.rados.rados.0.smithi042.stdout:2271: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2271: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 619 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2271: done (5 left) 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2272: done (4 left) 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2274: done (3 left) 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2275: read oid 44 snap -1 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2275: expect deleted 2024-09-20T19:33:05.494 INFO:tasks.rados.rados.0.smithi042.stdout:2276: write (excl) oid 36 current snap is 262 2024-09-20T19:33:05.495 INFO:tasks.rados.rados.0.smithi042.stdout:2276: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-20T19:33:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:05.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:05.503 INFO:tasks.rados.rados.0.smithi042.stdout:2276: writing smithi04268048-36 from 595179 to 1069344 tid 1 2024-09-20T19:33:05.512 INFO:tasks.rados.rados.0.smithi042.stdout:2276: writing smithi04268048-36 from 1622031 to 2186302 tid 2 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2276: writing smithi04268048-36 from 2737184 to 3366390 tid 3 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2277: delete oid 24 current snap is 262 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: finishing write tid 3 to smithi04268048-45 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: finishing write tid 4 to smithi04268048-45 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: finishing write tid 5 to smithi04268048-45 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: oid 45 updating version 0 to 1193 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: oid 45 version 1193 is already newer than 1191 2024-09-20T19:33:05.521 INFO:tasks.rados.rados.0.smithi042.stdout:2269: oid 45 updating version 1193 to 1194 2024-09-20T19:33:05.522 INFO:tasks.rados.rados.0.smithi042.stdout:2269: oid 45 version 1194 is already newer than 1192 2024-09-20T19:33:05.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1194 (ObjNum 795 snap 262 seq_num 795) dirty exists 2024-09-20T19:33:05.522 INFO:tasks.rados.rados.0.smithi042.stdout:2269: left oid 45 (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:05.522 INFO:tasks.rados.rados.0.smithi042.stdout:2273: expect (ObjNum 791 snap 260 seq_num 791) 2024-09-20T19:33:05.578 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3148490 2024-09-20T19:33:05.596 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.608 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[32075]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.609 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.610 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:05 smithi042 bash[22706]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:05.679 INFO:tasks.rados.rados.0.smithi042.stdout:2270: expect (ObjNum 768 snap 255 seq_num 768) 2024-09-20T19:33:05.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:05.739 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1280679 2024-09-20T19:33:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.113864+0000 osd.2 (osd.2) 207 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:33:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.117987+0000 osd.2 (osd.2) 208 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:33:05.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.317825+0000 osd.5 (osd.5) 207 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.320109+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.326720+0000 mgr.y (mgr.24491) 569 : cluster [DBG] pgmap v1011: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 273 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 18 op/s; 5.6 MiB/s, 19 objects/s recovering 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.364863+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.366446+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:05 smithi162 bash[24557]: cluster 2024-09-20T19:33:04.393214+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: finishing write tid 1 to smithi04268048-36 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: finishing write tid 2 to smithi04268048-36 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: finishing write tid 3 to smithi04268048-36 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: finishing write tid 4 to smithi04268048-36 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: finishing write tid 5 to smithi04268048-36 2024-09-20T19:33:05.826 INFO:tasks.rados.rados.0.smithi042.stdout:2276: oid 36 updating version 0 to 580 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2276: oid 36 updating version 580 to 581 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2276: oid 36 version 581 is already newer than 578 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2276: oid 36 version 581 is already newer than 579 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 581 (ObjNum 797 snap 262 seq_num 797) dirty exists 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2276: left oid 36 (ObjNum 797 snap 262 seq_num 797) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2269: done (5 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2270: done (4 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2273: done (3 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2275: done (2 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2276: done (1 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2277: done (0 left) 2024-09-20T19:33:05.827 INFO:tasks.rados.rados.0.smithi042.stdout:2278: snap_remove snap 247 2024-09-20T19:33:05.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:05.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:05.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:06.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:06.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:06.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:33:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:06.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:06.422 INFO:tasks.rados.rados.0.smithi042.stdout:2278: done (0 left) 2024-09-20T19:33:06.422 INFO:tasks.rados.rados.0.smithi042.stdout:2279: snap_create 2024-09-20T19:33:06.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:06 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:06.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:06.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:06.679 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.351545+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.356309+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:05.402558+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:06 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.384449+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.f scrub starts 2024-09-20T19:33:06.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:06.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:06.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:06.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:07.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:07.140 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:07.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:07.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:07.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:07.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:07.435 INFO:tasks.rados.rados.0.smithi042.stdout:2279: done (0 left) 2024-09-20T19:33:07.435 INFO:tasks.rados.rados.0.smithi042.stdout:2280: rollback oid 28 current snap is 263 2024-09-20T19:33:07.435 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 243 2024-09-20T19:33:07.436 INFO:tasks.rados.rados.0.smithi042.stdout:2281: snap_create 2024-09-20T19:33:07.448 INFO:tasks.rados.rados.0.smithi042.stdout:2280: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:33:07.449 INFO:tasks.rados.rados.0.smithi042.stdout:2280: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:33:07.449 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 732 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-20T19:33:07.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:07.552 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:07 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:07.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.109285+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.111745+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.327389+0000 mgr.y (mgr.24491) 570 : cluster [DBG] pgmap v1014: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 266 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 330 KiB/s wr, 7 op/s; 1.5 MiB/s, 5 objects/s recovering 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.388332+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.f scrub ok 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.417516+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.754530+0000 osd.6 (osd.6) 150 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.756954+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.363595+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:07 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.365581+0000 osd.1 (osd.1) 236 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:33:07.678 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:07.749 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:33:07.749 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:33:07.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:07.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:07.923 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:07.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:08.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:08.054 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:08.156 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:08.184 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:08.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:08.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:08.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:08.438 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:33:08.443 INFO:tasks.rados.rados.0.smithi042.stdout:2280: done (1 left) 2024-09-20T19:33:08.443 INFO:tasks.rados.rados.0.smithi042.stdout:2281: done (0 left) 2024-09-20T19:33:08.443 INFO:tasks.rados.rados.0.smithi042.stdout:2282: delete oid 34 current snap is 264 2024-09-20T19:33:08.455 INFO:tasks.rados.rados.0.smithi042.stdout:2282: done (0 left) 2024-09-20T19:33:08.455 INFO:tasks.rados.rados.0.smithi042.stdout:2283: read oid 10 snap 251 2024-09-20T19:33:08.455 INFO:tasks.rados.rados.0.smithi042.stdout:2283: expect (ObjNum 733 snap 242 seq_num 733) 2024-09-20T19:33:08.456 INFO:tasks.rados.rados.0.smithi042.stdout:2284: copy_from oid 1 from oid 25 current snap is 264 2024-09-20T19:33:08.456 INFO:tasks.rados.rados.0.smithi042.stdout:2285: write (excl) oid 31 current snap is 264 2024-09-20T19:33:08.456 INFO:tasks.rados.rados.0.smithi042.stdout:2285: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-20T19:33:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:08.462 INFO:tasks.rados.rados.0.smithi042.stdout:2285: writing smithi04268048-31 from 645600 to 1061865 tid 1 2024-09-20T19:33:08.471 INFO:tasks.rados.rados.0.smithi042.stdout:2285: writing smithi04268048-31 from 1797477 to 2509075 tid 2 2024-09-20T19:33:08.476 INFO:tasks.rados.rados.0.smithi042.stdout:2285: writing smithi04268048-31 from 3219453 to 3445600 tid 3 2024-09-20T19:33:08.476 INFO:tasks.rados.rados.0.smithi042.stdout:2286: delete oid 27 current snap is 264 2024-09-20T19:33:08.476 INFO:tasks.rados.rados.0.smithi042.stdout:2285: finishing write tid 1 to smithi04268048-31 2024-09-20T19:33:08.476 INFO:tasks.rados.rados.0.smithi042.stdout:2283: expect (ObjNum 733 snap 242 seq_num 733) 2024-09-20T19:33:08.476 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:08.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:08.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2284: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2284: finishing copy_from to smithi04268048-1 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 707 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2285: finishing write tid 2 to smithi04268048-31 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2285: finishing write tid 3 to smithi04268048-31 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2285: finishing write tid 4 to smithi04268048-31 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2283: done (3 left) 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2284: done (2 left) 2024-09-20T19:33:08.601 INFO:tasks.rados.rados.0.smithi042.stdout:2286: done (1 left) 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2287: snap_create 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: finishing write tid 5 to smithi04268048-31 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: oid 31 updating version 0 to 805 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: oid 31 updating version 805 to 807 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: oid 31 version 807 is already newer than 806 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: oid 31 version 807 is already newer than 802 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 807 (ObjNum 800 snap 264 seq_num 800) dirty exists 2024-09-20T19:33:08.602 INFO:tasks.rados.rados.0.smithi042.stdout:2285: left oid 31 (ObjNum 800 snap 264 seq_num 800) 2024-09-20T19:33:08.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[22706]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:08 smithi042 bash[32075]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:08.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.873506+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:08.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:06.876395+0000 osd.3 (osd.3) 40 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.134384+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.c scrub starts 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.136059+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.c scrub ok 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.310324+0000 osd.5 (osd.5) 209 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.312491+0000 osd.5 (osd.5) 210 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: cluster 2024-09-20T19:33:07.433278+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: audit 2024-09-20T19:33:07.893539+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.42:0/500072851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:08 smithi162 bash[24557]: audit 2024-09-20T19:33:07.894168+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25643 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:08.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:08.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:08.849 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:08.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:08.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:08.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:09.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:09.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:09.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:09.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:09.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:09.464 INFO:tasks.rados.rados.0.smithi042.stdout:2285: done (1 left) 2024-09-20T19:33:09.464 INFO:tasks.rados.rados.0.smithi042.stdout:2287: done (0 left) 2024-09-20T19:33:09.464 INFO:tasks.rados.rados.0.smithi042.stdout:2288: delete oid 13 current snap is 265 2024-09-20T19:33:09.473 INFO:tasks.rados.rados.0.smithi042.stdout:2288: done (0 left) 2024-09-20T19:33:09.473 INFO:tasks.rados.rados.0.smithi042.stdout:2289: read oid 48 snap -1 2024-09-20T19:33:09.473 INFO:tasks.rados.rados.0.smithi042.stdout:2289: expect deleted 2024-09-20T19:33:09.473 INFO:tasks.rados.rados.0.smithi042.stdout:2290: snap_remove snap 243 2024-09-20T19:33:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:09.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:09.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.723 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.244698+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.245759+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.328047+0000 mgr.y (mgr.24491) 571 : cluster [DBG] pgmap v1017: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.425118+0000 mon.a (mon.0) 2118 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.425161+0000 mon.a (mon.0) 2119 : cluster [INF] Cluster is now healthy 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: audit 2024-09-20T19:33:08.427391+0000 mon.a (mon.0) 2120 : audit [INF] from='client.25643 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:08.441308+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: audit 2024-09-20T19:33:09.239196+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.356753+0000 osd.1 (osd.1) 237 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:09 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.359551+0000 osd.1 (osd.1) 238 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:33:09.731 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:09.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:09.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:09.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:09.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:10.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:10.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.125 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:10.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:10.254 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:10.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:10.478 INFO:tasks.rados.rados.0.smithi042.stdout:2290: done (1 left) 2024-09-20T19:33:10.478 INFO:tasks.rados.rados.0.smithi042.stdout:2291: write oid 3 current snap is 265 2024-09-20T19:33:10.478 INFO:tasks.rados.rados.0.smithi042.stdout:2291: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-20T19:33:10.482 INFO:tasks.rados.rados.0.smithi042.stdout:2291: writing smithi04268048-3 from 696021 to 1338033 tid 1 2024-09-20T19:33:10.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.488 INFO:tasks.rados.rados.0.smithi042.stdout:2291: writing smithi04268048-3 from 1856570 to 2431848 tid 2 2024-09-20T19:33:10.495 INFO:tasks.rados.rados.0.smithi042.stdout:2291: writing smithi04268048-3 from 3185369 to 3496021 tid 3 2024-09-20T19:33:10.495 INFO:tasks.rados.rados.0.smithi042.stdout:2289: done (1 left) 2024-09-20T19:33:10.495 INFO:tasks.rados.rados.0.smithi042.stdout:2292: snap_remove snap 263 2024-09-20T19:33:10.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:10.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.635 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:10 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.738 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.462633+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-20T19:33:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.632682+0000 mgr.y (mgr.24491) 572 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.800159+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:09.802294+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.983261+0000 mon.a (mon.0) 2124 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.984116+0000 mon.a (mon.0) 2125 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: audit 2024-09-20T19:33:09.992038+0000 mon.a (mon.0) 2126 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.309893+0000 osd.1 (osd.1) 239 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:10 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.311546+0000 osd.1 (osd.1) 240 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:33:10.845 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:33:10.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:10.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:11.015 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:11.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:11.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:11.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:11.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:11.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:11.494 INFO:tasks.rados.rados.0.smithi042.stdout:2292: done (1 left) 2024-09-20T19:33:11.494 INFO:tasks.rados.rados.0.smithi042.stdout:2293: read oid 32 snap -1 2024-09-20T19:33:11.494 INFO:tasks.rados.rados.0.smithi042.stdout:2293: expect (ObjNum 755 snap 252 seq_num 755) 2024-09-20T19:33:11.494 INFO:tasks.rados.rados.0.smithi042.stdout:2294: write (excl) oid 4 current snap is 265 2024-09-20T19:33:11.494 INFO:tasks.rados.rados.0.smithi042.stdout:2294: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-20T19:33:11.504 INFO:tasks.rados.rados.0.smithi042.stdout:2294: writing smithi04268048-4 from 712828 to 1430089 tid 1 2024-09-20T19:33:11.517 INFO:tasks.rados.rados.0.smithi042.stdout:2294: writing smithi04268048-4 from 1915052 to 2633988 tid 2 2024-09-20T19:33:11.523 INFO:tasks.rados.rados.0.smithi042.stdout:2294: writing smithi04268048-4 from 3212792 to 3512828 tid 3 2024-09-20T19:33:11.524 INFO:tasks.rados.rados.0.smithi042.stdout:2295: read oid 50 snap -1 2024-09-20T19:33:11.524 INFO:tasks.rados.rados.0.smithi042.stdout:2295: expect deleted 2024-09-20T19:33:11.524 INFO:tasks.rados.rados.0.smithi042.stdout:2296: write (excl) oid 40 current snap is 265 2024-09-20T19:33:11.524 INFO:tasks.rados.rados.0.smithi042.stdout:2291: finishing write tid 1 to smithi04268048-3 2024-09-20T19:33:11.524 INFO:tasks.rados.rados.0.smithi042.stdout:2296: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-20T19:33:11.532 INFO:tasks.rados.rados.0.smithi042.stdout:2296: writing smithi04268048-40 from 729635 to 1522145 tid 1 2024-09-20T19:33:11.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:11.538 INFO:tasks.rados.rados.0.smithi042.stdout:2296: writing smithi04268048-40 from 1973534 to 2552481 tid 2 2024-09-20T19:33:11.544 INFO:tasks.rados.rados.0.smithi042.stdout:2296: writing smithi04268048-40 from 3240215 to 3529635 tid 3 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2297: snap_create 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: finishing write tid 2 to smithi04268048-3 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: finishing write tid 3 to smithi04268048-3 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: finishing write tid 4 to smithi04268048-3 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: finishing write tid 5 to smithi04268048-3 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: oid 3 updating version 0 to 734 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: oid 3 updating version 734 to 735 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: oid 3 updating version 735 to 736 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: oid 3 updating version 736 to 737 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 737 (ObjNum 803 snap 265 seq_num 803) dirty exists 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2291: left oid 3 (ObjNum 803 snap 265 seq_num 803) 2024-09-20T19:33:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2293: expect (ObjNum 755 snap 252 seq_num 755) 2024-09-20T19:33:11.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:11.628 INFO:tasks.rados.rados.0.smithi042.stdout:2294: finishing write tid 1 to smithi04268048-4 2024-09-20T19:33:11.628 INFO:tasks.rados.rados.0.smithi042.stdout:2294: finishing write tid 2 to smithi04268048-4 2024-09-20T19:33:11.628 INFO:tasks.rados.rados.0.smithi042.stdout:2294: finishing write tid 3 to smithi04268048-4 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: finishing write tid 4 to smithi04268048-4 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: finishing write tid 5 to smithi04268048-4 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: oid 4 updating version 0 to 972 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: oid 4 updating version 972 to 973 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: oid 4 version 973 is already newer than 971 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: oid 4 version 973 is already newer than 970 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 973 (ObjNum 804 snap 265 seq_num 804) dirty exists 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2294: left oid 4 (ObjNum 804 snap 265 seq_num 804) 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: finishing write tid 1 to smithi04268048-40 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: finishing write tid 2 to smithi04268048-40 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: finishing write tid 3 to smithi04268048-40 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: finishing write tid 4 to smithi04268048-40 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: finishing write tid 5 to smithi04268048-40 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: oid 40 updating version 0 to 612 2024-09-20T19:33:11.629 INFO:tasks.rados.rados.0.smithi042.stdout:2296: oid 40 version 612 is already newer than 611 2024-09-20T19:33:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:2296: oid 40 updating version 612 to 613 2024-09-20T19:33:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:2296: oid 40 updating version 613 to 614 2024-09-20T19:33:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 614 (ObjNum 805 snap 265 seq_num 805) dirty exists 2024-09-20T19:33:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:2296: left oid 40 (ObjNum 805 snap 265 seq_num 805) 2024-09-20T19:33:11.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:11.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:11 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:11 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:11 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:11 smithi162 bash[24557]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:11.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[32075]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.328525+0000 mgr.y (mgr.24491) 573 : cluster [DBG] pgmap v1020: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:11.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:11 smithi042 bash[22706]: cluster 2024-09-20T19:33:10.473178+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-20T19:33:11.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:11.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:11.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:11.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:12.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:12.070 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:12.174 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:33:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:33:12.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:12.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:12.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:12.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:12 smithi042 bash[22706]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:12 smithi042 bash[22706]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2291: done (5 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2293: done (4 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2294: done (3 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2295: done (2 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2296: done (1 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2297: done (0 left) 2024-09-20T19:33:12.499 INFO:tasks.rados.rados.0.smithi042.stdout:2298: snap_remove snap 255 2024-09-20T19:33:12.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.593 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:12 smithi042 bash[32075]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:12 smithi042 bash[32075]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:12 smithi162 bash[24557]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:12 smithi162 bash[24557]: cluster 2024-09-20T19:33:11.483887+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-20T19:33:12.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:12.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:12.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:13.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:13.186 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1022,"stamp":"2024-09-20T19:33:11.477920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242076159,"num_objects":323,"num_object_clones":79,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":11,"num_read":3996,"num_read_kb":1520464,"num_write":2167,"num_write_kb":538198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":112316581,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16551,"ondisk_log_size":16551,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":913104,"kb_used_data":571680,"kb_used_omap":143,"kb_used_meta":340912,"kb_avail":749048112,"statfs":{"total":767960285184,"available":767025266688,"internally_reserved":0,"allocated":585400320,"data_stored":755404950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147312,"internal_metadata":349094032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12191742,"num_objects":0,"num_object_clones":-1,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":12,"num_read_kb":8637,"num_write":4,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052259"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":1088,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442603+0000","last_change":"2024-09-20T19:31:37.502437+0000","last_active":"2024-09-20T19:33:07.442603+0000","last_peered":"2024-09-20T19:33:07.442603+0000","last_clean":"2024-09-20T19:33:07.442603+0000","last_became_active":"2024-09-20T19:30:59.423955+0000","last_became_peered":"2024-09-20T19:30:59.423955+0000","last_unstale":"2024-09-20T19:33:07.442603+0000","last_undegraded":"2024-09-20T19:33:07.442603+0000","last_fullsized":"2024-09-20T19:33:07.442603+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1065,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.437239+0000","last_change":"2024-09-20T19:32:13.393790+0000","last_active":"2024-09-20T19:33:07.437239+0000","last_peered":"2024-09-20T19:33:07.437239+0000","last_clean":"2024-09-20T19:33:07.437239+0000","last_became_active":"2024-09-20T19:30:31.353214+0000","last_became_peered":"2024-09-20T19:30:31.353214+0000","last_unstale":"2024-09-20T19:33:07.437239+0000","last_undegraded":"2024-09-20T19:33:07.437239+0000","last_fullsized":"2024-09-20T19:33:07.437239+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.908799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1152,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441627+0000","last_change":"2024-09-20T19:32:58.834661+0000","last_active":"2024-09-20T19:33:07.441627+0000","last_peered":"2024-09-20T19:33:07.441627+0000","last_clean":"2024-09-20T19:33:07.441627+0000","last_became_active":"2024-09-20T19:30:59.432572+0000","last_became_peered":"2024-09-20T19:30:59.432572+0000","last_unstale":"2024-09-20T19:33:07.441627+0000","last_undegraded":"2024-09-20T19:33:07.441627+0000","last_fullsized":"2024-09-20T19:33:07.441627+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:32:58.834586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:58.834586+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:20.149523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1038,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.436459+0000","last_change":"2024-09-20T19:30:44.918367+0000","last_active":"2024-09-20T19:33:08.436459+0000","last_peered":"2024-09-20T19:33:08.436459+0000","last_clean":"2024-09-20T19:33:08.436459+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:33:08.436459+0000","last_undegraded":"2024-09-20T19:33:08.436459+0000","last_fullsized":"2024-09-20T19:33:08.436459+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:44.918249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:44.918249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1143,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442702+0000","last_change":"2024-09-20T19:32:12.874027+0000","last_active":"2024-09-20T19:33:07.442702+0000","last_peered":"2024-09-20T19:33:07.442702+0000","last_clean":"2024-09-20T19:33:07.442702+0000","last_became_active":"2024-09-20T19:30:59.439274+0000","last_became_peered":"2024-09-20T19:30:59.439274+0000","last_unstale":"2024-09-20T19:33:07.442702+0000","last_undegraded":"2024-09-20T19:33:07.442702+0000","last_fullsized":"2024-09-20T19:33:07.442702+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.712357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1142,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.436921+0000","last_change":"2024-09-20T19:30:49.903222+0000","last_active":"2024-09-20T19:33:08.436921+0000","last_peered":"2024-09-20T19:33:08.436921+0000","last_clean":"2024-09-20T19:33:08.436921+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:33:08.436921+0000","last_undegraded":"2024-09-20T19:33:08.436921+0000","last_fullsized":"2024-09-20T19:33:08.436921+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:49.903144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:30:49.903144+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1265,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.415095+0000","last_change":"2024-09-20T19:29:00.755422+0000","last_active":"2024-09-20T19:33:06.415095+0000","last_peered":"2024-09-20T19:33:06.415095+0000","last_clean":"2024-09-20T19:33:06.415095+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:33:06.415095+0000","last_undegraded":"2024-09-20T19:33:06.415095+0000","last_fullsized":"2024-09-20T19:33:06.415095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:29:00.755361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.755361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1100,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.437793+0000","last_change":"2024-09-20T19:31:20.821748+0000","last_active":"2024-09-20T19:33:08.437793+0000","last_peered":"2024-09-20T19:33:08.437793+0000","last_clean":"2024-09-20T19:33:08.437793+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:33:08.437793+0000","last_undegraded":"2024-09-20T19:33:08.437793+0000","last_fullsized":"2024-09-20T19:33:08.437793+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:20.821598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.821598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"678'1061","reported_seq":3713,"reported_epoch":679,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:07.428848+0000","last_change":"2024-09-20T19:33:06.481929+0000","last_active":"2024-09-20T19:33:07.428848+0000","last_peered":"2024-09-20T19:33:07.428848+0000","last_clean":"2024-09-20T19:33:07.428848+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:33:07.428848+0000","last_undegraded":"2024-09-20T19:33:07.428848+0000","last_fullsized":"2024-09-20T19:33:07.428848+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:56.272270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.174814754,"stat_sum":{"num_bytes":6286115,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":36,"num_read_kb":31345,"num_write":36,"num_write_kb":10922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1279,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424371+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:33:06.424371+0000","last_peered":"2024-09-20T19:33:06.424371+0000","last_clean":"2024-09-20T19:33:06.424371+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:33:06.424371+0000","last_undegraded":"2024-09-20T19:33:06.424371+0000","last_fullsized":"2024-09-20T19:33:06.424371+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.958849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1106,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438403+0000","last_change":"2024-09-20T19:32:29.224300+0000","last_active":"2024-09-20T19:33:08.438403+0000","last_peered":"2024-09-20T19:33:08.438403+0000","last_clean":"2024-09-20T19:33:08.438403+0000","last_became_active":"2024-09-20T19:30:59.395790+0000","last_became_peered":"2024-09-20T19:30:59.395790+0000","last_unstale":"2024-09-20T19:33:08.438403+0000","last_undegraded":"2024-09-20T19:33:08.438403+0000","last_fullsized":"2024-09-20T19:33:08.438403+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:32:29.224169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.224169+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:53.488936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1263,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420993+0000","last_change":"2024-09-20T19:30:31.352779+0000","last_active":"2024-09-20T19:33:06.420993+0000","last_peered":"2024-09-20T19:33:06.420993+0000","last_clean":"2024-09-20T19:33:06.420993+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:33:06.420993+0000","last_undegraded":"2024-09-20T19:33:06.420993+0000","last_fullsized":"2024-09-20T19:33:06.420993+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:11.324315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:11.324315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.925224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1276,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422881+0000","last_change":"2024-09-20T19:32:28.436574+0000","last_active":"2024-09-20T19:33:06.422881+0000","last_peered":"2024-09-20T19:33:06.422881+0000","last_clean":"2024-09-20T19:33:06.422881+0000","last_became_active":"2024-09-20T19:30:59.422108+0000","last_became_peered":"2024-09-20T19:30:59.422108+0000","last_unstale":"2024-09-20T19:33:06.422881+0000","last_undegraded":"2024-09-20T19:33:06.422881+0000","last_fullsized":"2024-09-20T19:33:06.422881+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.196618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"678'376","reported_seq":3062,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.521700+0000","last_change":"2024-09-20T19:33:06.521700+0000","last_active":"2024-09-20T19:33:06.521700+0000","last_peered":"2024-09-20T19:33:06.521700+0000","last_clean":"2024-09-20T19:33:06.521700+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:33:06.521700+0000","last_undegraded":"2024-09-20T19:33:06.521700+0000","last_fullsized":"2024-09-20T19:33:06.521700+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'290","last_scrub_stamp":"2024-09-20T19:29:01.442771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:01.442771+0000","objects_scrubbed":4,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.100607327,"stat_sum":{"num_bytes":6507672,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":11060,"num_write":24,"num_write_kb":6268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1268,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423921+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:33:06.423921+0000","last_peered":"2024-09-20T19:33:06.423921+0000","last_clean":"2024-09-20T19:33:06.423921+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:33:06.423921+0000","last_undegraded":"2024-09-20T19:33:06.423921+0000","last_fullsized":"2024-09-20T19:33:06.423921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.698584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1152,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.436962+0000","last_change":"2024-09-20T19:32:15.452097+0000","last_active":"2024-09-20T19:33:08.436962+0000","last_peered":"2024-09-20T19:33:08.436962+0000","last_clean":"2024-09-20T19:33:08.436962+0000","last_became_active":"2024-09-20T19:30:59.403992+0000","last_became_peered":"2024-09-20T19:30:59.403992+0000","last_unstale":"2024-09-20T19:33:08.436962+0000","last_undegraded":"2024-09-20T19:33:08.436962+0000","last_fullsized":"2024-09-20T19:33:08.436962+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.451996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.451996+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:23.255186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1263,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425074+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:33:06.425074+0000","last_peered":"2024-09-20T19:33:06.425074+0000","last_clean":"2024-09-20T19:33:06.425074+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:33:06.425074+0000","last_undegraded":"2024-09-20T19:33:06.425074+0000","last_fullsized":"2024-09-20T19:33:06.425074+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.435769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":241,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.428186+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:33:07.428186+0000","last_peered":"2024-09-20T19:33:07.428186+0000","last_clean":"2024-09-20T19:33:07.428186+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:33:07.428186+0000","last_undegraded":"2024-09-20T19:33:07.428186+0000","last_fullsized":"2024-09-20T19:33:07.428186+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.693370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":880,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422720+0000","last_change":"2024-09-20T19:30:59.413612+0000","last_active":"2024-09-20T19:33:06.422720+0000","last_peered":"2024-09-20T19:33:06.422720+0000","last_clean":"2024-09-20T19:33:06.422720+0000","last_became_active":"2024-09-20T19:30:59.413239+0000","last_became_peered":"2024-09-20T19:30:59.413239+0000","last_unstale":"2024-09-20T19:33:06.422720+0000","last_undegraded":"2024-09-20T19:33:06.422720+0000","last_fullsized":"2024-09-20T19:33:06.422720+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:04.483789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:04.483789+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:38.360692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"678'236","reported_seq":2971,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.432014+0000","last_change":"2024-09-20T19:33:06.432407+0000","last_active":"2024-09-20T19:33:06.432014+0000","last_peered":"2024-09-20T19:33:06.432014+0000","last_clean":"2024-09-20T19:33:06.432014+0000","last_became_active":"2024-09-20T19:30:09.790218+0000","last_became_peered":"2024-09-20T19:30:09.790218+0000","last_unstale":"2024-09-20T19:33:06.432014+0000","last_undegraded":"2024-09-20T19:33:06.432014+0000","last_fullsized":"2024-09-20T19:33:06.432014+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'200","last_scrub_stamp":"2024-09-20T19:32:31.708023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.708023+0000","objects_scrubbed":2,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:39.642570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.019859890000000002,"stat_sum":{"num_bytes":2735439,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":45682,"num_write":87,"num_write_kb":31093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2317410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"4"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.d","version":"523'31","reported_seq":1128,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438811+0000","last_change":"2024-09-20T19:32:23.326867+0000","last_active":"2024-09-20T19:33:08.438811+0000","last_peered":"2024-09-20T19:33:08.438811+0000","last_clean":"2024-09-20T19:33:08.438811+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:33:08.438811+0000","last_undegraded":"2024-09-20T19:33:08.438811+0000","last_fullsized":"2024-09-20T19:33:08.438811+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:32:23.326721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.326721+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:26.897588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1264,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423518+0000","last_change":"2024-09-20T19:30:31.374792+0000","last_active":"2024-09-20T19:33:06.423518+0000","last_peered":"2024-09-20T19:33:06.423518+0000","last_clean":"2024-09-20T19:33:06.423518+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:33:06.423518+0000","last_undegraded":"2024-09-20T19:33:06.423518+0000","last_fullsized":"2024-09-20T19:33:06.423518+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:10.363587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:10.363587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:43.992389+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1283,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423738+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:33:06.423738+0000","last_peered":"2024-09-20T19:33:06.423738+0000","last_clean":"2024-09-20T19:33:06.423738+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:33:06.423738+0000","last_undegraded":"2024-09-20T19:33:06.423738+0000","last_fullsized":"2024-09-20T19:33:06.423738+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1038,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.445737+0000","last_change":"2024-09-20T19:32:57.337890+0000","last_active":"2024-09-20T19:33:07.445737+0000","last_peered":"2024-09-20T19:33:07.445737+0000","last_clean":"2024-09-20T19:33:07.445737+0000","last_became_active":"2024-09-20T19:32:57.337487+0000","last_became_peered":"2024-09-20T19:32:57.337487+0000","last_unstale":"2024-09-20T19:33:07.445737+0000","last_undegraded":"2024-09-20T19:33:07.445737+0000","last_fullsized":"2024-09-20T19:33:07.445737+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:31.525745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.525745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1351,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421265+0000","last_change":"2024-09-20T19:33:02.955491+0000","last_active":"2024-09-20T19:33:06.421265+0000","last_peered":"2024-09-20T19:33:06.421265+0000","last_clean":"2024-09-20T19:33:06.421265+0000","last_became_active":"2024-09-20T19:32:57.349778+0000","last_became_peered":"2024-09-20T19:32:57.349778+0000","last_unstale":"2024-09-20T19:33:06.421265+0000","last_undegraded":"2024-09-20T19:33:06.421265+0000","last_fullsized":"2024-09-20T19:33:06.421265+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:27.788847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"678'516","reported_seq":2931,"reported_epoch":679,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:07.436964+0000","last_change":"2024-09-20T19:33:05.382837+0000","last_active":"2024-09-20T19:33:07.436964+0000","last_peered":"2024-09-20T19:33:07.436964+0000","last_clean":"2024-09-20T19:33:07.436964+0000","last_became_active":"2024-09-20T19:30:59.433690+0000","last_became_peered":"2024-09-20T19:30:59.433690+0000","last_unstale":"2024-09-20T19:33:07.436964+0000","last_undegraded":"2024-09-20T19:33:07.436964+0000","last_fullsized":"2024-09-20T19:33:07.436964+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'449","last_scrub_stamp":"2024-09-20T19:32:09.344681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.344681+0000","objects_scrubbed":6,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.237070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.10321581000000001,"stat_sum":{"num_bytes":19278549,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":251,"num_read_kb":210201,"num_write":172,"num_write_kb":69767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17328954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"1"}]},{"pgid":"4.c","version":"523'20","reported_seq":1160,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438598+0000","last_change":"2024-09-20T19:32:10.444211+0000","last_active":"2024-09-20T19:33:08.438598+0000","last_peered":"2024-09-20T19:33:08.438598+0000","last_clean":"2024-09-20T19:33:08.438598+0000","last_became_active":"2024-09-20T19:30:59.412354+0000","last_became_peered":"2024-09-20T19:30:59.412354+0000","last_unstale":"2024-09-20T19:33:08.438598+0000","last_undegraded":"2024-09-20T19:33:08.438598+0000","last_fullsized":"2024-09-20T19:33:08.438598+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:32:10.444013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.444013+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:11.431206+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1284,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423083+0000","last_change":"2024-09-20T19:31:39.214509+0000","last_active":"2024-09-20T19:33:06.423083+0000","last_peered":"2024-09-20T19:33:06.423083+0000","last_clean":"2024-09-20T19:33:06.423083+0000","last_became_active":"2024-09-20T19:30:59.435459+0000","last_became_peered":"2024-09-20T19:30:59.435459+0000","last_unstale":"2024-09-20T19:33:06.423083+0000","last_undegraded":"2024-09-20T19:33:06.423083+0000","last_fullsized":"2024-09-20T19:33:06.423083+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:39.214431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.214431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1465,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421732+0000","last_change":"2024-09-20T19:29:54.085338+0000","last_active":"2024-09-20T19:33:06.421732+0000","last_peered":"2024-09-20T19:33:06.421732+0000","last_clean":"2024-09-20T19:33:06.421732+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:33:06.421732+0000","last_undegraded":"2024-09-20T19:33:06.421732+0000","last_fullsized":"2024-09-20T19:33:06.421732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:29:54.085246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:29:54.085246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:56.042847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1103,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438889+0000","last_change":"2024-09-20T19:32:57.956406+0000","last_active":"2024-09-20T19:33:08.438889+0000","last_peered":"2024-09-20T19:33:08.438889+0000","last_clean":"2024-09-20T19:33:08.438889+0000","last_became_active":"2024-09-20T19:32:57.344694+0000","last_became_peered":"2024-09-20T19:32:57.344694+0000","last_unstale":"2024-09-20T19:33:08.438889+0000","last_undegraded":"2024-09-20T19:33:08.438889+0000","last_fullsized":"2024-09-20T19:33:08.438889+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:57.956335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.956335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:05.511349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":862,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422796+0000","last_change":"2024-09-20T19:28:03.366431+0000","last_active":"2024-09-20T19:33:06.422796+0000","last_peered":"2024-09-20T19:33:06.422796+0000","last_clean":"2024-09-20T19:33:06.422796+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:33:06.422796+0000","last_undegraded":"2024-09-20T19:33:06.422796+0000","last_fullsized":"2024-09-20T19:33:06.422796+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:28:03.366356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.366356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.375144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"676'800","reported_seq":3514,"reported_epoch":678,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:06.521620+0000","last_change":"2024-09-20T19:33:06.521656+0000","last_active":"2024-09-20T19:33:06.521620+0000","last_peered":"2024-09-20T19:33:06.521620+0000","last_clean":"2024-09-20T19:33:06.521620+0000","last_became_active":"2024-09-20T19:30:59.437942+0000","last_became_peered":"2024-09-20T19:30:59.437942+0000","last_unstale":"2024-09-20T19:33:06.521620+0000","last_undegraded":"2024-09-20T19:33:06.521620+0000","last_fullsized":"2024-09-20T19:33:06.521620+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'237","last_scrub_stamp":"2024-09-20T19:24:35.542766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:35.542766+0000","objects_scrubbed":12,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.10139326799999999,"stat_sum":{"num_bytes":14305531,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":412,"num_read_kb":295976,"num_write":283,"num_write_kb":88614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":11179590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f6","length":"1"}]},{"pgid":"4.f","version":"523'27","reported_seq":1343,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.430879+0000","last_change":"2024-09-20T19:33:06.388487+0000","last_active":"2024-09-20T19:33:06.430879+0000","last_peered":"2024-09-20T19:33:06.430879+0000","last_clean":"2024-09-20T19:33:06.430879+0000","last_became_active":"2024-09-20T19:30:31.365973+0000","last_became_peered":"2024-09-20T19:30:31.365973+0000","last_unstale":"2024-09-20T19:33:06.430879+0000","last_undegraded":"2024-09-20T19:33:06.430879+0000","last_fullsized":"2024-09-20T19:33:06.430879+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:33:06.388344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.388344+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:15.100225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1292,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.415467+0000","last_change":"2024-09-20T19:32:57.249115+0000","last_active":"2024-09-20T19:33:06.415467+0000","last_peered":"2024-09-20T19:33:06.415467+0000","last_clean":"2024-09-20T19:33:06.415467+0000","last_became_active":"2024-09-20T19:30:59.434928+0000","last_became_peered":"2024-09-20T19:30:59.434928+0000","last_unstale":"2024-09-20T19:33:06.415467+0000","last_undegraded":"2024-09-20T19:33:06.415467+0000","last_fullsized":"2024-09-20T19:33:06.415467+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:57.249036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.249036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:16.844536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1256,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439225+0000","last_change":"2024-09-20T19:30:38.015879+0000","last_active":"2024-09-20T19:33:08.439225+0000","last_peered":"2024-09-20T19:33:08.439225+0000","last_clean":"2024-09-20T19:33:08.439225+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:33:08.439225+0000","last_undegraded":"2024-09-20T19:33:08.439225+0000","last_fullsized":"2024-09-20T19:33:08.439225+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:30:38.015815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:30:38.015815+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.989994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1268,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423887+0000","last_change":"2024-09-20T19:32:24.498998+0000","last_active":"2024-09-20T19:33:06.423887+0000","last_peered":"2024-09-20T19:33:06.423887+0000","last_clean":"2024-09-20T19:33:06.423887+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:33:06.423887+0000","last_undegraded":"2024-09-20T19:33:06.423887+0000","last_fullsized":"2024-09-20T19:33:06.423887+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:24.498942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:24.498942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.513248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"676'511","reported_seq":3221,"reported_epoch":678,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:06.536205+0000","last_change":"2024-09-20T19:33:06.536236+0000","last_active":"2024-09-20T19:33:06.536205+0000","last_peered":"2024-09-20T19:33:06.536205+0000","last_clean":"2024-09-20T19:33:06.536205+0000","last_became_active":"2024-09-20T19:30:09.785274+0000","last_became_peered":"2024-09-20T19:30:09.785274+0000","last_unstale":"2024-09-20T19:33:06.536205+0000","last_undegraded":"2024-09-20T19:33:06.536205+0000","last_fullsized":"2024-09-20T19:33:06.536205+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'149","last_scrub_stamp":"2024-09-20T19:24:44.564761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:24:44.564761+0000","objects_scrubbed":9,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.115762743,"stat_sum":{"num_bytes":8067822,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":272,"num_read_kb":230215,"num_write":178,"num_write_kb":69515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":15038144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"}]},{"pgid":"4.e","version":"523'19","reported_seq":1160,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439700+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:33:08.439700+0000","last_peered":"2024-09-20T19:33:08.439700+0000","last_clean":"2024-09-20T19:33:08.439700+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:33:08.439700+0000","last_undegraded":"2024-09-20T19:33:08.439700+0000","last_fullsized":"2024-09-20T19:33:08.439700+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1160,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.417603+0000","last_change":"2024-09-20T19:31:26.549536+0000","last_active":"2024-09-20T19:33:06.417603+0000","last_peered":"2024-09-20T19:33:06.417603+0000","last_clean":"2024-09-20T19:33:06.417603+0000","last_became_active":"2024-09-20T19:30:59.407964+0000","last_became_peered":"2024-09-20T19:30:59.407964+0000","last_unstale":"2024-09-20T19:33:06.417603+0000","last_undegraded":"2024-09-20T19:33:06.417603+0000","last_fullsized":"2024-09-20T19:33:06.417603+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.756600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1264,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425074+0000","last_change":"2024-09-20T19:32:13.572694+0000","last_active":"2024-09-20T19:33:06.425074+0000","last_peered":"2024-09-20T19:33:06.425074+0000","last_clean":"2024-09-20T19:33:06.425074+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:33:06.425074+0000","last_undegraded":"2024-09-20T19:33:06.425074+0000","last_fullsized":"2024-09-20T19:33:06.425074+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.572577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.572577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:15.678184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1075,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.444921+0000","last_change":"2024-09-20T19:30:59.413132+0000","last_active":"2024-09-20T19:33:07.444921+0000","last_peered":"2024-09-20T19:33:07.444921+0000","last_clean":"2024-09-20T19:33:07.444921+0000","last_became_active":"2024-09-20T19:30:59.412722+0000","last_became_peered":"2024-09-20T19:30:59.412722+0000","last_unstale":"2024-09-20T19:33:07.444921+0000","last_undegraded":"2024-09-20T19:33:07.444921+0000","last_fullsized":"2024-09-20T19:33:07.444921+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":708,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.446012+0000","last_change":"2024-09-20T19:33:06.876548+0000","last_active":"2024-09-20T19:33:07.446012+0000","last_peered":"2024-09-20T19:33:07.446012+0000","last_clean":"2024-09-20T19:33:07.446012+0000","last_became_active":"2024-09-20T19:30:59.426646+0000","last_became_peered":"2024-09-20T19:30:59.426646+0000","last_unstale":"2024-09-20T19:33:07.446012+0000","last_undegraded":"2024-09-20T19:33:07.446012+0000","last_fullsized":"2024-09-20T19:33:07.446012+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:06.876408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.876408+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:36.119343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":2684,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.443927+0000","last_change":"2024-09-20T19:33:06.477945+0000","last_active":"2024-09-20T19:33:07.443927+0000","last_peered":"2024-09-20T19:33:07.443927+0000","last_clean":"2024-09-20T19:33:07.443927+0000","last_became_active":"2024-09-20T19:30:59.420070+0000","last_became_peered":"2024-09-20T19:30:59.420070+0000","last_unstale":"2024-09-20T19:33:07.443927+0000","last_undegraded":"2024-09-20T19:33:07.443927+0000","last_fullsized":"2024-09-20T19:33:07.443927+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'186","last_scrub_stamp":"2024-09-20T19:31:31.606808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.606808+0000","objects_scrubbed":3,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.042797600999999998,"stat_sum":{"num_bytes":3075846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":4832,"num_write":5,"num_write_kb":1395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.1","version":"523'28","reported_seq":1129,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.437590+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:33:08.437590+0000","last_peered":"2024-09-20T19:33:08.437590+0000","last_clean":"2024-09-20T19:33:08.437590+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:33:08.437590+0000","last_undegraded":"2024-09-20T19:33:08.437590+0000","last_fullsized":"2024-09-20T19:33:08.437590+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.002901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1269,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421654+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:33:06.421654+0000","last_peered":"2024-09-20T19:33:06.421654+0000","last_clean":"2024-09-20T19:33:06.421654+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:33:06.421654+0000","last_undegraded":"2024-09-20T19:33:06.421654+0000","last_fullsized":"2024-09-20T19:33:06.421654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"554'15","reported_seq":1198,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441491+0000","last_change":"2024-09-20T19:30:59.419924+0000","last_active":"2024-09-20T19:33:07.441491+0000","last_peered":"2024-09-20T19:33:07.441491+0000","last_clean":"2024-09-20T19:33:07.441491+0000","last_became_active":"2024-09-20T19:30:59.418231+0000","last_became_peered":"2024-09-20T19:30:59.418231+0000","last_unstale":"2024-09-20T19:33:07.441491+0000","last_undegraded":"2024-09-20T19:33:07.441491+0000","last_fullsized":"2024-09-20T19:33:07.441491+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:24:32.089810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:24:32.089810+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.866533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1250,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423199+0000","last_change":"2024-09-20T19:30:03.184615+0000","last_active":"2024-09-20T19:33:06.423199+0000","last_peered":"2024-09-20T19:33:06.423199+0000","last_clean":"2024-09-20T19:33:06.423199+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:33:06.423199+0000","last_undegraded":"2024-09-20T19:33:06.423199+0000","last_fullsized":"2024-09-20T19:33:06.423199+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:03.184552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:30:03.184552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":668,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439239+0000","last_change":"2024-09-20T19:32:57.343108+0000","last_active":"2024-09-20T19:33:08.439239+0000","last_peered":"2024-09-20T19:33:08.439239+0000","last_clean":"2024-09-20T19:33:08.439239+0000","last_became_active":"2024-09-20T19:32:57.342079+0000","last_became_peered":"2024-09-20T19:32:57.342079+0000","last_unstale":"2024-09-20T19:33:08.439239+0000","last_undegraded":"2024-09-20T19:33:08.439239+0000","last_fullsized":"2024-09-20T19:33:08.439239+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:30:34.035776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:30:34.035776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"677'395","reported_seq":2892,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.444327+0000","last_change":"2024-09-20T19:33:06.477568+0000","last_active":"2024-09-20T19:33:07.444327+0000","last_peered":"2024-09-20T19:33:07.444327+0000","last_clean":"2024-09-20T19:33:07.444327+0000","last_became_active":"2024-09-20T19:30:59.407988+0000","last_became_peered":"2024-09-20T19:30:59.407988+0000","last_unstale":"2024-09-20T19:33:07.444327+0000","last_undegraded":"2024-09-20T19:33:07.444327+0000","last_fullsized":"2024-09-20T19:33:07.444327+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'350","last_scrub_stamp":"2024-09-20T19:31:19.546439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.546439+0000","objects_scrubbed":2,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.045313642000000001,"stat_sum":{"num_bytes":9915547,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":14,"num_read_kb":10454,"num_write":16,"num_write_kb":8433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.0","version":"523'30","reported_seq":1161,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.439111+0000","last_change":"2024-09-20T19:32:20.900983+0000","last_active":"2024-09-20T19:33:07.439111+0000","last_peered":"2024-09-20T19:33:07.439111+0000","last_clean":"2024-09-20T19:33:07.439111+0000","last_became_active":"2024-09-20T19:30:59.431399+0000","last_became_peered":"2024-09-20T19:30:59.431399+0000","last_unstale":"2024-09-20T19:33:07.439111+0000","last_undegraded":"2024-09-20T19:33:07.439111+0000","last_fullsized":"2024-09-20T19:33:07.439111+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.542941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1277,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.430630+0000","last_change":"2024-09-20T19:32:15.096541+0000","last_active":"2024-09-20T19:33:06.430630+0000","last_peered":"2024-09-20T19:33:06.430630+0000","last_clean":"2024-09-20T19:33:06.430630+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:33:06.430630+0000","last_undegraded":"2024-09-20T19:33:06.430630+0000","last_fullsized":"2024-09-20T19:33:06.430630+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:32:15.096450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.096450+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:25.106545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1081,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.436548+0000","last_change":"2024-09-20T19:32:18.400368+0000","last_active":"2024-09-20T19:33:08.436548+0000","last_peered":"2024-09-20T19:33:08.436548+0000","last_clean":"2024-09-20T19:33:08.436548+0000","last_became_active":"2024-09-20T19:30:59.403694+0000","last_became_peered":"2024-09-20T19:30:59.403694+0000","last_unstale":"2024-09-20T19:33:08.436548+0000","last_undegraded":"2024-09-20T19:33:08.436548+0000","last_fullsized":"2024-09-20T19:33:08.436548+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:18.400304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.400304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.263499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1035,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.437113+0000","last_change":"2024-09-20T19:33:02.904339+0000","last_active":"2024-09-20T19:33:08.437113+0000","last_peered":"2024-09-20T19:33:08.437113+0000","last_clean":"2024-09-20T19:33:08.437113+0000","last_became_active":"2024-09-20T19:30:59.403080+0000","last_became_peered":"2024-09-20T19:30:59.403080+0000","last_unstale":"2024-09-20T19:33:08.437113+0000","last_undegraded":"2024-09-20T19:33:08.437113+0000","last_fullsized":"2024-09-20T19:33:08.437113+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.904221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.904221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:15.160671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":975,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420999+0000","last_change":"2024-09-20T19:30:59.410472+0000","last_active":"2024-09-20T19:33:06.420999+0000","last_peered":"2024-09-20T19:33:06.420999+0000","last_clean":"2024-09-20T19:33:06.420999+0000","last_became_active":"2024-09-20T19:30:59.410103+0000","last_became_peered":"2024-09-20T19:30:59.410103+0000","last_unstale":"2024-09-20T19:33:06.420999+0000","last_undegraded":"2024-09-20T19:33:06.420999+0000","last_fullsized":"2024-09-20T19:33:06.420999+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:25:21.382463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:25:21.382463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"677'789","reported_seq":3529,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.422806+0000","last_change":"2024-09-20T19:33:05.486409+0000","last_active":"2024-09-20T19:33:06.422806+0000","last_peered":"2024-09-20T19:33:06.422806+0000","last_clean":"2024-09-20T19:33:06.422806+0000","last_became_active":"2024-09-20T19:30:59.413518+0000","last_became_peered":"2024-09-20T19:30:59.413518+0000","last_unstale":"2024-09-20T19:33:06.422806+0000","last_undegraded":"2024-09-20T19:33:06.422806+0000","last_fullsized":"2024-09-20T19:33:06.422806+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'685","last_scrub_stamp":"2024-09-20T19:32:09.614901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:09.614901+0000","objects_scrubbed":9,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:23.109348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.04347815,"stat_sum":{"num_bytes":23110401,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":37,"num_read_kb":33910,"num_write":23,"num_write_kb":17546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.3","version":"523'29","reported_seq":1090,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438685+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:33:07.438685+0000","last_peered":"2024-09-20T19:33:07.438685+0000","last_clean":"2024-09-20T19:33:07.438685+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:33:07.438685+0000","last_undegraded":"2024-09-20T19:33:07.438685+0000","last_fullsized":"2024-09-20T19:33:07.438685+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.517393+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1228,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421469+0000","last_change":"2024-09-20T19:30:31.348665+0000","last_active":"2024-09-20T19:33:06.421469+0000","last_peered":"2024-09-20T19:33:06.421469+0000","last_clean":"2024-09-20T19:33:06.421469+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:33:06.421469+0000","last_undegraded":"2024-09-20T19:33:06.421469+0000","last_fullsized":"2024-09-20T19:33:06.421469+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:26:22.169579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:26:22.169579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.962129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1273,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.757095+0000","last_change":"2024-09-20T19:33:06.757095+0000","last_active":"2024-09-20T19:33:06.757095+0000","last_peered":"2024-09-20T19:33:06.757095+0000","last_clean":"2024-09-20T19:33:06.757095+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:33:06.757095+0000","last_undegraded":"2024-09-20T19:33:06.757095+0000","last_fullsized":"2024-09-20T19:33:06.757095+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.756963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.756963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:13.553227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1261,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.414866+0000","last_change":"2024-09-20T19:33:00.335442+0000","last_active":"2024-09-20T19:33:06.414866+0000","last_peered":"2024-09-20T19:33:06.414866+0000","last_clean":"2024-09-20T19:33:06.414866+0000","last_became_active":"2024-09-20T19:32:57.335047+0000","last_became_peered":"2024-09-20T19:32:57.335047+0000","last_unstale":"2024-09-20T19:33:06.414866+0000","last_undegraded":"2024-09-20T19:33:06.414866+0000","last_fullsized":"2024-09-20T19:33:06.414866+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:00.335289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:00.335289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:21.283913+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":976,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420774+0000","last_change":"2024-09-20T19:30:59.409489+0000","last_active":"2024-09-20T19:33:06.420774+0000","last_peered":"2024-09-20T19:33:06.420774+0000","last_clean":"2024-09-20T19:33:06.420774+0000","last_became_active":"2024-09-20T19:30:59.409151+0000","last_became_peered":"2024-09-20T19:30:59.409151+0000","last_unstale":"2024-09-20T19:33:06.420774+0000","last_undegraded":"2024-09-20T19:33:06.420774+0000","last_fullsized":"2024-09-20T19:33:06.420774+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"678'584","reported_seq":3232,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.437423+0000","last_change":"2024-09-20T19:33:06.426030+0000","last_active":"2024-09-20T19:33:08.437423+0000","last_peered":"2024-09-20T19:33:08.437423+0000","last_clean":"2024-09-20T19:33:08.437423+0000","last_became_active":"2024-09-20T19:31:48.294687+0000","last_became_peered":"2024-09-20T19:31:48.294687+0000","last_unstale":"2024-09-20T19:33:08.437423+0000","last_undegraded":"2024-09-20T19:33:08.437423+0000","last_fullsized":"2024-09-20T19:33:08.437423+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'560","last_scrub_stamp":"2024-09-20T19:32:56.921141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:56.921141+0000","objects_scrubbed":5,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:17.407994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0095608210000000006,"stat_sum":{"num_bytes":6965992,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":46,"num_read_kb":53632,"num_write":31,"num_write_kb":6492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6670312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"4"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.2","version":"523'20","reported_seq":1322,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422918+0000","last_change":"2024-09-20T19:31:19.351200+0000","last_active":"2024-09-20T19:33:06.422918+0000","last_peered":"2024-09-20T19:33:06.422918+0000","last_clean":"2024-09-20T19:33:06.422918+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:33:06.422918+0000","last_undegraded":"2024-09-20T19:33:06.422918+0000","last_fullsized":"2024-09-20T19:33:06.422918+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:19.351134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.351134+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:36.023354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1277,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424220+0000","last_change":"2024-09-20T19:33:04.366510+0000","last_active":"2024-09-20T19:33:06.424220+0000","last_peered":"2024-09-20T19:33:06.424220+0000","last_clean":"2024-09-20T19:33:06.424220+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:33:06.424220+0000","last_undegraded":"2024-09-20T19:33:06.424220+0000","last_fullsized":"2024-09-20T19:33:06.424220+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:04.366451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.366451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:14.153349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1187,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438995+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:33:07.438995+0000","last_peered":"2024-09-20T19:33:07.438995+0000","last_clean":"2024-09-20T19:33:07.438995+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:33:07.438995+0000","last_undegraded":"2024-09-20T19:33:07.438995+0000","last_fullsized":"2024-09-20T19:33:07.438995+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:32.990270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1061,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.429014+0000","last_change":"2024-09-20T19:32:57.337690+0000","last_active":"2024-09-20T19:33:07.429014+0000","last_peered":"2024-09-20T19:33:07.429014+0000","last_clean":"2024-09-20T19:33:07.429014+0000","last_became_active":"2024-09-20T19:32:57.337416+0000","last_became_peered":"2024-09-20T19:32:57.337416+0000","last_unstale":"2024-09-20T19:33:07.429014+0000","last_undegraded":"2024-09-20T19:33:07.429014+0000","last_fullsized":"2024-09-20T19:33:07.429014+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.455584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.455584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":873,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422671+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:33:06.422671+0000","last_peered":"2024-09-20T19:33:06.422671+0000","last_clean":"2024-09-20T19:33:06.422671+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:33:06.422671+0000","last_undegraded":"2024-09-20T19:33:06.422671+0000","last_fullsized":"2024-09-20T19:33:06.422671+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.285306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"678'433","reported_seq":3178,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.536549+0000","last_change":"2024-09-20T19:33:06.536549+0000","last_active":"2024-09-20T19:33:06.536549+0000","last_peered":"2024-09-20T19:33:06.536549+0000","last_clean":"2024-09-20T19:33:06.536549+0000","last_became_active":"2024-09-20T19:30:59.409330+0000","last_became_peered":"2024-09-20T19:30:59.409330+0000","last_unstale":"2024-09-20T19:33:06.536549+0000","last_undegraded":"2024-09-20T19:33:06.536549+0000","last_fullsized":"2024-09-20T19:33:06.536549+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'400","last_scrub_stamp":"2024-09-20T19:31:36.808095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.808095+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.10014526,"stat_sum":{"num_bytes":7647680,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":20882,"num_write":23,"num_write_kb":7969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"}]},{"pgid":"4.4","version":"554'44","reported_seq":1549,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424116+0000","last_change":"2024-09-20T19:31:31.390088+0000","last_active":"2024-09-20T19:33:06.424116+0000","last_peered":"2024-09-20T19:33:06.424116+0000","last_clean":"2024-09-20T19:33:06.424116+0000","last_became_active":"2024-09-20T19:30:59.436599+0000","last_became_peered":"2024-09-20T19:30:59.436599+0000","last_unstale":"2024-09-20T19:33:06.424116+0000","last_undegraded":"2024-09-20T19:33:06.424116+0000","last_fullsized":"2024-09-20T19:33:06.424116+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'44","last_scrub_stamp":"2024-09-20T19:31:31.390009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:31.390009+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1278,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424857+0000","last_change":"2024-09-20T19:30:02.530981+0000","last_active":"2024-09-20T19:33:06.424857+0000","last_peered":"2024-09-20T19:33:06.424857+0000","last_clean":"2024-09-20T19:33:06.424857+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:33:06.424857+0000","last_undegraded":"2024-09-20T19:33:06.424857+0000","last_fullsized":"2024-09-20T19:33:06.424857+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:30:02.530860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:30:02.530860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:55.387954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1065,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.434355+0000","last_change":"2024-09-20T19:32:17.478298+0000","last_active":"2024-09-20T19:33:07.434355+0000","last_peered":"2024-09-20T19:33:07.434355+0000","last_clean":"2024-09-20T19:33:07.434355+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:33:07.434355+0000","last_undegraded":"2024-09-20T19:33:07.434355+0000","last_fullsized":"2024-09-20T19:33:07.434355+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:17.478143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:17.478143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.002346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1064,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441284+0000","last_change":"2024-09-20T19:32:14.826848+0000","last_active":"2024-09-20T19:33:07.441284+0000","last_peered":"2024-09-20T19:33:07.441284+0000","last_clean":"2024-09-20T19:33:07.441284+0000","last_became_active":"2024-09-20T19:30:59.424738+0000","last_became_peered":"2024-09-20T19:30:59.424738+0000","last_unstale":"2024-09-20T19:33:07.441284+0000","last_undegraded":"2024-09-20T19:33:07.441284+0000","last_fullsized":"2024-09-20T19:33:07.441284+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.259591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1343,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421512+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:33:06.421512+0000","last_peered":"2024-09-20T19:33:06.421512+0000","last_clean":"2024-09-20T19:33:06.421512+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:33:06.421512+0000","last_undegraded":"2024-09-20T19:33:06.421512+0000","last_fullsized":"2024-09-20T19:33:06.421512+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.269206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"675'858","reported_seq":3242,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.435744+0000","last_change":"2024-09-20T19:33:06.481008+0000","last_active":"2024-09-20T19:33:07.435744+0000","last_peered":"2024-09-20T19:33:07.435744+0000","last_clean":"2024-09-20T19:33:07.435744+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:33:07.435744+0000","last_undegraded":"2024-09-20T19:33:07.435744+0000","last_fullsized":"2024-09-20T19:33:07.435744+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'779","last_scrub_stamp":"2024-09-20T19:31:22.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.595437+0000","objects_scrubbed":4,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.059713135000000001,"stat_sum":{"num_bytes":12080657,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":53296,"num_write":22,"num_write_kb":10147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.7","version":"523'17","reported_seq":1247,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.426918+0000","last_change":"2024-09-20T19:33:04.118188+0000","last_active":"2024-09-20T19:33:06.426918+0000","last_peered":"2024-09-20T19:33:06.426918+0000","last_clean":"2024-09-20T19:33:06.426918+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:33:06.426918+0000","last_undegraded":"2024-09-20T19:33:06.426918+0000","last_fullsized":"2024-09-20T19:33:06.426918+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:33:04.117998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.117998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:15.863884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2156,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.413385+0000","last_change":"2024-09-20T19:28:48.600562+0000","last_active":"2024-09-20T19:33:06.413385+0000","last_peered":"2024-09-20T19:33:06.413385+0000","last_clean":"2024-09-20T19:33:06.413385+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:33:06.413385+0000","last_undegraded":"2024-09-20T19:33:06.413385+0000","last_fullsized":"2024-09-20T19:33:06.413385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:28:48.600384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.600384+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1283,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.415796+0000","last_change":"2024-09-20T19:31:23.521703+0000","last_active":"2024-09-20T19:33:06.415796+0000","last_peered":"2024-09-20T19:33:06.415796+0000","last_clean":"2024-09-20T19:33:06.415796+0000","last_became_active":"2024-09-20T19:30:59.416753+0000","last_became_peered":"2024-09-20T19:30:59.416753+0000","last_unstale":"2024-09-20T19:33:06.415796+0000","last_undegraded":"2024-09-20T19:33:06.415796+0000","last_fullsized":"2024-09-20T19:33:06.415796+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:23.521555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:23.521555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424512+0000","last_change":"2024-09-20T19:33:04.320247+0000","last_active":"2024-09-20T19:33:06.424512+0000","last_peered":"2024-09-20T19:33:06.424512+0000","last_clean":"2024-09-20T19:33:06.424512+0000","last_became_active":"2024-09-20T19:32:57.343760+0000","last_became_peered":"2024-09-20T19:32:57.343760+0000","last_unstale":"2024-09-20T19:33:06.424512+0000","last_undegraded":"2024-09-20T19:33:06.424512+0000","last_fullsized":"2024-09-20T19:33:06.424512+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:04.320119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.320119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:05.309537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1269,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424092+0000","last_change":"2024-09-20T19:32:21.799367+0000","last_active":"2024-09-20T19:33:06.424092+0000","last_peered":"2024-09-20T19:33:06.424092+0000","last_clean":"2024-09-20T19:33:06.424092+0000","last_became_active":"2024-09-20T19:30:59.437740+0000","last_became_peered":"2024-09-20T19:30:59.437740+0000","last_unstale":"2024-09-20T19:33:06.424092+0000","last_undegraded":"2024-09-20T19:33:06.424092+0000","last_fullsized":"2024-09-20T19:33:06.424092+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.069051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":875,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423446+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:33:06.423446+0000","last_peered":"2024-09-20T19:33:06.423446+0000","last_clean":"2024-09-20T19:33:06.423446+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:33:06.423446+0000","last_undegraded":"2024-09-20T19:33:06.423446+0000","last_fullsized":"2024-09-20T19:33:06.423446+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"677'876","reported_seq":3656,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.424077+0000","last_change":"2024-09-20T19:33:06.365910+0000","last_active":"2024-09-20T19:33:06.424077+0000","last_peered":"2024-09-20T19:33:06.424077+0000","last_clean":"2024-09-20T19:33:06.424077+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:33:06.424077+0000","last_undegraded":"2024-09-20T19:33:06.424077+0000","last_fullsized":"2024-09-20T19:33:06.424077+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:08.077024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.94745183600000005,"stat_sum":{"num_bytes":10598297,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":39,"num_read_kb":27160,"num_write":17,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"3"},{"start":"ff","length":"1"},{"start":"102","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":1084,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438565+0000","last_change":"2024-09-20T19:32:57.497035+0000","last_active":"2024-09-20T19:33:07.438565+0000","last_peered":"2024-09-20T19:33:07.438565+0000","last_clean":"2024-09-20T19:33:07.438565+0000","last_became_active":"2024-09-20T19:32:57.348829+0000","last_became_peered":"2024-09-20T19:32:57.348829+0000","last_unstale":"2024-09-20T19:33:07.438565+0000","last_undegraded":"2024-09-20T19:33:07.438565+0000","last_fullsized":"2024-09-20T19:33:07.438565+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:32:57.496961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.496961+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:19.904544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1000,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.435994+0000","last_change":"2024-09-20T19:30:59.438831+0000","last_active":"2024-09-20T19:33:07.435994+0000","last_peered":"2024-09-20T19:33:07.435994+0000","last_clean":"2024-09-20T19:33:07.435994+0000","last_became_active":"2024-09-20T19:30:59.438639+0000","last_became_peered":"2024-09-20T19:30:59.438639+0000","last_unstale":"2024-09-20T19:33:07.435994+0000","last_undegraded":"2024-09-20T19:33:07.435994+0000","last_fullsized":"2024-09-20T19:33:07.435994+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:24:39.000765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:24:39.000765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.493054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1281,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421291+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:33:06.421291+0000","last_peered":"2024-09-20T19:33:06.421291+0000","last_clean":"2024-09-20T19:33:06.421291+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:33:06.421291+0000","last_undegraded":"2024-09-20T19:33:06.421291+0000","last_fullsized":"2024-09-20T19:33:06.421291+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:41.273146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1272,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425127+0000","last_change":"2024-09-20T19:32:23.505615+0000","last_active":"2024-09-20T19:33:06.425127+0000","last_peered":"2024-09-20T19:33:06.425127+0000","last_clean":"2024-09-20T19:33:06.425127+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:33:06.425127+0000","last_undegraded":"2024-09-20T19:33:06.425127+0000","last_fullsized":"2024-09-20T19:33:06.425127+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.505475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.505475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:49.486718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1281,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423534+0000","last_change":"2024-09-20T19:30:59.436100+0000","last_active":"2024-09-20T19:33:06.423534+0000","last_peered":"2024-09-20T19:33:06.423534+0000","last_clean":"2024-09-20T19:33:06.423534+0000","last_became_active":"2024-09-20T19:30:59.435560+0000","last_became_peered":"2024-09-20T19:30:59.435560+0000","last_unstale":"2024-09-20T19:33:06.423534+0000","last_undegraded":"2024-09-20T19:33:06.423534+0000","last_fullsized":"2024-09-20T19:33:06.423534+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:14.942057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:14.942057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":2657,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.439102+0000","last_change":"2024-09-20T19:33:04.475897+0000","last_active":"2024-09-20T19:33:07.439102+0000","last_peered":"2024-09-20T19:33:07.439102+0000","last_clean":"2024-09-20T19:33:07.439102+0000","last_became_active":"2024-09-20T19:30:59.449561+0000","last_became_peered":"2024-09-20T19:30:59.449561+0000","last_unstale":"2024-09-20T19:33:07.439102+0000","last_undegraded":"2024-09-20T19:33:07.439102+0000","last_fullsized":"2024-09-20T19:33:07.439102+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:31:29.529433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.529433+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1765713309999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.5","version":"523'30","reported_seq":1490,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420709+0000","last_change":"2024-09-20T19:30:31.397491+0000","last_active":"2024-09-20T19:33:06.420709+0000","last_peered":"2024-09-20T19:33:06.420709+0000","last_clean":"2024-09-20T19:33:06.420709+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:33:06.420709+0000","last_undegraded":"2024-09-20T19:33:06.420709+0000","last_fullsized":"2024-09-20T19:33:06.420709+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'16","last_scrub_stamp":"2024-09-20T19:24:43.584143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:24:43.584143+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:53.784578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":1095,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442398+0000","last_change":"2024-09-20T19:32:57.875922+0000","last_active":"2024-09-20T19:33:07.442398+0000","last_peered":"2024-09-20T19:33:07.442398+0000","last_clean":"2024-09-20T19:33:07.442398+0000","last_became_active":"2024-09-20T19:30:59.411724+0000","last_became_peered":"2024-09-20T19:30:59.411724+0000","last_unstale":"2024-09-20T19:33:07.442398+0000","last_undegraded":"2024-09-20T19:33:07.442398+0000","last_fullsized":"2024-09-20T19:33:07.442398+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:32:57.875788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.875788+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:20.926597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1262,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425288+0000","last_change":"2024-09-20T19:28:48.425075+0000","last_active":"2024-09-20T19:33:06.425288+0000","last_peered":"2024-09-20T19:33:06.425288+0000","last_clean":"2024-09-20T19:33:06.425288+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:33:06.425288+0000","last_undegraded":"2024-09-20T19:33:06.425288+0000","last_fullsized":"2024-09-20T19:33:06.425288+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:48.424987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:48.424987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.399134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1322,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421509+0000","last_change":"2024-09-20T19:30:09.775314+0000","last_active":"2024-09-20T19:33:06.421509+0000","last_peered":"2024-09-20T19:33:06.421509+0000","last_clean":"2024-09-20T19:33:06.421509+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:33:06.421509+0000","last_undegraded":"2024-09-20T19:33:06.421509+0000","last_fullsized":"2024-09-20T19:33:06.421509+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:26:13.424049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.424049+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:39.829392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1254,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423708+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:33:06.423708+0000","last_peered":"2024-09-20T19:33:06.423708+0000","last_clean":"2024-09-20T19:33:06.423708+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:33:06.423708+0000","last_undegraded":"2024-09-20T19:33:06.423708+0000","last_fullsized":"2024-09-20T19:33:06.423708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:50.790580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1267,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425293+0000","last_change":"2024-09-20T19:32:10.638734+0000","last_active":"2024-09-20T19:33:06.425293+0000","last_peered":"2024-09-20T19:33:06.425293+0000","last_clean":"2024-09-20T19:33:06.425293+0000","last_became_active":"2024-09-20T19:30:59.410315+0000","last_became_peered":"2024-09-20T19:30:59.410315+0000","last_unstale":"2024-09-20T19:33:06.425293+0000","last_undegraded":"2024-09-20T19:33:06.425293+0000","last_fullsized":"2024-09-20T19:33:06.425293+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:10.638671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.638671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.768982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"675'341","reported_seq":2774,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.430253+0000","last_change":"2024-09-20T19:33:05.382750+0000","last_active":"2024-09-20T19:33:07.430253+0000","last_peered":"2024-09-20T19:33:07.430253+0000","last_clean":"2024-09-20T19:33:07.430253+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:33:07.430253+0000","last_undegraded":"2024-09-20T19:33:07.430253+0000","last_fullsized":"2024-09-20T19:33:07.430253+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'300","last_scrub_stamp":"2024-09-20T19:31:19.575078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.575078+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.90009156199999996,"stat_sum":{"num_bytes":3533830,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":20,"num_write_kb":5041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.a","version":"523'35","reported_seq":1312,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.419968+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:33:06.419968+0000","last_peered":"2024-09-20T19:33:06.419968+0000","last_clean":"2024-09-20T19:33:06.419968+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:33:06.419968+0000","last_undegraded":"2024-09-20T19:33:06.419968+0000","last_fullsized":"2024-09-20T19:33:06.419968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1283,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.426842+0000","last_change":"2024-09-20T19:31:40.539723+0000","last_active":"2024-09-20T19:33:06.426842+0000","last_peered":"2024-09-20T19:33:06.426842+0000","last_clean":"2024-09-20T19:33:06.426842+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:33:06.426842+0000","last_undegraded":"2024-09-20T19:33:06.426842+0000","last_fullsized":"2024-09-20T19:33:06.426842+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:40.539596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.539596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":935,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438242+0000","last_change":"2024-09-20T19:32:57.330746+0000","last_active":"2024-09-20T19:33:07.438242+0000","last_peered":"2024-09-20T19:33:07.438242+0000","last_clean":"2024-09-20T19:33:07.438242+0000","last_became_active":"2024-09-20T19:32:57.330228+0000","last_became_peered":"2024-09-20T19:32:57.330228+0000","last_unstale":"2024-09-20T19:33:07.438242+0000","last_undegraded":"2024-09-20T19:33:07.438242+0000","last_fullsized":"2024-09-20T19:33:07.438242+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1252,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421397+0000","last_change":"2024-09-20T19:32:15.777865+0000","last_active":"2024-09-20T19:33:06.421397+0000","last_peered":"2024-09-20T19:33:06.421397+0000","last_clean":"2024-09-20T19:33:06.421397+0000","last_became_active":"2024-09-20T19:30:59.440497+0000","last_became_peered":"2024-09-20T19:30:59.440497+0000","last_unstale":"2024-09-20T19:33:06.421397+0000","last_undegraded":"2024-09-20T19:33:06.421397+0000","last_fullsized":"2024-09-20T19:33:06.421397+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:21.419161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"678'1198","reported_seq":3895,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.440452+0000","last_change":"2024-09-20T19:33:06.441338+0000","last_active":"2024-09-20T19:33:06.440452+0000","last_peered":"2024-09-20T19:33:06.440452+0000","last_clean":"2024-09-20T19:33:06.440452+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:33:06.440452+0000","last_undegraded":"2024-09-20T19:33:06.440452+0000","last_fullsized":"2024-09-20T19:33:06.440452+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'1120","last_scrub_stamp":"2024-09-20T19:32:08.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:08.554870+0000","objects_scrubbed":7,"log_size":1198,"log_dups_size":0,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:26.286827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.023168998999999999,"stat_sum":{"num_bytes":16245639,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":35518,"num_write":67,"num_write_kb":28251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"4"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.b","version":"523'15","reported_seq":1063,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438874+0000","last_change":"2024-09-20T19:32:15.464944+0000","last_active":"2024-09-20T19:33:07.438874+0000","last_peered":"2024-09-20T19:33:07.438874+0000","last_clean":"2024-09-20T19:33:07.438874+0000","last_became_active":"2024-09-20T19:30:31.349098+0000","last_became_peered":"2024-09-20T19:30:31.349098+0000","last_unstale":"2024-09-20T19:33:07.438874+0000","last_undegraded":"2024-09-20T19:33:07.438874+0000","last_fullsized":"2024-09-20T19:33:07.438874+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.464807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.464807+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.851627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1291,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425239+0000","last_change":"2024-09-20T19:31:29.374819+0000","last_active":"2024-09-20T19:33:06.425239+0000","last_peered":"2024-09-20T19:33:06.425239+0000","last_clean":"2024-09-20T19:33:06.425239+0000","last_became_active":"2024-09-20T19:30:59.434929+0000","last_became_peered":"2024-09-20T19:30:59.434929+0000","last_unstale":"2024-09-20T19:33:06.425239+0000","last_undegraded":"2024-09-20T19:33:06.425239+0000","last_fullsized":"2024-09-20T19:33:06.425239+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1282,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.415207+0000","last_change":"2024-09-20T19:31:36.575855+0000","last_active":"2024-09-20T19:33:06.415207+0000","last_peered":"2024-09-20T19:33:06.415207+0000","last_clean":"2024-09-20T19:33:06.415207+0000","last_became_active":"2024-09-20T19:30:59.407432+0000","last_became_peered":"2024-09-20T19:30:59.407432+0000","last_unstale":"2024-09-20T19:33:06.415207+0000","last_undegraded":"2024-09-20T19:33:06.415207+0000","last_fullsized":"2024-09-20T19:33:06.415207+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.575803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.575803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1059,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.436104+0000","last_change":"2024-09-20T19:32:57.354330+0000","last_active":"2024-09-20T19:33:07.436104+0000","last_peered":"2024-09-20T19:33:07.436104+0000","last_clean":"2024-09-20T19:33:07.436104+0000","last_became_active":"2024-09-20T19:32:57.342426+0000","last_became_peered":"2024-09-20T19:32:57.342426+0000","last_unstale":"2024-09-20T19:33:07.436104+0000","last_undegraded":"2024-09-20T19:33:07.436104+0000","last_fullsized":"2024-09-20T19:33:07.436104+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:32:10.330783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.330783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:26.679949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"678'463","reported_seq":3166,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.437343+0000","last_change":"2024-09-20T19:33:06.424269+0000","last_active":"2024-09-20T19:33:06.437343+0000","last_peered":"2024-09-20T19:33:06.437343+0000","last_clean":"2024-09-20T19:33:06.437343+0000","last_became_active":"2024-09-20T19:21:30.255126+0000","last_became_peered":"2024-09-20T19:21:30.255126+0000","last_unstale":"2024-09-20T19:33:06.437343+0000","last_undegraded":"2024-09-20T19:33:06.437343+0000","last_fullsized":"2024-09-20T19:33:06.437343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'405","last_scrub_stamp":"2024-09-20T19:32:29.036109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.036109+0000","objects_scrubbed":7,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.364414+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.94351996599999999,"stat_sum":{"num_bytes":9503861,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":186,"num_read_kb":141002,"num_write":111,"num_write_kb":51402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"3"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.8","version":"523'27","reported_seq":1308,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422930+0000","last_change":"2024-09-20T19:29:49.731520+0000","last_active":"2024-09-20T19:33:06.422930+0000","last_peered":"2024-09-20T19:33:06.422930+0000","last_clean":"2024-09-20T19:33:06.422930+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:33:06.422930+0000","last_undegraded":"2024-09-20T19:33:06.422930+0000","last_fullsized":"2024-09-20T19:33:06.422930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'17","last_scrub_stamp":"2024-09-20T19:29:49.731377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:29:49.731377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":47,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.443835+0000","last_change":"2024-09-20T19:32:57.353798+0000","last_active":"2024-09-20T19:33:07.443835+0000","last_peered":"2024-09-20T19:33:07.443835+0000","last_clean":"2024-09-20T19:33:07.443835+0000","last_became_active":"2024-09-20T19:32:57.347922+0000","last_became_peered":"2024-09-20T19:32:57.347922+0000","last_unstale":"2024-09-20T19:33:07.443835+0000","last_undegraded":"2024-09-20T19:33:07.443835+0000","last_fullsized":"2024-09-20T19:33:07.443835+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:35.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.568627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1465,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421075+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:33:06.421075+0000","last_peered":"2024-09-20T19:33:06.421075+0000","last_clean":"2024-09-20T19:33:06.421075+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:33:06.421075+0000","last_undegraded":"2024-09-20T19:33:06.421075+0000","last_fullsized":"2024-09-20T19:33:06.421075+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.376352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1260,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423006+0000","last_change":"2024-09-20T19:32:12.619664+0000","last_active":"2024-09-20T19:33:06.423006+0000","last_peered":"2024-09-20T19:33:06.423006+0000","last_clean":"2024-09-20T19:33:06.423006+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:33:06.423006+0000","last_undegraded":"2024-09-20T19:33:06.423006+0000","last_fullsized":"2024-09-20T19:33:06.423006+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.619550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.619550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:19.382228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"676'441","reported_seq":2867,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.442538+0000","last_change":"2024-09-20T19:33:06.476988+0000","last_active":"2024-09-20T19:33:07.442538+0000","last_peered":"2024-09-20T19:33:07.442538+0000","last_clean":"2024-09-20T19:33:07.442538+0000","last_became_active":"2024-09-20T19:30:59.413000+0000","last_became_peered":"2024-09-20T19:30:59.413000+0000","last_unstale":"2024-09-20T19:33:07.442538+0000","last_undegraded":"2024-09-20T19:33:07.442538+0000","last_fullsized":"2024-09-20T19:33:07.442538+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'344","last_scrub_stamp":"2024-09-20T19:27:23.499201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:27:23.499201+0000","objects_scrubbed":6,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.042294865000000001,"stat_sum":{"num_bytes":10790424,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":191,"num_read_kb":157196,"num_write":134,"num_write_kb":56239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4084895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.9","version":"523'18","reported_seq":1137,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.437375+0000","last_change":"2024-09-20T19:32:26.295823+0000","last_active":"2024-09-20T19:33:08.437375+0000","last_peered":"2024-09-20T19:33:08.437375+0000","last_clean":"2024-09-20T19:33:08.437375+0000","last_became_active":"2024-09-20T19:30:59.404224+0000","last_became_peered":"2024-09-20T19:30:59.404224+0000","last_unstale":"2024-09-20T19:33:08.437375+0000","last_undegraded":"2024-09-20T19:33:08.437375+0000","last_fullsized":"2024-09-20T19:33:08.437375+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:52.872180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1136,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439586+0000","last_change":"2024-09-20T19:32:19.382969+0000","last_active":"2024-09-20T19:33:08.439586+0000","last_peered":"2024-09-20T19:33:08.439586+0000","last_clean":"2024-09-20T19:33:08.439586+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:33:08.439586+0000","last_undegraded":"2024-09-20T19:33:08.439586+0000","last_fullsized":"2024-09-20T19:33:08.439586+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:32:19.382850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:19.382850+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:24.971441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1283,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.419767+0000","last_change":"2024-09-20T19:33:06.111877+0000","last_active":"2024-09-20T19:33:06.419767+0000","last_peered":"2024-09-20T19:33:06.419767+0000","last_clean":"2024-09-20T19:33:06.419767+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:33:06.419767+0000","last_undegraded":"2024-09-20T19:33:06.419767+0000","last_fullsized":"2024-09-20T19:33:06.419767+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.111753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.111753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:07.777802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1101,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.439779+0000","last_change":"2024-09-20T19:32:11.827975+0000","last_active":"2024-09-20T19:33:07.439779+0000","last_peered":"2024-09-20T19:33:07.439779+0000","last_clean":"2024-09-20T19:33:07.439779+0000","last_became_active":"2024-09-20T19:30:59.425091+0000","last_became_peered":"2024-09-20T19:30:59.425091+0000","last_unstale":"2024-09-20T19:33:07.439779+0000","last_undegraded":"2024-09-20T19:33:07.439779+0000","last_fullsized":"2024-09-20T19:33:07.439779+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:30.305742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"676'962","reported_seq":3876,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:06.424499+0000","last_change":"2024-09-20T19:33:05.384045+0000","last_active":"2024-09-20T19:33:06.424499+0000","last_peered":"2024-09-20T19:33:06.424499+0000","last_clean":"2024-09-20T19:33:06.424499+0000","last_became_active":"2024-09-20T19:33:00.327443+0000","last_became_peered":"2024-09-20T19:33:00.327443+0000","last_unstale":"2024-09-20T19:33:06.424499+0000","last_undegraded":"2024-09-20T19:33:06.424499+0000","last_fullsized":"2024-09-20T19:33:06.424499+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":5,"last_scrub":"483'743","last_scrub_stamp":"2024-09-20T19:29:15.367390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:15.367390+0000","objects_scrubbed":10,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.86386823700000004,"stat_sum":{"num_bytes":30084293,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":70,"num_read_kb":43689,"num_write":68,"num_write_kb":19792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":17400655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"f6","length":"2"}]},{"pgid":"4.16","version":"523'15","reported_seq":1065,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438253+0000","last_change":"2024-09-20T19:32:16.493984+0000","last_active":"2024-09-20T19:33:07.438253+0000","last_peered":"2024-09-20T19:33:07.438253+0000","last_clean":"2024-09-20T19:33:07.438253+0000","last_became_active":"2024-09-20T19:30:59.447014+0000","last_became_peered":"2024-09-20T19:30:59.447014+0000","last_unstale":"2024-09-20T19:33:07.438253+0000","last_undegraded":"2024-09-20T19:33:07.438253+0000","last_fullsized":"2024-09-20T19:33:07.438253+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:18.681371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1299,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420025+0000","last_change":"2024-09-20T19:31:41.498323+0000","last_active":"2024-09-20T19:33:06.420025+0000","last_peered":"2024-09-20T19:33:06.420025+0000","last_clean":"2024-09-20T19:33:06.420025+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:33:06.420025+0000","last_undegraded":"2024-09-20T19:33:06.420025+0000","last_fullsized":"2024-09-20T19:33:06.420025+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:41.498237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.498237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1079,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442246+0000","last_change":"2024-09-20T19:30:59.429090+0000","last_active":"2024-09-20T19:33:07.442246+0000","last_peered":"2024-09-20T19:33:07.442246+0000","last_clean":"2024-09-20T19:33:07.442246+0000","last_became_active":"2024-09-20T19:30:59.426634+0000","last_became_peered":"2024-09-20T19:30:59.426634+0000","last_unstale":"2024-09-20T19:33:07.442246+0000","last_undegraded":"2024-09-20T19:33:07.442246+0000","last_fullsized":"2024-09-20T19:33:07.442246+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:01.260486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:01.260486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1271,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.426337+0000","last_change":"2024-09-20T19:33:02.129851+0000","last_active":"2024-09-20T19:33:06.426337+0000","last_peered":"2024-09-20T19:33:06.426337+0000","last_clean":"2024-09-20T19:33:06.426337+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:33:06.426337+0000","last_undegraded":"2024-09-20T19:33:06.426337+0000","last_fullsized":"2024-09-20T19:33:06.426337+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.129712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.129712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:22.807234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"678'987","reported_seq":3602,"reported_epoch":679,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:07.446234+0000","last_change":"2024-09-20T19:33:06.477983+0000","last_active":"2024-09-20T19:33:07.446234+0000","last_peered":"2024-09-20T19:33:07.446234+0000","last_clean":"2024-09-20T19:33:07.446234+0000","last_became_active":"2024-09-20T19:31:02.399794+0000","last_became_peered":"2024-09-20T19:31:02.399794+0000","last_unstale":"2024-09-20T19:33:07.446234+0000","last_undegraded":"2024-09-20T19:33:07.446234+0000","last_fullsized":"2024-09-20T19:33:07.446234+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'769","last_scrub_stamp":"2024-09-20T19:28:51.165603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:51.165603+0000","objects_scrubbed":9,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.070342647999999994,"stat_sum":{"num_bytes":13597589,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":79,"num_read_kb":47687,"num_write":53,"num_write_kb":19830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":13361098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"}]},{"pgid":"2.11","version":"0'0","reported_seq":1272,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421880+0000","last_change":"2024-09-20T19:30:31.386421+0000","last_active":"2024-09-20T19:33:06.421880+0000","last_peered":"2024-09-20T19:33:06.421880+0000","last_clean":"2024-09-20T19:33:06.421880+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:33:06.421880+0000","last_undegraded":"2024-09-20T19:33:06.421880+0000","last_fullsized":"2024-09-20T19:33:06.421880+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:38.575807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:38.575807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:55.820149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1118,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441531+0000","last_change":"2024-09-20T19:31:41.454838+0000","last_active":"2024-09-20T19:33:07.441531+0000","last_peered":"2024-09-20T19:33:07.441531+0000","last_clean":"2024-09-20T19:33:07.441531+0000","last_became_active":"2024-09-20T19:30:59.429434+0000","last_became_peered":"2024-09-20T19:30:59.429434+0000","last_unstale":"2024-09-20T19:33:07.441531+0000","last_undegraded":"2024-09-20T19:33:07.441531+0000","last_fullsized":"2024-09-20T19:33:07.441531+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1272,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424699+0000","last_change":"2024-09-20T19:30:59.412406+0000","last_active":"2024-09-20T19:33:06.424699+0000","last_peered":"2024-09-20T19:33:06.424699+0000","last_clean":"2024-09-20T19:33:06.424699+0000","last_became_active":"2024-09-20T19:30:59.411374+0000","last_became_peered":"2024-09-20T19:30:59.411374+0000","last_unstale":"2024-09-20T19:33:06.424699+0000","last_undegraded":"2024-09-20T19:33:06.424699+0000","last_fullsized":"2024-09-20T19:33:06.424699+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:59.345399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:27:59.345399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.306547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1252,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424017+0000","last_change":"2024-09-20T19:30:31.376526+0000","last_active":"2024-09-20T19:33:06.424017+0000","last_peered":"2024-09-20T19:33:06.424017+0000","last_clean":"2024-09-20T19:33:06.424017+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:33:06.424017+0000","last_undegraded":"2024-09-20T19:33:06.424017+0000","last_fullsized":"2024-09-20T19:33:06.424017+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:27.296694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:25:27.296694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.879308+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"677'563","reported_seq":3200,"reported_epoch":678,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:06.424233+0000","last_change":"2024-09-20T19:33:06.365929+0000","last_active":"2024-09-20T19:33:06.424233+0000","last_peered":"2024-09-20T19:33:06.424233+0000","last_clean":"2024-09-20T19:33:06.424233+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:33:06.424233+0000","last_undegraded":"2024-09-20T19:33:06.424233+0000","last_fullsized":"2024-09-20T19:33:06.424233+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:35.793500+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.94780993300000005,"stat_sum":{"num_bytes":2285917,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":4546,"num_write":8,"num_write_kb":948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"3"},{"start":"ff","length":"1"},{"start":"102","length":"1"}]},{"pgid":"4.14","version":"523'20","reported_seq":1128,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441329+0000","last_change":"2024-09-20T19:31:40.419678+0000","last_active":"2024-09-20T19:33:07.441329+0000","last_peered":"2024-09-20T19:33:07.441329+0000","last_clean":"2024-09-20T19:33:07.441329+0000","last_became_active":"2024-09-20T19:30:59.428000+0000","last_became_peered":"2024-09-20T19:30:59.428000+0000","last_unstale":"2024-09-20T19:33:07.441329+0000","last_undegraded":"2024-09-20T19:33:07.441329+0000","last_fullsized":"2024-09-20T19:33:07.441329+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1280,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423357+0000","last_change":"2024-09-20T19:30:59.412019+0000","last_active":"2024-09-20T19:33:06.423357+0000","last_peered":"2024-09-20T19:33:06.423357+0000","last_clean":"2024-09-20T19:33:06.423357+0000","last_became_active":"2024-09-20T19:30:59.411617+0000","last_became_peered":"2024-09-20T19:30:59.411617+0000","last_unstale":"2024-09-20T19:33:06.423357+0000","last_undegraded":"2024-09-20T19:33:06.423357+0000","last_fullsized":"2024-09-20T19:33:06.423357+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.424475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.424475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:48.399868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1465,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424043+0000","last_change":"2024-09-20T19:32:11.594974+0000","last_active":"2024-09-20T19:33:06.424043+0000","last_peered":"2024-09-20T19:33:06.424043+0000","last_clean":"2024-09-20T19:33:06.424043+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:33:06.424043+0000","last_undegraded":"2024-09-20T19:33:06.424043+0000","last_fullsized":"2024-09-20T19:33:06.424043+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:11.594902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.594902+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:28.976985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1047,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.439714+0000","last_change":"2024-09-20T19:32:11.370896+0000","last_active":"2024-09-20T19:33:07.439714+0000","last_peered":"2024-09-20T19:33:07.439714+0000","last_clean":"2024-09-20T19:33:07.439714+0000","last_became_active":"2024-09-20T19:30:59.447481+0000","last_became_peered":"2024-09-20T19:30:59.447481+0000","last_unstale":"2024-09-20T19:33:07.439714+0000","last_undegraded":"2024-09-20T19:33:07.439714+0000","last_fullsized":"2024-09-20T19:33:07.439714+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:37.301214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"678'382","reported_seq":2856,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439650+0000","last_change":"2024-09-20T19:33:06.434706+0000","last_active":"2024-09-20T19:33:08.439650+0000","last_peered":"2024-09-20T19:33:08.439650+0000","last_clean":"2024-09-20T19:33:08.439650+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:33:08.439650+0000","last_undegraded":"2024-09-20T19:33:08.439650+0000","last_fullsized":"2024-09-20T19:33:08.439650+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"670'374","last_scrub_stamp":"2024-09-20T19:32:58.919240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:58.919240+0000","objects_scrubbed":5,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:25.513726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016995878999999998,"stat_sum":{"num_bytes":6415874,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":42970,"num_write":27,"num_write_kb":10113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"4"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.15","version":"523'15","reported_seq":1308,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424192+0000","last_change":"2024-09-20T19:30:59.427602+0000","last_active":"2024-09-20T19:33:06.424192+0000","last_peered":"2024-09-20T19:33:06.424192+0000","last_clean":"2024-09-20T19:33:06.424192+0000","last_became_active":"2024-09-20T19:30:59.404598+0000","last_became_peered":"2024-09-20T19:30:59.404598+0000","last_unstale":"2024-09-20T19:33:06.424192+0000","last_undegraded":"2024-09-20T19:33:06.424192+0000","last_fullsized":"2024-09-20T19:33:06.424192+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-20T19:27:22.506346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:27:22.506346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:40.357492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1020,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.439102+0000","last_change":"2024-09-20T19:32:26.439201+0000","last_active":"2024-09-20T19:33:07.439102+0000","last_peered":"2024-09-20T19:33:07.439102+0000","last_clean":"2024-09-20T19:33:07.439102+0000","last_became_active":"2024-09-20T19:30:59.448308+0000","last_became_peered":"2024-09-20T19:30:59.448308+0000","last_unstale":"2024-09-20T19:33:07.439102+0000","last_undegraded":"2024-09-20T19:33:07.439102+0000","last_fullsized":"2024-09-20T19:33:07.439102+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.836552+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"669'14","reported_seq":1264,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.445814+0000","last_change":"2024-09-20T19:32:57.357095+0000","last_active":"2024-09-20T19:33:07.445814+0000","last_peered":"2024-09-20T19:33:07.445814+0000","last_clean":"2024-09-20T19:33:07.445814+0000","last_became_active":"2024-09-20T19:32:57.348336+0000","last_became_peered":"2024-09-20T19:32:57.348336+0000","last_unstale":"2024-09-20T19:33:07.445814+0000","last_undegraded":"2024-09-20T19:33:07.445814+0000","last_fullsized":"2024-09-20T19:33:07.445814+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:28:18.243973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:18.243973+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1098,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439355+0000","last_change":"2024-09-20T19:31:19.806432+0000","last_active":"2024-09-20T19:33:08.439355+0000","last_peered":"2024-09-20T19:33:08.439355+0000","last_clean":"2024-09-20T19:33:08.439355+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:33:08.439355+0000","last_undegraded":"2024-09-20T19:33:08.439355+0000","last_fullsized":"2024-09-20T19:33:08.439355+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:19.806369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:19.806369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"677'459","reported_seq":3172,"reported_epoch":678,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:06.420893+0000","last_change":"2024-09-20T19:33:06.420863+0000","last_active":"2024-09-20T19:33:06.420893+0000","last_peered":"2024-09-20T19:33:06.420893+0000","last_clean":"2024-09-20T19:33:06.420893+0000","last_became_active":"2024-09-20T19:31:00.440451+0000","last_became_peered":"2024-09-20T19:31:00.440451+0000","last_unstale":"2024-09-20T19:33:06.420893+0000","last_undegraded":"2024-09-20T19:33:06.420893+0000","last_fullsized":"2024-09-20T19:33:06.420893+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.86774154400000003,"stat_sum":{"num_bytes":3428727,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":12214,"num_write":3,"num_write_kb":3349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"3"},{"start":"ff","length":"1"},{"start":"102","length":"1"},{"start":"104","length":"2"}]},{"pgid":"4.12","version":"523'15","reported_seq":1302,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422739+0000","last_change":"2024-09-20T19:32:31.080255+0000","last_active":"2024-09-20T19:33:06.422739+0000","last_peered":"2024-09-20T19:33:06.422739+0000","last_clean":"2024-09-20T19:33:06.422739+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:33:06.422739+0000","last_undegraded":"2024-09-20T19:33:06.422739+0000","last_fullsized":"2024-09-20T19:33:06.422739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:31.080102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.080102+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.008745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1264,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420535+0000","last_change":"2024-09-20T19:30:59.412393+0000","last_active":"2024-09-20T19:33:06.420535+0000","last_peered":"2024-09-20T19:33:06.420535+0000","last_clean":"2024-09-20T19:33:06.420535+0000","last_became_active":"2024-09-20T19:30:59.412120+0000","last_became_peered":"2024-09-20T19:30:59.412120+0000","last_unstale":"2024-09-20T19:33:06.420535+0000","last_undegraded":"2024-09-20T19:33:06.420535+0000","last_fullsized":"2024-09-20T19:33:06.420535+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:24:42.625450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:42.625450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:45.638553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1117,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.440087+0000","last_change":"2024-09-20T19:31:35.565401+0000","last_active":"2024-09-20T19:33:07.440087+0000","last_peered":"2024-09-20T19:33:07.440087+0000","last_clean":"2024-09-20T19:33:07.440087+0000","last_became_active":"2024-09-20T19:30:59.432219+0000","last_became_peered":"2024-09-20T19:30:59.432219+0000","last_unstale":"2024-09-20T19:33:07.440087+0000","last_undegraded":"2024-09-20T19:33:07.440087+0000","last_fullsized":"2024-09-20T19:33:07.440087+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1037,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.429278+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:33:07.429278+0000","last_peered":"2024-09-20T19:33:07.429278+0000","last_clean":"2024-09-20T19:33:07.429278+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:33:07.429278+0000","last_undegraded":"2024-09-20T19:33:07.429278+0000","last_fullsized":"2024-09-20T19:33:07.429278+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:25.008535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"677'607","reported_seq":3230,"reported_epoch":679,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:33:07.442508+0000","last_change":"2024-09-20T19:33:05.478591+0000","last_active":"2024-09-20T19:33:07.442508+0000","last_peered":"2024-09-20T19:33:07.442508+0000","last_clean":"2024-09-20T19:33:07.442508+0000","last_became_active":"2024-09-20T19:31:48.284326+0000","last_became_peered":"2024-09-20T19:31:48.284326+0000","last_unstale":"2024-09-20T19:33:07.442508+0000","last_undegraded":"2024-09-20T19:33:07.442508+0000","last_fullsized":"2024-09-20T19:33:07.442508+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'561","last_scrub_stamp":"2024-09-20T19:28:49.166999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:28:49.166999+0000","objects_scrubbed":1,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.095701337999999997,"stat_sum":{"num_bytes":5681063,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":5243,"num_write":16,"num_write_kb":3533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3019792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.13","version":"523'19","reported_seq":1113,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438108+0000","last_change":"2024-09-20T19:32:32.189200+0000","last_active":"2024-09-20T19:33:08.438108+0000","last_peered":"2024-09-20T19:33:08.438108+0000","last_clean":"2024-09-20T19:33:08.438108+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:33:08.438108+0000","last_undegraded":"2024-09-20T19:33:08.438108+0000","last_fullsized":"2024-09-20T19:33:08.438108+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:32.189122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.189122+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:44.082120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1292,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.425122+0000","last_change":"2024-09-20T19:31:20.343942+0000","last_active":"2024-09-20T19:33:06.425122+0000","last_peered":"2024-09-20T19:33:06.425122+0000","last_clean":"2024-09-20T19:33:06.425122+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:33:06.425122+0000","last_undegraded":"2024-09-20T19:33:06.425122+0000","last_fullsized":"2024-09-20T19:33:06.425122+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:20.343882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.343882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1271,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.422247+0000","last_change":"2024-09-20T19:31:45.587402+0000","last_active":"2024-09-20T19:33:06.422247+0000","last_peered":"2024-09-20T19:33:06.422247+0000","last_clean":"2024-09-20T19:33:06.422247+0000","last_became_active":"2024-09-20T19:31:45.586887+0000","last_became_peered":"2024-09-20T19:31:45.586887+0000","last_unstale":"2024-09-20T19:33:06.422247+0000","last_undegraded":"2024-09-20T19:33:06.422247+0000","last_fullsized":"2024-09-20T19:33:06.422247+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:47.571921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:28:47.571921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1113,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438698+0000","last_change":"2024-09-20T19:31:36.528455+0000","last_active":"2024-09-20T19:33:07.438698+0000","last_peered":"2024-09-20T19:33:07.438698+0000","last_clean":"2024-09-20T19:33:07.438698+0000","last_became_active":"2024-09-20T19:30:59.431451+0000","last_became_peered":"2024-09-20T19:30:59.431451+0000","last_unstale":"2024-09-20T19:33:07.438698+0000","last_undegraded":"2024-09-20T19:33:07.438698+0000","last_fullsized":"2024-09-20T19:33:07.438698+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:36.528327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.528327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"677'437","reported_seq":3080,"reported_epoch":679,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:07.438465+0000","last_change":"2024-09-20T19:33:06.481926+0000","last_active":"2024-09-20T19:33:07.438465+0000","last_peered":"2024-09-20T19:33:07.438465+0000","last_clean":"2024-09-20T19:33:07.438465+0000","last_became_active":"2024-09-20T19:33:00.324250+0000","last_became_peered":"2024-09-20T19:33:00.324250+0000","last_unstale":"2024-09-20T19:33:07.438465+0000","last_undegraded":"2024-09-20T19:33:07.438465+0000","last_fullsized":"2024-09-20T19:33:07.438465+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":5,"last_scrub":"524'363","last_scrub_stamp":"2024-09-20T19:30:33.692750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:33.692750+0000","objects_scrubbed":3,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.053478667000000001,"stat_sum":{"num_bytes":9336887,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":21,"num_write_kb":7910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":6438138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e7","length":"1"},{"start":"f6","length":"2"},{"start":"102","length":"1"}]},{"pgid":"4.10","version":"523'8","reported_seq":1121,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441964+0000","last_change":"2024-09-20T19:32:18.830939+0000","last_active":"2024-09-20T19:33:07.441964+0000","last_peered":"2024-09-20T19:33:07.441964+0000","last_clean":"2024-09-20T19:33:07.441964+0000","last_became_active":"2024-09-20T19:30:59.418597+0000","last_became_peered":"2024-09-20T19:30:59.418597+0000","last_unstale":"2024-09-20T19:33:07.441964+0000","last_undegraded":"2024-09-20T19:33:07.441964+0000","last_fullsized":"2024-09-20T19:33:07.441964+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:32:18.830843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.830843+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.107094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1268,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424778+0000","last_change":"2024-09-20T19:33:03.325570+0000","last_active":"2024-09-20T19:33:06.424778+0000","last_peered":"2024-09-20T19:33:06.424778+0000","last_clean":"2024-09-20T19:33:06.424778+0000","last_became_active":"2024-09-20T19:32:57.338936+0000","last_became_peered":"2024-09-20T19:32:57.338936+0000","last_unstale":"2024-09-20T19:33:06.424778+0000","last_undegraded":"2024-09-20T19:33:06.424778+0000","last_fullsized":"2024-09-20T19:33:06.424778+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:03.325428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:03.325428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:24.020132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1264,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421646+0000","last_change":"2024-09-20T19:30:31.387063+0000","last_active":"2024-09-20T19:33:06.421646+0000","last_peered":"2024-09-20T19:33:06.421646+0000","last_clean":"2024-09-20T19:33:06.421646+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:33:06.421646+0000","last_undegraded":"2024-09-20T19:33:06.421646+0000","last_fullsized":"2024-09-20T19:33:06.421646+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:25:32.544996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:25:32.544996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1252,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420802+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:33:06.420802+0000","last_peered":"2024-09-20T19:33:06.420802+0000","last_clean":"2024-09-20T19:33:06.420802+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:33:06.420802+0000","last_undegraded":"2024-09-20T19:33:06.420802+0000","last_fullsized":"2024-09-20T19:33:06.420802+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:00.258729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":2765,"reported_epoch":678,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:33:06.536498+0000","last_change":"2024-09-20T19:33:06.536531+0000","last_active":"2024-09-20T19:33:06.536498+0000","last_peered":"2024-09-20T19:33:06.536498+0000","last_clean":"2024-09-20T19:33:06.536498+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:33:06.536498+0000","last_undegraded":"2024-09-20T19:33:06.536498+0000","last_fullsized":"2024-09-20T19:33:06.536498+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:29:00.472909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.472909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.116802829,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"cf","length":"1"},{"start":"d6","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"e7","length":"1"},{"start":"ef","length":"1"},{"start":"f6","length":"3"}]},{"pgid":"4.11","version":"523'19","reported_seq":1136,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.444987+0000","last_change":"2024-09-20T19:32:16.785202+0000","last_active":"2024-09-20T19:33:07.444987+0000","last_peered":"2024-09-20T19:33:07.444987+0000","last_clean":"2024-09-20T19:33:07.444987+0000","last_became_active":"2024-09-20T19:30:59.426703+0000","last_became_peered":"2024-09-20T19:30:59.426703+0000","last_unstale":"2024-09-20T19:33:07.444987+0000","last_undegraded":"2024-09-20T19:33:07.444987+0000","last_fullsized":"2024-09-20T19:33:07.444987+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:42.001001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1269,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421552+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:33:06.421552+0000","last_peered":"2024-09-20T19:33:06.421552+0000","last_clean":"2024-09-20T19:33:06.421552+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:33:06.421552+0000","last_undegraded":"2024-09-20T19:33:06.421552+0000","last_fullsized":"2024-09-20T19:33:06.421552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1272,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420695+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:33:06.420695+0000","last_peered":"2024-09-20T19:33:06.420695+0000","last_clean":"2024-09-20T19:33:06.420695+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:33:06.420695+0000","last_undegraded":"2024-09-20T19:33:06.420695+0000","last_fullsized":"2024-09-20T19:33:06.420695+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:51.725052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1129,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.440658+0000","last_change":"2024-09-20T19:33:02.767940+0000","last_active":"2024-09-20T19:33:07.440658+0000","last_peered":"2024-09-20T19:33:07.440658+0000","last_clean":"2024-09-20T19:33:07.440658+0000","last_became_active":"2024-09-20T19:30:59.431821+0000","last_became_peered":"2024-09-20T19:30:59.431821+0000","last_unstale":"2024-09-20T19:33:07.440658+0000","last_undegraded":"2024-09-20T19:33:07.440658+0000","last_fullsized":"2024-09-20T19:33:07.440658+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.767876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.767876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:32.273419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":1250,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423543+0000","last_change":"2024-09-20T19:28:13.311223+0000","last_active":"2024-09-20T19:33:06.423543+0000","last_peered":"2024-09-20T19:33:06.423543+0000","last_clean":"2024-09-20T19:33:06.423543+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:33:06.423543+0000","last_undegraded":"2024-09-20T19:33:06.423543+0000","last_fullsized":"2024-09-20T19:33:06.423543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:28:13.311036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:28:13.311036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:47.856777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1288,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.423023+0000","last_change":"2024-09-20T19:32:31.401371+0000","last_active":"2024-09-20T19:33:06.423023+0000","last_peered":"2024-09-20T19:33:06.423023+0000","last_clean":"2024-09-20T19:33:06.423023+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:33:06.423023+0000","last_undegraded":"2024-09-20T19:33:06.423023+0000","last_fullsized":"2024-09-20T19:33:06.423023+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:31.401257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.401257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.241223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":1054,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438455+0000","last_change":"2024-09-20T19:31:36.551849+0000","last_active":"2024-09-20T19:33:07.438455+0000","last_peered":"2024-09-20T19:33:07.438455+0000","last_clean":"2024-09-20T19:33:07.438455+0000","last_became_active":"2024-09-20T19:30:59.447209+0000","last_became_peered":"2024-09-20T19:30:59.447209+0000","last_unstale":"2024-09-20T19:33:07.438455+0000","last_undegraded":"2024-09-20T19:33:07.438455+0000","last_fullsized":"2024-09-20T19:33:07.438455+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:36.551731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.551731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1469,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421370+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:33:06.421370+0000","last_peered":"2024-09-20T19:33:06.421370+0000","last_clean":"2024-09-20T19:33:06.421370+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:33:06.421370+0000","last_undegraded":"2024-09-20T19:33:06.421370+0000","last_fullsized":"2024-09-20T19:33:06.421370+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1271,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424135+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:33:06.424135+0000","last_peered":"2024-09-20T19:33:06.424135+0000","last_clean":"2024-09-20T19:33:06.424135+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:33:06.424135+0000","last_undegraded":"2024-09-20T19:33:06.424135+0000","last_fullsized":"2024-09-20T19:33:06.424135+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:41.117115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":1134,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.441386+0000","last_change":"2024-09-20T19:30:59.433439+0000","last_active":"2024-09-20T19:33:07.441386+0000","last_peered":"2024-09-20T19:33:07.441386+0000","last_clean":"2024-09-20T19:33:07.441386+0000","last_became_active":"2024-09-20T19:30:59.432846+0000","last_became_peered":"2024-09-20T19:30:59.432846+0000","last_unstale":"2024-09-20T19:33:07.441386+0000","last_undegraded":"2024-09-20T19:33:07.441386+0000","last_fullsized":"2024-09-20T19:33:07.441386+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:24:30.106245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:24:30.106245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:43.925898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1319,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.421546+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:33:06.421546+0000","last_peered":"2024-09-20T19:33:06.421546+0000","last_clean":"2024-09-20T19:33:06.421546+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:33:06.421546+0000","last_undegraded":"2024-09-20T19:33:06.421546+0000","last_fullsized":"2024-09-20T19:33:06.421546+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1059,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.438844+0000","last_change":"2024-09-20T19:32:57.349141+0000","last_active":"2024-09-20T19:33:07.438844+0000","last_peered":"2024-09-20T19:33:07.438844+0000","last_clean":"2024-09-20T19:33:07.438844+0000","last_became_active":"2024-09-20T19:32:57.348743+0000","last_became_peered":"2024-09-20T19:32:57.348743+0000","last_unstale":"2024-09-20T19:33:07.438844+0000","last_undegraded":"2024-09-20T19:33:07.438844+0000","last_fullsized":"2024-09-20T19:33:07.438844+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.455533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.455533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:39.000546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1095,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.439229+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:33:08.439229+0000","last_peered":"2024-09-20T19:33:08.439229+0000","last_clean":"2024-09-20T19:33:08.439229+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:33:08.439229+0000","last_undegraded":"2024-09-20T19:33:08.439229+0000","last_fullsized":"2024-09-20T19:33:08.439229+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1280,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.420184+0000","last_change":"2024-09-20T19:32:15.866641+0000","last_active":"2024-09-20T19:33:06.420184+0000","last_peered":"2024-09-20T19:33:06.420184+0000","last_clean":"2024-09-20T19:33:06.420184+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:33:06.420184+0000","last_undegraded":"2024-09-20T19:33:06.420184+0000","last_fullsized":"2024-09-20T19:33:06.420184+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.866572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.866572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:20.064376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":1056,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424008+0000","last_change":"2024-09-20T19:33:05.356476+0000","last_active":"2024-09-20T19:33:06.424008+0000","last_peered":"2024-09-20T19:33:06.424008+0000","last_clean":"2024-09-20T19:33:06.424008+0000","last_became_active":"2024-09-20T19:32:57.339768+0000","last_became_peered":"2024-09-20T19:32:57.339768+0000","last_unstale":"2024-09-20T19:33:06.424008+0000","last_undegraded":"2024-09-20T19:33:06.424008+0000","last_fullsized":"2024-09-20T19:33:06.424008+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:33:05.356320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:05.356320+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:34:25.324662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1282,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-20T19:33:06.424575+0000","last_change":"2024-09-20T19:32:16.075391+0000","last_active":"2024-09-20T19:33:06.424575+0000","last_peered":"2024-09-20T19:33:06.424575+0000","last_clean":"2024-09-20T19:33:06.424575+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:33:06.424575+0000","last_undegraded":"2024-09-20T19:33:06.424575+0000","last_fullsized":"2024-09-20T19:33:06.424575+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.075336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.075336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:33.476752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1076,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.445100+0000","last_change":"2024-09-20T19:30:59.414333+0000","last_active":"2024-09-20T19:33:07.445100+0000","last_peered":"2024-09-20T19:33:07.445100+0000","last_clean":"2024-09-20T19:33:07.445100+0000","last_became_active":"2024-09-20T19:30:59.413650+0000","last_became_peered":"2024-09-20T19:30:59.413650+0000","last_unstale":"2024-09-20T19:33:07.445100+0000","last_undegraded":"2024-09-20T19:33:07.445100+0000","last_fullsized":"2024-09-20T19:33:07.445100+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:54.253853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.253853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1086,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442956+0000","last_change":"2024-09-20T19:30:59.425869+0000","last_active":"2024-09-20T19:33:07.442956+0000","last_peered":"2024-09-20T19:33:07.442956+0000","last_clean":"2024-09-20T19:33:07.442956+0000","last_became_active":"2024-09-20T19:30:59.424644+0000","last_became_peered":"2024-09-20T19:30:59.424644+0000","last_unstale":"2024-09-20T19:33:07.442956+0000","last_undegraded":"2024-09-20T19:33:07.442956+0000","last_fullsized":"2024-09-20T19:33:07.442956+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:58.281416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:27:58.281416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":1094,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-20T19:33:07.442225+0000","last_change":"2024-09-20T19:32:10.800441+0000","last_active":"2024-09-20T19:33:07.442225+0000","last_peered":"2024-09-20T19:33:07.442225+0000","last_clean":"2024-09-20T19:33:07.442225+0000","last_became_active":"2024-09-20T19:30:59.418441+0000","last_became_peered":"2024-09-20T19:30:59.418441+0000","last_unstale":"2024-09-20T19:33:07.442225+0000","last_undegraded":"2024-09-20T19:33:07.442225+0000","last_fullsized":"2024-09-20T19:33:07.442225+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:10.800347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.800347+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:34.402765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1094,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-20T19:33:08.438503+0000","last_change":"2024-09-20T19:32:33.198084+0000","last_active":"2024-09-20T19:33:08.438503+0000","last_peered":"2024-09-20T19:33:08.438503+0000","last_clean":"2024-09-20T19:33:08.438503+0000","last_became_active":"2024-09-20T19:30:59.409071+0000","last_became_peered":"2024-09-20T19:30:59.409071+0000","last_unstale":"2024-09-20T19:33:08.438503+0000","last_undegraded":"2024-09-20T19:33:08.438503+0000","last_fullsized":"2024-09-20T19:33:08.438503+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:33.198003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:33.198003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:33:46.209339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":241479652,"num_objects":124,"num_object_clones":79,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":11,"num_read":2002,"num_read_kb":1518718,"num_write":1445,"num_write_kb":538153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":112315472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536748032,"data_stored":725468770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":14874,"ondisk_log_size":14874,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1099,"num_read_kb":856,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907133,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147620,"kb_used_data":90064,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93597532,"statfs":{"total":95995035648,"available":95843872768,"internally_reserved":0,"allocated":92225536,"data_stored":127398262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234184,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81136,"kb_used_data":49940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664016,"statfs":{"total":95995035648,"available":95911952384,"internally_reserved":0,"allocated":51138560,"data_stored":67556031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561234,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115924,"kb_used_data":84732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93629228,"statfs":{"total":95995035648,"available":95876329472,"internally_reserved":0,"allocated":86765568,"data_stored":107845231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928545,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79200,"kb_used_data":42840,"kb_used_omap":35,"kb_used_meta":36124,"kb_avail":93665952,"statfs":{"total":95995035648,"available":95913934848,"internally_reserved":0,"allocated":43868160,"data_stored":54053373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":36991921},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":550,"seq":2362232012827,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128148,"kb_used_data":91260,"kb_used_omap":51,"kb_used_meta":36812,"kb_avail":93617004,"statfs":{"total":95995035648,"available":95863812096,"internally_reserved":0,"allocated":93450240,"data_stored":126909634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52777,"internal_metadata":37695959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378870,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117140,"kb_used_data":67896,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93628012,"statfs":{"total":95995035648,"available":95875084288,"internally_reserved":0,"allocated":69525504,"data_stored":91254076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131772,"kb_used_data":74216,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93613380,"statfs":{"total":95995035648,"available":95860101120,"internally_reserved":0,"allocated":75997184,"data_stored":88328683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648037,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112164,"kb_used_data":70732,"kb_used_omap":49,"kb_used_meta":41358,"kb_avail":93632988,"statfs":{"total":95995035648,"available":95880179712,"internally_reserved":0,"allocated":72429568,"data_stored":92059660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":42351145},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67334144,"data_stored":88999314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70922240,"data_stored":85282837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60329984,"data_stored":83587690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":87760896,"data_stored":126308846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38952960,"data_stored":51097079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":78553088,"data_stored":101671127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46071808,"data_stored":64503777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86822912,"data_stored":124018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:33:13.193 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T19:33:13.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 36 2024-09-20T19:33:13.223 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:13.310 INFO:tasks.rados.rados.0.smithi042.stdout:2298: done (0 left) 2024-09-20T19:33:13.310 INFO:tasks.rados.rados.0.smithi042.stdout:2299: rollback oid 19 current snap is 266 2024-09-20T19:33:13.310 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 251 2024-09-20T19:33:13.310 INFO:tasks.rados.rados.0.smithi042.stdout:2300: snap_remove snap 254 2024-09-20T19:33:13.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:13.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:13.511 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:13.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:33:13.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:13.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[22706]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:13 smithi042 bash[32075]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:13.740 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.158018+0000 mgr.y (mgr.24491) 574 : audit [DBG] from='client.15807 v1:172.21.15.42:0/4281788262' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.329481+0000 mgr.y (mgr.24491) 575 : cluster [DBG] pgmap v1023: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 231 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.492170+0000 osd.7 (osd.7) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.492949+0000 osd.7 (osd.7) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:12.495914+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.990143+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: audit 2024-09-20T19:33:12.992300+0000 mon.a (mon.0) 2131 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.136149+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:33:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.137317+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:33:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:13 smithi162 bash[24557]: cluster 2024-09-20T19:33:13.306641+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-20T19:33:13.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:13.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:13.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:14.022 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:14.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:14.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:14.253 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:14.311 INFO:tasks.rados.rados.0.smithi042.stdout:2300: done (1 left) 2024-09-20T19:33:14.311 INFO:tasks.rados.rados.0.smithi042.stdout:2301: copy_from oid 16 from oid 37 current snap is 266 2024-09-20T19:33:14.311 INFO:tasks.rados.rados.0.smithi042.stdout:2302: rollback oid 2 current snap is 266 2024-09-20T19:33:14.312 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 241 2024-09-20T19:33:14.312 INFO:tasks.rados.rados.0.smithi042.stdout:2299: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:33:14.312 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 0 (ObjNum 33 snap 0 seq_num 1415318944) dirty dne 2024-09-20T19:33:14.312 INFO:tasks.rados.rados.0.smithi042.stdout:2299: done (2 left) 2024-09-20T19:33:14.312 INFO:tasks.rados.rados.0.smithi042.stdout:2303: snap_remove snap 261 2024-09-20T19:33:14.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:14.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:14.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:14.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:14.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:14.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:14.691 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:14.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:14.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:14.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:14.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:15.032 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:15.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.193 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:15.298 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.325 INFO:tasks.rados.rados.0.smithi042.stdout:2303: done (2 left) 2024-09-20T19:33:15.325 INFO:tasks.rados.rados.0.smithi042.stdout:2304: snap_remove snap 248 2024-09-20T19:33:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:15.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:15.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:15.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[32075]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:15 smithi042 bash[22706]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:15.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.720 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:15.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.174898+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.a scrub starts 2024-09-20T19:33:15.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.177116+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.a scrub ok 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.309302+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.330036+0000 mgr.y (mgr.24491) 576 : cluster [DBG] pgmap v1027: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.642193+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:15 smithi162 bash[24557]: cluster 2024-09-20T19:33:14.644260+0000 osd.4 (osd.4) 58 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:33:15.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:15.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:15.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:15.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:16.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:16.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:16.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:16.243 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:16.337 INFO:tasks.rados.rados.0.smithi042.stdout:2304: done (2 left) 2024-09-20T19:33:16.337 INFO:tasks.rados.rados.0.smithi042.stdout:2305: rollback oid 46 current snap is 266 2024-09-20T19:33:16.337 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 262 2024-09-20T19:33:16.337 INFO:tasks.rados.rados.0.smithi042.stdout:2301: finishing copy_from to smithi04268048-16 2024-09-20T19:33:16.337 INFO:tasks.rados.rados.0.smithi042.stdout:2301: got expected ENOENT (src dne) 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2301: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2302: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2302: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2302: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 622 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2301: done (2 left) 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2302: done (1 left) 2024-09-20T19:33:16.338 INFO:tasks.rados.rados.0.smithi042.stdout:2306: snap_remove snap 252 2024-09-20T19:33:16.346 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:16.371 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:16.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:16.485 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:16 smithi162 bash[24557]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.486 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:16 smithi162 bash[24557]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.500 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:16.544 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:33:16.545 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:33:16.563 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:33:16.563 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:33:16.579 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:33:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:33:16.597 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:33:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:33:16.603 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:16.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:16 smithi042 bash[32075]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.613 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:16 smithi042 bash[32075]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.613 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:16.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:16 smithi042 bash[22706]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:16 smithi042 bash[22706]: cluster 2024-09-20T19:33:15.323165+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-20T19:33:16.618 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:33:16.618 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:33:16.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:16.640 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:33:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:33:16.661 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:33:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:33:16.682 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:33:16.682 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:33:16.727 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:16.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:16.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:16.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:16.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:16.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:16.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:17.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:17.122 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:17.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:17.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:17.350 INFO:tasks.rados.rados.0.smithi042.stdout:2306: done (1 left) 2024-09-20T19:33:17.351 INFO:tasks.rados.rados.0.smithi042.stdout:2307: snap_create 2024-09-20T19:33:17.351 INFO:tasks.rados.rados.0.smithi042.stdout:2305: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:33:17.351 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1059 (ObjNum 129 snap 0 seq_num 1415260456) dirty dne 2024-09-20T19:33:17.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[22706]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:17 smithi042 bash[32075]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.295502+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.f scrub starts 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.297773+0000 osd.5 (osd.5) 212 : cluster [DBG] 5.f scrub ok 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.334546+0000 mgr.y (mgr.24491) 577 : cluster [DBG] pgmap v1030: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 226 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.336004+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-20T19:33:17.525 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:17 smithi162 bash[24557]: audit 2024-09-20T19:33:16.525915+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.42:0/1412707835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:17.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:17.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.802 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:17.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.933 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:17.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:18.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:18.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:18.197 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:18.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:18.319 INFO:tasks.rados.rados.0.smithi042.stdout:2305: done (1 left) 2024-09-20T19:33:18.319 INFO:tasks.rados.rados.0.smithi042.stdout:2307: done (0 left) 2024-09-20T19:33:18.319 INFO:tasks.rados.rados.0.smithi042.stdout:2308: snap_remove snap 265 2024-09-20T19:33:18.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:18.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:18.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:18.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:18 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.739138+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:16.739766+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.158733+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.159927+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.346283+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.352695+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.355708+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.642711+0000 osd.4 (osd.4) 59 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:17.645130+0000 osd.4 (osd.4) 60 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:18 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.312542+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-20T19:33:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:18.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:18.849 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:18.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:18.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:18.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:19.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:19.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:19.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:19.240 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:19.315 INFO:tasks.rados.rados.0.smithi042.stdout:2308: done (0 left) 2024-09-20T19:33:19.315 INFO:tasks.rados.rados.0.smithi042.stdout:2309: rollback oid 42 current snap is 267 2024-09-20T19:33:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 264 2024-09-20T19:33:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:2310: read oid 40 snap -1 2024-09-20T19:33:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:2310: expect (ObjNum 805 snap 265 seq_num 805) 2024-09-20T19:33:19.316 INFO:tasks.rados.rados.0.smithi042.stdout:2311: snap_create 2024-09-20T19:33:19.325 INFO:tasks.rados.rados.0.smithi042.stdout:2309: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:33:19.325 INFO:tasks.rados.rados.0.smithi042.stdout:2309: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:33:19.326 INFO:tasks.rados.rados.0.smithi042.stdout:2309: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:33:19.326 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 979 (ObjNum 682 snap 221 seq_num 682) dirty exists 2024-09-20T19:33:19.331 INFO:tasks.rados.rados.0.smithi042.stdout:2310: expect (ObjNum 805 snap 265 seq_num 805) 2024-09-20T19:33:19.331 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 695777 2024-09-20T19:33:19.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:19.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:19.423 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:19.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:19.481 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.491 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:19 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.503 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:19.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:19.632 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.335323+0000 mgr.y (mgr.24491) 578 : cluster [DBG] pgmap v1033: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.615849+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:18.618187+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.313085+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-20T19:33:19.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:19.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:19.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:19.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:19.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:19.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:20.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:20.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:20.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:20.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:20.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:2309: done (2 left) 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:2310: done (1 left) 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:2311: done (0 left) 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:2312: rollback oid 5 current snap is 268 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 250 2024-09-20T19:33:20.320 INFO:tasks.rados.rados.0.smithi042.stdout:2313: snap_remove snap 267 2024-09-20T19:33:20.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:20.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:20 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.345067+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.a scrub starts 2024-09-20T19:33:20.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:19.347377+0000 osd.1 (osd.1) 244 : cluster [DBG] 2.a scrub ok 2024-09-20T19:33:20.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: audit 2024-09-20T19:33:19.640804+0000 mgr.y (mgr.24491) 579 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:20.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:20 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.318401+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-20T19:33:20.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:20.729 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:33:20.730 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:33:20.730 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:33:20.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:33:20.730 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:33:20.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:20.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:20.929 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:20.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:21.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:21.161 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:21.291 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:21.322 INFO:tasks.rados.rados.0.smithi042.stdout:2313: done (1 left) 2024-09-20T19:33:21.322 INFO:tasks.rados.rados.0.smithi042.stdout:2314: write oid 21 current snap is 268 2024-09-20T19:33:21.322 INFO:tasks.rados.rados.0.smithi042.stdout:2314: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-20T19:33:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:21.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:21.327 INFO:tasks.rados.rados.0.smithi042.stdout:2314: writing smithi04268048-21 from 746442 to 1330554 tid 1 2024-09-20T19:33:21.333 INFO:tasks.rados.rados.0.smithi042.stdout:2314: writing smithi04268048-21 from 1748369 to 2470974 tid 2 2024-09-20T19:33:21.340 INFO:tasks.rados.rados.0.smithi042.stdout:2314: writing smithi04268048-21 from 2983991 to 3546442 tid 3 2024-09-20T19:33:21.340 INFO:tasks.rados.rados.0.smithi042.stdout:2315: write oid 28 current snap is 268 2024-09-20T19:33:21.340 INFO:tasks.rados.rados.0.smithi042.stdout:2312: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:33:21.340 INFO:tasks.rados.rados.0.smithi042.stdout:2315: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-20T19:33:21.349 INFO:tasks.rados.rados.0.smithi042.stdout:2315: writing smithi04268048-28 from 763249 to 1422610 tid 1 2024-09-20T19:33:21.356 INFO:tasks.rados.rados.0.smithi042.stdout:2315: writing smithi04268048-28 from 2090498 to 2673114 tid 2 2024-09-20T19:33:21.361 INFO:tasks.rados.rados.0.smithi042.stdout:2315: writing smithi04268048-28 from 3295061 to 3563249 tid 3 2024-09-20T19:33:21.361 INFO:tasks.rados.rados.0.smithi042.stdout:2316: rollback oid 2 current snap is 268 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 260 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2312: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2312: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 802 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2314: finishing write tid 1 to smithi04268048-21 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2314: finishing write tid 2 to smithi04268048-21 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2312: done (3 left) 2024-09-20T19:33:21.362 INFO:tasks.rados.rados.0.smithi042.stdout:2317: snap_remove snap 264 2024-09-20T19:33:21.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:21.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:21.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:21.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:21.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:21.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:21 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:21.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:21 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:21.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:21 smithi162 bash[24557]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:21.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:21 smithi162 bash[24557]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:21.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:21.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:21.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:21.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:21.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:21.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:22.005 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:22.005 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:22.005 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[32075]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:22.006 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[32075]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:22.006 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:22.006 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.336301+0000 mgr.y (mgr.24491) 580 : cluster [DBG] pgmap v1036: 168 pgs: 12 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 146 active+clean; 215 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T19:33:22.006 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[22706]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:22.006 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:21 smithi042 bash[22706]: cluster 2024-09-20T19:33:21.316869+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-20T19:33:22.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:22.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:22.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:22.224 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:22.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:22.360 INFO:tasks.rados.rados.0.smithi042.stdout:2317: done (3 left) 2024-09-20T19:33:22.360 INFO:tasks.rados.rados.0.smithi042.stdout:2318: write (excl) oid 5 current snap is 268 2024-09-20T19:33:22.360 INFO:tasks.rados.rados.0.smithi042.stdout:2318: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-20T19:33:22.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:22.368 INFO:tasks.rados.rados.0.smithi042.stdout:2318: writing smithi04268048-5 from 780056 to 1514666 tid 1 2024-09-20T19:33:22.377 INFO:tasks.rados.rados.0.smithi042.stdout:2318: writing smithi04268048-5 from 2148980 to 2875254 tid 2 2024-09-20T19:33:22.381 INFO:tasks.rados.rados.0.smithi042.stdout:2318: writing smithi04268048-5 from 3322484 to 3580056 tid 3 2024-09-20T19:33:22.381 INFO:tasks.rados.rados.0.smithi042.stdout:2319: write oid 18 current snap is 268 2024-09-20T19:33:22.381 INFO:tasks.rados.rados.0.smithi042.stdout:2319: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-20T19:33:22.388 INFO:tasks.rados.rados.0.smithi042.stdout:2319: writing smithi04268048-18 from 796863 to 1206722 tid 1 2024-09-20T19:33:22.395 INFO:tasks.rados.rados.0.smithi042.stdout:2319: writing smithi04268048-18 from 1807462 to 2393747 tid 2 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2319: writing smithi04268048-18 from 3066260 to 3596863 tid 3 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2320: write (excl) oid 41 current snap is 268 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: finishing write tid 3 to smithi04268048-21 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: finishing write tid 4 to smithi04268048-21 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: finishing write tid 5 to smithi04268048-21 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: oid 21 updating version 0 to 997 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: oid 21 updating version 997 to 998 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: oid 21 updating version 998 to 1000 2024-09-20T19:33:22.402 INFO:tasks.rados.rados.0.smithi042.stdout:2314: oid 21 version 1000 is already newer than 999 2024-09-20T19:33:22.403 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1000 (ObjNum 806 snap 268 seq_num 806) dirty exists 2024-09-20T19:33:22.403 INFO:tasks.rados.rados.0.smithi042.stdout:2314: left oid 21 (ObjNum 806 snap 268 seq_num 806) 2024-09-20T19:33:22.403 INFO:tasks.rados.rados.0.smithi042.stdout:2315: finishing write tid 1 to smithi04268048-28 2024-09-20T19:33:22.403 INFO:tasks.rados.rados.0.smithi042.stdout:2315: finishing write tid 2 to smithi04268048-28 2024-09-20T19:33:22.403 INFO:tasks.rados.rados.0.smithi042.stdout:2320: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-20T19:33:22.407 INFO:tasks.rados.rados.0.smithi042.stdout:2320: writing smithi04268048-41 from 413670 to 898778 tid 1 2024-09-20T19:33:22.414 INFO:tasks.rados.rados.0.smithi042.stdout:2320: writing smithi04268048-41 from 1465944 to 2195887 tid 2 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2320: writing smithi04268048-41 from 2977330 to 3502131 tid 3 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2314: done (5 left) 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2321: write (excl) oid 15 current snap is 268 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2315: finishing write tid 3 to smithi04268048-28 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2315: finishing write tid 4 to smithi04268048-28 2024-09-20T19:33:22.422 INFO:tasks.rados.rados.0.smithi042.stdout:2315: finishing write tid 5 to smithi04268048-28 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2315: oid 28 updating version 0 to 736 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2315: oid 28 version 736 is already newer than 735 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2315: oid 28 version 736 is already newer than 733 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2315: oid 28 version 736 is already newer than 734 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 736 (ObjNum 807 snap 268 seq_num 807) dirty exists 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2315: left oid 28 (ObjNum 807 snap 268 seq_num 807) 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2316: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2316: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2316: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 625 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2318: finishing write tid 1 to smithi04268048-5 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2318: finishing write tid 2 to smithi04268048-5 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2318: finishing write tid 3 to smithi04268048-5 2024-09-20T19:33:22.423 INFO:tasks.rados.rados.0.smithi042.stdout:2318: finishing write tid 4 to smithi04268048-5 2024-09-20T19:33:22.424 INFO:tasks.rados.rados.0.smithi042.stdout:2321: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-09-20T19:33:22.428 INFO:tasks.rados.rados.0.smithi042.stdout:2321: writing smithi04268048-15 from 430477 to 990834 tid 1 2024-09-20T19:33:22.433 INFO:tasks.rados.rados.0.smithi042.stdout:2321: writing smithi04268048-15 from 1408073 to 1881674 tid 2 2024-09-20T19:33:22.441 INFO:tasks.rados.rados.0.smithi042.stdout:2321: writing smithi04268048-15 from 2488400 to 3224473 tid 3 2024-09-20T19:33:22.441 INFO:tasks.rados.rados.0.smithi042.stdout:2315: done (5 left) 2024-09-20T19:33:22.441 INFO:tasks.rados.rados.0.smithi042.stdout:2316: done (4 left) 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2322: write oid 44 current snap is 268 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: finishing write tid 5 to smithi04268048-5 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: oid 5 updating version 0 to 803 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: oid 5 updating version 803 to 805 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: oid 5 version 805 is already newer than 804 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: oid 5 updating version 805 to 806 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 806 (ObjNum 808 snap 268 seq_num 808) dirty exists 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2318: left oid 5 (ObjNum 808 snap 268 seq_num 808) 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2319: finishing write tid 1 to smithi04268048-18 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2319: finishing write tid 2 to smithi04268048-18 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2319: finishing write tid 3 to smithi04268048-18 2024-09-20T19:33:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2322: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-20T19:33:22.447 INFO:tasks.rados.rados.0.smithi042.stdout:2322: writing smithi04268048-44 from 447284 to 1082890 tid 1 2024-09-20T19:33:22.455 INFO:tasks.rados.rados.0.smithi042.stdout:2322: writing smithi04268048-44 from 1866555 to 2600167 tid 2 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2322: writing smithi04268048-44 from 3315823 to 3647284 tid 3 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2318: done (4 left) 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2323: read oid 34 snap -1 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2323: expect deleted 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2324: copy_from oid 3 from oid 47 current snap is 268 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: finishing write tid 4 to smithi04268048-18 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: finishing write tid 5 to smithi04268048-18 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: oid 18 updating version 0 to 1003 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: oid 18 version 1003 is already newer than 1002 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: oid 18 updating version 1003 to 1004 2024-09-20T19:33:22.462 INFO:tasks.rados.rados.0.smithi042.stdout:2319: oid 18 updating version 1004 to 1005 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1005 (ObjNum 809 snap 268 seq_num 809) dirty exists 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2319: left oid 18 (ObjNum 809 snap 268 seq_num 809) 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: finishing write tid 1 to smithi04268048-41 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: finishing write tid 2 to smithi04268048-41 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: finishing write tid 3 to smithi04268048-41 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2319: done (5 left) 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2325: write oid 21 current snap is 268 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: finishing write tid 4 to smithi04268048-41 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: finishing write tid 5 to smithi04268048-41 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: oid 41 updating version 0 to 715 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: oid 41 updating version 715 to 716 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: oid 41 version 716 is already newer than 714 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:2320: oid 41 updating version 716 to 717 2024-09-20T19:33:22.463 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 717 (ObjNum 810 snap 268 seq_num 810) dirty exists 2024-09-20T19:33:22.464 INFO:tasks.rados.rados.0.smithi042.stdout:2320: left oid 41 (ObjNum 810 snap 268 seq_num 810) 2024-09-20T19:33:22.464 INFO:tasks.rados.rados.0.smithi042.stdout:2321: finishing write tid 1 to smithi04268048-15 2024-09-20T19:33:22.464 INFO:tasks.rados.rados.0.smithi042.stdout:2325: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-09-20T19:33:22.468 INFO:tasks.rados.rados.0.smithi042.stdout:2325: writing smithi04268048-21 from 464091 to 1174946 tid 1 2024-09-20T19:33:22.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:22.475 INFO:tasks.rados.rados.0.smithi042.stdout:2325: writing smithi04268048-21 from 1925037 to 2402307 tid 2 2024-09-20T19:33:22.481 INFO:tasks.rados.rados.0.smithi042.stdout:2325: writing smithi04268048-21 from 2943246 to 3418216 tid 3 2024-09-20T19:33:22.481 INFO:tasks.rados.rados.0.smithi042.stdout:2320: done (5 left) 2024-09-20T19:33:22.481 INFO:tasks.rados.rados.0.smithi042.stdout:2326: write (excl) oid 39 current snap is 268 2024-09-20T19:33:22.481 INFO:tasks.rados.rados.0.smithi042.stdout:2326: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-20T19:33:22.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:22 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:22 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:22 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:22 smithi162 bash[24557]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.487 INFO:tasks.rados.rados.0.smithi042.stdout:2326: writing smithi04268048-39 from 480898 to 983355 tid 1 2024-09-20T19:33:22.495 INFO:tasks.rados.rados.0.smithi042.stdout:2326: writing smithi04268048-39 from 1699872 to 2437153 tid 2 2024-09-20T19:33:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:22.502 INFO:tasks.rados.rados.0.smithi042.stdout:2326: writing smithi04268048-39 from 3087022 to 3680898 tid 3 2024-09-20T19:33:22.502 INFO:tasks.rados.rados.0.smithi042.stdout:2327: read oid 9 snap 266 2024-09-20T19:33:22.503 INFO:tasks.rados.rados.0.smithi042.stdout:2327: expect deleted 2024-09-20T19:33:22.503 INFO:tasks.rados.rados.0.smithi042.stdout:2328: snap_remove snap 241 2024-09-20T19:33:22.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:22.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[22706]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.751265+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.d scrub starts 2024-09-20T19:33:22.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:22 smithi042 bash[32075]: cluster 2024-09-20T19:33:20.758111+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.d scrub ok 2024-09-20T19:33:22.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:22.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:22.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:33:22.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:22.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:22.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:22.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:23.007 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:23.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:23.134 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:23.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:23.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:2328: done (7 left) 2024-09-20T19:33:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:2329: write (excl) oid 18 current snap is 268 2024-09-20T19:33:23.322 INFO:tasks.rados.rados.0.smithi042.stdout:2329: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-20T19:33:23.327 INFO:tasks.rados.rados.0.smithi042.stdout:2329: writing smithi04268048-18 from 497705 to 1075411 tid 1 2024-09-20T19:33:23.335 INFO:tasks.rados.rados.0.smithi042.stdout:2329: writing smithi04268048-18 from 1642001 to 2122940 tid 2 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:2329: writing smithi04268048-18 from 2598092 to 3095606 tid 3 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:2330: rollback oid 50 current snap is 268 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 250 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:2321: finishing write tid 2 to smithi04268048-15 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:2321: finishing write tid 3 to smithi04268048-15 2024-09-20T19:33:23.343 INFO:tasks.rados.rados.0.smithi042.stdout:2321: finishing write tid 4 to smithi04268048-15 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: finishing write tid 5 to smithi04268048-15 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: oid 15 updating version 0 to 720 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: oid 15 updating version 720 to 723 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: oid 15 version 723 is already newer than 722 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: oid 15 version 723 is already newer than 721 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 723 (ObjNum 811 snap 268 seq_num 811) dirty exists 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: left oid 15 (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2322: finishing write tid 1 to smithi04268048-44 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2322: finishing write tid 2 to smithi04268048-44 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2321: done (8 left) 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2331: snap_create 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2322: finishing write tid 3 to smithi04268048-44 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2322: finishing write tid 4 to smithi04268048-44 2024-09-20T19:33:23.344 INFO:tasks.rados.rados.0.smithi042.stdout:2322: finishing write tid 5 to smithi04268048-44 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2322: oid 44 updating version 0 to 1070 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2322: oid 44 version 1070 is already newer than 1067 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2322: oid 44 version 1070 is already newer than 1069 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2322: oid 44 version 1070 is already newer than 1068 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1070 (ObjNum 812 snap 268 seq_num 812) dirty exists 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2322: left oid 44 (ObjNum 812 snap 268 seq_num 812) 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2325: finishing write tid 1 to smithi04268048-21 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2325: finishing write tid 2 to smithi04268048-21 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2324: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2324: finishing copy_from to smithi04268048-3 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 948 (ObjNum 787 snap 260 seq_num 787) dirty exists 2024-09-20T19:33:23.345 INFO:tasks.rados.rados.0.smithi042.stdout:2325: finishing write tid 3 to smithi04268048-21 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: finishing write tid 4 to smithi04268048-21 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: finishing write tid 5 to smithi04268048-21 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: oid 21 updating version 0 to 1009 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: oid 21 version 1009 is already newer than 1008 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: oid 21 version 1009 is already newer than 1006 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: oid 21 version 1009 is already newer than 1007 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1009 (ObjNum 813 snap 268 seq_num 813) dirty exists 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2325: left oid 21 (ObjNum 813 snap 268 seq_num 813) 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: finishing write tid 1 to smithi04268048-39 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: finishing write tid 2 to smithi04268048-39 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: finishing write tid 3 to smithi04268048-39 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: finishing write tid 4 to smithi04268048-39 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: finishing write tid 5 to smithi04268048-39 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: oid 39 updating version 0 to 720 2024-09-20T19:33:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2326: oid 39 updating version 720 to 721 2024-09-20T19:33:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2326: oid 39 version 721 is already newer than 719 2024-09-20T19:33:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2326: oid 39 version 721 is already newer than 718 2024-09-20T19:33:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 721 (ObjNum 814 snap 268 seq_num 814) dirty exists 2024-09-20T19:33:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2326: left oid 39 (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:23.359 INFO:tasks.rados.rados.0.smithi042.stdout:2330: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:33:23.359 INFO:tasks.rados.rados.0.smithi042.stdout:2330: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:33:23.359 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 892 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:33:23.360 INFO:tasks.rados.rados.0.smithi042.stdout:2329: finishing write tid 1 to smithi04268048-18 2024-09-20T19:33:23.360 INFO:tasks.rados.rados.0.smithi042.stdout:2329: finishing write tid 2 to smithi04268048-18 2024-09-20T19:33:23.360 INFO:tasks.rados.rados.0.smithi042.stdout:2329: finishing write tid 3 to smithi04268048-18 2024-09-20T19:33:23.363 INFO:tasks.rados.rados.0.smithi042.stdout:2329: finishing write tid 4 to smithi04268048-18 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: finishing write tid 5 to smithi04268048-18 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: oid 18 updating version 0 to 1013 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: oid 18 version 1013 is already newer than 1010 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: oid 18 version 1013 is already newer than 1011 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: oid 18 version 1013 is already newer than 1012 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1013 (ObjNum 815 snap 268 seq_num 815) dirty exists 2024-09-20T19:33:23.364 INFO:tasks.rados.rados.0.smithi042.stdout:2329: left oid 18 (ObjNum 815 snap 268 seq_num 815) 2024-09-20T19:33:23.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.373 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.374 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.375 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:23 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.388 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:23.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:23.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:22.337880+0000 mgr.y (mgr.24491) 581 : cluster [DBG] pgmap v1038: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 211 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T19:33:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:22.358811+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-20T19:33:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:23 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.316520+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-20T19:33:23.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:23.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:23.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.776 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:23.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:23.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:24.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:24.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:24.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:24.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:24.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:24.342 INFO:teuthology.orchestra.run.smithi042.stderr:set pool 7 pg_num to 36 2024-09-20T19:33:24.342 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2322: done (8 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2323: done (7 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2324: done (6 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2325: done (5 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2326: done (4 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2327: done (3 left) 2024-09-20T19:33:24.350 INFO:tasks.rados.rados.0.smithi042.stdout:2329: done (2 left) 2024-09-20T19:33:24.351 INFO:tasks.rados.rados.0.smithi042.stdout:2330: done (1 left) 2024-09-20T19:33:24.351 INFO:tasks.rados.rados.0.smithi042.stdout:2331: done (0 left) 2024-09-20T19:33:24.351 INFO:tasks.rados.rados.0.smithi042.stdout:2332: read oid 3 snap -1 2024-09-20T19:33:24.351 INFO:tasks.rados.rados.0.smithi042.stdout:2332: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:33:24.351 INFO:tasks.rados.rados.0.smithi042.stdout:2333: snap_create 2024-09-20T19:33:24.363 INFO:tasks.rados.rados.0.smithi042.stdout:2332: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:33:24.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:24.399 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2600602 2024-09-20T19:33:24.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.412 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.413 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:24 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:24.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:24.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.234322+0000 osd.2 (osd.2) 219 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.240152+0000 osd.2 (osd.2) 220 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.367001+0000 osd.5 (osd.5) 213 : cluster [DBG] 6.a scrub starts 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.370080+0000 osd.5 (osd.5) 214 : cluster [DBG] 6.a scrub ok 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:23.421464+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.673 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:23.453877+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.479021+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:23.480442+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.338986+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:24.341366+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? v1:172.21.15.42:0/1718103094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: audit 2024-09-20T19:33:24.341533+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.42:0/2320992250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:24 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.349491+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-20T19:33:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:24.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:24.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:24.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:24.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:24.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:25.049 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.076 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:25.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:25.310 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:25.340 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:25.348 INFO:tasks.rados.rados.0.smithi042.stdout:2332: done (1 left) 2024-09-20T19:33:25.348 INFO:tasks.rados.rados.0.smithi042.stdout:2333: done (0 left) 2024-09-20T19:33:25.348 INFO:tasks.rados.rados.0.smithi042.stdout:2334: read oid 30 snap -1 2024-09-20T19:33:25.348 INFO:tasks.rados.rados.0.smithi042.stdout:2334: expect deleted 2024-09-20T19:33:25.348 INFO:tasks.rados.rados.0.smithi042.stdout:2335: snap_create 2024-09-20T19:33:25.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:25 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:25 smithi162 bash[24557]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:25 smithi162 bash[24557]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:25 smithi162 bash[24557]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.472 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:25.575 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[32075]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[32075]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[32075]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[22706]: cluster 2024-09-20T19:33:24.338740+0000 mgr.y (mgr.24491) 582 : cluster [DBG] pgmap v1041: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[22706]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:25 smithi042 bash[22706]: cluster 2024-09-20T19:33:25.347274+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-20T19:33:25.603 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:25.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.734 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:25.836 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.862 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:25.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:25.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:25.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:26.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:26.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:26.256 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:26.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:2334: done (1 left) 2024-09-20T19:33:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:2335: done (0 left) 2024-09-20T19:33:26.363 INFO:tasks.rados.rados.0.smithi042.stdout:2336: delete oid 8 current snap is 271 2024-09-20T19:33:26.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:26.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:26.512 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[32075]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:26 smithi042 bash[22706]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.524 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:26.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: audit 2024-09-20T19:33:26.340821+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: audit 2024-09-20T19:33:26.345639+0000 mon.a (mon.0) 2151 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:26 smithi162 bash[24557]: cluster 2024-09-20T19:33:26.353655+0000 mon.a (mon.0) 2152 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-20T19:33:26.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:26.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:26.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:26.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:26.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:27.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:27.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:27.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:27.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:27.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:27.402 INFO:tasks.rados.rados.0.smithi042.stdout:2336: done (0 left) 2024-09-20T19:33:27.402 INFO:tasks.rados.rados.0.smithi042.stdout:2337: snap_create 2024-09-20T19:33:27.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:27.429 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:27.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.544 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:27 smithi162 bash[24557]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:27 smithi162 bash[24557]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:27 smithi162 bash[24557]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:27 smithi162 bash[24557]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:27.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[32075]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[32075]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[32075]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[32075]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[22706]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[22706]: cluster 2024-09-20T19:33:26.339497+0000 mgr.y (mgr.24491) 583 : cluster [DBG] pgmap v1044: 168 pgs: 9 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 151 active+clean; 223 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[22706]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:27 smithi042 bash[22706]: cluster 2024-09-20T19:33:27.353187+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-20T19:33:27.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:27.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:27.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:27.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:28.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:28.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:28.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:28.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2337: done (0 left) 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2338: read oid 9 snap -1 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2338: expect deleted 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2339: copy_from oid 31 from oid 33 current snap is 272 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2340: copy_from oid 27 from oid 30 current snap is 272 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2341: write (excl) oid 42 current snap is 272 2024-09-20T19:33:28.326 INFO:tasks.rados.rados.0.smithi042.stdout:2341: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2024-09-20T19:33:28.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:28.345 INFO:tasks.rados.rados.0.smithi042.stdout:2341: writing smithi04268048-42 from 531319 to 1259523 tid 1 2024-09-20T19:33:28.362 INFO:tasks.rados.rados.0.smithi042.stdout:2341: writing smithi04268048-42 from 1758965 to 2243573 tid 2 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2341: writing smithi04268048-42 from 2652938 to 3172996 tid 3 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2342: rollback oid 35 current snap is 272 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 271 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2339: finishing copy_from to smithi04268048-31 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2339: got expected ENOENT (src dne) 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2339: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2340: finishing copy_from to smithi04268048-27 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2340: got expected ENOENT (src dne) 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2340: finishing copy_from racing read to smithi04268048-27 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2341: finishing write tid 1 to smithi04268048-42 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2341: finishing write tid 2 to smithi04268048-42 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2338: done (4 left) 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2339: done (3 left) 2024-09-20T19:33:28.377 INFO:tasks.rados.rados.0.smithi042.stdout:2340: done (2 left) 2024-09-20T19:33:28.378 INFO:tasks.rados.rados.0.smithi042.stdout:2343: read oid 36 snap -1 2024-09-20T19:33:28.378 INFO:tasks.rados.rados.0.smithi042.stdout:2343: expect (ObjNum 797 snap 262 seq_num 797) 2024-09-20T19:33:28.378 INFO:tasks.rados.rados.0.smithi042.stdout:2344: snap_remove snap 257 2024-09-20T19:33:28.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:33:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:28.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:28.592 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:28.692 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:33:28.692 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:33:28.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.710 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:33:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:33:28.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:28.727 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:33:28.727 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:33:28.744 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:33:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:33:28.761 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:33:28.761 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:33:28.779 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:33:28.779 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:33:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:33:28.799 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:33:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:33:28.821 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:33:28.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:28.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:28.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:29.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:29.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[32075]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:29 smithi042 bash[22706]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:29.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.028749+0000 mon.a (mon.0) 2154 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:33:29.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.030975+0000 mon.a (mon.0) 2155 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:29.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: cluster 2024-09-20T19:33:28.319421+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: cluster 2024-09-20T19:33:28.340161+0000 mgr.y (mgr.24491) 584 : cluster [DBG] pgmap v1048: 178 pgs: 6 unknown, 5 peering, 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 206 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.341992+0000 mon.a (mon.0) 2157 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[24557]: audit 2024-09-20T19:33:28.672928+0000 mon.a (mon.0) 2158 : audit [DBG] from='client.? v1:172.21.15.42:0/540671081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:29.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2344: done (3 left) 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2345: read oid 7 snap 266 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2345: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2346: write oid 27 current snap is 272 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:2342: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:33:29.332 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: finishing write tid 3 to smithi04268048-42 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: finishing write tid 4 to smithi04268048-42 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: finishing write tid 5 to smithi04268048-42 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: oid 42 updating version 0 to 997 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: oid 42 updating version 997 to 998 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: oid 42 version 998 is already newer than 996 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: oid 42 version 998 is already newer than 995 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 998 (ObjNum 817 snap 272 seq_num 817) dirty exists 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2341: left oid 42 (ObjNum 817 snap 272 seq_num 817) 2024-09-20T19:33:29.333 INFO:tasks.rados.rados.0.smithi042.stdout:2343: expect (ObjNum 797 snap 262 seq_num 797) 2024-09-20T19:33:29.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:29.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:29.430 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2365504 2024-09-20T19:33:29.454 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:29.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:29.546 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1365297 2024-09-20T19:33:29.591 INFO:tasks.rados.rados.0.smithi042.stdout:2345: expect (ObjNum 631 snap 198 seq_num 631) 2024-09-20T19:33:29.596 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.596 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 848323 2024-09-20T19:33:29.622 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:29.651 INFO:tasks.rados.rados.0.smithi042.stdout:2346: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-20T19:33:29.657 INFO:tasks.rados.rados.0.smithi042.stdout:2346: writing smithi04268048-27 from 548126 to 951579 tid 1 2024-09-20T19:33:29.664 INFO:tasks.rados.rados.0.smithi042.stdout:2346: writing smithi04268048-27 from 1417447 to 2162066 tid 2 2024-09-20T19:33:29.669 INFO:tasks.rados.rados.0.smithi042.stdout:2346: writing smithi04268048-27 from 2680361 to 3128044 tid 3 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2346: writing smithi04268048-27 from 3680450 to 3748126 tid 4 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2341: done (4 left) 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2342: done (3 left) 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2343: done (2 left) 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2345: done (1 left) 2024-09-20T19:33:29.673 INFO:tasks.rados.rados.0.smithi042.stdout:2347: copy_from oid 8 from oid 48 current snap is 272 2024-09-20T19:33:29.674 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 1 to smithi04268048-27 2024-09-20T19:33:29.674 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 2 to smithi04268048-27 2024-09-20T19:33:29.674 INFO:tasks.rados.rados.0.smithi042.stdout:2348: read oid 22 snap 249 2024-09-20T19:33:29.674 INFO:tasks.rados.rados.0.smithi042.stdout:2348: expect deleted 2024-09-20T19:33:29.674 INFO:tasks.rados.rados.0.smithi042.stdout:2349: delete oid 50 current snap is 272 2024-09-20T19:33:29.677 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 3 to smithi04268048-27 2024-09-20T19:33:29.677 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 4 to smithi04268048-27 2024-09-20T19:33:29.678 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 5 to smithi04268048-27 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: finishing write tid 6 to smithi04268048-27 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: oid 27 updating version 0 to 1002 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: oid 27 updating version 1002 to 1003 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: oid 27 version 1003 is already newer than 1000 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: oid 27 version 1003 is already newer than 1001 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: oid 27 version 1003 is already newer than 999 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1003 (ObjNum 818 snap 272 seq_num 818) dirty exists 2024-09-20T19:33:29.679 INFO:tasks.rados.rados.0.smithi042.stdout:2346: left oid 27 (ObjNum 818 snap 272 seq_num 818) 2024-09-20T19:33:29.681 INFO:tasks.rados.rados.0.smithi042.stdout:2346: done (3 left) 2024-09-20T19:33:29.681 INFO:tasks.rados.rados.0.smithi042.stdout:2349: done (2 left) 2024-09-20T19:33:29.681 INFO:tasks.rados.rados.0.smithi042.stdout:2350: rollback oid 47 current snap is 272 2024-09-20T19:33:29.681 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 47 to 270 2024-09-20T19:33:29.681 INFO:tasks.rados.rados.0.smithi042.stdout:2351: delete oid 16 current snap is 272 2024-09-20T19:33:29.687 INFO:tasks.rados.rados.0.smithi042.stdout:2350: finishing rollback tid 0 to smithi04268048-47 2024-09-20T19:33:29.688 INFO:tasks.rados.rados.0.smithi042.stdout:2350: finishing rollback tid 1 to smithi04268048-47 2024-09-20T19:33:29.688 INFO:tasks.rados.rados.0.smithi042.stdout:2350: finishing rollback tid 2 to smithi04268048-47 2024-09-20T19:33:29.688 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1007 (ObjNum 787 snap 260 seq_num 787) dirty exists 2024-09-20T19:33:29.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:29.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:29.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:29.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:29.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:29.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:29.993 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:30.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:30.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:30.140 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:30.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:30.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:30.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:30.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:30.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:30.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:30.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:30 smithi162 bash[24557]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:30.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[32075]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: audit 2024-09-20T19:33:29.320604+0000 mon.a (mon.0) 2159 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: cluster 2024-09-20T19:33:29.327446+0000 mon.a (mon.0) 2160 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-20T19:33:30.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:30 smithi042 bash[22706]: audit 2024-09-20T19:33:29.652064+0000 mgr.y (mgr.24491) 585 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:30.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:30.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:30.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:30.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:30.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:31.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:31.054 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:31.157 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.186 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:31.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:31.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:31.315 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:31.403 INFO:tasks.rados.rados.0.smithi042.stdout:2347: finishing copy_from to smithi04268048-8 2024-09-20T19:33:31.403 INFO:tasks.rados.rados.0.smithi042.stdout:2347: got expected ENOENT (src dne) 2024-09-20T19:33:31.403 INFO:tasks.rados.rados.0.smithi042.stdout:2347: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:33:31.415 INFO:tasks.rados.rados.0.smithi042.stdout:2347: done (3 left) 2024-09-20T19:33:31.415 INFO:tasks.rados.rados.0.smithi042.stdout:2348: done (2 left) 2024-09-20T19:33:31.415 INFO:tasks.rados.rados.0.smithi042.stdout:2350: done (1 left) 2024-09-20T19:33:31.415 INFO:tasks.rados.rados.0.smithi042.stdout:2351: done (0 left) 2024-09-20T19:33:31.416 INFO:tasks.rados.rados.0.smithi042.stdout:2352: copy_from oid 46 from oid 31 current snap is 272 2024-09-20T19:33:31.416 INFO:tasks.rados.rados.0.smithi042.stdout:2353: write (excl) oid 6 current snap is 272 2024-09-20T19:33:31.416 INFO:tasks.rados.rados.0.smithi042.stdout:2353: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-20T19:33:31.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:31.419 INFO:tasks.rados.rados.0.smithi042.stdout:2353: writing smithi04268048-6 from 598547 to 1227747 tid 1 2024-09-20T19:33:31.427 INFO:tasks.rados.rados.0.smithi042.stdout:2353: writing smithi04268048-6 from 1876540 to 2368486 tid 2 2024-09-20T19:33:31.429 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[32075]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:31 smithi042 bash[22706]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2353: writing smithi04268048-6 from 3046277 to 3727776 tid 3 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2354: read oid 39 snap -1 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2354: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2355: delete oid 17 current snap is 272 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2352: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2353: finishing write tid 1 to smithi04268048-6 2024-09-20T19:33:31.437 INFO:tasks.rados.rados.0.smithi042.stdout:2352: finishing copy_from to smithi04268048-46 2024-09-20T19:33:31.438 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 807 (ObjNum 800 snap 264 seq_num 800) dirty exists 2024-09-20T19:33:31.438 INFO:tasks.rados.rados.0.smithi042.stdout:2353: finishing write tid 2 to smithi04268048-6 2024-09-20T19:33:31.444 INFO:tasks.rados.rados.0.smithi042.stdout:2352: done (3 left) 2024-09-20T19:33:31.444 INFO:tasks.rados.rados.0.smithi042.stdout:2355: done (2 left) 2024-09-20T19:33:31.444 INFO:tasks.rados.rados.0.smithi042.stdout:2356: snap_create 2024-09-20T19:33:31.445 INFO:tasks.rados.rados.0.smithi042.stdout:2354: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:31.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:31.460 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:33:31.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:33:31.460 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T19:33:31.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd out 3 2024-09-20T19:33:31.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:31.566 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3587461 2024-09-20T19:33:31.569 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:31.600 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:33:31.600 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: finishing write tid 3 to smithi04268048-6 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: finishing write tid 4 to smithi04268048-6 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: finishing write tid 5 to smithi04268048-6 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: oid 6 updating version 0 to 830 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: oid 6 version 830 is already newer than 829 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: oid 6 version 830 is already newer than 827 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: oid 6 version 830 is already newer than 828 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 830 (ObjNum 821 snap 272 seq_num 821) dirty exists 2024-09-20T19:33:31.648 INFO:tasks.rados.rados.0.smithi042.stdout:2353: left oid 6 (ObjNum 821 snap 272 seq_num 821) 2024-09-20T19:33:31.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:30.332294+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:30.341082+0000 mgr.y (mgr.24491) 586 : cluster [DBG] pgmap v1051: 178 pgs: 8 unknown, 5 peering, 11 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 147 active+clean; 198 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:31 smithi162 bash[24557]: cluster 2024-09-20T19:33:31.335560+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-20T19:33:31.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:31.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:31.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:31.956 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:31.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:32.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:32.086 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:32.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:32.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:32.344 INFO:tasks.rados.rados.0.smithi042.stdout:2353: done (2 left) 2024-09-20T19:33:32.344 INFO:tasks.rados.rados.0.smithi042.stdout:2354: done (1 left) 2024-09-20T19:33:32.345 INFO:tasks.rados.rados.0.smithi042.stdout:2356: done (0 left) 2024-09-20T19:33:32.345 INFO:tasks.rados.rados.0.smithi042.stdout:2357: read oid 46 snap -1 2024-09-20T19:33:32.345 INFO:tasks.rados.rados.0.smithi042.stdout:2357: expect (ObjNum 800 snap 264 seq_num 800) 2024-09-20T19:33:32.345 INFO:tasks.rados.rados.0.smithi042.stdout:2358: write oid 41 current snap is 273 2024-09-20T19:33:32.345 INFO:tasks.rados.rados.0.smithi042.stdout:2358: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-20T19:33:32.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:32.357 INFO:tasks.rados.rados.0.smithi042.stdout:2358: writing smithi04268048-41 from 632161 to 1128212 tid 1 2024-09-20T19:33:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:32.369 INFO:tasks.rados.rados.0.smithi042.stdout:2358: writing smithi04268048-41 from 1593504 to 2089119 tid 2 2024-09-20T19:33:32.380 INFO:tasks.rados.rados.0.smithi042.stdout:2358: writing smithi04268048-41 from 2701123 to 3121519 tid 3 2024-09-20T19:33:32.387 INFO:tasks.rados.rados.0.smithi042.stdout:2358: writing smithi04268048-41 from 3611645 to 3832161 tid 4 2024-09-20T19:33:32.388 INFO:tasks.rados.rados.0.smithi042.stdout:2359: read oid 8 snap 271 2024-09-20T19:33:32.388 INFO:tasks.rados.rados.0.smithi042.stdout:2359: expect deleted 2024-09-20T19:33:32.388 INFO:tasks.rados.rados.0.smithi042.stdout:2360: copy_from oid 28 from oid 31 current snap is 273 2024-09-20T19:33:32.388 INFO:tasks.rados.rados.0.smithi042.stdout:2357: expect (ObjNum 800 snap 264 seq_num 800) 2024-09-20T19:33:32.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:32.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:32.538 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2494241 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 1 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 2 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 3 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 4 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 5 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: finishing write tid 6 to smithi04268048-41 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: oid 41 updating version 0 to 725 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: oid 41 updating version 725 to 726 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: oid 41 version 726 is already newer than 723 2024-09-20T19:33:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:2358: oid 41 version 726 is already newer than 722 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:2358: oid 41 version 726 is already newer than 724 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 726 (ObjNum 823 snap 273 seq_num 823) dirty exists 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:2358: left oid 41 (ObjNum 823 snap 273 seq_num 823) 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:2357: done (3 left) 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:2358: done (2 left) 2024-09-20T19:33:32.566 INFO:tasks.rados.rados.0.smithi042.stdout:2361: snap_remove snap 250 2024-09-20T19:33:32.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:32.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:33:32.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:33:32.715 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:32.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:32.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:32.980 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:33.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:33.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:33.330 INFO:tasks.rados.rados.0.smithi042.stdout:2361: done (2 left) 2024-09-20T19:33:33.330 INFO:tasks.rados.rados.0.smithi042.stdout:2362: write (excl) oid 7 current snap is 273 2024-09-20T19:33:33.330 INFO:tasks.rados.rados.0.smithi042.stdout:2362: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2024-09-20T19:33:33.330 INFO:tasks.rados.rados.0.smithi042.stdout:2362: writing smithi04268048-7 from 648968 to 1220268 tid 1 2024-09-20T19:33:33.335 INFO:tasks.rados.rados.0.smithi042.stdout:2362: writing smithi04268048-7 from 1651986 to 2291259 tid 2 2024-09-20T19:33:33.343 INFO:tasks.rados.rados.0.smithi042.stdout:2362: writing smithi04268048-7 from 2728546 to 3360214 tid 3 2024-09-20T19:33:33.343 INFO:tasks.rados.rados.0.smithi042.stdout:2363: snap_remove snap 269 2024-09-20T19:33:33.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.390 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:33.492 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.504 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.504 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.504 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.504 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[32075]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:33 smithi042 bash[22706]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:33.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.651 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:33.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:33.765 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.765 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:32.340505+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-20T19:33:33.765 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.765 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:32.344025+0000 mgr.y (mgr.24491) 587 : cluster [DBG] pgmap v1054: 178 pgs: 5 active+undersized+remapped, 3 unknown, 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 152 active+clean; 203 MiB data, 857 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s; 3.2 MiB/s, 16 objects/s recovering 2024-09-20T19:33:33.765 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:33 smithi162 bash[24557]: cluster 2024-09-20T19:33:33.317786+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-20T19:33:33.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:33.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:33.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:34.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:34.044 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:34.146 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:34.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:34.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:34.355 INFO:tasks.rados.rados.0.smithi042.stdout:2363: done (3 left) 2024-09-20T19:33:34.355 INFO:tasks.rados.rados.0.smithi042.stdout:2364: delete oid 16 current snap is 273 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: finishing write tid 1 to smithi04268048-7 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: finishing write tid 2 to smithi04268048-7 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: finishing write tid 3 to smithi04268048-7 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: finishing write tid 4 to smithi04268048-7 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: finishing write tid 5 to smithi04268048-7 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: oid 7 updating version 0 to 870 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: oid 7 updating version 870 to 871 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: oid 7 updating version 871 to 872 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: oid 7 version 872 is already newer than 869 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 872 (ObjNum 824 snap 273 seq_num 824) dirty exists 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2362: left oid 7 (ObjNum 824 snap 273 seq_num 824) 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2360: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:33:34.356 INFO:tasks.rados.rados.0.smithi042.stdout:2360: finishing copy_from to smithi04268048-28 2024-09-20T19:33:34.357 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 807 (ObjNum 800 snap 264 seq_num 800) dirty exists 2024-09-20T19:33:34.367 INFO:tasks.rados.rados.0.smithi042.stdout:2359: done (3 left) 2024-09-20T19:33:34.367 INFO:tasks.rados.rados.0.smithi042.stdout:2360: done (2 left) 2024-09-20T19:33:34.367 INFO:tasks.rados.rados.0.smithi042.stdout:2362: done (1 left) 2024-09-20T19:33:34.367 INFO:tasks.rados.rados.0.smithi042.stdout:2364: done (0 left) 2024-09-20T19:33:34.367 INFO:tasks.rados.rados.0.smithi042.stdout:2365: snap_remove snap 251 2024-09-20T19:33:34.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:34.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:34.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:34.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:34.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:34.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:34.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:34.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:34.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:35.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:35.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:35.178 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:35.185 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:35.207 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:35.310 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:35.360 INFO:tasks.rados.rados.0.smithi042.stdout:2365: done (0 left) 2024-09-20T19:33:35.361 INFO:tasks.rados.rados.0.smithi042.stdout:2366: rollback oid 34 current snap is 273 2024-09-20T19:33:35.361 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 271 2024-09-20T19:33:35.361 INFO:tasks.rados.rados.0.smithi042.stdout:2367: write (excl) oid 32 current snap is 273 2024-09-20T19:33:35.361 INFO:tasks.rados.rados.0.smithi042.stdout:2367: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-09-20T19:33:35.369 INFO:tasks.rados.rados.0.smithi042.stdout:2367: writing smithi04268048-32 from 682582 to 1404380 tid 1 2024-09-20T19:33:35.377 INFO:tasks.rados.rados.0.smithi042.stdout:2367: writing smithi04268048-32 from 2168950 to 2811892 tid 2 2024-09-20T19:33:35.384 INFO:tasks.rados.rados.0.smithi042.stdout:2367: writing smithi04268048-32 from 3583392 to 3882582 tid 3 2024-09-20T19:33:35.384 INFO:tasks.rados.rados.0.smithi042.stdout:2368: write oid 14 current snap is 273 2024-09-20T19:33:35.384 INFO:tasks.rados.rados.0.smithi042.stdout:2368: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-20T19:33:35.393 INFO:tasks.rados.rados.0.smithi042.stdout:2368: writing smithi04268048-14 from 699389 to 1496436 tid 1 2024-09-20T19:33:35.399 INFO:tasks.rados.rados.0.smithi042.stdout:2368: writing smithi04268048-14 from 2111079 to 2614032 tid 2 2024-09-20T19:33:35.406 INFO:tasks.rados.rados.0.smithi042.stdout:2368: writing smithi04268048-14 from 3094462 to 3676299 tid 3 2024-09-20T19:33:35.406 INFO:tasks.rados.rados.0.smithi042.stdout:2369: copy_from oid 28 from oid 1 current snap is 273 2024-09-20T19:33:35.406 INFO:tasks.rados.rados.0.smithi042.stdout:2366: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:33:35.406 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1053 (ObjNum 129 snap 0 seq_num 1415942672) dirty dne 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: finishing write tid 1 to smithi04268048-32 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: finishing write tid 2 to smithi04268048-32 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: finishing write tid 3 to smithi04268048-32 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: finishing write tid 4 to smithi04268048-32 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: finishing write tid 5 to smithi04268048-32 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: oid 32 updating version 0 to 1234 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: oid 32 updating version 1234 to 1237 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: oid 32 version 1237 is already newer than 1235 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: oid 32 version 1237 is already newer than 1236 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1237 (ObjNum 826 snap 273 seq_num 826) dirty exists 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: left oid 32 (ObjNum 826 snap 273 seq_num 826) 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2366: done (3 left) 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2367: done (2 left) 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2370: copy_from oid 21 from oid 24 current snap is 273 2024-09-20T19:33:35.407 INFO:tasks.rados.rados.0.smithi042.stdout:2371: snap_remove snap 271 2024-09-20T19:33:35.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:35.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:35.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:35.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[22706]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[22706]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[22706]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[22706]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[32075]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[32075]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[32075]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:35 smithi042 bash[32075]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:35.707 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:35 smithi162 bash[24557]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.707 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:35 smithi162 bash[24557]: cluster 2024-09-20T19:33:34.344959+0000 mgr.y (mgr.24491) 588 : cluster [DBG] pgmap v1056: 178 pgs: 4 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 204 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 9.6 MiB/s wr, 64 op/s; 6.1 MiB/s, 32 objects/s recovering 2024-09-20T19:33:35.707 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:35 smithi162 bash[24557]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.707 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:35 smithi162 bash[24557]: cluster 2024-09-20T19:33:34.351926+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-20T19:33:35.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:35.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:35.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:35.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:35.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:35.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:36.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:36.113 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:36.207 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:36.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:36.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:36.378 INFO:tasks.rados.rados.0.smithi042.stdout:2371: done (3 left) 2024-09-20T19:33:36.378 INFO:tasks.rados.rados.0.smithi042.stdout:2372: snap_create 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2370: finishing copy_from to smithi04268048-21 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2370: got expected ENOENT (src dne) 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2370: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: finishing write tid 1 to smithi04268048-14 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: finishing write tid 2 to smithi04268048-14 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: finishing write tid 3 to smithi04268048-14 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: finishing write tid 4 to smithi04268048-14 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: finishing write tid 5 to smithi04268048-14 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: oid 14 updating version 0 to 810 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: oid 14 version 810 is already newer than 808 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: oid 14 version 810 is already newer than 807 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: oid 14 version 810 is already newer than 809 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 810 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:33:36.379 INFO:tasks.rados.rados.0.smithi042.stdout:2368: left oid 14 (ObjNum 827 snap 273 seq_num 827) 2024-09-20T19:33:36.380 INFO:tasks.rados.rados.0.smithi042.stdout:2369: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:33:36.380 INFO:tasks.rados.rados.0.smithi042.stdout:2369: finishing copy_from to smithi04268048-28 2024-09-20T19:33:36.380 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 707 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:33:36.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:36.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:36.505 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:36.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:36 smithi042 bash[32075]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:36 smithi042 bash[32075]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:36 smithi042 bash[22706]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:36 smithi042 bash[22706]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:36.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:36 smithi162 bash[24557]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:36 smithi162 bash[24557]: cluster 2024-09-20T19:33:35.359523+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-20T19:33:36.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:36.737 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:36.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:36.893 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:36.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:36.996 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:37.024 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:37.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:37.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:37.227 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:37.228 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:37.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:37.373 INFO:tasks.rados.rados.0.smithi042.stdout:2368: done (3 left) 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2369: done (2 left) 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2370: done (1 left) 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2372: done (0 left) 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2373: read oid 3 snap 273 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2373: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:33:37.374 INFO:tasks.rados.rados.0.smithi042.stdout:2374: snap_create 2024-09-20T19:33:37.383 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:37.388 INFO:tasks.rados.rados.0.smithi042.stdout:2373: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:33:37.412 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:37.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:37.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:37 smithi162 bash[24557]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[32075]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.003392+0000 osd.6 (osd.6) 154 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.004366+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.345948+0000 mgr.y (mgr.24491) 589 : cluster [DBG] pgmap v1059: 178 pgs: 2 active+undersized+remapped, 2 unknown, 2 peering, 7 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 209 MiB data, 869 MiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 2.9 MiB/s wr, 14 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:37 smithi042 bash[22706]: cluster 2024-09-20T19:33:36.372692+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-20T19:33:37.776 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:37.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:37.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:37.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:37.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:38.039 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:38.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:38.303 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:38.375 INFO:tasks.rados.rados.0.smithi042.stdout:2373: done (1 left) 2024-09-20T19:33:38.375 INFO:tasks.rados.rados.0.smithi042.stdout:2374: done (0 left) 2024-09-20T19:33:38.375 INFO:tasks.rados.rados.0.smithi042.stdout:2375: delete oid 44 current snap is 275 2024-09-20T19:33:38.412 INFO:tasks.rados.rados.0.smithi042.stdout:2375: done (0 left) 2024-09-20T19:33:38.413 INFO:tasks.rados.rados.0.smithi042.stdout:2376: delete oid 27 current snap is 275 2024-09-20T19:33:38.423 INFO:tasks.rados.rados.0.smithi042.stdout:2376: done (0 left) 2024-09-20T19:33:38.423 INFO:tasks.rados.rados.0.smithi042.stdout:2377: copy_from oid 33 from oid 41 current snap is 275 2024-09-20T19:33:38.423 INFO:tasks.rados.rados.0.smithi042.stdout:2378: snap_remove snap 260 2024-09-20T19:33:38.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:38.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:38.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:38.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:38.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[22706]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.674 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.675 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:38 smithi042 bash[32075]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.013675+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.014580+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:37.371112+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.703 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:38 smithi162 bash[24557]: cluster 2024-09-20T19:33:38.374594+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-20T19:33:38.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:38.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:38.845 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:38.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:38.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:38.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:39.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:39.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:39.234 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:39.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:39.362 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:39.390 INFO:tasks.rados.rados.0.smithi042.stdout:2378: done (1 left) 2024-09-20T19:33:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:2379: read oid 39 snap -1 2024-09-20T19:33:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:2379: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:2380: write (excl) oid 4 current snap is 275 2024-09-20T19:33:39.391 INFO:tasks.rados.rados.0.smithi042.stdout:2380: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-20T19:33:39.391 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:33:39.399 INFO:tasks.rados.rados.0.smithi042.stdout:2380: writing smithi04268048-4 from 749810 to 1488957 tid 1 2024-09-20T19:33:39.406 INFO:tasks.rados.rados.0.smithi042.stdout:2380: writing smithi04268048-4 from 2002878 to 2653158 tid 2 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:2380: writing smithi04268048-4 from 3293084 to 3708737 tid 3 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:2381: rollback oid 30 current snap is 275 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 268 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:2377: finishing copy_from racing read to smithi04268048-33 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:2377: finishing copy_from to smithi04268048-33 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 726 (ObjNum 823 snap 273 seq_num 823) dirty exists 2024-09-20T19:33:39.413 INFO:tasks.rados.rados.0.smithi042.stdout:2379: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:39.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:39.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:39.545 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1057665 2024-09-20T19:33:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[24557]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: finishing write tid 1 to smithi04268048-4 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2381: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 955 (ObjNum 225 snap 0 seq_num 1415374880) dirty dne 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: finishing write tid 2 to smithi04268048-4 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: finishing write tid 3 to smithi04268048-4 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: finishing write tid 4 to smithi04268048-4 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: finishing write tid 5 to smithi04268048-4 2024-09-20T19:33:39.585 INFO:tasks.rados.rados.0.smithi042.stdout:2380: oid 4 updating version 0 to 1018 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2380: oid 4 updating version 1018 to 1019 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2380: oid 4 version 1019 is already newer than 1017 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2380: oid 4 updating version 1019 to 1020 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1020 (ObjNum 830 snap 275 seq_num 830) dirty exists 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2380: left oid 4 (ObjNum 830 snap 275 seq_num 830) 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2377: done (3 left) 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2379: done (2 left) 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2380: done (1 left) 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2381: done (0 left) 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2382: rollback oid 21 current snap is 275 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 272 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2383: write (excl) oid 11 current snap is 275 2024-09-20T19:33:39.586 INFO:tasks.rados.rados.0.smithi042.stdout:2383: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-20T19:33:39.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:39.597 INFO:tasks.rados.rados.0.smithi042.stdout:2383: writing smithi04268048-11 from 766617 to 1181013 tid 1 2024-09-20T19:33:39.609 INFO:tasks.rados.rados.0.smithi042.stdout:2383: writing smithi04268048-11 from 1945007 to 2455298 tid 2 2024-09-20T19:33:39.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:39.622 INFO:tasks.rados.rados.0.smithi042.stdout:2383: writing smithi04268048-11 from 3204154 to 3831079 tid 3 2024-09-20T19:33:39.622 INFO:tasks.rados.rados.0.smithi042.stdout:2384: write oid 25 current snap is 275 2024-09-20T19:33:39.622 INFO:tasks.rados.rados.0.smithi042.stdout:2382: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:33:39.622 INFO:tasks.rados.rados.0.smithi042.stdout:2382: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:33:39.622 INFO:tasks.rados.rados.0.smithi042.stdout:2382: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:33:39.623 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1032 (ObjNum 813 snap 268 seq_num 813) dirty exists 2024-09-20T19:33:39.623 INFO:tasks.rados.rados.0.smithi042.stdout:2383: finishing write tid 1 to smithi04268048-11 2024-09-20T19:33:39.623 INFO:tasks.rados.rados.0.smithi042.stdout:2384: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-20T19:33:39.636 INFO:tasks.rados.rados.0.smithi042.stdout:2384: writing smithi04268048-25 from 783424 to 1273069 tid 1 2024-09-20T19:33:39.651 INFO:tasks.rados.rados.0.smithi042.stdout:2384: writing smithi04268048-25 from 2003489 to 2657438 tid 2 2024-09-20T19:33:39.662 INFO:tasks.rados.rados.0.smithi042.stdout:2384: writing smithi04268048-25 from 3231577 to 3786127 tid 3 2024-09-20T19:33:39.662 INFO:tasks.rados.rados.0.smithi042.stdout:2382: done (2 left) 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:2385: rollback oid 46 current snap is 275 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 266 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:2383: finishing write tid 2 to smithi04268048-11 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:2383: finishing write tid 3 to smithi04268048-11 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:2383: finishing write tid 4 to smithi04268048-11 2024-09-20T19:33:39.663 INFO:tasks.rados.rados.0.smithi042.stdout:2383: finishing write tid 5 to smithi04268048-11 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:2383: oid 11 updating version 0 to 910 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:2383: oid 11 updating version 910 to 911 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:2383: oid 11 updating version 911 to 912 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:2383: oid 11 updating version 912 to 913 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 913 (ObjNum 831 snap 275 seq_num 831) dirty exists 2024-09-20T19:33:39.664 INFO:tasks.rados.rados.0.smithi042.stdout:2383: left oid 11 (ObjNum 831 snap 275 seq_num 831) 2024-09-20T19:33:39.665 INFO:tasks.rados.rados.0.smithi042.stdout:2384: finishing write tid 1 to smithi04268048-25 2024-09-20T19:33:39.665 INFO:tasks.rados.rados.0.smithi042.stdout:2384: finishing write tid 2 to smithi04268048-25 2024-09-20T19:33:39.665 INFO:tasks.rados.rados.0.smithi042.stdout:2383: done (2 left) 2024-09-20T19:33:39.665 INFO:tasks.rados.rados.0.smithi042.stdout:2386: snap_remove snap 262 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[22706]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: cluster 2024-09-20T19:33:38.346717+0000 mgr.y (mgr.24491) 590 : cluster [DBG] pgmap v1062: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 212 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.9 MiB/s wr, 10 op/s; 456 KiB/s, 0 objects/s recovering 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: audit 2024-09-20T19:33:38.781648+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.42:0/2496971489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:39 smithi042 bash[32075]: audit 2024-09-20T19:33:38.782234+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25679 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:33:39.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:39.742 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:39.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:39.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:39.873 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:39.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:39.976 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:40.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:40.194 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:40.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2386: done (2 left) 2024-09-20T19:33:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2387: rollback oid 1 current snap is 275 2024-09-20T19:33:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 272 2024-09-20T19:33:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2384: finishing write tid 3 to smithi04268048-25 2024-09-20T19:33:40.415 INFO:tasks.rados.rados.0.smithi042.stdout:2384: finishing write tid 4 to smithi04268048-25 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: finishing write tid 5 to smithi04268048-25 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: oid 25 updating version 0 to 874 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: oid 25 version 874 is already newer than 872 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: oid 25 version 874 is already newer than 873 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: oid 25 updating version 874 to 875 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 875 (ObjNum 832 snap 275 seq_num 832) dirty exists 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: left oid 25 (ObjNum 832 snap 275 seq_num 832) 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2385: finishing rollback tid 0 to smithi04268048-46 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2385: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1086 (ObjNum 33 snap 0 seq_num 1390893360) dirty dne 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2384: done (2 left) 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2385: done (1 left) 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2388: read oid 2 snap -1 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2388: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:33:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2389: rollback oid 5 current snap is 275 2024-09-20T19:33:40.417 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 270 2024-09-20T19:33:40.417 INFO:tasks.rados.rados.0.smithi042.stdout:2390: snap_remove snap 249 2024-09-20T19:33:40.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:40.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:40 smithi162 bash[24557]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:40.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.568 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[22706]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: cluster 2024-09-20T19:33:39.382355+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: audit 2024-09-20T19:33:39.383950+0000 mon.a (mon.0) 2172 : audit [INF] from='client.25679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: cluster 2024-09-20T19:33:39.388338+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.569 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:40 smithi042 bash[32075]: audit 2024-09-20T19:33:39.661037+0000 mgr.y (mgr.24491) 591 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:40.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:40.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:40.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:40.846 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:40.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.977 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:40.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:41.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:41.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:41.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:41.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:41.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:41.368 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:2390: done (3 left) 2024-09-20T19:33:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:2391: write oid 30 current snap is 275 2024-09-20T19:33:41.438 INFO:tasks.rados.rados.0.smithi042.stdout:2391: seq_num 833 ranges {400231=564894,1661971=338260} 2024-09-20T19:33:41.439 INFO:tasks.rados.rados.0.smithi042.stdout:2391: writing smithi04268048-30 from 400231 to 965125 tid 1 2024-09-20T19:33:41.445 INFO:tasks.rados.rados.0.smithi042.stdout:2391: writing smithi04268048-30 from 1661971 to 2000231 tid 2 2024-09-20T19:33:41.445 INFO:tasks.rados.rados.0.smithi042.stdout:2392: snap_remove snap 273 2024-09-20T19:33:41.471 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:41.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:41.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[32075]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[32075]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[32075]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.610 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[32075]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[22706]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[22706]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[22706]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.611 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:41 smithi042 bash[22706]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:41.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:41.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:41 smithi162 bash[24557]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:41 smithi162 bash[24557]: cluster 2024-09-20T19:33:40.347394+0000 mgr.y (mgr.24491) 592 : cluster [DBG] pgmap v1065: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 211 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 8 op/s; 804 KiB/s, 2 objects/s recovering 2024-09-20T19:33:41.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:41 smithi162 bash[24557]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:41 smithi162 bash[24557]: cluster 2024-09-20T19:33:40.412299+0000 mon.a (mon.0) 2174 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-20T19:33:41.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:41.898 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:41.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:42.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:42.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:42.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:42.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:42.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:42.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: debug 2024-09-20T19:33:42.215+0000 7fc23bce4640 -1 mon.a@0(leader).osd e713 definitely_dead 0 2024-09-20T19:33:42.409 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:2392: done (4 left) 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:2393: read oid 21 snap -1 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:2393: expect (ObjNum 813 snap 268 seq_num 813) 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:2394: rollback oid 40 current snap is 275 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 270 2024-09-20T19:33:42.432 INFO:tasks.rados.rados.0.smithi042.stdout:2387: finishing rollback tid 0 to smithi04268048-1 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2387: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2389: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2389: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2389: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 813 (ObjNum 808 snap 268 seq_num 808) dirty exists 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2387: finishing rollback tid 2 to smithi04268048-1 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 831 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:33:42.433 INFO:tasks.rados.rados.0.smithi042.stdout:2388: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:33:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:marked out osd.3. 2024-09-20T19:33:42.443 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 948194 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.473 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.474 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.474 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.474 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.474 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.474 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:42 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.484 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:42.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:42.541 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: finishing write tid 1 to smithi04268048-30 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: finishing write tid 2 to smithi04268048-30 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: finishing write tid 3 to smithi04268048-30 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: finishing write tid 4 to smithi04268048-30 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: oid 30 updating version 0 to 1038 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: oid 30 updating version 1038 to 1039 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: oid 30 version 1039 is already newer than 1037 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1039 (ObjNum 833 snap 275 seq_num 833) dirty exists 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2391: left oid 30 (ObjNum 833 snap 275 seq_num 833) 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2387: done (5 left) 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2388: done (4 left) 2024-09-20T19:33:42.595 INFO:tasks.rados.rados.0.smithi042.stdout:2389: done (3 left) 2024-09-20T19:33:42.596 INFO:tasks.rados.rados.0.smithi042.stdout:2391: done (2 left) 2024-09-20T19:33:42.596 INFO:tasks.rados.rados.0.smithi042.stdout:2395: copy_from oid 42 from oid 38 current snap is 275 2024-09-20T19:33:42.596 INFO:tasks.rados.rados.0.smithi042.stdout:2396: copy_from oid 16 from oid 39 current snap is 275 2024-09-20T19:33:42.596 INFO:tasks.rados.rados.0.smithi042.stdout:2397: snap_create 2024-09-20T19:33:42.611 INFO:tasks.rados.rados.0.smithi042.stdout:2395: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:33:42.612 INFO:tasks.rados.rados.0.smithi042.stdout:2395: finishing copy_from to smithi04268048-42 2024-09-20T19:33:42.613 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 616 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:42.638 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:33:42.638 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:33:42.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:42.646 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:33:42.646 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:33:42.656 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:33:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:33:42.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:42.674 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:33:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:33:42.692 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:33:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:33:42.713 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:33:42.714 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:33:42.733 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:33:42.733 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:33:42.753 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:33:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:33:42.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:42.774 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:33:42.774 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:33:42.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: cluster 2024-09-20T19:33:41.421370+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-20T19:33:42.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: audit 2024-09-20T19:33:42.220871+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.42:0/1103177433' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: audit 2024-09-20T19:33:42.221378+0000 mon.a (mon.0) 2176 : audit [INF] from='client.25685 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-20T19:33:42.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:42 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.221437+0000 mon.a (mon.0) 2177 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-09-20T19:33:42.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:42.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:42.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:43.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:43.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:43.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:43.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:43.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2395: done (4 left) 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2397: done (3 left) 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2398: read oid 27 snap -1 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2398: expect deleted 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2399: write oid 33 current snap is 276 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2399: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-20T19:33:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2399: writing smithi04268048-33 from 417038 to 1057181 tid 1 2024-09-20T19:33:43.356 INFO:tasks.rados.rados.0.smithi042.stdout:2399: writing smithi04268048-33 from 1720453 to 2017038 tid 2 2024-09-20T19:33:43.356 INFO:tasks.rados.rados.0.smithi042.stdout:2400: read oid 15 snap -1 2024-09-20T19:33:43.357 INFO:tasks.rados.rados.0.smithi042.stdout:2400: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:33:43.357 INFO:tasks.rados.rados.0.smithi042.stdout:2401: read oid 18 snap -1 2024-09-20T19:33:43.357 INFO:tasks.rados.rados.0.smithi042.stdout:2401: expect (ObjNum 815 snap 268 seq_num 815) 2024-09-20T19:33:43.357 INFO:tasks.rados.rados.0.smithi042.stdout:2402: delete oid 20 current snap is 276 2024-09-20T19:33:43.370 INFO:tasks.rados.rados.0.smithi042.stdout:2399: finishing write tid 1 to smithi04268048-33 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: finishing write tid 2 to smithi04268048-33 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: finishing write tid 3 to smithi04268048-33 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: finishing write tid 4 to smithi04268048-33 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: oid 33 updating version 0 to 728 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: oid 33 version 728 is already newer than 727 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: oid 33 updating version 728 to 729 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 729 (ObjNum 834 snap 276 seq_num 834) dirty exists 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: left oid 33 (ObjNum 834 snap 276 seq_num 834) 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2398: done (7 left) 2024-09-20T19:33:43.372 INFO:tasks.rados.rados.0.smithi042.stdout:2399: done (6 left) 2024-09-20T19:33:43.373 INFO:tasks.rados.rados.0.smithi042.stdout:2402: done (5 left) 2024-09-20T19:33:43.373 INFO:tasks.rados.rados.0.smithi042.stdout:2403: snap_remove snap 266 2024-09-20T19:33:43.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:43 smithi162 bash[24557]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:43.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:43.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[32075]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.348980+0000 mgr.y (mgr.24491) 593 : cluster [DBG] pgmap v1068: 178 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 149 active+clean; 200 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:42.423342+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:42.429160+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:42.616820+0000 mon.a (mon.0) 2180 : audit [DBG] from='client.? v1:172.21.15.42:0/3078645107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: audit 2024-09-20T19:33:43.061406+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:43 smithi042 bash[22706]: cluster 2024-09-20T19:33:43.328084+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2024-09-20T19:33:43.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:43.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:43.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:43.978 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:43.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:44.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:33:44.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.109 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:44.212 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:44.332 INFO:tasks.rados.rados.0.smithi042.stdout:2403: done (5 left) 2024-09-20T19:33:44.332 INFO:tasks.rados.rados.0.smithi042.stdout:2404: snap_remove snap 274 2024-09-20T19:33:44.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:44.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:44.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:44.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:44.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:44.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:44.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:44.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:45.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:45.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:45.291 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:45.344 INFO:tasks.rados.rados.0.smithi042.stdout:2404: done (5 left) 2024-09-20T19:33:45.344 INFO:tasks.rados.rados.0.smithi042.stdout:2405: write (excl) oid 22 current snap is 276 2024-09-20T19:33:45.345 INFO:tasks.rados.rados.0.smithi042.stdout:2405: seq_num 836 ranges {450652=790641,1721064=329588} 2024-09-20T19:33:45.353 INFO:tasks.rados.rados.0.smithi042.stdout:2405: writing smithi04268048-22 from 450652 to 1241293 tid 1 2024-09-20T19:33:45.358 INFO:tasks.rados.rados.0.smithi042.stdout:2405: writing smithi04268048-22 from 1721064 to 2050652 tid 2 2024-09-20T19:33:45.359 INFO:tasks.rados.rados.0.smithi042.stdout:2406: copy_from oid 26 from oid 45 current snap is 276 2024-09-20T19:33:45.359 INFO:tasks.rados.rados.0.smithi042.stdout:2393: expect (ObjNum 813 snap 268 seq_num 813) 2024-09-20T19:33:45.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:45.416 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:45.475 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2836623 2024-09-20T19:33:45.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:45.528 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[32075]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[32075]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[32075]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[32075]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[22706]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[22706]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[22706]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.529 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:45 smithi042 bash[22706]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.536 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2860550 2024-09-20T19:33:45.542 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:45.556 INFO:tasks.rados.rados.0.smithi042.stdout:2401: expect (ObjNum 815 snap 268 seq_num 815) 2024-09-20T19:33:45.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:33:45.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:45.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:45 smithi162 bash[24557]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:45 smithi162 bash[24557]: cluster 2024-09-20T19:33:44.325400+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2024-09-20T19:33:45.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:45 smithi162 bash[24557]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:45 smithi162 bash[24557]: cluster 2024-09-20T19:33:44.350257+0000 mgr.y (mgr.24491) 594 : cluster [DBG] pgmap v1072: 178 pgs: 14 peering, 14 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 138 active+clean; 195 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-20T19:33:45.673 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:45.675 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3651276 2024-09-20T19:33:45.748 INFO:tasks.rados.rados.0.smithi042.stdout:2394: finishing rollback tid 0 to smithi04268048-40 2024-09-20T19:33:45.748 INFO:tasks.rados.rados.0.smithi042.stdout:2396: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:33:45.748 INFO:tasks.rados.rados.0.smithi042.stdout:2400: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:33:45.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:45.801 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:45.851 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:45.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2396: finishing copy_from to smithi04268048-16 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 721 (ObjNum 814 snap 268 seq_num 814) dirty exists 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2394: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2394: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 630 (ObjNum 805 snap 265 seq_num 805) dirty exists 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2393: done (6 left) 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2394: done (5 left) 2024-09-20T19:33:45.906 INFO:tasks.rados.rados.0.smithi042.stdout:2396: done (4 left) 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2400: done (3 left) 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2401: done (2 left) 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2407: copy_from oid 31 from oid 42 current snap is 276 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: finishing write tid 1 to smithi04268048-22 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: finishing write tid 2 to smithi04268048-22 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: finishing write tid 3 to smithi04268048-22 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: finishing write tid 4 to smithi04268048-22 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: oid 22 updating version 0 to 1016 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: oid 22 updating version 1016 to 1017 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: oid 22 version 1017 is already newer than 1015 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1017 (ObjNum 836 snap 276 seq_num 836) dirty exists 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2405: left oid 22 (ObjNum 836 snap 276 seq_num 836) 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2406: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:2406: finishing copy_from to smithi04268048-26 2024-09-20T19:33:45.907 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1194 (ObjNum 795 snap 262 seq_num 795) dirty exists 2024-09-20T19:33:45.908 INFO:tasks.rados.rados.0.smithi042.stdout:2405: done (2 left) 2024-09-20T19:33:45.908 INFO:tasks.rados.rados.0.smithi042.stdout:2406: done (1 left) 2024-09-20T19:33:45.908 INFO:tasks.rados.rados.0.smithi042.stdout:2408: snap_remove snap 272 2024-09-20T19:33:45.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:45.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:46.033 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:46.058 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:46.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:46.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:46.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:46.333 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:46.359 INFO:tasks.rados.rados.0.smithi042.stdout:2408: done (1 left) 2024-09-20T19:33:46.359 INFO:tasks.rados.rados.0.smithi042.stdout:2409: copy_from oid 10 from oid 9 current snap is 276 2024-09-20T19:33:46.359 INFO:tasks.rados.rados.0.smithi042.stdout:2410: read oid 6 snap 268 2024-09-20T19:33:46.359 INFO:tasks.rados.rados.0.smithi042.stdout:2410: expect deleted 2024-09-20T19:33:46.359 INFO:tasks.rados.rados.0.smithi042.stdout:2411: snap_remove snap 275 2024-09-20T19:33:46.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:46.439 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[22706]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[22706]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[22706]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[22706]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[32075]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[32075]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.441 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[32075]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.442 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:46 smithi042 bash[32075]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:46.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:46.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:46 smithi162 bash[24557]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:46 smithi162 bash[24557]: cluster 2024-09-20T19:33:45.322129+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T19:33:46.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:46 smithi162 bash[24557]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:46 smithi162 bash[24557]: cluster 2024-09-20T19:33:45.336521+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2024-09-20T19:33:46.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:46.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:46.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:46.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:46.959 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:46.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:47.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:47.191 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:47.220 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:47.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:47.350 INFO:tasks.rados.rados.0.smithi042.stdout:2411: done (3 left) 2024-09-20T19:33:47.350 INFO:tasks.rados.rados.0.smithi042.stdout:2412: write (excl) oid 20 current snap is 276 2024-09-20T19:33:47.350 INFO:tasks.rados.rados.0.smithi042.stdout:2412: seq_num 837 ranges {467459=582243,1495899=404945} 2024-09-20T19:33:47.359 INFO:tasks.rados.rados.0.smithi042.stdout:2412: writing smithi04268048-20 from 467459 to 1049702 tid 1 2024-09-20T19:33:47.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:47.364 INFO:tasks.rados.rados.0.smithi042.stdout:2412: writing smithi04268048-20 from 1495899 to 1900844 tid 2 2024-09-20T19:33:47.364 INFO:tasks.rados.rados.0.smithi042.stdout:2413: write (excl) oid 36 current snap is 276 2024-09-20T19:33:47.364 INFO:tasks.rados.rados.0.smithi042.stdout:2413: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-20T19:33:47.371 INFO:tasks.rados.rados.0.smithi042.stdout:2413: writing smithi04268048-36 from 484266 to 1141758 tid 1 2024-09-20T19:33:47.376 INFO:tasks.rados.rados.0.smithi042.stdout:2413: writing smithi04268048-36 from 1554381 to 2084266 tid 2 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2414: delete oid 28 current snap is 276 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2409: finishing copy_from to smithi04268048-10 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2409: got expected ENOENT (src dne) 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2409: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2407: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2407: finishing copy_from to smithi04268048-31 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 616 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: finishing write tid 1 to smithi04268048-20 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: finishing write tid 2 to smithi04268048-20 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: finishing write tid 3 to smithi04268048-20 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: finishing write tid 4 to smithi04268048-20 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: oid 20 updating version 0 to 990 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: oid 20 updating version 990 to 991 2024-09-20T19:33:47.377 INFO:tasks.rados.rados.0.smithi042.stdout:2412: oid 20 updating version 991 to 992 2024-09-20T19:33:47.378 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 992 (ObjNum 837 snap 276 seq_num 837) dirty exists 2024-09-20T19:33:47.378 INFO:tasks.rados.rados.0.smithi042.stdout:2412: left oid 20 (ObjNum 837 snap 276 seq_num 837) 2024-09-20T19:33:47.388 INFO:tasks.rados.rados.0.smithi042.stdout:2413: finishing write tid 1 to smithi04268048-36 2024-09-20T19:33:47.388 INFO:tasks.rados.rados.0.smithi042.stdout:2413: finishing write tid 2 to smithi04268048-36 2024-09-20T19:33:47.389 INFO:tasks.rados.rados.0.smithi042.stdout:2413: finishing write tid 3 to smithi04268048-36 2024-09-20T19:33:47.389 INFO:tasks.rados.rados.0.smithi042.stdout:2413: finishing write tid 4 to smithi04268048-36 2024-09-20T19:33:47.390 INFO:tasks.rados.rados.0.smithi042.stdout:2413: oid 36 updating version 0 to 584 2024-09-20T19:33:47.390 INFO:tasks.rados.rados.0.smithi042.stdout:2413: oid 36 version 584 is already newer than 583 2024-09-20T19:33:47.390 INFO:tasks.rados.rados.0.smithi042.stdout:2413: oid 36 version 584 is already newer than 582 2024-09-20T19:33:47.390 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 584 (ObjNum 838 snap 276 seq_num 838) dirty exists 2024-09-20T19:33:47.390 INFO:tasks.rados.rados.0.smithi042.stdout:2413: left oid 36 (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2407: done (5 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2409: done (4 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2410: done (3 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2412: done (2 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2413: done (1 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2414: done (0 left) 2024-09-20T19:33:47.393 INFO:tasks.rados.rados.0.smithi042.stdout:2415: delete oid 41 current snap is 276 2024-09-20T19:33:47.396 INFO:tasks.rados.rados.0.smithi042.stdout:2415: done (0 left) 2024-09-20T19:33:47.397 INFO:tasks.rados.rados.0.smithi042.stdout:2416: delete oid 6 current snap is 276 2024-09-20T19:33:47.399 INFO:tasks.rados.rados.0.smithi042.stdout:2416: done (0 left) 2024-09-20T19:33:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:2417: write oid 19 current snap is 276 2024-09-20T19:33:47.400 INFO:tasks.rados.rados.0.smithi042.stdout:2417: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-20T19:33:47.406 INFO:tasks.rados.rados.0.smithi042.stdout:2417: writing smithi04268048-19 from 551494 to 1109982 tid 1 2024-09-20T19:33:47.413 INFO:tasks.rados.rados.0.smithi042.stdout:2417: writing smithi04268048-19 from 1671956 to 2151494 tid 2 2024-09-20T19:33:47.413 INFO:tasks.rados.rados.0.smithi042.stdout:2418: write (excl) oid 26 current snap is 276 2024-09-20T19:33:47.413 INFO:tasks.rados.rados.0.smithi042.stdout:2417: finishing write tid 1 to smithi04268048-19 2024-09-20T19:33:47.414 INFO:tasks.rados.rados.0.smithi042.stdout:2418: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-20T19:33:47.421 INFO:tasks.rados.rados.0.smithi042.stdout:2418: writing smithi04268048-26 from 568301 to 1202038 tid 1 2024-09-20T19:33:47.428 INFO:tasks.rados.rados.0.smithi042.stdout:2418: writing smithi04268048-26 from 1614085 to 2030037 tid 2 2024-09-20T19:33:47.428 INFO:tasks.rados.rados.0.smithi042.stdout:2419: write (excl) oid 38 current snap is 276 2024-09-20T19:33:47.429 INFO:tasks.rados.rados.0.smithi042.stdout:2419: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-20T19:33:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:2419: writing smithi04268048-38 from 585108 to 1010447 tid 1 2024-09-20T19:33:47.444 INFO:tasks.rados.rados.0.smithi042.stdout:2419: writing smithi04268048-38 from 1788920 to 2185108 tid 2 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2420: copy_from oid 40 from oid 30 current snap is 276 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: finishing write tid 2 to smithi04268048-19 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: finishing write tid 3 to smithi04268048-19 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: finishing write tid 4 to smithi04268048-19 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: oid 19 updating version 0 to 811 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: oid 19 version 811 is already newer than 810 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: oid 19 version 811 is already newer than 809 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 811 (ObjNum 842 snap 276 seq_num 842) dirty exists 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2417: left oid 19 (ObjNum 842 snap 276 seq_num 842) 2024-09-20T19:33:47.445 INFO:tasks.rados.rados.0.smithi042.stdout:2418: finishing write tid 1 to smithi04268048-26 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: finishing write tid 2 to smithi04268048-26 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: finishing write tid 3 to smithi04268048-26 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: finishing write tid 4 to smithi04268048-26 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: oid 26 updating version 0 to 1196 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: oid 26 updating version 1196 to 1197 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: oid 26 version 1197 is already newer than 1195 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1197 (ObjNum 843 snap 276 seq_num 843) dirty exists 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: left oid 26 (ObjNum 843 snap 276 seq_num 843) 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2419: finishing write tid 1 to smithi04268048-38 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2417: done (3 left) 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2418: done (2 left) 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2421: write oid 19 current snap is 276 2024-09-20T19:33:47.446 INFO:tasks.rados.rados.0.smithi042.stdout:2421: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-20T19:33:47.452 INFO:tasks.rados.rados.0.smithi042.stdout:2421: writing smithi04268048-19 from 601915 to 1102503 tid 1 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2421: writing smithi04268048-19 from 1847402 to 2201915 tid 2 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2422: read oid 6 snap -1 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2422: expect deleted 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2423: copy_from oid 5 from oid 33 current snap is 276 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2419: finishing write tid 2 to smithi04268048-38 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2419: finishing write tid 3 to smithi04268048-38 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2419: finishing write tid 4 to smithi04268048-38 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2419: oid 38 updating version 0 to 834 2024-09-20T19:33:47.461 INFO:tasks.rados.rados.0.smithi042.stdout:2419: oid 38 version 834 is already newer than 833 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2419: oid 38 updating version 834 to 835 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 835 (ObjNum 844 snap 276 seq_num 844) dirty exists 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2419: left oid 38 (ObjNum 844 snap 276 seq_num 844) 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2420: finishing copy_from racing read to smithi04268048-40 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2421: finishing write tid 1 to smithi04268048-19 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2419: done (4 left) 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2424: copy_from oid 23 from oid 24 current snap is 276 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2425: snap_create 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:2420: finishing copy_from to smithi04268048-40 2024-09-20T19:33:47.462 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1039 (ObjNum 833 snap 275 seq_num 833) dirty exists 2024-09-20T19:33:47.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2424: finishing copy_from to smithi04268048-23 2024-09-20T19:33:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2424: got expected ENOENT (src dne) 2024-09-20T19:33:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:2424: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:33:47.469 INFO:tasks.rados.rados.0.smithi042.stdout:2421: finishing write tid 2 to smithi04268048-19 2024-09-20T19:33:47.472 INFO:tasks.rados.rados.0.smithi042.stdout:2421: finishing write tid 3 to smithi04268048-19 2024-09-20T19:33:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2421: finishing write tid 4 to smithi04268048-19 2024-09-20T19:33:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2421: oid 19 updating version 0 to 813 2024-09-20T19:33:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2421: oid 19 updating version 813 to 814 2024-09-20T19:33:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:2421: oid 19 version 814 is already newer than 812 2024-09-20T19:33:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 814 (ObjNum 845 snap 276 seq_num 845) dirty exists 2024-09-20T19:33:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:2421: left oid 19 (ObjNum 845 snap 276 seq_num 845) 2024-09-20T19:33:47.474 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[32075]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.474 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[32075]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.474 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[32075]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.474 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[32075]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[22706]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[22706]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[22706]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:47 smithi042 bash[22706]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.480 INFO:tasks.rados.rados.0.smithi042.stdout:2423: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:33:47.483 INFO:tasks.rados.rados.0.smithi042.stdout:2423: finishing copy_from to smithi04268048-5 2024-09-20T19:33:47.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 729 (ObjNum 834 snap 276 seq_num 834) dirty exists 2024-09-20T19:33:47.486 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:47.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:47.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:33:47.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:47.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:47 smithi162 bash[24557]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:47 smithi162 bash[24557]: cluster 2024-09-20T19:33:46.347780+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2024-09-20T19:33:47.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:47 smithi162 bash[24557]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:47 smithi162 bash[24557]: cluster 2024-09-20T19:33:46.351498+0000 mgr.y (mgr.24491) 595 : cluster [DBG] pgmap v1075: 178 pgs: 1 active+undersized+remapped, 14 peering, 14 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 135 active+clean; 191 MiB data, 736 MiB used, 625 GiB / 626 GiB avail; 260 B/s rd, 423 KiB/s wr, 2 op/s; 112 B/s, 4 objects/s recovering 2024-09-20T19:33:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:47.720 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:47.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:47.870 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:47.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:47.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:47.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:48.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:48.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:48.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:48.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:48.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2420: done (5 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2421: done (4 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2422: done (3 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2423: done (2 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2424: done (1 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2425: done (0 left) 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2426: copy_from oid 3 from oid 28 current snap is 277 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2427: write (excl) oid 18 current snap is 277 2024-09-20T19:33:48.378 INFO:tasks.rados.rados.0.smithi042.stdout:2427: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-20T19:33:48.380 INFO:tasks.rados.rados.0.smithi042.stdout:2427: writing smithi04268048-18 from 618722 to 1194559 tid 1 2024-09-20T19:33:48.387 INFO:tasks.rados.rados.0.smithi042.stdout:2427: writing smithi04268048-18 from 1905884 to 2218722 tid 2 2024-09-20T19:33:48.387 INFO:tasks.rados.rados.0.smithi042.stdout:2428: read oid 39 snap -1 2024-09-20T19:33:48.387 INFO:tasks.rados.rados.0.smithi042.stdout:2428: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:48.387 INFO:tasks.rados.rados.0.smithi042.stdout:2429: snap_remove snap 270 2024-09-20T19:33:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:48.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:48.524 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:48.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:48 smithi162 bash[24557]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.553 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:48 smithi162 bash[24557]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:48.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:48 smithi042 bash[22706]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:48 smithi042 bash[22706]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:48 smithi042 bash[32075]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:48 smithi042 bash[32075]: cluster 2024-09-20T19:33:47.349110+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2024-09-20T19:33:48.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:48.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:48.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:48.908 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:48.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:49.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:49.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:49.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:49.269 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:49.297 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:49.371 INFO:tasks.rados.rados.0.smithi042.stdout:2429: done (3 left) 2024-09-20T19:33:49.371 INFO:tasks.rados.rados.0.smithi042.stdout:2430: snap_create 2024-09-20T19:33:49.371 INFO:tasks.rados.rados.0.smithi042.stdout:2426: finishing copy_from to smithi04268048-3 2024-09-20T19:33:49.371 INFO:tasks.rados.rados.0.smithi042.stdout:2426: got expected ENOENT (src dne) 2024-09-20T19:33:49.371 INFO:tasks.rados.rados.0.smithi042.stdout:2426: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:33:49.372 INFO:tasks.rados.rados.0.smithi042.stdout:2427: finishing write tid 1 to smithi04268048-18 2024-09-20T19:33:49.372 INFO:tasks.rados.rados.0.smithi042.stdout:2427: finishing write tid 2 to smithi04268048-18 2024-09-20T19:33:49.372 INFO:tasks.rados.rados.0.smithi042.stdout:2428: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:49.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:49.403 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2752916 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.411 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[24557]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:49.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.529 INFO:tasks.rados.rados.0.smithi042.stdout:2427: finishing write tid 3 to smithi04268048-18 2024-09-20T19:33:49.529 INFO:tasks.rados.rados.0.smithi042.stdout:2427: finishing write tid 4 to smithi04268048-18 2024-09-20T19:33:49.530 INFO:tasks.rados.rados.0.smithi042.stdout:2427: oid 18 updating version 0 to 1053 2024-09-20T19:33:49.530 INFO:tasks.rados.rados.0.smithi042.stdout:2427: oid 18 updating version 1053 to 1055 2024-09-20T19:33:49.530 INFO:tasks.rados.rados.0.smithi042.stdout:2427: oid 18 version 1055 is already newer than 1054 2024-09-20T19:33:49.530 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1055 (ObjNum 846 snap 277 seq_num 846) dirty exists 2024-09-20T19:33:49.530 INFO:tasks.rados.rados.0.smithi042.stdout:2427: left oid 18 (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:33:49.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:49.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[32075]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.352586+0000 mgr.y (mgr.24491) 596 : cluster [DBG] pgmap v1077: 178 pgs: 1 active+undersized+remapped, 11 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 159 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 16 MiB/s, 98 objects/s recovering 2024-09-20T19:33:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.355181+0000 mon.a (mon.0) 2188 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-20T19:33:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:49 smithi042 bash[22706]: cluster 2024-09-20T19:33:48.370688+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2024-09-20T19:33:49.681 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:49.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:49.861 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:49.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:49.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:49.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:49.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:50.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:50.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:50.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:50.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:50.389 INFO:tasks.rados.rados.0.smithi042.stdout:2426: done (3 left) 2024-09-20T19:33:50.389 INFO:tasks.rados.rados.0.smithi042.stdout:2427: done (2 left) 2024-09-20T19:33:50.389 INFO:tasks.rados.rados.0.smithi042.stdout:2428: done (1 left) 2024-09-20T19:33:50.389 INFO:tasks.rados.rados.0.smithi042.stdout:2430: done (0 left) 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2431: read oid 32 snap 268 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2431: expect (ObjNum 755 snap 252 seq_num 755) 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2432: read oid 45 snap -1 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2432: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2433: copy_from oid 46 from oid 24 current snap is 278 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2434: write (excl) oid 8 current snap is 278 2024-09-20T19:33:50.390 INFO:tasks.rados.rados.0.smithi042.stdout:2434: seq_num 847 ranges {635529=651086,1848013=387516} 2024-09-20T19:33:50.404 INFO:tasks.rados.rados.0.smithi042.stdout:2434: writing smithi04268048-8 from 635529 to 1286615 tid 1 2024-09-20T19:33:50.414 INFO:tasks.rados.rados.0.smithi042.stdout:2434: writing smithi04268048-8 from 1848013 to 2235529 tid 2 2024-09-20T19:33:50.415 INFO:tasks.rados.rados.0.smithi042.stdout:2435: read oid 16 snap -1 2024-09-20T19:33:50.415 INFO:tasks.rados.rados.0.smithi042.stdout:2435: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:50.415 INFO:tasks.rados.rados.0.smithi042.stdout:2436: delete oid 19 current snap is 278 2024-09-20T19:33:50.415 INFO:tasks.rados.rados.0.smithi042.stdout:2431: expect (ObjNum 755 snap 252 seq_num 755) 2024-09-20T19:33:50.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:33:50.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T19:33:50.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:50.430 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2563370 2024-09-20T19:33:50.436 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1602329 2024-09-20T19:33:50.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:50.457 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:50.487 INFO:tasks.rados.rados.0.smithi042.stdout:2432: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:50.536 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:50.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:50.581 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:50.607 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1001229 2024-09-20T19:33:50.649 INFO:tasks.rados.rados.0.smithi042.stdout:2433: finishing copy_from to smithi04268048-46 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2433: got expected ENOENT (src dne) 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2433: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2431: done (5 left) 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2432: done (4 left) 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2433: done (3 left) 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2436: done (2 left) 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2437: snap_create 2024-09-20T19:33:50.650 INFO:tasks.rados.rados.0.smithi042.stdout:2435: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:33:50.663 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1485930 2024-09-20T19:33:50.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[32075]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[32075]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[32075]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[32075]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[22706]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[22706]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[22706]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:50 smithi042 bash[22706]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:50 smithi162 bash[24557]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:50 smithi162 bash[24557]: cluster 2024-09-20T19:33:49.369680+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2024-09-20T19:33:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:50 smithi162 bash[24557]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:50 smithi162 bash[24557]: audit 2024-09-20T19:33:49.669249+0000 mgr.y (mgr.24491) 597 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:33:50.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: finishing write tid 1 to smithi04268048-8 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: finishing write tid 2 to smithi04268048-8 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: finishing write tid 3 to smithi04268048-8 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: finishing write tid 4 to smithi04268048-8 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: oid 8 updating version 0 to 855 2024-09-20T19:33:50.814 INFO:tasks.rados.rados.0.smithi042.stdout:2434: oid 8 version 855 is already newer than 854 2024-09-20T19:33:50.815 INFO:tasks.rados.rados.0.smithi042.stdout:2434: oid 8 updating version 855 to 856 2024-09-20T19:33:50.815 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:33:50.815 INFO:tasks.rados.rados.0.smithi042.stdout:2434: left oid 8 (ObjNum 847 snap 278 seq_num 847) 2024-09-20T19:33:50.832 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:50.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:50.962 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:50.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:51.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:51.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:51.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:51.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:51.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:51.353 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:51.391 INFO:tasks.rados.rados.0.smithi042.stdout:2434: done (2 left) 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2435: done (1 left) 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2437: done (0 left) 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2438: read oid 42 snap -1 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2438: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2439: rollback oid 14 current snap is 279 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 277 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2440: read oid 1 snap -1 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2440: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2441: write (excl) oid 35 current snap is 279 2024-09-20T19:33:51.392 INFO:tasks.rados.rados.0.smithi042.stdout:2441: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-20T19:33:51.408 INFO:tasks.rados.rados.0.smithi042.stdout:2441: writing smithi04268048-35 from 669143 to 1070727 tid 1 2024-09-20T19:33:51.417 INFO:tasks.rados.rados.0.smithi042.stdout:2441: writing smithi04268048-35 from 1564977 to 1991936 tid 2 2024-09-20T19:33:51.417 INFO:tasks.rados.rados.0.smithi042.stdout:2442: copy_from oid 28 from oid 36 current snap is 279 2024-09-20T19:33:51.417 INFO:tasks.rados.rados.0.smithi042.stdout:2440: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:33:51.456 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.459 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2152551 2024-09-20T19:33:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:51.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 653619 2024-09-20T19:33:51.560 INFO:tasks.rados.rados.0.smithi042.stdout:2439: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:33:51.560 INFO:tasks.rados.rados.0.smithi042.stdout:2439: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:33:51.560 INFO:tasks.rados.rados.0.smithi042.stdout:2439: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:33:51.560 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 816 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:33:51.560 INFO:tasks.rados.rados.0.smithi042.stdout:2438: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:33:51.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.610 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:51.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[32075]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[32075]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.690 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[32075]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.691 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[32075]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[22706]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[22706]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[22706]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.691 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:51 smithi042 bash[22706]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: finishing write tid 1 to smithi04268048-35 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: finishing write tid 2 to smithi04268048-35 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: finishing write tid 3 to smithi04268048-35 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: finishing write tid 4 to smithi04268048-35 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: oid 35 updating version 0 to 1056 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: oid 35 version 1056 is already newer than 1055 2024-09-20T19:33:51.712 INFO:tasks.rados.rados.0.smithi042.stdout:2441: oid 35 version 1056 is already newer than 1054 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1056 (ObjNum 849 snap 279 seq_num 849) dirty exists 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2441: left oid 35 (ObjNum 849 snap 279 seq_num 849) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2442: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2442: finishing copy_from to smithi04268048-28 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 584 (ObjNum 838 snap 276 seq_num 838) dirty exists 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2438: done (4 left) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2439: done (3 left) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2440: done (2 left) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2441: done (1 left) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2442: done (0 left) 2024-09-20T19:33:51.713 INFO:tasks.rados.rados.0.smithi042.stdout:2443: snap_create 2024-09-20T19:33:51.713 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:51.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:51.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:51 smithi162 bash[24557]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:51 smithi162 bash[24557]: cluster 2024-09-20T19:33:50.353572+0000 mgr.y (mgr.24491) 598 : cluster [DBG] pgmap v1080: 178 pgs: 12 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 155 active+clean; 161 MiB data, 741 MiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 4.2 MiB/s wr, 24 op/s; 16 MiB/s, 94 objects/s recovering 2024-09-20T19:33:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:51 smithi162 bash[24557]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:51 smithi162 bash[24557]: cluster 2024-09-20T19:33:50.386530+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2024-09-20T19:33:51.842 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:51.935 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:51.974 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:51.994 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:51.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:52.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:52.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:52.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:52.357 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:33:52.406 INFO:tasks.rados.rados.0.smithi042.stdout:2443: done (0 left) 2024-09-20T19:33:52.406 INFO:tasks.rados.rados.0.smithi042.stdout:2444: write (excl) oid 34 current snap is 280 2024-09-20T19:33:52.406 INFO:tasks.rados.rados.0.smithi042.stdout:2444: seq_num 850 ranges {685950=476833,1623459=570617} 2024-09-20T19:33:52.412 INFO:tasks.rados.rados.0.smithi042.stdout:2444: writing smithi04268048-34 from 685950 to 1162783 tid 1 2024-09-20T19:33:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:2444: writing smithi04268048-34 from 1623459 to 2194076 tid 2 2024-09-20T19:33:52.418 INFO:tasks.rados.rados.0.smithi042.stdout:2445: snap_create 2024-09-20T19:33:52.421 INFO:tasks.rados.rados.0.smithi042.stdout:2444: finishing write tid 1 to smithi04268048-34 2024-09-20T19:33:52.430 INFO:tasks.rados.rados.0.smithi042.stdout:2444: finishing write tid 2 to smithi04268048-34 2024-09-20T19:33:52.430 INFO:tasks.rados.rados.0.smithi042.stdout:2444: finishing write tid 3 to smithi04268048-34 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:2444: finishing write tid 4 to smithi04268048-34 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:2444: oid 34 updating version 0 to 1058 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:2444: oid 34 updating version 1058 to 1059 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:2444: oid 34 version 1059 is already newer than 1057 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1059 (ObjNum 850 snap 280 seq_num 850) dirty exists 2024-09-20T19:33:52.431 INFO:tasks.rados.rados.0.smithi042.stdout:2444: left oid 34 (ObjNum 850 snap 280 seq_num 850) 2024-09-20T19:33:52.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:52.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:52.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[22706]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:52 smithi042 bash[32075]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:52.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: cluster 2024-09-20T19:33:51.386876+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2024-09-20T19:33:52.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: audit 2024-09-20T19:33:52.353563+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.42:0/2184536182' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.754 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:52 smithi162 bash[24557]: audit 2024-09-20T19:33:52.354015+0000 mon.a (mon.0) 2193 : audit [INF] from='client.25697 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:33:52.770 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:52.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:52.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:52.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:53.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:53.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:53.254 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:53.283 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:53.332 INFO:tasks.rados.rados.0.smithi042.stdout:2444: done (1 left) 2024-09-20T19:33:53.332 INFO:tasks.rados.rados.0.smithi042.stdout:2445: done (0 left) 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:2446: read oid 21 snap 279 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:2446: expect (ObjNum 813 snap 268 seq_num 813) 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:2447: rollback oid 34 current snap is 281 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 279 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:2448: write oid 47 current snap is 281 2024-09-20T19:33:53.333 INFO:tasks.rados.rados.0.smithi042.stdout:2448: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-20T19:33:53.340 INFO:tasks.rados.rados.0.smithi042.stdout:2448: writing smithi04268048-47 from 702757 to 1254839 tid 1 2024-09-20T19:33:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:2448: writing smithi04268048-47 from 1681941 to 2112569 tid 2 2024-09-20T19:33:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:2449: rollback oid 22 current snap is 281 2024-09-20T19:33:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 278 2024-09-20T19:33:53.347 INFO:tasks.rados.rados.0.smithi042.stdout:2446: expect (ObjNum 813 snap 268 seq_num 813) 2024-09-20T19:33:53.358 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1735976 2024-09-20T19:33:53.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:53.408 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:53.461 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3458851 2024-09-20T19:33:53.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:53.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:2447: finishing rollback tid 0 to smithi04268048-34 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:2447: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1061 (ObjNum 129 snap 0 seq_num 1415933432) dirty dne 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:2449: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:2449: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:33:53.540 INFO:tasks.rados.rados.0.smithi042.stdout:2449: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1020 (ObjNum 836 snap 276 seq_num 836) dirty exists 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: finishing write tid 1 to smithi04268048-47 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: finishing write tid 2 to smithi04268048-47 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: finishing write tid 3 to smithi04268048-47 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: finishing write tid 4 to smithi04268048-47 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: oid 47 updating version 0 to 1036 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: oid 47 updating version 1036 to 1037 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: oid 47 updating version 1037 to 1038 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1038 (ObjNum 851 snap 281 seq_num 851) dirty exists 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: left oid 47 (ObjNum 851 snap 281 seq_num 851) 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2446: done (3 left) 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2447: done (2 left) 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2448: done (1 left) 2024-09-20T19:33:53.541 INFO:tasks.rados.rados.0.smithi042.stdout:2449: done (0 left) 2024-09-20T19:33:53.542 INFO:tasks.rados.rados.0.smithi042.stdout:2450: read oid 45 snap -1 2024-09-20T19:33:53.542 INFO:tasks.rados.rados.0.smithi042.stdout:2450: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:53.542 INFO:tasks.rados.rados.0.smithi042.stdout:2451: write (excl) oid 11 current snap is 281 2024-09-20T19:33:53.542 INFO:tasks.rados.rados.0.smithi042.stdout:2451: seq_num 852 ranges {719564=743684,2140423=179141} 2024-09-20T19:33:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:53.564 INFO:tasks.rados.rados.0.smithi042.stdout:2451: writing smithi04268048-11 from 719564 to 1463248 tid 1 2024-09-20T19:33:53.579 INFO:tasks.rados.rados.0.smithi042.stdout:2451: writing smithi04268048-11 from 2140423 to 2319564 tid 2 2024-09-20T19:33:53.580 INFO:tasks.rados.rados.0.smithi042.stdout:2452: rollback oid 40 current snap is 281 2024-09-20T19:33:53.580 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 278 2024-09-20T19:33:53.580 INFO:tasks.rados.rados.0.smithi042.stdout:2450: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:53.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:53.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.654 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.655 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:53 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:53.743 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2452804 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: finishing write tid 1 to smithi04268048-11 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: finishing write tid 2 to smithi04268048-11 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: finishing write tid 3 to smithi04268048-11 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: finishing write tid 4 to smithi04268048-11 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: oid 11 updating version 0 to 925 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: oid 11 updating version 925 to 927 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: oid 11 version 927 is already newer than 926 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 927 (ObjNum 852 snap 281 seq_num 852) dirty exists 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2451: left oid 11 (ObjNum 852 snap 281 seq_num 852) 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2452: finishing rollback tid 0 to smithi04268048-40 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2452: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:2452: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:33:53.757 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1042 (ObjNum 833 snap 275 seq_num 833) dirty exists 2024-09-20T19:33:53.758 INFO:tasks.rados.rados.0.smithi042.stdout:2450: done (2 left) 2024-09-20T19:33:53.758 INFO:tasks.rados.rados.0.smithi042.stdout:2451: done (1 left) 2024-09-20T19:33:53.758 INFO:tasks.rados.rados.0.smithi042.stdout:2452: done (0 left) 2024-09-20T19:33:53.758 INFO:tasks.rados.rados.0.smithi042.stdout:2453: write oid 39 current snap is 281 2024-09-20T19:33:53.758 INFO:tasks.rados.rados.0.smithi042.stdout:2453: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-20T19:33:53.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:53.776 INFO:tasks.rados.rados.0.smithi042.stdout:2453: writing smithi04268048-39 from 736371 to 1155304 tid 1 2024-09-20T19:33:53.788 INFO:tasks.rados.rados.0.smithi042.stdout:2453: writing smithi04268048-39 from 1798905 to 2233202 tid 2 2024-09-20T19:33:53.788 INFO:tasks.rados.rados.0.smithi042.stdout:2454: snap_create 2024-09-20T19:33:53.788 INFO:tasks.rados.rados.0.smithi042.stdout:2453: finishing write tid 1 to smithi04268048-39 2024-09-20T19:33:53.789 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:53.793 INFO:tasks.rados.rados.0.smithi042.stdout:2453: finishing write tid 2 to smithi04268048-39 2024-09-20T19:33:53.794 INFO:tasks.rados.rados.0.smithi042.stdout:2453: finishing write tid 3 to smithi04268048-39 2024-09-20T19:33:53.794 INFO:tasks.rados.rados.0.smithi042.stdout:2453: finishing write tid 4 to smithi04268048-39 2024-09-20T19:33:53.794 INFO:tasks.rados.rados.0.smithi042.stdout:2453: oid 39 updating version 0 to 730 2024-09-20T19:33:53.794 INFO:tasks.rados.rados.0.smithi042.stdout:2453: oid 39 version 730 is already newer than 728 2024-09-20T19:33:53.794 INFO:tasks.rados.rados.0.smithi042.stdout:2453: oid 39 version 730 is already newer than 729 2024-09-20T19:33:53.795 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 730 (ObjNum 853 snap 281 seq_num 853) dirty exists 2024-09-20T19:33:53.795 INFO:tasks.rados.rados.0.smithi042.stdout:2453: left oid 39 (ObjNum 853 snap 281 seq_num 853) 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.053333+0000 osd.2 (osd.2) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.054031+0000 osd.2 (osd.2) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.354798+0000 mgr.y (mgr.24491) 599 : cluster [DBG] pgmap v1083: 178 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 158 active+clean; 160 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 31 op/s 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.393176+0000 mon.a (mon.0) 2194 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: audit 2024-09-20T19:33:52.400579+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25697 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.404845+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.431912+0000 osd.1 (osd.1) 245 : cluster [DBG] 7.b scrub starts 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.583050+0000 osd.4 (osd.4) 65 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.586083+0000 osd.4 (osd.4) 66 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.220510+0000 osd.7 (osd.7) 179 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:33:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.222427+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:33:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:53 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.327187+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2024-09-20T19:33:53.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:53.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:53.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:53.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:54.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:54.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:54.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:54.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:54.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:54.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:54.424 INFO:tasks.rados.rados.0.smithi042.stdout:2453: done (1 left) 2024-09-20T19:33:54.424 INFO:tasks.rados.rados.0.smithi042.stdout:2454: done (0 left) 2024-09-20T19:33:54.424 INFO:tasks.rados.rados.0.smithi042.stdout:2455: snap_remove snap 281 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:54 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:54.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:52.436710+0000 osd.1 (osd.1) 246 : cluster [DBG] 7.b scrub ok 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.176871+0000 osd.2 (osd.2) 223 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.179740+0000 osd.2 (osd.2) 224 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:33:54.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.563981+0000 osd.0 (osd.0) 51 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.566282+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.606974+0000 osd.4 (osd.4) 67 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:53.608537+0000 osd.4 (osd.4) 68 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.207159+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:33:54.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:54 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.210000+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:33:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:54.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:54.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:54.800 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:54.820 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:54.897 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:33:54.897 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:33:54.898 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:33:54.898 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:33:54.898 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:33:54.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:54.942 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:54.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:55.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:55.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:55.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.334 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:55.434 INFO:tasks.rados.rados.0.smithi042.stdout:2455: done (0 left) 2024-09-20T19:33:55.434 INFO:tasks.rados.rados.0.smithi042.stdout:2456: write (excl) oid 35 current snap is 282 2024-09-20T19:33:55.434 INFO:tasks.rados.rados.0.smithi042.stdout:2456: seq_num 854 ranges {753178=494182,1857387=495791} 2024-09-20T19:33:55.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.439 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[22706]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.441 INFO:tasks.rados.rados.0.smithi042.stdout:2456: writing smithi04268048-35 from 753178 to 1247360 tid 1 2024-09-20T19:33:55.447 INFO:tasks.rados.rados.0.smithi042.stdout:2456: writing smithi04268048-35 from 1857387 to 2353178 tid 2 2024-09-20T19:33:55.448 INFO:tasks.rados.rados.0.smithi042.stdout:2457: rollback oid 9 current snap is 282 2024-09-20T19:33:55.448 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 277 2024-09-20T19:33:55.448 INFO:tasks.rados.rados.0.smithi042.stdout:2458: snap_remove snap 276 2024-09-20T19:33:55.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:55.557 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.558 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:55 smithi042 bash[32075]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:55.590 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:55.687 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:33:55.687 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:33:55.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.705 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:33:55.705 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:33:55.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.355513+0000 mgr.y (mgr.24491) 600 : cluster [DBG] pgmap v1086: 178 pgs: 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 165 active+clean; 160 MiB data, 670 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.421808+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.613241+0000 osd.0 (osd.0) 53 : cluster [DBG] 7.f scrub starts 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.706 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:55 smithi162 bash[24557]: cluster 2024-09-20T19:33:54.617333+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.f scrub ok 2024-09-20T19:33:55.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:33:55.722 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:33:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:55.739 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:33:55.739 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:33:55.757 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:33:55.757 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:33:55.776 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:33:55.777 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:33:55.794 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:33:55.794 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:33:55.813 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:33:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:33:55.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:55.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:55.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:56.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:56.097 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:56.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:56.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:56.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:33:56.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:56.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:56.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:33:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:33:56.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:56.470 INFO:tasks.rados.rados.0.smithi042.stdout:2458: done (2 left) 2024-09-20T19:33:56.470 INFO:tasks.rados.rados.0.smithi042.stdout:2459: copy_from oid 4 from oid 2 current snap is 282 2024-09-20T19:33:56.470 INFO:tasks.rados.rados.0.smithi042.stdout:2460: write (excl) oid 28 current snap is 282 2024-09-20T19:33:56.470 INFO:tasks.rados.rados.0.smithi042.stdout:2460: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-20T19:33:56.472 INFO:tasks.rados.rados.0.smithi042.stdout:2460: writing smithi04268048-28 from 769985 to 1339416 tid 1 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2460: writing smithi04268048-28 from 1915869 to 2353835 tid 2 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: finishing write tid 1 to smithi04268048-35 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: finishing write tid 2 to smithi04268048-35 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2457: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 1415933432) dirty dne 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: finishing write tid 3 to smithi04268048-35 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: finishing write tid 4 to smithi04268048-35 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: oid 35 updating version 0 to 1062 2024-09-20T19:33:56.479 INFO:tasks.rados.rados.0.smithi042.stdout:2456: oid 35 updating version 1062 to 1063 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:2456: oid 35 updating version 1063 to 1064 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1064 (ObjNum 854 snap 282 seq_num 854) dirty exists 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:2456: left oid 35 (ObjNum 854 snap 282 seq_num 854) 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:2456: done (3 left) 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:2457: done (2 left) 2024-09-20T19:33:56.480 INFO:tasks.rados.rados.0.smithi042.stdout:2461: delete oid 46 current snap is 282 2024-09-20T19:33:56.488 INFO:tasks.rados.rados.0.smithi042.stdout:2459: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:33:56.488 INFO:tasks.rados.rados.0.smithi042.stdout:2461: done (2 left) 2024-09-20T19:33:56.488 INFO:tasks.rados.rados.0.smithi042.stdout:2462: snap_create 2024-09-20T19:33:56.489 INFO:tasks.rados.rados.0.smithi042.stdout:2459: finishing copy_from to smithi04268048-4 2024-09-20T19:33:56.489 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 625 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:33:56.491 INFO:tasks.rados.rados.0.smithi042.stdout:2460: finishing write tid 1 to smithi04268048-28 2024-09-20T19:33:56.491 INFO:tasks.rados.rados.0.smithi042.stdout:2460: finishing write tid 2 to smithi04268048-28 2024-09-20T19:33:56.492 INFO:tasks.rados.rados.0.smithi042.stdout:2460: finishing write tid 3 to smithi04268048-28 2024-09-20T19:33:56.493 INFO:tasks.rados.rados.0.smithi042.stdout:2460: finishing write tid 4 to smithi04268048-28 2024-09-20T19:33:56.493 INFO:tasks.rados.rados.0.smithi042.stdout:2460: oid 28 updating version 0 to 816 2024-09-20T19:33:56.493 INFO:tasks.rados.rados.0.smithi042.stdout:2460: oid 28 updating version 816 to 818 2024-09-20T19:33:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:2460: oid 28 version 818 is already newer than 817 2024-09-20T19:33:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 818 (ObjNum 855 snap 282 seq_num 855) dirty exists 2024-09-20T19:33:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:2460: left oid 28 (ObjNum 855 snap 282 seq_num 855) 2024-09-20T19:33:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:56.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:56 smithi162 bash[24557]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:56.724 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.735 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.736 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.210107+0000 osd.2 (osd.2) 225 : cluster [DBG] 7.a scrub starts 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.211662+0000 osd.2 (osd.2) 226 : cluster [DBG] 7.a scrub ok 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.426402+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.624460+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.c scrub starts 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.628306+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.c scrub ok 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.647809+0000 osd.4 (osd.4) 69 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: cluster 2024-09-20T19:33:55.653111+0000 osd.4 (osd.4) 70 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:55.668208+0000 mon.a (mon.0) 2200 : audit [DBG] from='client.? v1:172.21.15.42:0/3006434493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.341751+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.341994+0000 mon.a (mon.0) 2202 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.342168+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.738 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[32075]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:56 smithi042 bash[22706]: audit 2024-09-20T19:33:56.342382+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]} : dispatch 2024-09-20T19:33:56.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:56.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:56.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:56.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:56.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:57.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:33:57.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:57.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:57.244 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:57.270 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:33:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:33:57.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:57.463 INFO:tasks.rados.rados.0.smithi042.stdout:2459: done (2 left) 2024-09-20T19:33:57.463 INFO:tasks.rados.rados.0.smithi042.stdout:2460: done (1 left) 2024-09-20T19:33:57.463 INFO:tasks.rados.rados.0.smithi042.stdout:2462: done (0 left) 2024-09-20T19:33:57.463 INFO:tasks.rados.rados.0.smithi042.stdout:2463: snap_create 2024-09-20T19:33:57.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.509 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.510 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.511 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:57 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:57.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:57.650 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:57.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:56.356028+0000 mgr.y (mgr.24491) 601 : cluster [DBG] pgmap v1089: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 160 MiB data, 674 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.442746+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.442913+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.443060+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: audit 2024-09-20T19:33:56.443270+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.e", "id": [2, 4]}]': finished 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:56.456436+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.259388+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.260090+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.389078+0000 osd.1 (osd.1) 249 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:57 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.391137+0000 osd.1 (osd.1) 250 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:33:57.781 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:57.884 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:57.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:58.015 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:58.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:58.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:58.300 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:58.335 INFO:tasks.rados.rados.0.smithi042.stdout:2463: done (0 left) 2024-09-20T19:33:58.335 INFO:tasks.rados.rados.0.smithi042.stdout:2464: write oid 13 current snap is 284 2024-09-20T19:33:58.335 INFO:tasks.rados.rados.0.smithi042.stdout:2464: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-20T19:33:58.342 INFO:tasks.rados.rados.0.smithi042.stdout:2464: writing smithi04268048-13 from 403599 to 1123528 tid 1 2024-09-20T19:33:58.349 INFO:tasks.rados.rados.0.smithi042.stdout:2464: writing smithi04268048-13 from 1916480 to 2358115 tid 2 2024-09-20T19:33:58.349 INFO:tasks.rados.rados.0.smithi042.stdout:2465: write (excl) oid 20 current snap is 284 2024-09-20T19:33:58.349 INFO:tasks.rados.rados.0.smithi042.stdout:2464: finishing write tid 1 to smithi04268048-13 2024-09-20T19:33:58.349 INFO:tasks.rados.rados.0.smithi042.stdout:2465: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-20T19:33:58.356 INFO:tasks.rados.rados.0.smithi042.stdout:2465: writing smithi04268048-20 from 420406 to 1215584 tid 1 2024-09-20T19:33:58.363 INFO:tasks.rados.rados.0.smithi042.stdout:2465: writing smithi04268048-20 from 1974962 to 2420406 tid 2 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2466: copy_from oid 47 from oid 44 current snap is 284 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: finishing write tid 2 to smithi04268048-13 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: finishing write tid 3 to smithi04268048-13 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: finishing write tid 4 to smithi04268048-13 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: oid 13 updating version 0 to 586 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: oid 13 updating version 586 to 587 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: oid 13 version 587 is already newer than 585 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 587 (ObjNum 857 snap 284 seq_num 857) dirty exists 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: left oid 13 (ObjNum 857 snap 284 seq_num 857) 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2464: done (2 left) 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2467: delete oid 42 current snap is 284 2024-09-20T19:33:58.364 INFO:tasks.rados.rados.0.smithi042.stdout:2465: finishing write tid 1 to smithi04268048-20 2024-09-20T19:33:58.369 INFO:tasks.rados.rados.0.smithi042.stdout:2465: finishing write tid 2 to smithi04268048-20 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:2465: finishing write tid 3 to smithi04268048-20 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:2465: finishing write tid 4 to smithi04268048-20 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:2465: oid 20 updating version 0 to 995 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:2465: oid 20 version 995 is already newer than 994 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:2465: oid 20 version 995 is already newer than 993 2024-09-20T19:33:58.370 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 995 (ObjNum 858 snap 284 seq_num 858) dirty exists 2024-09-20T19:33:58.371 INFO:tasks.rados.rados.0.smithi042.stdout:2465: left oid 20 (ObjNum 858 snap 284 seq_num 858) 2024-09-20T19:33:58.378 INFO:tasks.rados.rados.0.smithi042.stdout:2466: finishing copy_from to smithi04268048-47 2024-09-20T19:33:58.378 INFO:tasks.rados.rados.0.smithi042.stdout:2466: got expected ENOENT (src dne) 2024-09-20T19:33:58.379 INFO:tasks.rados.rados.0.smithi042.stdout:2466: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:33:58.381 INFO:tasks.rados.rados.0.smithi042.stdout:2465: done (2 left) 2024-09-20T19:33:58.381 INFO:tasks.rados.rados.0.smithi042.stdout:2466: done (1 left) 2024-09-20T19:33:58.381 INFO:tasks.rados.rados.0.smithi042.stdout:2467: done (0 left) 2024-09-20T19:33:58.381 INFO:tasks.rados.rados.0.smithi042.stdout:2468: write oid 13 current snap is 284 2024-09-20T19:33:58.381 INFO:tasks.rados.rados.0.smithi042.stdout:2468: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-20T19:33:58.387 INFO:tasks.rados.rados.0.smithi042.stdout:2468: writing smithi04268048-13 from 454020 to 1116049 tid 1 2024-09-20T19:33:58.394 INFO:tasks.rados.rados.0.smithi042.stdout:2468: writing smithi04268048-13 from 1691926 to 2280888 tid 2 2024-09-20T19:33:58.395 INFO:tasks.rados.rados.0.smithi042.stdout:2469: copy_from oid 4 from oid 12 current snap is 284 2024-09-20T19:33:58.395 INFO:tasks.rados.rados.0.smithi042.stdout:2468: finishing write tid 1 to smithi04268048-13 2024-09-20T19:33:58.395 INFO:tasks.rados.rados.0.smithi042.stdout:2470: read oid 45 snap -1 2024-09-20T19:33:58.395 INFO:tasks.rados.rados.0.smithi042.stdout:2470: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:58.395 INFO:tasks.rados.rados.0.smithi042.stdout:2471: snap_remove snap 280 2024-09-20T19:33:58.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:58.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:58.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:58.562 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:33:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:33:58.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:58 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:58.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: cluster 2024-09-20T19:33:57.456455+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: audit 2024-09-20T19:33:58.093141+0000 mon.a (mon.0) 2211 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:58 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.333496+0000 mon.a (mon.0) 2212 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2024-09-20T19:33:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:58.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:58.952 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:33:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:58.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:33:59.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:33:59.055 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:33:59.094 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:59.197 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:59.220 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:33:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:33:59.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:59.353 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:33:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:33:59.370 INFO:tasks.rados.rados.0.smithi042.stdout:2471: done (3 left) 2024-09-20T19:33:59.370 INFO:tasks.rados.rados.0.smithi042.stdout:2472: write (excl) oid 44 current snap is 284 2024-09-20T19:33:59.370 INFO:tasks.rados.rados.0.smithi042.stdout:2472: seq_num 861 ranges {470827=737278,1750408=720419} 2024-09-20T19:33:59.377 INFO:tasks.rados.rados.0.smithi042.stdout:2472: writing smithi04268048-44 from 470827 to 1208105 tid 1 2024-09-20T19:33:59.388 INFO:tasks.rados.rados.0.smithi042.stdout:2472: writing smithi04268048-44 from 1750408 to 2470827 tid 2 2024-09-20T19:33:59.388 INFO:tasks.rados.rados.0.smithi042.stdout:2470: expect (ObjNum 795 snap 262 seq_num 795) 2024-09-20T19:33:59.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:59.462 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:59.484 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:59.522 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: finishing write tid 2 to smithi04268048-13 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: finishing write tid 3 to smithi04268048-13 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: finishing write tid 4 to smithi04268048-13 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: oid 13 updating version 0 to 590 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: oid 13 version 590 is already newer than 589 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: oid 13 version 590 is already newer than 588 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 590 (ObjNum 860 snap 284 seq_num 860) dirty exists 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2468: left oid 13 (ObjNum 860 snap 284 seq_num 860) 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2469: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2469: finishing copy_from to smithi04268048-4 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 725 (ObjNum 794 snap 262 seq_num 794) dirty exists 2024-09-20T19:33:59.583 INFO:tasks.rados.rados.0.smithi042.stdout:2472: finishing write tid 1 to smithi04268048-44 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2468: done (3 left) 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2469: done (2 left) 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2470: done (1 left) 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2473: snap_create 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: finishing write tid 2 to smithi04268048-44 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: finishing write tid 3 to smithi04268048-44 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: finishing write tid 4 to smithi04268048-44 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: oid 44 updating version 0 to 1100 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: oid 44 updating version 1100 to 1102 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: oid 44 version 1102 is already newer than 1101 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1102 (ObjNum 861 snap 284 seq_num 861) dirty exists 2024-09-20T19:33:59.584 INFO:tasks.rados.rados.0.smithi042.stdout:2472: left oid 44 (ObjNum 861 snap 284 seq_num 861) 2024-09-20T19:33:59.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:59.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:33:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:33:59.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:59.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:33:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:33:59.744 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:33:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:33:59.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.859 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.207229+0000 osd.5 (osd.5) 215 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.210126+0000 osd.5 (osd.5) 216 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.356747+0000 mgr.y (mgr.24491) 602 : cluster [DBG] pgmap v1093: 178 pgs: 1 remapped+peering, 1 peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 170 active+clean; 150 MiB data, 699 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.559226+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:58.560817+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:33:59.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.339669+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.347185+0000 osd.2 (osd.2) 227 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.352080+0000 osd.2 (osd.2) 228 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.361013+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.392184+0000 osd.1 (osd.1) 251 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:33:59 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.395277+0000 osd.1 (osd.1) 252 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:33:59.875 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:33:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:33:59.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:33:59.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:00.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:00.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2472: done (1 left) 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2473: done (0 left) 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2474: read oid 25 snap 268 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2474: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2475: read oid 27 snap 277 2024-09-20T19:34:00.364 INFO:tasks.rados.rados.0.smithi042.stdout:2475: expect deleted 2024-09-20T19:34:00.365 INFO:tasks.rados.rados.0.smithi042.stdout:2476: write oid 32 current snap is 285 2024-09-20T19:34:00.365 INFO:tasks.rados.rados.0.smithi042.stdout:2476: seq_num 862 ranges {487634=412527,1408890=592631} 2024-09-20T19:34:00.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:00.377 INFO:tasks.rados.rados.0.smithi042.stdout:2476: writing smithi04268048-32 from 487634 to 900161 tid 1 2024-09-20T19:34:00.391 INFO:tasks.rados.rados.0.smithi042.stdout:2476: writing smithi04268048-32 from 1408890 to 2001521 tid 2 2024-09-20T19:34:00.391 INFO:tasks.rados.rados.0.smithi042.stdout:2477: read oid 41 snap -1 2024-09-20T19:34:00.391 INFO:tasks.rados.rados.0.smithi042.stdout:2477: expect deleted 2024-09-20T19:34:00.391 INFO:tasks.rados.rados.0.smithi042.stdout:2478: rollback oid 1 current snap is 285 2024-09-20T19:34:00.398 INFO:tasks.rados.rados.0.smithi042.stdout:2474: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:34:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:00.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:00.399 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 721025 2024-09-20T19:34:00.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:00.547 INFO:tasks.rados.rados.0.smithi042.stdout:2476: finishing write tid 1 to smithi04268048-32 2024-09-20T19:34:00.547 INFO:tasks.rados.rados.0.smithi042.stdout:2476: finishing write tid 2 to smithi04268048-32 2024-09-20T19:34:00.547 INFO:tasks.rados.rados.0.smithi042.stdout:2476: finishing write tid 3 to smithi04268048-32 2024-09-20T19:34:00.547 INFO:tasks.rados.rados.0.smithi042.stdout:2476: finishing write tid 4 to smithi04268048-32 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2476: oid 32 updating version 0 to 1259 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2476: oid 32 version 1259 is already newer than 1258 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2476: oid 32 version 1259 is already newer than 1257 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1259 (ObjNum 862 snap 285 seq_num 862) dirty exists 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2476: left oid 32 (ObjNum 862 snap 285 seq_num 862) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 278 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2474: done (4 left) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2475: done (3 left) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2476: done (2 left) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2477: done (1 left) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2479: read oid 18 snap 282 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2479: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2480: read oid 40 snap -1 2024-09-20T19:34:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2480: expect (ObjNum 833 snap 275 seq_num 833) 2024-09-20T19:34:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2481: read oid 3 snap -1 2024-09-20T19:34:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2481: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:34:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2482: delete oid 12 current snap is 285 2024-09-20T19:34:00.551 INFO:tasks.rados.rados.0.smithi042.stdout:2482: done (4 left) 2024-09-20T19:34:00.551 INFO:tasks.rados.rados.0.smithi042.stdout:2483: copy_from oid 47 from oid 14 current snap is 285 2024-09-20T19:34:00.551 INFO:tasks.rados.rados.0.smithi042.stdout:2484: snap_create 2024-09-20T19:34:00.553 INFO:tasks.rados.rados.0.smithi042.stdout:2480: expect (ObjNum 833 snap 275 seq_num 833) 2024-09-20T19:34:00.581 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:00.605 INFO:tasks.rados.rados.0.smithi042.stdout:2479: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:00.613 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 933784 2024-09-20T19:34:00.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.505634+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e scrub starts 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:33:59.507342+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.e scrub ok 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: audit 2024-09-20T19:33:59.674877+0000 mgr.y (mgr.24491) 603 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.361420+0000 mon.a (mon.0) 2215 : cluster [DBG] osdmap e732: 8 total, 8 up, 7 in 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.382700+0000 osd.1 (osd.1) 253 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:34:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.387008+0000 osd.1 (osd.1) 254 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:34:00.736 INFO:tasks.rados.rados.0.smithi042.stdout:2481: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:34:00.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:00.791 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 447922 2024-09-20T19:34:00.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:00.890 INFO:tasks.rados.rados.0.smithi042.stdout:2478: finishing rollback tid 0 to smithi04268048-1 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:2478: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:2478: finishing rollback tid 2 to smithi04268048-1 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 839 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:2483: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:2483: finishing copy_from to smithi04268048-47 2024-09-20T19:34:00.891 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 816 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:34:00.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:00.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:01.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:01.105 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:01.105 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:01.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:01.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:01.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.288 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:01.391 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.417 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T19:34:01.476 INFO:tasks.rados.rados.0.smithi042.stdout:2478: done (5 left) 2024-09-20T19:34:01.476 INFO:tasks.rados.rados.0.smithi042.stdout:2479: done (4 left) 2024-09-20T19:34:01.476 INFO:tasks.rados.rados.0.smithi042.stdout:2480: done (3 left) 2024-09-20T19:34:01.476 INFO:tasks.rados.rados.0.smithi042.stdout:2481: done (2 left) 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2483: done (1 left) 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2484: done (0 left) 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2485: copy_from oid 36 from oid 27 current snap is 286 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2486: read oid 7 snap -1 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2486: expect (ObjNum 824 snap 273 seq_num 824) 2024-09-20T19:34:01.477 INFO:tasks.rados.rados.0.smithi042.stdout:2487: snap_remove snap 277 2024-09-20T19:34:01.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:01.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:01.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[22706]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[22706]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[22706]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[32075]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[32075]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:01 smithi042 bash[32075]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:01.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:01.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:01.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:01 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:01 smithi162 bash[24557]: cluster 2024-09-20T19:34:00.357496+0000 mgr.y (mgr.24491) 604 : cluster [DBG] pgmap v1096: 178 pgs: 1 creating+peering, 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 169 active+clean; 155 MiB data, 706 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:01 smithi162 bash[24557]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:01 smithi162 bash[24557]: audit 2024-09-20T19:34:01.226147+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:34:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:01.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:01.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:02.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:02.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:02.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:02.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:02.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:02.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:02.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:02.484 INFO:tasks.rados.rados.0.smithi042.stdout:2487: done (2 left) 2024-09-20T19:34:02.485 INFO:tasks.rados.rados.0.smithi042.stdout:2488: rollback oid 50 current snap is 286 2024-09-20T19:34:02.485 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 268 2024-09-20T19:34:02.485 INFO:tasks.rados.rados.0.smithi042.stdout:2489: snap_remove snap 279 2024-09-20T19:34:02.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:02.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:02.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:02 smithi162 bash[24557]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[22706]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.116506+0000 osd.5 (osd.5) 217 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.119884+0000 osd.5 (osd.5) 218 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: audit 2024-09-20T19:34:01.468628+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.42:0/2208034192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:02 smithi042 bash[32075]: cluster 2024-09-20T19:34:01.472479+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2024-09-20T19:34:02.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:02.981 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:02.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:03.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:03.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:03.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:03.245 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2489: done (3 left) 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2490: snap_create 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2485: finishing copy_from to smithi04268048-36 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2485: got expected ENOENT (src dne) 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2485: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:34:03.339 INFO:tasks.rados.rados.0.smithi042.stdout:2486: expect (ObjNum 824 snap 273 seq_num 824) 2024-09-20T19:34:03.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:03.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:03.465 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:03.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:03.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:03.525 INFO:tasks.rados.rados.0.smithi042.stdout:2488: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:34:03.525 INFO:tasks.rados.rados.0.smithi042.stdout:2488: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:34:03.525 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 933 (ObjNum 429 snap 133 seq_num 429) dirty exists 2024-09-20T19:34:03.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:03 smithi162 bash[24557]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T19:34:03.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:03.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:03.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[32075]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.754 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.358709+0000 mgr.y (mgr.24491) 605 : cluster [DBG] pgmap v1098: 178 pgs: 1 creating+peering, 2 active+clean+snaptrim, 175 active+clean; 152 MiB data, 727 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.482977+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.753382+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:02.755532+0000 osd.6 (osd.6) 159 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:03 smithi042 bash[22706]: cluster 2024-09-20T19:34:03.335368+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2024-09-20T19:34:03.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:03.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:03.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:03.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:03.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:04.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:04.169 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:34:04.169 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:34:04.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:04.281 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:34:04.353 INFO:tasks.rados.rados.0.smithi042.stdout:2485: done (3 left) 2024-09-20T19:34:04.353 INFO:tasks.rados.rados.0.smithi042.stdout:2486: done (2 left) 2024-09-20T19:34:04.353 INFO:tasks.rados.rados.0.smithi042.stdout:2488: done (1 left) 2024-09-20T19:34:04.353 INFO:tasks.rados.rados.0.smithi042.stdout:2490: done (0 left) 2024-09-20T19:34:04.353 INFO:tasks.rados.rados.0.smithi042.stdout:2491: snap_create 2024-09-20T19:34:04.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:04.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:04.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.516 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.517 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:04 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:04.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:04.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:04.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.314968+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.42:0/1449187384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.315645+0000 mon.a (mon.0) 2221 : audit [INF] from='client.25715 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.342333+0000 mon.a (mon.0) 2222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: audit 2024-09-20T19:34:04.343955+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25715 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:04 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.348435+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2024-09-20T19:34:04.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:04.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:04.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:04.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:04.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:05.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:05.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:05.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:05.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:05.308 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:05.363 INFO:tasks.rados.rados.0.smithi042.stdout:2491: done (0 left) 2024-09-20T19:34:05.363 INFO:tasks.rados.rados.0.smithi042.stdout:2492: snap_remove snap 287 2024-09-20T19:34:05.411 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:05.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:05.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:05 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:05.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:04.359100+0000 mgr.y (mgr.24491) 606 : cluster [DBG] pgmap v1102: 178 pgs: 1 creating+peering, 3 active+clean+snaptrim, 174 active+clean; 157 MiB data, 730 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.251512+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.253016+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:05 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.360708+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2024-09-20T19:34:05.697 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:05.800 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:05.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:34:05.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:05.959 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:05.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:06.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:06.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:06.218 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:06.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:06.369 INFO:tasks.rados.rados.0.smithi042.stdout:2492: done (0 left) 2024-09-20T19:34:06.369 INFO:tasks.rados.rados.0.smithi042.stdout:2493: read oid 2 snap -1 2024-09-20T19:34:06.369 INFO:tasks.rados.rados.0.smithi042.stdout:2493: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:34:06.369 INFO:tasks.rados.rados.0.smithi042.stdout:2494: write oid 48 current snap is 288 2024-09-20T19:34:06.369 INFO:tasks.rados.rados.0.smithi042.stdout:2494: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-20T19:34:06.381 INFO:tasks.rados.rados.0.smithi042.stdout:2494: writing smithi04268048-48 from 521248 to 1084273 tid 1 2024-09-20T19:34:06.404 INFO:tasks.rados.rados.0.smithi042.stdout:2494: writing smithi04268048-48 from 1809501 to 2405801 tid 2 2024-09-20T19:34:06.404 INFO:tasks.rados.rados.0.smithi042.stdout:2495: delete oid 41 current snap is 288 2024-09-20T19:34:06.404 INFO:tasks.rados.rados.0.smithi042.stdout:2493: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:34:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:06.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:06.471 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:06.474 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:06.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2146025 2024-09-20T19:34:06.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.531 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.532 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.533 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:06 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: finishing write tid 1 to smithi04268048-48 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: finishing write tid 2 to smithi04268048-48 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: finishing write tid 3 to smithi04268048-48 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: finishing write tid 4 to smithi04268048-48 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: oid 48 updating version 0 to 884 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: oid 48 version 884 is already newer than 882 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: oid 48 version 884 is already newer than 883 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 884 (ObjNum 864 snap 288 seq_num 864) dirty exists 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: left oid 48 (ObjNum 864 snap 288 seq_num 864) 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2493: done (2 left) 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2494: done (1 left) 2024-09-20T19:34:06.570 INFO:tasks.rados.rados.0.smithi042.stdout:2495: done (0 left) 2024-09-20T19:34:06.571 INFO:tasks.rados.rados.0.smithi042.stdout:2496: delete oid 50 current snap is 288 2024-09-20T19:34:06.575 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:06.575 INFO:tasks.rados.rados.0.smithi042.stdout:2496: done (0 left) 2024-09-20T19:34:06.576 INFO:tasks.rados.rados.0.smithi042.stdout:2497: delete oid 4 current snap is 288 2024-09-20T19:34:06.579 INFO:tasks.rados.rados.0.smithi042.stdout:2497: done (0 left) 2024-09-20T19:34:06.579 INFO:tasks.rados.rados.0.smithi042.stdout:2498: write (excl) oid 48 current snap is 288 2024-09-20T19:34:06.579 INFO:tasks.rados.rados.0.smithi042.stdout:2498: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-20T19:34:06.585 INFO:tasks.rados.rados.0.smithi042.stdout:2498: writing smithi04268048-48 from 588476 to 1168850 tid 1 2024-09-20T19:34:06.591 INFO:tasks.rados.rados.0.smithi042.stdout:2498: writing smithi04268048-48 from 1643429 to 2247067 tid 2 2024-09-20T19:34:06.591 INFO:tasks.rados.rados.0.smithi042.stdout:2499: delete oid 27 current snap is 288 2024-09-20T19:34:06.591 INFO:tasks.rados.rados.0.smithi042.stdout:2498: finishing write tid 1 to smithi04268048-48 2024-09-20T19:34:06.595 INFO:tasks.rados.rados.0.smithi042.stdout:2499: done (1 left) 2024-09-20T19:34:06.595 INFO:tasks.rados.rados.0.smithi042.stdout:2500: read oid 47 snap -1 2024-09-20T19:34:06.595 INFO:tasks.rados.rados.0.smithi042.stdout:2500: expect (ObjNum 827 snap 273 seq_num 827) 2024-09-20T19:34:06.595 INFO:tasks.rados.rados.0.smithi042.stdout:2501: delete oid 7 current snap is 288 2024-09-20T19:34:06.596 INFO:tasks.rados.rados.0.smithi042.stdout:2498: finishing write tid 2 to smithi04268048-48 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: finishing write tid 3 to smithi04268048-48 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: finishing write tid 4 to smithi04268048-48 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: oid 48 updating version 0 to 885 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: oid 48 updating version 885 to 886 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: oid 48 updating version 886 to 887 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 887 (ObjNum 868 snap 288 seq_num 868) dirty exists 2024-09-20T19:34:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:2498: left oid 48 (ObjNum 868 snap 288 seq_num 868) 2024-09-20T19:34:06.598 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:06.600 INFO:tasks.rados.rados.0.smithi042.stdout:2498: done (2 left) 2024-09-20T19:34:06.600 INFO:tasks.rados.rados.0.smithi042.stdout:2501: done (1 left) 2024-09-20T19:34:06.600 INFO:tasks.rados.rados.0.smithi042.stdout:2502: copy_from oid 36 from oid 46 current snap is 288 2024-09-20T19:34:06.600 INFO:tasks.rados.rados.0.smithi042.stdout:2503: delete oid 40 current snap is 288 2024-09-20T19:34:06.602 INFO:tasks.rados.rados.0.smithi042.stdout:2502: finishing copy_from to smithi04268048-36 2024-09-20T19:34:06.602 INFO:tasks.rados.rados.0.smithi042.stdout:2502: got expected ENOENT (src dne) 2024-09-20T19:34:06.602 INFO:tasks.rados.rados.0.smithi042.stdout:2502: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:34:06.610 INFO:tasks.rados.rados.0.smithi042.stdout:2500: expect (ObjNum 827 snap 273 seq_num 827) 2024-09-20T19:34:06.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.045454+0000 osd.5 (osd.5) 219 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.048338+0000 osd.5 (osd.5) 220 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.560721+0000 osd.4 (osd.4) 73 : cluster [DBG] 7.e scrub starts 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:05.563296+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.e scrub ok 2024-09-20T19:34:06.713 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.713 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:06 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.364190+0000 mon.a (mon.0) 2226 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2024-09-20T19:34:06.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2500: done (2 left) 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2502: done (1 left) 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2503: done (0 left) 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2504: read oid 22 snap -1 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2504: expect (ObjNum 836 snap 276 seq_num 836) 2024-09-20T19:34:06.803 INFO:tasks.rados.rados.0.smithi042.stdout:2505: delete oid 41 current snap is 288 2024-09-20T19:34:06.806 INFO:tasks.rados.rados.0.smithi042.stdout:2505: done (1 left) 2024-09-20T19:34:06.806 INFO:tasks.rados.rados.0.smithi042.stdout:2506: read oid 37 snap -1 2024-09-20T19:34:06.806 INFO:tasks.rados.rados.0.smithi042.stdout:2506: expect deleted 2024-09-20T19:34:06.806 INFO:tasks.rados.rados.0.smithi042.stdout:2507: write oid 26 current snap is 288 2024-09-20T19:34:06.807 INFO:tasks.rados.rados.0.smithi042.stdout:2507: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-20T19:34:06.823 INFO:tasks.rados.rados.0.smithi042.stdout:2507: writing smithi04268048-26 from 672511 to 1229130 tid 1 2024-09-20T19:34:06.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.833 INFO:tasks.rados.rados.0.smithi042.stdout:2507: writing smithi04268048-26 from 1819486 to 2574120 tid 2 2024-09-20T19:34:06.833 INFO:tasks.rados.rados.0.smithi042.stdout:2508: delete oid 21 current snap is 288 2024-09-20T19:34:06.833 INFO:tasks.rados.rados.0.smithi042.stdout:2504: expect (ObjNum 836 snap 276 seq_num 836) 2024-09-20T19:34:06.833 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: finishing write tid 1 to smithi04268048-26 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2504: done (3 left) 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2506: done (2 left) 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2508: done (1 left) 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2509: snap_create 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: finishing write tid 2 to smithi04268048-26 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: finishing write tid 3 to smithi04268048-26 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: finishing write tid 4 to smithi04268048-26 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: oid 26 updating version 0 to 1200 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: oid 26 updating version 1200 to 1201 2024-09-20T19:34:06.957 INFO:tasks.rados.rados.0.smithi042.stdout:2507: oid 26 version 1201 is already newer than 1199 2024-09-20T19:34:06.958 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1201 (ObjNum 873 snap 288 seq_num 873) dirty exists 2024-09-20T19:34:06.958 INFO:tasks.rados.rados.0.smithi042.stdout:2507: left oid 26 (ObjNum 873 snap 288 seq_num 873) 2024-09-20T19:34:06.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:06.989 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:06.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:07.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:07.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:07.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:07.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:2507: done (1 left) 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:2509: done (0 left) 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:2510: rollback oid 42 current snap is 289 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 285 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:2511: rollback oid 4 current snap is 289 2024-09-20T19:34:07.491 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 4 to 284 2024-09-20T19:34:07.492 INFO:tasks.rados.rados.0.smithi042.stdout:2512: snap_create 2024-09-20T19:34:07.505 INFO:tasks.rados.rados.0.smithi042.stdout:2510: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:34:07.505 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1046 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:34:07.505 INFO:tasks.rados.rados.0.smithi042.stdout:2511: finishing rollback tid 1 to smithi04268048-4 2024-09-20T19:34:07.505 INFO:tasks.rados.rados.0.smithi042.stdout:2511: finishing rollback tid 2 to smithi04268048-4 2024-09-20T19:34:07.506 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1072 (ObjNum 794 snap 262 seq_num 794) dirty exists 2024-09-20T19:34:07.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:07.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:07.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:07.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:07 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:07.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.072583+0000 osd.5 (osd.5) 221 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:34:07.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.076540+0000 osd.5 (osd.5) 222 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:34:07.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.359975+0000 mgr.y (mgr.24491) 607 : cluster [DBG] pgmap v1105: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 154 MiB data, 729 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 974 KiB/s wr, 9 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.542119+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.f scrub starts 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.544179+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.f scrub ok 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.653560+0000 osd.6 (osd.6) 160 : cluster [DBG] 7.d scrub starts 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:06.656430+0000 osd.6 (osd.6) 161 : cluster [DBG] 7.d scrub ok 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.356033+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:07 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.356075+0000 mon.a (mon.0) 2228 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:07.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:07.899 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:07.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:07.970 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:34:08.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:08.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:08.043 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:34:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:34:08.062 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:34:08.062 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:34:08.079 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:34:08.079 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:34:08.097 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:34:08.097 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:34:08.117 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:34:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:34:08.132 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:08.138 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:34:08.138 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:34:08.155 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:08.165 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:34:08.165 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:34:08.186 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:34:08.187 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:34:08.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:08.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:08.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:08.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:2510: done (2 left) 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:2511: done (1 left) 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:2512: done (0 left) 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:2513: rollback oid 9 current snap is 290 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 268 2024-09-20T19:34:08.492 INFO:tasks.rados.rados.0.smithi042.stdout:2514: delete oid 23 current snap is 290 2024-09-20T19:34:08.497 INFO:tasks.rados.rados.0.smithi042.stdout:2514: done (1 left) 2024-09-20T19:34:08.497 INFO:tasks.rados.rados.0.smithi042.stdout:2515: read oid 1 snap -1 2024-09-20T19:34:08.498 INFO:tasks.rados.rados.0.smithi042.stdout:2515: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:34:08.498 INFO:tasks.rados.rados.0.smithi042.stdout:2516: read oid 37 snap -1 2024-09-20T19:34:08.498 INFO:tasks.rados.rados.0.smithi042.stdout:2516: expect deleted 2024-09-20T19:34:08.498 INFO:tasks.rados.rados.0.smithi042.stdout:2517: snap_remove snap 282 2024-09-20T19:34:08.501 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.501 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.501 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.502 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:08 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:08.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:08.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:08.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:08.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:08.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:08.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.054800+0000 osd.5 (osd.5) 223 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:34:08.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.057070+0000 osd.5 (osd.5) 224 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:34:08.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:07.489959+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: audit 2024-09-20T19:34:08.025099+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.42:0/1315856586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.197949+0000 osd.1 (osd.1) 255 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.199966+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.918 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:08 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.490688+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-09-20T19:34:08.933 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:08.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:09.036 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:09.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:09.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:09.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:09.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:09.506 INFO:tasks.rados.rados.0.smithi042.stdout:2517: done (3 left) 2024-09-20T19:34:09.506 INFO:tasks.rados.rados.0.smithi042.stdout:2518: snap_create 2024-09-20T19:34:09.506 INFO:tasks.rados.rados.0.smithi042.stdout:2513: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:34:09.506 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 0 (ObjNum 113 snap 0 seq_num 1415004112) dirty dne 2024-09-20T19:34:09.506 INFO:tasks.rados.rados.0.smithi042.stdout:2515: expect (ObjNum 737 snap 243 seq_num 737) 2024-09-20T19:34:09.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:09.581 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[22706]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:09 smithi042 bash[32075]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.639 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1567547 2024-09-20T19:34:09.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:09.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.007675+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.010900+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.361268+0000 mgr.y (mgr.24491) 608 : cluster [DBG] pgmap v1107: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 162 MiB data, 728 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.544584+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.546347+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.026531+0000 osd.7 (osd.7) 183 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.034760+0000 osd.7 (osd.7) 184 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:09 smithi162 bash[24557]: cluster 2024-09-20T19:34:09.503291+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-09-20T19:34:09.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:09.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:09.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:09.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:09.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:10.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:10.098 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:10.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:10.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:10.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.487 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2513: done (3 left) 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2515: done (2 left) 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2516: done (1 left) 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2518: done (0 left) 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2519: write (excl) oid 30 current snap is 291 2024-09-20T19:34:10.505 INFO:tasks.rados.rados.0.smithi042.stdout:2519: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-20T19:34:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:10.512 INFO:tasks.rados.rados.0.smithi042.stdout:2519: writing smithi04268048-30 from 722932 to 1221651 tid 1 2024-09-20T19:34:10.521 INFO:tasks.rados.rados.0.smithi042.stdout:2519: writing smithi04268048-30 from 1994932 to 2496893 tid 2 2024-09-20T19:34:10.522 INFO:tasks.rados.rados.0.smithi042.stdout:2520: write oid 29 current snap is 291 2024-09-20T19:34:10.522 INFO:tasks.rados.rados.0.smithi042.stdout:2520: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-20T19:34:10.538 INFO:tasks.rados.rados.0.smithi042.stdout:2520: writing smithi04268048-29 from 739739 to 1313707 tid 1 2024-09-20T19:34:10.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: writing smithi04268048-29 from 2053414 to 2739739 tid 2 2024-09-20T19:34:10.546 INFO:tasks.rados.rados.0.smithi042.stdout:2521: delete oid 13 current snap is 291 2024-09-20T19:34:10.546 INFO:tasks.rados.rados.0.smithi042.stdout:2519: finishing write tid 1 to smithi04268048-30 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: finishing write tid 2 to smithi04268048-30 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: finishing write tid 3 to smithi04268048-30 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: finishing write tid 4 to smithi04268048-30 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: oid 30 updating version 0 to 1069 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: oid 30 version 1069 is already newer than 1068 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: oid 30 version 1069 is already newer than 1067 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1069 (ObjNum 876 snap 291 seq_num 876) dirty exists 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2519: left oid 30 (ObjNum 876 snap 291 seq_num 876) 2024-09-20T19:34:10.547 INFO:tasks.rados.rados.0.smithi042.stdout:2520: finishing write tid 1 to smithi04268048-29 2024-09-20T19:34:10.549 INFO:tasks.rados.rados.0.smithi042.stdout:2519: done (2 left) 2024-09-20T19:34:10.549 INFO:tasks.rados.rados.0.smithi042.stdout:2521: done (1 left) 2024-09-20T19:34:10.549 INFO:tasks.rados.rados.0.smithi042.stdout:2522: write (excl) oid 3 current snap is 291 2024-09-20T19:34:10.549 INFO:tasks.rados.rados.0.smithi042.stdout:2522: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-20T19:34:10.558 INFO:tasks.rados.rados.0.smithi042.stdout:2522: writing smithi04268048-3 from 773353 to 1497819 tid 1 2024-09-20T19:34:10.566 INFO:tasks.rados.rados.0.smithi042.stdout:2522: writing smithi04268048-3 from 2170378 to 2773353 tid 2 2024-09-20T19:34:10.566 INFO:tasks.rados.rados.0.smithi042.stdout:2523: read oid 37 snap -1 2024-09-20T19:34:10.566 INFO:tasks.rados.rados.0.smithi042.stdout:2523: expect deleted 2024-09-20T19:34:10.566 INFO:tasks.rados.rados.0.smithi042.stdout:2524: snap_remove snap 268 2024-09-20T19:34:10.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:10.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.599 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:10 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.610 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:10.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:10.774 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.989644+0000 osd.5 (osd.5) 227 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:08.991810+0000 osd.5 (osd.5) 228 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: audit 2024-09-20T19:34:09.686323+0000 mgr.y (mgr.24491) 609 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.217622+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.220594+0000 osd.1 (osd.1) 258 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: audit 2024-09-20T19:34:10.455403+0000 mon.a (mon.0) 2232 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:10 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.504304+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-09-20T19:34:10.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:10.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:11.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:11.023 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:11.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:34:11.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:11.159 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:11.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:11.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:11.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:11.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:11.544 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2524: done (3 left) 2024-09-20T19:34:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2525: rollback oid 40 current snap is 291 2024-09-20T19:34:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 289 2024-09-20T19:34:11.545 INFO:tasks.rados.rados.0.smithi042.stdout:2520: finishing write tid 2 to smithi04268048-29 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: finishing write tid 3 to smithi04268048-29 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: finishing write tid 4 to smithi04268048-29 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: oid 29 updating version 0 to 831 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: oid 29 version 831 is already newer than 830 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: oid 29 updating version 831 to 832 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 832 (ObjNum 877 snap 291 seq_num 877) dirty exists 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2520: left oid 29 (ObjNum 877 snap 291 seq_num 877) 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2522: finishing write tid 1 to smithi04268048-3 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2522: finishing write tid 2 to smithi04268048-3 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2522: finishing write tid 3 to smithi04268048-3 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2522: finishing write tid 4 to smithi04268048-3 2024-09-20T19:34:11.546 INFO:tasks.rados.rados.0.smithi042.stdout:2522: oid 3 updating version 0 to 950 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2522: oid 3 updating version 950 to 951 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2522: oid 3 version 951 is already newer than 949 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 951 (ObjNum 879 snap 291 seq_num 879) dirty exists 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2522: left oid 3 (ObjNum 879 snap 291 seq_num 879) 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2520: done (3 left) 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2522: done (2 left) 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2523: done (1 left) 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2526: copy_from oid 4 from oid 12 current snap is 291 2024-09-20T19:34:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:2527: snap_remove snap 285 2024-09-20T19:34:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:11.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:11.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:11.778 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.778 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.778 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.778 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[32075]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:11 smithi042 bash[22706]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.010402+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.a scrub starts 2024-09-20T19:34:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.011630+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.a scrub ok 2024-09-20T19:34:11.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:10.361853+0000 mgr.y (mgr.24491) 610 : cluster [DBG] pgmap v1110: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 165 MiB data, 731 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.165936+0000 osd.2 (osd.2) 231 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.169521+0000 osd.2 (osd.2) 232 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.176565+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.179846+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.193375+0000 mon.a (mon.0) 2234 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.194131+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:34:11.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:11 smithi162 bash[24557]: audit 2024-09-20T19:34:11.199438+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:11.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:11.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:11.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:12.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:12.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:12.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:12.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:12.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T19:34:12.544 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:12.566 INFO:tasks.rados.rados.0.smithi042.stdout:2527: done (2 left) 2024-09-20T19:34:12.566 INFO:tasks.rados.rados.0.smithi042.stdout:2528: copy_from oid 49 from oid 32 current snap is 291 2024-09-20T19:34:12.566 INFO:tasks.rados.rados.0.smithi042.stdout:2529: write oid 25 current snap is 291 2024-09-20T19:34:12.566 INFO:tasks.rados.rados.0.smithi042.stdout:2529: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-20T19:34:12.567 INFO:tasks.rados.rados.0.smithi042.stdout:2529: writing smithi04268048-25 from 790160 to 1589875 tid 1 2024-09-20T19:34:12.575 INFO:tasks.rados.rados.0.smithi042.stdout:2529: writing smithi04268048-25 from 2112507 to 2621806 tid 2 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2525: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1043 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2526: finishing copy_from to smithi04268048-4 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2526: got expected ENOENT (src dne) 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2526: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2525: done (3 left) 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2526: done (2 left) 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2530: write (excl) oid 21 current snap is 291 2024-09-20T19:34:12.576 INFO:tasks.rados.rados.0.smithi042.stdout:2530: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-20T19:34:12.581 INFO:tasks.rados.rados.0.smithi042.stdout:2530: writing smithi04268048-21 from 406967 to 881931 tid 1 2024-09-20T19:34:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:12.590 INFO:tasks.rados.rados.0.smithi042.stdout:2530: writing smithi04268048-21 from 1370989 to 2140299 tid 2 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2530: writing smithi04268048-21 from 2677186 to 2806967 tid 3 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2531: read oid 14 snap -1 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2531: expect (ObjNum 827 snap 273 seq_num 827) 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2532: read oid 50 snap -1 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2532: expect deleted 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2528: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: finishing write tid 1 to smithi04268048-25 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: finishing write tid 2 to smithi04268048-25 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: finishing write tid 3 to smithi04268048-25 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: finishing write tid 4 to smithi04268048-25 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: oid 25 updating version 0 to 889 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: oid 25 updating version 889 to 891 2024-09-20T19:34:12.594 INFO:tasks.rados.rados.0.smithi042.stdout:2529: oid 25 version 891 is already newer than 890 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 891 (ObjNum 880 snap 291 seq_num 880) dirty exists 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2529: left oid 25 (ObjNum 880 snap 291 seq_num 880) 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2528: finishing copy_from to smithi04268048-49 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1259 (ObjNum 862 snap 285 seq_num 862) dirty exists 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2530: finishing write tid 1 to smithi04268048-21 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2528: done (4 left) 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2529: done (3 left) 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2533: write oid 28 current snap is 291 2024-09-20T19:34:12.595 INFO:tasks.rados.rados.0.smithi042.stdout:2533: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-20T19:34:12.599 INFO:tasks.rados.rados.0.smithi042.stdout:2533: writing smithi04268048-28 from 423774 to 1090340 tid 1 2024-09-20T19:34:12.605 INFO:tasks.rados.rados.0.smithi042.stdout:2533: writing smithi04268048-28 from 1545824 to 2058792 tid 2 2024-09-20T19:34:12.610 INFO:tasks.rados.rados.0.smithi042.stdout:2533: writing smithi04268048-28 from 2704609 to 2823774 tid 3 2024-09-20T19:34:12.610 INFO:tasks.rados.rados.0.smithi042.stdout:2530: finishing write tid 2 to smithi04268048-21 2024-09-20T19:34:12.610 INFO:tasks.rados.rados.0.smithi042.stdout:2530: finishing write tid 3 to smithi04268048-21 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: finishing write tid 4 to smithi04268048-21 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: finishing write tid 5 to smithi04268048-21 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: oid 21 updating version 0 to 1079 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: oid 21 version 1079 is already newer than 1078 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: oid 21 updating version 1079 to 1081 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: oid 21 version 1081 is already newer than 1080 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1081 (ObjNum 881 snap 291 seq_num 881) dirty exists 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2530: left oid 21 (ObjNum 881 snap 291 seq_num 881) 2024-09-20T19:34:12.611 INFO:tasks.rados.rados.0.smithi042.stdout:2531: expect (ObjNum 827 snap 273 seq_num 827) 2024-09-20T19:34:12.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:12.730 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:34:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:34:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:34:12.731 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:34:12.763 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2019297 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2533: finishing write tid 1 to smithi04268048-28 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2530: done (3 left) 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2531: done (2 left) 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2532: done (1 left) 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2534: read oid 17 snap 278 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2534: expect deleted 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2535: copy_from oid 37 from oid 27 current snap is 291 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2533: finishing write tid 2 to smithi04268048-28 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2533: finishing write tid 3 to smithi04268048-28 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2533: finishing write tid 4 to smithi04268048-28 2024-09-20T19:34:12.802 INFO:tasks.rados.rados.0.smithi042.stdout:2533: finishing write tid 5 to smithi04268048-28 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: oid 28 updating version 0 to 821 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: oid 28 version 821 is already newer than 819 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: oid 28 updating version 821 to 822 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: oid 28 version 822 is already newer than 820 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 822 (ObjNum 882 snap 291 seq_num 882) dirty exists 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: left oid 28 (ObjNum 882 snap 291 seq_num 882) 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2533: done (2 left) 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2536: write (excl) oid 11 current snap is 291 2024-09-20T19:34:12.803 INFO:tasks.rados.rados.0.smithi042.stdout:2536: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-20T19:34:12.807 INFO:tasks.rados.rados.0.smithi042.stdout:2536: writing smithi04268048-11 from 440581 to 1182396 tid 1 2024-09-20T19:34:12.814 INFO:tasks.rados.rados.0.smithi042.stdout:2536: writing smithi04268048-11 from 1604306 to 2260932 tid 2 2024-09-20T19:34:12.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:12.817 INFO:tasks.rados.rados.0.smithi042.stdout:2536: writing smithi04268048-11 from 2732032 to 2840581 tid 3 2024-09-20T19:34:12.817 INFO:tasks.rados.rados.0.smithi042.stdout:2537: write (excl) oid 4 current snap is 291 2024-09-20T19:34:12.817 INFO:tasks.rados.rados.0.smithi042.stdout:2537: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-09-20T19:34:12.822 INFO:tasks.rados.rados.0.smithi042.stdout:2537: writing smithi04268048-4 from 457388 to 874452 tid 1 2024-09-20T19:34:12.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:12 smithi162 bash[24557]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.829 INFO:tasks.rados.rados.0.smithi042.stdout:2537: writing smithi04268048-4 from 1546435 to 2063072 tid 2 2024-09-20T19:34:12.834 INFO:tasks.rados.rados.0.smithi042.stdout:2537: writing smithi04268048-4 from 2643102 to 2857388 tid 3 2024-09-20T19:34:12.834 INFO:tasks.rados.rados.0.smithi042.stdout:2538: copy_from oid 23 from oid 40 current snap is 291 2024-09-20T19:34:12.834 INFO:tasks.rados.rados.0.smithi042.stdout:2535: finishing copy_from to smithi04268048-37 2024-09-20T19:34:12.834 INFO:tasks.rados.rados.0.smithi042.stdout:2535: got expected ENOENT (src dne) 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2535: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: finishing write tid 1 to smithi04268048-11 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: finishing write tid 2 to smithi04268048-11 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: finishing write tid 3 to smithi04268048-11 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: finishing write tid 4 to smithi04268048-11 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: finishing write tid 5 to smithi04268048-11 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: oid 11 updating version 0 to 947 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: oid 11 version 947 is already newer than 946 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: oid 11 updating version 947 to 948 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: oid 11 updating version 948 to 949 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 949 (ObjNum 883 snap 291 seq_num 883) dirty exists 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2536: left oid 11 (ObjNum 883 snap 291 seq_num 883) 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2537: finishing write tid 1 to smithi04268048-4 2024-09-20T19:34:12.835 INFO:tasks.rados.rados.0.smithi042.stdout:2537: finishing write tid 2 to smithi04268048-4 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2534: done (4 left) 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2535: done (3 left) 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2536: done (2 left) 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2539: read oid 10 snap -1 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2539: expect (ObjNum 774 snap 257 seq_num 774) 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2540: write oid 7 current snap is 291 2024-09-20T19:34:12.836 INFO:tasks.rados.rados.0.smithi042.stdout:2540: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-09-20T19:34:12.841 INFO:tasks.rados.rados.0.smithi042.stdout:2540: writing smithi04268048-7 from 474195 to 966508 tid 1 2024-09-20T19:34:12.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:12.848 INFO:tasks.rados.rados.0.smithi042.stdout:2540: writing smithi04268048-7 from 1604917 to 2265212 tid 2 2024-09-20T19:34:12.852 INFO:tasks.rados.rados.0.smithi042.stdout:2540: writing smithi04268048-7 from 2670525 to 2874195 tid 3 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2541: rollback oid 12 current snap is 291 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 283 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2538: finishing copy_from to smithi04268048-23 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2538: got expected ENOENT (src dne) 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2538: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: finishing write tid 3 to smithi04268048-4 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: finishing write tid 4 to smithi04268048-4 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: finishing write tid 5 to smithi04268048-4 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: oid 4 updating version 0 to 1083 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: oid 4 version 1083 is already newer than 1081 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: oid 4 version 1083 is already newer than 1082 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: oid 4 version 1083 is already newer than 1080 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1083 (ObjNum 884 snap 291 seq_num 884) dirty exists 2024-09-20T19:34:12.853 INFO:tasks.rados.rados.0.smithi042.stdout:2537: left oid 4 (ObjNum 884 snap 291 seq_num 884) 2024-09-20T19:34:12.854 INFO:tasks.rados.rados.0.smithi042.stdout:2539: expect (ObjNum 774 snap 257 seq_num 774) 2024-09-20T19:34:12.908 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 303888 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[32075]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.056260+0000 osd.5 (osd.5) 231 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.058048+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: cluster 2024-09-20T19:34:11.542866+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:12 smithi042 bash[22706]: audit 2024-09-20T19:34:11.998766+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:34:12.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:12.965 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 526069 2024-09-20T19:34:12.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:12.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: finishing write tid 1 to smithi04268048-7 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: finishing write tid 2 to smithi04268048-7 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: finishing write tid 3 to smithi04268048-7 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: finishing write tid 4 to smithi04268048-7 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: finishing write tid 5 to smithi04268048-7 2024-09-20T19:34:13.009 INFO:tasks.rados.rados.0.smithi042.stdout:2540: oid 7 updating version 0 to 1203 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2540: oid 7 updating version 1203 to 1205 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2540: oid 7 version 1205 is already newer than 1202 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2540: oid 7 version 1205 is already newer than 1204 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1205 (ObjNum 885 snap 291 seq_num 885) dirty exists 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2540: left oid 7 (ObjNum 885 snap 291 seq_num 885) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2541: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2541: finishing rollback tid 2 to smithi04268048-12 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 728 (ObjNum 794 snap 262 seq_num 794) dirty exists 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2537: done (4 left) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2538: done (3 left) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2539: done (2 left) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2540: done (1 left) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2541: done (0 left) 2024-09-20T19:34:13.010 INFO:tasks.rados.rados.0.smithi042.stdout:2542: snap_create 2024-09-20T19:34:13.077 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:13.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:13.231 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:13.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2542: done (0 left) 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2543: copy_from oid 44 from oid 48 current snap is 292 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2544: copy_from oid 19 from oid 13 current snap is 292 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2545: read oid 32 snap -1 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2545: expect (ObjNum 862 snap 285 seq_num 862) 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2546: write oid 40 current snap is 292 2024-09-20T19:34:13.342 INFO:tasks.rados.rados.0.smithi042.stdout:2546: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-20T19:34:13.350 INFO:tasks.rados.rados.0.smithi042.stdout:2546: writing smithi04268048-40 from 491002 to 1058564 tid 1 2024-09-20T19:34:13.359 INFO:tasks.rados.rados.0.smithi042.stdout:2546: writing smithi04268048-40 from 1663399 to 2183705 tid 2 2024-09-20T19:34:13.364 INFO:tasks.rados.rados.0.smithi042.stdout:2546: writing smithi04268048-40 from 2814301 to 2891002 tid 3 2024-09-20T19:34:13.364 INFO:tasks.rados.rados.0.smithi042.stdout:2547: snap_create 2024-09-20T19:34:13.365 INFO:tasks.rados.rados.0.smithi042.stdout:2544: finishing copy_from to smithi04268048-19 2024-09-20T19:34:13.365 INFO:tasks.rados.rados.0.smithi042.stdout:2544: got expected ENOENT (src dne) 2024-09-20T19:34:13.365 INFO:tasks.rados.rados.0.smithi042.stdout:2544: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:34:13.365 INFO:tasks.rados.rados.0.smithi042.stdout:2545: expect (ObjNum 862 snap 285 seq_num 862) 2024-09-20T19:34:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:13.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:13.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:13.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: finishing write tid 1 to smithi04268048-40 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2543: finishing copy_from racing read to smithi04268048-44 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: finishing write tid 2 to smithi04268048-40 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: finishing write tid 3 to smithi04268048-40 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: finishing write tid 4 to smithi04268048-40 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: finishing write tid 5 to smithi04268048-40 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: oid 40 updating version 0 to 1046 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: oid 40 version 1046 is already newer than 1045 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: oid 40 updating version 1046 to 1047 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: oid 40 version 1047 is already newer than 1044 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1047 (ObjNum 886 snap 292 seq_num 886) dirty exists 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2546: left oid 40 (ObjNum 886 snap 292 seq_num 886) 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:2543: finishing copy_from to smithi04268048-44 2024-09-20T19:34:13.512 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 887 (ObjNum 868 snap 288 seq_num 868) dirty exists 2024-09-20T19:34:13.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:13.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:13 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.732 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:13.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.049083+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.051189+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.362879+0000 mgr.y (mgr.24491) 611 : cluster [DBG] pgmap v1113: 178 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 168 active+clean; 150 MiB data, 717 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:12.541635+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? v1:172.21.15.42:0/816351087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.548766+0000 mon.a (mon.0) 2240 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.569996+0000 osd.4 (osd.4) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:13.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:12.571385+0000 osd.4 (osd.4) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.119518+0000 osd.2 (osd.2) 233 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.122167+0000 osd.2 (osd.2) 234 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:13.125486+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: audit 2024-09-20T19:34:13.126701+0000 mon.a (mon.0) 2242 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:13 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.337578+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-09-20T19:34:13.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:13.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:13.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:13.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:14.018 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:14.121 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:14.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:14.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:14.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2543: done (4 left) 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2544: done (3 left) 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2545: done (2 left) 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2546: done (1 left) 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2547: done (0 left) 2024-09-20T19:34:14.359 INFO:tasks.rados.rados.0.smithi042.stdout:2548: delete oid 3 current snap is 293 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2548: done (0 left) 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2549: rollback oid 9 current snap is 293 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 288 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2550: read oid 46 snap -1 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2550: expect deleted 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2551: write oid 33 current snap is 293 2024-09-20T19:34:14.368 INFO:tasks.rados.rados.0.smithi042.stdout:2551: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-20T19:34:14.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.384 INFO:tasks.rados.rados.0.smithi042.stdout:2551: writing smithi04268048-33 from 524616 to 1242676 tid 1 2024-09-20T19:34:14.397 INFO:tasks.rados.rados.0.smithi042.stdout:2551: writing smithi04268048-33 from 1664010 to 2187985 tid 2 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2551: writing smithi04268048-33 from 2752794 to 2924616 tid 3 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2552: rollback oid 4 current snap is 293 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 4 to 291 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2549: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2551: finishing write tid 1 to smithi04268048-33 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2551: finishing write tid 2 to smithi04268048-33 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2549: done (3 left) 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2550: done (2 left) 2024-09-20T19:34:14.407 INFO:tasks.rados.rados.0.smithi042.stdout:2553: snap_remove snap 288 2024-09-20T19:34:14.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:14.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.537 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:14.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:14.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:14.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:14 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.795 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:14.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.910 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.517605+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:13.521610+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.911 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:14 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.352749+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-09-20T19:34:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:14.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:15.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:15.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:15.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:15.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:15.370 INFO:tasks.rados.rados.0.smithi042.stdout:2553: done (2 left) 2024-09-20T19:34:15.370 INFO:tasks.rados.rados.0.smithi042.stdout:2554: snap_create 2024-09-20T19:34:15.370 INFO:tasks.rados.rados.0.smithi042.stdout:2551: finishing write tid 3 to smithi04268048-33 2024-09-20T19:34:15.370 INFO:tasks.rados.rados.0.smithi042.stdout:2551: finishing write tid 4 to smithi04268048-33 2024-09-20T19:34:15.370 INFO:tasks.rados.rados.0.smithi042.stdout:2551: finishing write tid 5 to smithi04268048-33 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2551: oid 33 updating version 0 to 733 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2551: oid 33 version 733 is already newer than 730 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2551: oid 33 version 733 is already newer than 731 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2551: oid 33 version 733 is already newer than 732 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 733 (ObjNum 888 snap 293 seq_num 888) dirty exists 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2551: left oid 33 (ObjNum 888 snap 293 seq_num 888) 2024-09-20T19:34:15.371 INFO:tasks.rados.rados.0.smithi042.stdout:2552: finishing rollback tid 0 to smithi04268048-4 2024-09-20T19:34:15.372 INFO:tasks.rados.rados.0.smithi042.stdout:2552: finishing rollback tid 1 to smithi04268048-4 2024-09-20T19:34:15.372 INFO:tasks.rados.rados.0.smithi042.stdout:2552: finishing rollback tid 2 to smithi04268048-4 2024-09-20T19:34:15.372 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1091 (ObjNum 884 snap 291 seq_num 884) dirty exists 2024-09-20T19:34:15.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:15.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:15.456 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T19:34:15.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:15.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:15.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:15 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:34:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:15.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:15.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.037677+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.039608+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:14.363611+0000 mgr.y (mgr.24491) 612 : cluster [DBG] pgmap v1117: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 148 MiB data, 708 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:15 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.363167+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-09-20T19:34:15.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:15.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:15.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:15.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:16.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:16.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:16.222 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:16.325 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:16.348 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:16.349 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:16.364 INFO:tasks.rados.rados.0.smithi042.stdout:2551: done (2 left) 2024-09-20T19:34:16.364 INFO:tasks.rados.rados.0.smithi042.stdout:2552: done (1 left) 2024-09-20T19:34:16.364 INFO:tasks.rados.rados.0.smithi042.stdout:2554: done (0 left) 2024-09-20T19:34:16.364 INFO:tasks.rados.rados.0.smithi042.stdout:2555: snap_remove snap 291 2024-09-20T19:34:16.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:16.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:16.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:16.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:16 smithi162 bash[24557]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:16.841 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[22706]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.084526+0000 osd.5 (osd.5) 237 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:34:16.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:15.086889+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.143223+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.144469+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.185661+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.187415+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.362769+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: audit 2024-09-20T19:34:16.466609+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.42:0/2286412783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.856 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.857 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:16 smithi042 bash[32075]: audit 2024-09-20T19:34:16.467073+0000 mon.a (mon.0) 2247 : audit [INF] from='client.25733 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:34:16.869 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:16.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:16.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:17.000 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:17.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:17.132 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:17.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:17.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:34:17.379 INFO:tasks.rados.rados.0.smithi042.stdout:2555: done (0 left) 2024-09-20T19:34:17.379 INFO:tasks.rados.rados.0.smithi042.stdout:2556: delete oid 33 current snap is 294 2024-09-20T19:34:17.391 INFO:tasks.rados.rados.0.smithi042.stdout:2556: done (0 left) 2024-09-20T19:34:17.391 INFO:tasks.rados.rados.0.smithi042.stdout:2557: snap_remove snap 284 2024-09-20T19:34:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:17.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:17.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:17.630 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.662 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.663 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.664 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.664 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:17 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:17.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.049646+0000 osd.5 (osd.5) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.050961+0000 osd.5 (osd.5) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:16.364299+0000 mgr.y (mgr.24491) 613 : cluster [DBG] pgmap v1120: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 141 MiB data, 703 MiB used, 625 GiB / 626 GiB avail; 804 B/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:34:17.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.099936+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.d scrub starts 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.102798+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.d scrub ok 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.140006+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.143375+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.358426+0000 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: audit 2024-09-20T19:34:17.365761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.25733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:34:17.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:17 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.374049+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-09-20T19:34:17.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:17.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:17.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:17.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:18.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:18.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:18.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:18.375 INFO:tasks.rados.rados.0.smithi042.stdout:2557: done (0 left) 2024-09-20T19:34:18.375 INFO:tasks.rados.rados.0.smithi042.stdout:2558: read oid 9 snap 289 2024-09-20T19:34:18.375 INFO:tasks.rados.rados.0.smithi042.stdout:2558: expect deleted 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:2559: rollback oid 12 current snap is 294 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 290 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:2560: read oid 49 snap -1 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:2560: expect (ObjNum 862 snap 285 seq_num 862) 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:2561: write (excl) oid 22 current snap is 294 2024-09-20T19:34:18.378 INFO:tasks.rados.rados.0.smithi042.stdout:2561: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-20T19:34:18.385 INFO:tasks.rados.rados.0.smithi042.stdout:2561: writing smithi04268048-22 from 558230 to 1143141 tid 1 2024-09-20T19:34:18.397 INFO:tasks.rados.rados.0.smithi042.stdout:2561: writing smithi04268048-22 from 1897327 to 2424971 tid 2 2024-09-20T19:34:18.401 INFO:tasks.rados.rados.0.smithi042.stdout:2561: writing smithi04268048-22 from 2923993 to 2958230 tid 3 2024-09-20T19:34:18.401 INFO:tasks.rados.rados.0.smithi042.stdout:2562: read oid 25 snap -1 2024-09-20T19:34:18.401 INFO:tasks.rados.rados.0.smithi042.stdout:2562: expect (ObjNum 880 snap 291 seq_num 880) 2024-09-20T19:34:18.401 INFO:tasks.rados.rados.0.smithi042.stdout:2563: snap_create 2024-09-20T19:34:18.401 INFO:tasks.rados.rados.0.smithi042.stdout:2560: expect (ObjNum 862 snap 285 seq_num 862) 2024-09-20T19:34:18.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:18.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:18.472 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:18.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.564 INFO:tasks.rados.rados.0.smithi042.stdout:2562: expect (ObjNum 880 snap 291 seq_num 880) 2024-09-20T19:34:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:18.641 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:18.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:18 smithi162 bash[24557]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:18.719 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:18.797 INFO:tasks.rados.rados.0.smithi042.stdout:2559: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:34:18.797 INFO:tasks.rados.rados.0.smithi042.stdout:2559: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:34:18.797 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 730 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:34:18.797 INFO:tasks.rados.rados.0.smithi042.stdout:2561: finishing write tid 1 to smithi04268048-22 2024-09-20T19:34:18.797 INFO:tasks.rados.rados.0.smithi042.stdout:2561: finishing write tid 2 to smithi04268048-22 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: finishing write tid 3 to smithi04268048-22 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: finishing write tid 4 to smithi04268048-22 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: finishing write tid 5 to smithi04268048-22 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: oid 22 updating version 0 to 1024 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: oid 22 version 1024 is already newer than 1023 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: oid 22 version 1024 is already newer than 1022 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: oid 22 version 1024 is already newer than 1021 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1024 (ObjNum 890 snap 294 seq_num 890) dirty exists 2024-09-20T19:34:18.798 INFO:tasks.rados.rados.0.smithi042.stdout:2561: left oid 22 (ObjNum 890 snap 294 seq_num 890) 2024-09-20T19:34:18.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.820 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[22706]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.110866+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.d scrub starts 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:17.113158+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.d scrub ok 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.821 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:18 smithi042 bash[32075]: cluster 2024-09-20T19:34:18.374141+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-09-20T19:34:18.834 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:18.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:18.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:19.005 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:19.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:19.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:19.198 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:19.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:19.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:19.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:19.381 INFO:tasks.rados.rados.0.smithi042.stdout:2558: done (5 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2559: done (4 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2560: done (3 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2561: done (2 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2562: done (1 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2563: done (0 left) 2024-09-20T19:34:19.382 INFO:tasks.rados.rados.0.smithi042.stdout:2564: read oid 47 snap 278 2024-09-20T19:34:19.383 INFO:tasks.rados.rados.0.smithi042.stdout:2564: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:34:19.383 INFO:tasks.rados.rados.0.smithi042.stdout:2565: read oid 6 snap -1 2024-09-20T19:34:19.383 INFO:tasks.rados.rados.0.smithi042.stdout:2565: expect deleted 2024-09-20T19:34:19.383 INFO:tasks.rados.rados.0.smithi042.stdout:2566: delete oid 27 current snap is 295 2024-09-20T19:34:19.395 INFO:tasks.rados.rados.0.smithi042.stdout:2564: expect (ObjNum 787 snap 260 seq_num 787) 2024-09-20T19:34:19.431 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2738559 2024-09-20T19:34:19.435 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:19.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:19.529 INFO:tasks.rados.rados.0.smithi042.stdout:2564: done (2 left) 2024-09-20T19:34:19.529 INFO:tasks.rados.rados.0.smithi042.stdout:2565: done (1 left) 2024-09-20T19:34:19.529 INFO:tasks.rados.rados.0.smithi042.stdout:2566: done (0 left) 2024-09-20T19:34:19.529 INFO:tasks.rados.rados.0.smithi042.stdout:2567: write (excl) oid 49 current snap is 295 2024-09-20T19:34:19.530 INFO:tasks.rados.rados.0.smithi042.stdout:2567: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-20T19:34:19.536 INFO:tasks.rados.rados.0.smithi042.stdout:2567: writing smithi04268048-49 from 591844 to 1327253 tid 1 2024-09-20T19:34:19.542 INFO:tasks.rados.rados.0.smithi042.stdout:2567: writing smithi04268048-49 from 1897938 to 2429251 tid 2 2024-09-20T19:34:19.546 INFO:tasks.rados.rados.0.smithi042.stdout:2567: writing smithi04268048-49 from 2862486 to 2991844 tid 3 2024-09-20T19:34:19.546 INFO:tasks.rados.rados.0.smithi042.stdout:2568: read oid 35 snap 292 2024-09-20T19:34:19.546 INFO:tasks.rados.rados.0.smithi042.stdout:2568: expect (ObjNum 854 snap 282 seq_num 854) 2024-09-20T19:34:19.546 INFO:tasks.rados.rados.0.smithi042.stdout:2569: write (excl) oid 37 current snap is 295 2024-09-20T19:34:19.546 INFO:tasks.rados.rados.0.smithi042.stdout:2569: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-20T19:34:19.552 INFO:tasks.rados.rados.0.smithi042.stdout:2569: writing smithi04268048-37 from 608651 to 1019309 tid 1 2024-09-20T19:34:19.559 INFO:tasks.rados.rados.0.smithi042.stdout:2569: writing smithi04268048-37 from 1556420 to 2231391 tid 2 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2569: writing smithi04268048-37 from 2773556 to 3008651 tid 3 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2570: read oid 11 snap -1 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2570: expect (ObjNum 883 snap 291 seq_num 883) 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2571: write oid 38 current snap is 295 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2567: finishing write tid 1 to smithi04268048-49 2024-09-20T19:34:19.564 INFO:tasks.rados.rados.0.smithi042.stdout:2571: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-09-20T19:34:19.570 INFO:tasks.rados.rados.0.smithi042.stdout:2571: writing smithi04268048-38 from 625458 to 1111365 tid 1 2024-09-20T19:34:19.576 INFO:tasks.rados.rados.0.smithi042.stdout:2571: writing smithi04268048-38 from 1614902 to 2149884 tid 2 2024-09-20T19:34:19.581 INFO:tasks.rados.rados.0.smithi042.stdout:2571: writing smithi04268048-38 from 2917332 to 3025458 tid 3 2024-09-20T19:34:19.581 INFO:tasks.rados.rados.0.smithi042.stdout:2572: write (excl) oid 12 current snap is 295 2024-09-20T19:34:19.581 INFO:tasks.rados.rados.0.smithi042.stdout:2572: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-09-20T19:34:19.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:19.588 INFO:tasks.rados.rados.0.smithi042.stdout:2572: writing smithi04268048-12 from 642265 to 1203421 tid 1 2024-09-20T19:34:19.595 INFO:tasks.rados.rados.0.smithi042.stdout:2572: writing smithi04268048-12 from 1673384 to 2352024 tid 2 2024-09-20T19:34:19.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:19 smithi162 bash[24557]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:19 smithi162 bash[24557]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:19 smithi162 bash[24557]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:19 smithi162 bash[24557]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.599 INFO:tasks.rados.rados.0.smithi042.stdout:2572: writing smithi04268048-12 from 2828402 to 3042265 tid 3 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2573: read oid 41 snap -1 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2573: expect deleted 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2574: delete oid 40 current snap is 295 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: finishing write tid 2 to smithi04268048-49 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: finishing write tid 3 to smithi04268048-49 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: finishing write tid 4 to smithi04268048-49 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: finishing write tid 5 to smithi04268048-49 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: oid 49 updating version 0 to 1279 2024-09-20T19:34:19.600 INFO:tasks.rados.rados.0.smithi042.stdout:2567: oid 49 updating version 1279 to 1280 2024-09-20T19:34:19.601 INFO:tasks.rados.rados.0.smithi042.stdout:2567: oid 49 version 1280 is already newer than 1277 2024-09-20T19:34:19.601 INFO:tasks.rados.rados.0.smithi042.stdout:2567: oid 49 version 1280 is already newer than 1278 2024-09-20T19:34:19.602 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1280 (ObjNum 892 snap 295 seq_num 892) dirty exists 2024-09-20T19:34:19.602 INFO:tasks.rados.rados.0.smithi042.stdout:2567: left oid 49 (ObjNum 892 snap 295 seq_num 892) 2024-09-20T19:34:19.603 INFO:tasks.rados.rados.0.smithi042.stdout:2568: expect (ObjNum 854 snap 282 seq_num 854) 2024-09-20T19:34:19.613 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:19.670 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:19.704 INFO:tasks.rados.rados.0.smithi042.stdout:2569: finishing write tid 1 to smithi04268048-37 2024-09-20T19:34:19.705 INFO:tasks.rados.rados.0.smithi042.stdout:2569: finishing write tid 2 to smithi04268048-37 2024-09-20T19:34:19.705 INFO:tasks.rados.rados.0.smithi042.stdout:2569: finishing write tid 3 to smithi04268048-37 2024-09-20T19:34:19.705 INFO:tasks.rados.rados.0.smithi042.stdout:2569: finishing write tid 4 to smithi04268048-37 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: finishing write tid 5 to smithi04268048-37 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: oid 37 updating version 0 to 808 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: oid 37 version 808 is already newer than 807 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: oid 37 updating version 808 to 809 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: oid 37 updating version 809 to 810 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 810 (ObjNum 893 snap 295 seq_num 893) dirty exists 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2569: left oid 37 (ObjNum 893 snap 295 seq_num 893) 2024-09-20T19:34:19.706 INFO:tasks.rados.rados.0.smithi042.stdout:2570: expect (ObjNum 883 snap 291 seq_num 883) 2024-09-20T19:34:19.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:19.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[32075]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[32075]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[32075]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[32075]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[22706]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[22706]: cluster 2024-09-20T19:34:18.365260+0000 mgr.y (mgr.24491) 614 : cluster [DBG] pgmap v1122: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 145 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:19.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[22706]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[22706]: cluster 2024-09-20T19:34:19.378367+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-09-20T19:34:19.727 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:34:19 smithi042 bash[109938]: debug 2024-09-20T19:34:19.599+0000 7fdcd6a8a640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 11e after trimming was started 2024-09-20T19:34:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:19.796 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2832400 2024-09-20T19:34:19.797 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:19.843 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:19.849 INFO:tasks.rados.rados.0.smithi042.stdout:2571: finishing write tid 1 to smithi04268048-38 2024-09-20T19:34:19.849 INFO:tasks.rados.rados.0.smithi042.stdout:2571: finishing write tid 2 to smithi04268048-38 2024-09-20T19:34:19.849 INFO:tasks.rados.rados.0.smithi042.stdout:2571: finishing write tid 3 to smithi04268048-38 2024-09-20T19:34:19.849 INFO:tasks.rados.rados.0.smithi042.stdout:2571: finishing write tid 4 to smithi04268048-38 2024-09-20T19:34:19.849 INFO:tasks.rados.rados.0.smithi042.stdout:2571: finishing write tid 5 to smithi04268048-38 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2571: oid 38 updating version 0 to 842 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2571: oid 38 version 842 is already newer than 841 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2571: oid 38 updating version 842 to 843 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2571: oid 38 updating version 843 to 844 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 844 (ObjNum 894 snap 295 seq_num 894) dirty exists 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2571: left oid 38 (ObjNum 894 snap 295 seq_num 894) 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: finishing write tid 1 to smithi04268048-12 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: finishing write tid 2 to smithi04268048-12 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: finishing write tid 3 to smithi04268048-12 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: finishing write tid 4 to smithi04268048-12 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: finishing write tid 5 to smithi04268048-12 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: oid 12 updating version 0 to 731 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: oid 12 updating version 731 to 732 2024-09-20T19:34:19.850 INFO:tasks.rados.rados.0.smithi042.stdout:2572: oid 12 updating version 732 to 733 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2572: oid 12 updating version 733 to 734 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 734 (ObjNum 895 snap 295 seq_num 895) dirty exists 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2572: left oid 12 (ObjNum 895 snap 295 seq_num 895) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2567: done (7 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2568: done (6 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2569: done (5 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2570: done (4 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2571: done (3 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2572: done (2 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2573: done (1 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2574: done (0 left) 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2575: copy_from oid 19 from oid 14 current snap is 295 2024-09-20T19:34:19.851 INFO:tasks.rados.rados.0.smithi042.stdout:2576: write (excl) oid 40 current snap is 295 2024-09-20T19:34:19.852 INFO:tasks.rados.rados.0.smithi042.stdout:2576: seq_num 897 ranges {675879=711654,2073995=682309} 2024-09-20T19:34:19.859 INFO:tasks.rados.rados.0.smithi042.stdout:2576: writing smithi04268048-40 from 675879 to 1387533 tid 1 2024-09-20T19:34:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:19.871 INFO:tasks.rados.rados.0.smithi042.stdout:2576: writing smithi04268048-40 from 2073995 to 2756304 tid 2 2024-09-20T19:34:19.871 INFO:tasks.rados.rados.0.smithi042.stdout:2577: rollback oid 17 current snap is 295 2024-09-20T19:34:19.872 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 289 2024-09-20T19:34:19.872 INFO:tasks.rados.rados.0.smithi042.stdout:2576: finishing write tid 1 to smithi04268048-40 2024-09-20T19:34:19.872 INFO:tasks.rados.rados.0.smithi042.stdout:2578: write oid 23 current snap is 295 2024-09-20T19:34:19.872 INFO:tasks.rados.rados.0.smithi042.stdout:2578: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-20T19:34:19.879 INFO:tasks.rados.rados.0.smithi042.stdout:2578: writing smithi04268048-23 from 692686 to 1195942 tid 1 2024-09-20T19:34:19.887 INFO:tasks.rados.rados.0.smithi042.stdout:2578: writing smithi04268048-23 from 1848830 to 2274797 tid 2 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2578: writing smithi04268048-23 from 2910671 to 3092686 tid 3 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2579: snap_create 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2577: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 0 (ObjNum 49 snap 0 seq_num 1390888752) dirty dne 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2575: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2576: finishing write tid 2 to smithi04268048-40 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2575: finishing copy_from to smithi04268048-19 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 816 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:34:19.893 INFO:tasks.rados.rados.0.smithi042.stdout:2576: finishing write tid 3 to smithi04268048-40 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2576: finishing write tid 4 to smithi04268048-40 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2576: oid 40 updating version 0 to 1049 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2576: oid 40 updating version 1049 to 1051 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2576: oid 40 version 1051 is already newer than 1050 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1051 (ObjNum 897 snap 295 seq_num 897) dirty exists 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2576: left oid 40 (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2578: finishing write tid 1 to smithi04268048-23 2024-09-20T19:34:19.894 INFO:tasks.rados.rados.0.smithi042.stdout:2578: finishing write tid 2 to smithi04268048-23 2024-09-20T19:34:19.895 INFO:tasks.rados.rados.0.smithi042.stdout:2578: finishing write tid 3 to smithi04268048-23 2024-09-20T19:34:19.896 INFO:tasks.rados.rados.0.smithi042.stdout:2578: finishing write tid 4 to smithi04268048-23 2024-09-20T19:34:19.896 INFO:tasks.rados.rados.0.smithi042.stdout:2578: finishing write tid 5 to smithi04268048-23 2024-09-20T19:34:19.896 INFO:tasks.rados.rados.0.smithi042.stdout:2578: oid 23 updating version 0 to 953 2024-09-20T19:34:19.898 INFO:tasks.rados.rados.0.smithi042.stdout:2578: oid 23 updating version 953 to 954 2024-09-20T19:34:19.898 INFO:tasks.rados.rados.0.smithi042.stdout:2578: oid 23 version 954 is already newer than 952 2024-09-20T19:34:19.898 INFO:tasks.rados.rados.0.smithi042.stdout:2578: oid 23 version 954 is already newer than 951 2024-09-20T19:34:19.898 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 954 (ObjNum 898 snap 295 seq_num 898) dirty exists 2024-09-20T19:34:19.898 INFO:tasks.rados.rados.0.smithi042.stdout:2578: left oid 23 (ObjNum 898 snap 295 seq_num 898) 2024-09-20T19:34:19.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:19.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:19.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:20.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:20.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:20.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:20.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:20.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:20.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:20.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:20.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:20.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:34:20.584 INFO:tasks.rados.rados.0.smithi042.stdout:2575: done (4 left) 2024-09-20T19:34:20.584 INFO:tasks.rados.rados.0.smithi042.stdout:2576: done (3 left) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2577: done (2 left) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2578: done (1 left) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2579: done (0 left) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2580: read oid 15 snap 290 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2580: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2581: read oid 38 snap -1 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2581: expect (ObjNum 894 snap 295 seq_num 894) 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2582: write oid 10 current snap is 296 2024-09-20T19:34:20.585 INFO:tasks.rados.rados.0.smithi042.stdout:2582: seq_num 899 ranges {709493=578505,1907312=685978} 2024-09-20T19:34:20.593 INFO:tasks.rados.rados.0.smithi042.stdout:2582: writing smithi04268048-10 from 709493 to 1287998 tid 1 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2582: writing smithi04268048-10 from 1907312 to 2593290 tid 2 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2583: read oid 43 snap 283 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2583: expect deleted 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2584: read oid 8 snap -1 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2584: expect (ObjNum 847 snap 278 seq_num 847) 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2585: write (excl) oid 39 current snap is 296 2024-09-20T19:34:20.603 INFO:tasks.rados.rados.0.smithi042.stdout:2581: expect (ObjNum 894 snap 295 seq_num 894) 2024-09-20T19:34:20.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:20.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:20.713 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2783891 2024-09-20T19:34:20.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:20.746 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:34:20.746 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:34:20.754 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:20 smithi162 bash[24557]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.754 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:20 smithi162 bash[24557]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.766 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:34:20.766 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:34:20.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:20.783 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:34:20.784 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:34:20.794 INFO:tasks.rados.rados.0.smithi042.stdout:2580: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:34:20.800 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:34:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:34:20.819 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:34:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:34:20.839 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:34:20.840 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:34:20.857 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:34:20.858 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:34:20.870 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2848637 2024-09-20T19:34:20.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:20.878 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:34:20.878 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:34:20.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:20 smithi042 bash[22706]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:20 smithi042 bash[22706]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:20 smithi042 bash[32075]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:20 smithi042 bash[32075]: audit 2024-09-20T19:34:19.697746+0000 mgr.y (mgr.24491) 615 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:20.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:20.941 INFO:tasks.rados.rados.0.smithi042.stdout:2582: finishing write tid 1 to smithi04268048-10 2024-09-20T19:34:20.941 INFO:tasks.rados.rados.0.smithi042.stdout:2584: expect (ObjNum 847 snap 278 seq_num 847) 2024-09-20T19:34:20.941 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:20.969 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 780745 2024-09-20T19:34:21.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:21.008 INFO:tasks.rados.rados.0.smithi042.stdout:2582: finishing write tid 2 to smithi04268048-10 2024-09-20T19:34:21.008 INFO:tasks.rados.rados.0.smithi042.stdout:2582: finishing write tid 3 to smithi04268048-10 2024-09-20T19:34:21.008 INFO:tasks.rados.rados.0.smithi042.stdout:2582: finishing write tid 4 to smithi04268048-10 2024-09-20T19:34:21.009 INFO:tasks.rados.rados.0.smithi042.stdout:2582: oid 10 updating version 0 to 836 2024-09-20T19:34:21.009 INFO:tasks.rados.rados.0.smithi042.stdout:2582: oid 10 version 836 is already newer than 834 2024-09-20T19:34:21.009 INFO:tasks.rados.rados.0.smithi042.stdout:2582: oid 10 version 836 is already newer than 835 2024-09-20T19:34:21.009 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 836 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:34:21.010 INFO:tasks.rados.rados.0.smithi042.stdout:2582: left oid 10 (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:34:21.010 INFO:tasks.rados.rados.0.smithi042.stdout:2585: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-09-20T19:34:21.021 INFO:tasks.rados.rados.0.smithi042.stdout:2585: writing smithi04268048-39 from 726300 to 1380054 tid 1 2024-09-20T19:34:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:21.034 INFO:tasks.rados.rados.0.smithi042.stdout:2585: writing smithi04268048-39 from 1965794 to 2395430 tid 2 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2585: writing smithi04268048-39 from 2965517 to 3126300 tid 3 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2580: done (5 left) 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2581: done (4 left) 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2582: done (3 left) 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2583: done (2 left) 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2584: done (1 left) 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2586: delete oid 12 current snap is 296 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2585: finishing write tid 1 to smithi04268048-39 2024-09-20T19:34:21.041 INFO:tasks.rados.rados.0.smithi042.stdout:2585: finishing write tid 2 to smithi04268048-39 2024-09-20T19:34:21.044 INFO:tasks.rados.rados.0.smithi042.stdout:2586: done (1 left) 2024-09-20T19:34:21.044 INFO:tasks.rados.rados.0.smithi042.stdout:2587: read oid 38 snap -1 2024-09-20T19:34:21.044 INFO:tasks.rados.rados.0.smithi042.stdout:2587: expect (ObjNum 894 snap 295 seq_num 894) 2024-09-20T19:34:21.044 INFO:tasks.rados.rados.0.smithi042.stdout:2588: delete oid 35 current snap is 296 2024-09-20T19:34:21.045 INFO:tasks.rados.rados.0.smithi042.stdout:2585: finishing write tid 3 to smithi04268048-39 2024-09-20T19:34:21.045 INFO:tasks.rados.rados.0.smithi042.stdout:2585: finishing write tid 4 to smithi04268048-39 2024-09-20T19:34:21.046 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:21.046 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:21.047 INFO:tasks.rados.rados.0.smithi042.stdout:2587: expect (ObjNum 894 snap 295 seq_num 894) 2024-09-20T19:34:21.059 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1029749 2024-09-20T19:34:21.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:21.157 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:21.161 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2822086 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: finishing write tid 5 to smithi04268048-39 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: oid 39 updating version 0 to 732 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: oid 39 updating version 732 to 734 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: oid 39 version 734 is already newer than 733 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: oid 39 version 734 is already newer than 731 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 734 (ObjNum 900 snap 296 seq_num 900) dirty exists 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: left oid 39 (ObjNum 900 snap 296 seq_num 900) 2024-09-20T19:34:21.235 INFO:tasks.rados.rados.0.smithi042.stdout:2585: done (2 left) 2024-09-20T19:34:21.236 INFO:tasks.rados.rados.0.smithi042.stdout:2587: done (1 left) 2024-09-20T19:34:21.236 INFO:tasks.rados.rados.0.smithi042.stdout:2588: done (0 left) 2024-09-20T19:34:21.236 INFO:tasks.rados.rados.0.smithi042.stdout:2589: snap_create 2024-09-20T19:34:21.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:21.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:21.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:21.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:21.539 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:21.642 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:21.677 INFO:tasks.rados.rados.0.smithi042.stdout:2589: done (0 left) 2024-09-20T19:34:21.677 INFO:tasks.rados.rados.0.smithi042.stdout:2590: write oid 45 current snap is 297 2024-09-20T19:34:21.677 INFO:tasks.rados.rados.0.smithi042.stdout:2590: seq_num 903 ranges {776721=479501,2024887=693316} 2024-09-20T19:34:21.683 INFO:tasks.rados.rados.0.smithi042.stdout:2590: writing smithi04268048-45 from 776721 to 1256222 tid 1 2024-09-20T19:34:21.692 INFO:tasks.rados.rados.0.smithi042.stdout:2590: writing smithi04268048-45 from 2024887 to 2718203 tid 2 2024-09-20T19:34:21.692 INFO:tasks.rados.rados.0.smithi042.stdout:2591: snap_create 2024-09-20T19:34:21.694 INFO:tasks.rados.rados.0.smithi042.stdout:2590: finishing write tid 1 to smithi04268048-45 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: finishing write tid 2 to smithi04268048-45 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: finishing write tid 3 to smithi04268048-45 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: finishing write tid 4 to smithi04268048-45 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: oid 45 updating version 0 to 1282 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: oid 45 updating version 1282 to 1283 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: oid 45 updating version 1283 to 1284 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1284 (ObjNum 903 snap 297 seq_num 903) dirty exists 2024-09-20T19:34:21.699 INFO:tasks.rados.rados.0.smithi042.stdout:2590: left oid 45 (ObjNum 903 snap 297 seq_num 903) 2024-09-20T19:34:21.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:21.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:21.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:21.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:21.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:21.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:21.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:21 smithi162 bash[24557]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:21.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:21.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:21.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:21.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:21.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:21.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:21.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:21.863 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[22706]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:21.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:21.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:21.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:22.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:22.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:22.142 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: cluster 2024-09-20T19:34:20.366111+0000 mgr.y (mgr.24491) 616 : cluster [DBG] pgmap v1125: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 147 MiB data, 685 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-20T19:34:22.142 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:22.142 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: cluster 2024-09-20T19:34:20.581988+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-09-20T19:34:22.142 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:22.142 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:21 smithi042 bash[32075]: audit 2024-09-20T19:34:20.726292+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.42:0/2857147837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:22.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:22.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:22.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:22.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:22.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:22.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:22.584 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:22.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:22.691 INFO:tasks.rados.rados.0.smithi042.stdout:2590: done (1 left) 2024-09-20T19:34:22.691 INFO:tasks.rados.rados.0.smithi042.stdout:2591: done (0 left) 2024-09-20T19:34:22.691 INFO:tasks.rados.rados.0.smithi042.stdout:2592: copy_from oid 37 from oid 20 current snap is 298 2024-09-20T19:34:22.691 INFO:tasks.rados.rados.0.smithi042.stdout:2593: write (excl) oid 38 current snap is 298 2024-09-20T19:34:22.691 INFO:tasks.rados.rados.0.smithi042.stdout:2593: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-09-20T19:34:22.710 INFO:tasks.rados.rados.0.smithi042.stdout:2593: writing smithi04268048-38 from 793528 to 1348278 tid 1 2024-09-20T19:34:22.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:22.728 INFO:tasks.rados.rados.0.smithi042.stdout:2593: writing smithi04268048-38 from 2083369 to 2520343 tid 2 2024-09-20T19:34:22.737 INFO:tasks.rados.rados.0.smithi042.stdout:2593: writing smithi04268048-38 from 2958856 to 3193528 tid 3 2024-09-20T19:34:22.737 INFO:tasks.rados.rados.0.smithi042.stdout:2594: snap_remove snap 283 2024-09-20T19:34:22.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:22.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[32075]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:22.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:22 smithi042 bash[22706]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:22.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:22.979 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:22.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:23.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:23.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.014478+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:23.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:23.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.017267+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:23.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:23.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:22 smithi162 bash[24557]: cluster 2024-09-20T19:34:21.675928+0000 mon.a (mon.0) 2255 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-09-20T19:34:23.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:23.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:23.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:23.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2594: done (2 left) 2024-09-20T19:34:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2595: delete oid 6 current snap is 298 2024-09-20T19:34:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2592: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:34:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:2592: finishing copy_from to smithi04268048-37 2024-09-20T19:34:23.346 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 995 (ObjNum 858 snap 284 seq_num 858) dirty exists 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: finishing write tid 1 to smithi04268048-38 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: finishing write tid 2 to smithi04268048-38 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: finishing write tid 3 to smithi04268048-38 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: finishing write tid 4 to smithi04268048-38 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: finishing write tid 5 to smithi04268048-38 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: oid 38 updating version 0 to 849 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: oid 38 version 849 is already newer than 848 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: oid 38 version 849 is already newer than 846 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: oid 38 version 849 is already newer than 847 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 849 (ObjNum 904 snap 298 seq_num 904) dirty exists 2024-09-20T19:34:23.347 INFO:tasks.rados.rados.0.smithi042.stdout:2593: left oid 38 (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:34:23.386 INFO:tasks.rados.rados.0.smithi042.stdout:2592: done (2 left) 2024-09-20T19:34:23.386 INFO:tasks.rados.rados.0.smithi042.stdout:2593: done (1 left) 2024-09-20T19:34:23.386 INFO:tasks.rados.rados.0.smithi042.stdout:2595: done (0 left) 2024-09-20T19:34:23.387 INFO:tasks.rados.rados.0.smithi042.stdout:2596: read oid 38 snap -1 2024-09-20T19:34:23.387 INFO:tasks.rados.rados.0.smithi042.stdout:2596: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:34:23.387 INFO:tasks.rados.rados.0.smithi042.stdout:2597: write (excl) oid 7 current snap is 298 2024-09-20T19:34:23.387 INFO:tasks.rados.rados.0.smithi042.stdout:2597: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-09-20T19:34:23.387 INFO:tasks.rados.rados.0.smithi042.stdout:2597: writing smithi04268048-7 from 427142 to 848743 tid 1 2024-09-20T19:34:23.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:23.391 INFO:tasks.rados.rados.0.smithi042.stdout:2597: writing smithi04268048-7 from 1400333 to 1840976 tid 2 2024-09-20T19:34:23.398 INFO:tasks.rados.rados.0.smithi042.stdout:2597: writing smithi04268048-7 from 2613702 to 3064028 tid 3 2024-09-20T19:34:23.398 INFO:tasks.rados.rados.0.smithi042.stdout:2598: snap_create 2024-09-20T19:34:23.398 INFO:tasks.rados.rados.0.smithi042.stdout:2597: finishing write tid 1 to smithi04268048-7 2024-09-20T19:34:23.398 INFO:tasks.rados.rados.0.smithi042.stdout:2597: finishing write tid 2 to smithi04268048-7 2024-09-20T19:34:23.398 INFO:tasks.rados.rados.0.smithi042.stdout:2596: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:34:23.399 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:23.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:23.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:23.583 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1220940 2024-09-20T19:34:23.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:23.636 INFO:tasks.rados.rados.0.smithi042.stdout:2597: finishing write tid 3 to smithi04268048-7 2024-09-20T19:34:23.636 INFO:tasks.rados.rados.0.smithi042.stdout:2597: finishing write tid 4 to smithi04268048-7 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: finishing write tid 5 to smithi04268048-7 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: oid 7 updating version 0 to 1206 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: oid 7 updating version 1206 to 1209 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: oid 7 version 1209 is already newer than 1208 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: oid 7 version 1209 is already newer than 1207 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1209 (ObjNum 906 snap 298 seq_num 906) dirty exists 2024-09-20T19:34:23.637 INFO:tasks.rados.rados.0.smithi042.stdout:2597: left oid 7 (ObjNum 906 snap 298 seq_num 906) 2024-09-20T19:34:23.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:23.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:23.773 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:23.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:23 smithi162 bash[24557]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:23.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:23.905 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:23.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:24.008 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:24.020 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[32075]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:22.367683+0000 mgr.y (mgr.24491) 617 : cluster [DBG] pgmap v1128: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 160 MiB data, 718 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 31 op/s 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:22.687695+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:24.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:23 smithi042 bash[22706]: cluster 2024-09-20T19:34:23.341865+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-09-20T19:34:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:24.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:24.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:24.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2596: done (2 left) 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2597: done (1 left) 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2598: done (0 left) 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2599: read oid 18 snap 298 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2599: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:24.364 INFO:tasks.rados.rados.0.smithi042.stdout:2600: snap_remove snap 294 2024-09-20T19:34:24.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:24.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:24.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:24.537 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:34:24.537 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:34:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:24.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:24.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:24.802 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:24.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:24.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:24.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:25.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:25.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:25.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:25.189 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:25.288 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:34:25.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:2600: done (1 left) 2024-09-20T19:34:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:2601: read oid 15 snap 278 2024-09-20T19:34:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:2601: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:34:25.371 INFO:tasks.rados.rados.0.smithi042.stdout:2602: snap_remove snap 297 2024-09-20T19:34:25.372 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T19:34:25.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:25.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:25.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:34:25.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.573 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[32075]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:25 smithi042 bash[22706]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:25.708 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.732 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: cluster 2024-09-20T19:34:24.361428+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: cluster 2024-09-20T19:34:24.368763+0000 mgr.y (mgr.24491) 618 : cluster [DBG] pgmap v1132: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 168 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.5 MiB/s wr, 39 op/s 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: audit 2024-09-20T19:34:24.780477+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.42:0/1084692757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:25 smithi162 bash[24557]: audit 2024-09-20T19:34:24.780878+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25745 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:34:25.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:25.862 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:25.965 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:25.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:26.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:26.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:26.351 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T19:34:26.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.375 INFO:tasks.rados.rados.0.smithi042.stdout:2602: done (2 left) 2024-09-20T19:34:26.375 INFO:tasks.rados.rados.0.smithi042.stdout:2603: snap_remove snap 293 2024-09-20T19:34:26.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:26.496 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[32075]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[32075]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[32075]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[32075]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[22706]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[22706]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[22706]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:26 smithi042 bash[22706]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:26.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:26 smithi162 bash[24557]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:26 smithi162 bash[24557]: audit 2024-09-20T19:34:25.361212+0000 mon.a (mon.0) 2260 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:34:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:26 smithi162 bash[24557]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:26 smithi162 bash[24557]: cluster 2024-09-20T19:34:25.366560+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-09-20T19:34:26.645 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:26.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:26.778 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:26.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:26.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:26.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:27.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:27.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:27.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:27.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:27.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:27.394 INFO:tasks.rados.rados.0.smithi042.stdout:2603: done (2 left) 2024-09-20T19:34:27.394 INFO:tasks.rados.rados.0.smithi042.stdout:2604: write (excl) oid 37 current snap is 299 2024-09-20T19:34:27.394 INFO:tasks.rados.rados.0.smithi042.stdout:2604: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-20T19:34:27.396 INFO:tasks.rados.rados.0.smithi042.stdout:2604: writing smithi04268048-37 from 443949 to 940799 tid 1 2024-09-20T19:34:27.404 INFO:tasks.rados.rados.0.smithi042.stdout:2604: writing smithi04268048-37 from 1458815 to 2043116 tid 2 2024-09-20T19:34:27.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:27.414 INFO:tasks.rados.rados.0.smithi042.stdout:2604: writing smithi04268048-37 from 2641125 to 3243949 tid 3 2024-09-20T19:34:27.426 INFO:tasks.rados.rados.0.smithi042.stdout:2599: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:27.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:27.451 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1894319 2024-09-20T19:34:27.464 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:27.524 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1033773 2024-09-20T19:34:27.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:27.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[22706]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[22706]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[22706]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[22706]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[32075]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[32075]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[32075]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:27 smithi042 bash[32075]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.559 INFO:tasks.rados.rados.0.smithi042.stdout:2601: expect (ObjNum 811 snap 268 seq_num 811) 2024-09-20T19:34:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:27.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:27.665 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 815238 2024-09-20T19:34:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:27 smithi162 bash[24557]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:27 smithi162 bash[24557]: cluster 2024-09-20T19:34:26.368473+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-09-20T19:34:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:27 smithi162 bash[24557]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:27 smithi162 bash[24557]: cluster 2024-09-20T19:34:26.369477+0000 mgr.y (mgr.24491) 619 : cluster [DBG] pgmap v1135: 178 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 173 active+clean; 174 MiB data, 734 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-20T19:34:27.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:27.711 INFO:tasks.rados.rados.0.smithi042.stdout:2604: finishing write tid 1 to smithi04268048-37 2024-09-20T19:34:27.711 INFO:tasks.rados.rados.0.smithi042.stdout:2604: finishing write tid 2 to smithi04268048-37 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2599: done (2 left) 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2601: done (1 left) 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2605: rollback oid 9 current snap is 299 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 296 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: finishing write tid 3 to smithi04268048-37 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: finishing write tid 4 to smithi04268048-37 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: finishing write tid 5 to smithi04268048-37 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: oid 37 updating version 0 to 997 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: oid 37 updating version 997 to 998 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: oid 37 version 998 is already newer than 996 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:2604: oid 37 updating version 998 to 999 2024-09-20T19:34:27.712 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 999 (ObjNum 907 snap 299 seq_num 907) dirty exists 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2604: left oid 37 (ObjNum 907 snap 299 seq_num 907) 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2604: done (1 left) 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2606: rollback oid 44 current snap is 299 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 292 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2607: rollback oid 16 current snap is 299 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 296 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2608: write (excl) oid 25 current snap is 299 2024-09-20T19:34:27.713 INFO:tasks.rados.rados.0.smithi042.stdout:2608: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-20T19:34:27.717 INFO:tasks.rados.rados.0.smithi042.stdout:2608: writing smithi04268048-25 from 460756 to 1032855 tid 1 2024-09-20T19:34:27.722 INFO:tasks.rados.rados.0.smithi042.stdout:2608: writing smithi04268048-25 from 1517297 to 1961609 tid 2 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2608: writing smithi04268048-25 from 2668548 to 3141418 tid 3 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2609: delete oid 7 current snap is 299 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2605: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2607: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2607: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:34:27.729 INFO:tasks.rados.rados.0.smithi042.stdout:2607: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 750 (ObjNum 814 snap 268 seq_num 814) dirty exists 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:2606: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:2606: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:2606: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1116 (ObjNum 868 snap 288 seq_num 868) dirty exists 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:2608: finishing write tid 1 to smithi04268048-25 2024-09-20T19:34:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:2608: finishing write tid 2 to smithi04268048-25 2024-09-20T19:34:27.731 INFO:tasks.rados.rados.0.smithi042.stdout:2605: done (4 left) 2024-09-20T19:34:27.731 INFO:tasks.rados.rados.0.smithi042.stdout:2606: done (3 left) 2024-09-20T19:34:27.731 INFO:tasks.rados.rados.0.smithi042.stdout:2607: done (2 left) 2024-09-20T19:34:27.732 INFO:tasks.rados.rados.0.smithi042.stdout:2609: done (1 left) 2024-09-20T19:34:27.732 INFO:tasks.rados.rados.0.smithi042.stdout:2610: snap_remove snap 292 2024-09-20T19:34:27.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:27.821 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:27.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:27.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:28.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:28.077 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:28.179 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:28.207 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:28.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:28.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2610: done (1 left) 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2611: read oid 16 snap -1 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2611: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2612: copy_from oid 2 from oid 45 current snap is 299 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2608: finishing write tid 3 to smithi04268048-25 2024-09-20T19:34:28.346 INFO:tasks.rados.rados.0.smithi042.stdout:2613: snap_remove snap 289 2024-09-20T19:34:28.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:28.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.580 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:28 smithi162 bash[24557]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:28.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[22706]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: cluster 2024-09-20T19:34:27.385181+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: audit 2024-09-20T19:34:28.149155+0000 mon.a (mon.0) 2264 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:28.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: audit 2024-09-20T19:34:28.150390+0000 mon.a (mon.0) 2265 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:28 smithi042 bash[32075]: cluster 2024-09-20T19:34:28.344111+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2024-09-20T19:34:28.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:28.717 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:28.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:28.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:28.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:28.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:29.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:29.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:29.238 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:29.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:34:29.358 INFO:tasks.rados.rados.0.smithi042.stdout:2613: done (3 left) 2024-09-20T19:34:29.358 INFO:tasks.rados.rados.0.smithi042.stdout:2614: read oid 13 snap -1 2024-09-20T19:34:29.358 INFO:tasks.rados.rados.0.smithi042.stdout:2614: expect deleted 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2615: delete oid 49 current snap is 299 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: finishing write tid 4 to smithi04268048-25 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: finishing write tid 5 to smithi04268048-25 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: oid 25 updating version 0 to 903 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: oid 25 version 903 is already newer than 901 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: oid 25 updating version 903 to 904 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:2608: oid 25 version 904 is already newer than 902 2024-09-20T19:34:29.359 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 904 (ObjNum 908 snap 299 seq_num 908) dirty exists 2024-09-20T19:34:29.360 INFO:tasks.rados.rados.0.smithi042.stdout:2608: left oid 25 (ObjNum 908 snap 299 seq_num 908) 2024-09-20T19:34:29.360 INFO:tasks.rados.rados.0.smithi042.stdout:2611: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:29.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2299304 2024-09-20T19:34:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:29.437 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2509821 2024-09-20T19:34:29.477 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2098993 2024-09-20T19:34:29.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[24557]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.501 INFO:tasks.rados.rados.0.smithi042.stdout:2612: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2612: finishing copy_from to smithi04268048-2 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1284 (ObjNum 903 snap 297 seq_num 903) dirty exists 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2608: done (4 left) 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2611: done (3 left) 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2612: done (2 left) 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2614: done (1 left) 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2615: done (0 left) 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2616: write oid 34 current snap is 299 2024-09-20T19:34:29.502 INFO:tasks.rados.rados.0.smithi042.stdout:2616: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-20T19:34:29.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:29.523 INFO:tasks.rados.rados.0.smithi042.stdout:2616: writing smithi04268048-34 from 511177 to 1309023 tid 1 2024-09-20T19:34:29.544 INFO:tasks.rados.rados.0.smithi042.stdout:2616: writing smithi04268048-34 from 1976390 to 2568029 tid 2 2024-09-20T19:34:29.554 INFO:tasks.rados.rados.0.smithi042.stdout:2616: writing smithi04268048-34 from 3034464 to 3311177 tid 3 2024-09-20T19:34:29.554 INFO:tasks.rados.rados.0.smithi042.stdout:2617: delete oid 45 current snap is 299 2024-09-20T19:34:29.554 INFO:tasks.rados.rados.0.smithi042.stdout:2616: finishing write tid 1 to smithi04268048-34 2024-09-20T19:34:29.554 INFO:tasks.rados.rados.0.smithi042.stdout:2616: finishing write tid 2 to smithi04268048-34 2024-09-20T19:34:29.557 INFO:tasks.rados.rados.0.smithi042.stdout:2617: done (1 left) 2024-09-20T19:34:29.557 INFO:tasks.rados.rados.0.smithi042.stdout:2618: write (excl) oid 48 current snap is 299 2024-09-20T19:34:29.557 INFO:tasks.rados.rados.0.smithi042.stdout:2618: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-20T19:34:29.567 INFO:tasks.rados.rados.0.smithi042.stdout:2618: writing smithi04268048-48 from 544791 to 1209488 tid 1 2024-09-20T19:34:29.577 INFO:tasks.rados.rados.0.smithi042.stdout:2618: writing smithi04268048-48 from 1693354 to 2288662 tid 2 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2618: writing smithi04268048-48 from 2689310 to 3134893 tid 3 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2619: snap_create 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: finishing write tid 3 to smithi04268048-34 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: finishing write tid 4 to smithi04268048-34 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: finishing write tid 5 to smithi04268048-34 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: oid 34 updating version 0 to 1068 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: oid 34 version 1068 is already newer than 1067 2024-09-20T19:34:29.585 INFO:tasks.rados.rados.0.smithi042.stdout:2616: oid 34 updating version 1068 to 1069 2024-09-20T19:34:29.586 INFO:tasks.rados.rados.0.smithi042.stdout:2616: oid 34 version 1069 is already newer than 1066 2024-09-20T19:34:29.586 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1069 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:34:29.586 INFO:tasks.rados.rados.0.smithi042.stdout:2616: left oid 34 (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:34:29.586 INFO:tasks.rados.rados.0.smithi042.stdout:2618: finishing write tid 1 to smithi04268048-48 2024-09-20T19:34:29.586 INFO:tasks.rados.rados.0.smithi042.stdout:2618: finishing write tid 2 to smithi04268048-48 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: finishing write tid 3 to smithi04268048-48 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: finishing write tid 4 to smithi04268048-48 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: finishing write tid 5 to smithi04268048-48 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: oid 48 updating version 0 to 910 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: oid 48 version 910 is already newer than 909 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: oid 48 updating version 910 to 911 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: oid 48 version 911 is already newer than 908 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 911 (ObjNum 913 snap 299 seq_num 913) dirty exists 2024-09-20T19:34:29.588 INFO:tasks.rados.rados.0.smithi042.stdout:2618: left oid 48 (ObjNum 913 snap 299 seq_num 913) 2024-09-20T19:34:29.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:29.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[22706]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:28.370151+0000 mgr.y (mgr.24491) 620 : cluster [DBG] pgmap v1138: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 749 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: audit 2024-09-20T19:34:28.839608+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:29.339236+0000 mon.a (mon.0) 2268 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: audit 2024-09-20T19:34:29.341507+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.42:0/2669979309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:29 smithi042 bash[32075]: cluster 2024-09-20T19:34:29.347668+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2024-09-20T19:34:29.640 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:29.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:29.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:29.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:29.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:29.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:30.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:30.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:30.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:30.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:30.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:30.296 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:30.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:30.418 INFO:tasks.rados.rados.0.smithi042.stdout:2616: done (2 left) 2024-09-20T19:34:30.418 INFO:tasks.rados.rados.0.smithi042.stdout:2618: done (1 left) 2024-09-20T19:34:30.418 INFO:tasks.rados.rados.0.smithi042.stdout:2619: done (0 left) 2024-09-20T19:34:30.419 INFO:tasks.rados.rados.0.smithi042.stdout:2620: delete oid 3 current snap is 300 2024-09-20T19:34:30.424 INFO:tasks.rados.rados.0.smithi042.stdout:2620: done (0 left) 2024-09-20T19:34:30.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: write (excl) oid 23 current snap is 300 2024-09-20T19:34:30.425 INFO:tasks.rados.rados.0.smithi042.stdout:2621: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-20T19:34:30.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:30.437 INFO:tasks.rados.rados.0.smithi042.stdout:2621: writing smithi04268048-23 from 578405 to 993600 tid 1 2024-09-20T19:34:30.449 INFO:tasks.rados.rados.0.smithi042.stdout:2621: writing smithi04268048-23 from 1410318 to 2009295 tid 2 2024-09-20T19:34:30.463 INFO:tasks.rados.rados.0.smithi042.stdout:2621: writing smithi04268048-23 from 2744156 to 3328636 tid 3 2024-09-20T19:34:30.463 INFO:tasks.rados.rados.0.smithi042.stdout:2622: snap_remove snap 298 2024-09-20T19:34:30.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:30.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:30.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:30.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:30 smithi042 bash[32075]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:30 smithi042 bash[32075]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:30 smithi042 bash[22706]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:30 smithi042 bash[22706]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:30.792 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:30.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:30 smithi162 bash[24557]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:30 smithi162 bash[24557]: audit 2024-09-20T19:34:29.709209+0000 mgr.y (mgr.24491) 621 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:30.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:30.952 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:30.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:31.054 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.083 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:31.185 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:31.211 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:31.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:31.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:31.423 INFO:tasks.rados.rados.0.smithi042.stdout:2622: done (1 left) 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2623: read oid 18 snap -1 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2623: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2624: snap_create 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: finishing write tid 1 to smithi04268048-23 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: finishing write tid 2 to smithi04268048-23 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: finishing write tid 3 to smithi04268048-23 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: finishing write tid 4 to smithi04268048-23 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: finishing write tid 5 to smithi04268048-23 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: oid 23 updating version 0 to 958 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: oid 23 version 958 is already newer than 955 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: oid 23 version 958 is already newer than 957 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:2621: oid 23 version 958 is already newer than 956 2024-09-20T19:34:31.424 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 958 (ObjNum 915 snap 300 seq_num 915) dirty exists 2024-09-20T19:34:31.425 INFO:tasks.rados.rados.0.smithi042.stdout:2621: left oid 23 (ObjNum 915 snap 300 seq_num 915) 2024-09-20T19:34:31.442 INFO:tasks.rados.rados.0.smithi042.stdout:2623: expect (ObjNum 846 snap 277 seq_num 846) 2024-09-20T19:34:31.442 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:31.445 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:31.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.578 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1733409 2024-09-20T19:34:31.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[22706]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[22706]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[22706]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[22706]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[32075]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[32075]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[32075]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:31 smithi042 bash[32075]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:31.729 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:31 smithi162 bash[24557]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:31 smithi162 bash[24557]: cluster 2024-09-20T19:34:30.371271+0000 mgr.y (mgr.24491) 622 : cluster [DBG] pgmap v1140: 178 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 active+clean+snaptrim, 172 active+clean; 178 MiB data, 750 MiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-20T19:34:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:31 smithi162 bash[24557]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:31 smithi162 bash[24557]: cluster 2024-09-20T19:34:30.414852+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2024-09-20T19:34:31.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:31.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:31.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:31.991 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:31.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:32.094 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:32.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:32.254 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:32.357 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:32.432 INFO:tasks.rados.rados.0.smithi042.stdout:2621: done (2 left) 2024-09-20T19:34:32.432 INFO:tasks.rados.rados.0.smithi042.stdout:2623: done (1 left) 2024-09-20T19:34:32.432 INFO:tasks.rados.rados.0.smithi042.stdout:2624: done (0 left) 2024-09-20T19:34:32.433 INFO:tasks.rados.rados.0.smithi042.stdout:2625: snap_create 2024-09-20T19:34:32.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:34:32.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:32.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:32 smithi162 bash[24557]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:32 smithi162 bash[24557]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:32.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:32.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:32 smithi042 bash[32075]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:32 smithi042 bash[32075]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:32 smithi042 bash[22706]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:32 smithi042 bash[22706]: cluster 2024-09-20T19:34:31.419953+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2024-09-20T19:34:32.642 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:32.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:32.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:32.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:32.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:32.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:33.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:33.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:33.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:33.266 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:33.350 INFO:tasks.rados.rados.0.smithi042.stdout:2625: done (0 left) 2024-09-20T19:34:33.350 INFO:tasks.rados.rados.0.smithi042.stdout:2626: copy_from oid 46 from oid 34 current snap is 302 2024-09-20T19:34:33.350 INFO:tasks.rados.rados.0.smithi042.stdout:2627: write (excl) oid 25 current snap is 302 2024-09-20T19:34:33.351 INFO:tasks.rados.rados.0.smithi042.stdout:2627: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-09-20T19:34:33.352 INFO:tasks.rados.rados.0.smithi042.stdout:2627: writing smithi04268048-25 from 595212 to 1085656 tid 1 2024-09-20T19:34:33.360 INFO:tasks.rados.rados.0.smithi042.stdout:2627: writing smithi04268048-25 from 1868800 to 2327788 tid 2 2024-09-20T19:34:33.366 INFO:tasks.rados.rados.0.smithi042.stdout:2627: writing smithi04268048-25 from 2887932 to 3395212 tid 3 2024-09-20T19:34:33.366 INFO:tasks.rados.rados.0.smithi042.stdout:2628: snap_create 2024-09-20T19:34:33.366 INFO:tasks.rados.rados.0.smithi042.stdout:2627: finishing write tid 1 to smithi04268048-25 2024-09-20T19:34:33.367 INFO:tasks.rados.rados.0.smithi042.stdout:2627: finishing write tid 2 to smithi04268048-25 2024-09-20T19:34:33.378 INFO:tasks.rados.rados.0.smithi042.stdout:2626: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: finishing write tid 3 to smithi04268048-25 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: finishing write tid 4 to smithi04268048-25 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: finishing write tid 5 to smithi04268048-25 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: oid 25 updating version 0 to 915 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: oid 25 updating version 915 to 917 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: oid 25 updating version 917 to 918 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: oid 25 version 918 is already newer than 916 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 918 (ObjNum 916 snap 302 seq_num 916) dirty exists 2024-09-20T19:34:33.380 INFO:tasks.rados.rados.0.smithi042.stdout:2627: left oid 25 (ObjNum 916 snap 302 seq_num 916) 2024-09-20T19:34:33.389 INFO:tasks.rados.rados.0.smithi042.stdout:2626: finishing copy_from to smithi04268048-46 2024-09-20T19:34:33.389 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1069 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:34:33.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.421 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:33.524 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:33.535 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[32075]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:33 smithi042 bash[22706]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.551 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:33.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:32.373098+0000 mgr.y (mgr.24491) 623 : cluster [DBG] pgmap v1143: 178 pgs: 2 active+clean+snaptrim_wait, 1 active+undersized+remapped, 6 active+clean+snaptrim, 169 active+clean; 174 MiB data, 764 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 6.7 MiB/s, 28 objects/s recovering 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:32.431221+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:33 smithi162 bash[24557]: cluster 2024-09-20T19:34:33.345248+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2024-09-20T19:34:33.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:33.785 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:33.834 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:34:33.834 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:34:33.910 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:33.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:34.025 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:34:34.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:34.046 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:34:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:34:34.055 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:34.080 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:34:34.081 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:34:34.111 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:34:34.111 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:34:34.133 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:34:34.133 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:34:34.154 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:34:34.154 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:34:34.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:34.176 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:34:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:34:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:34.197 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:34:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:34:34.279 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:34.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:34.343 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T19:34:34.349 INFO:tasks.rados.rados.0.smithi042.stdout:2626: done (2 left) 2024-09-20T19:34:34.349 INFO:tasks.rados.rados.0.smithi042.stdout:2627: done (1 left) 2024-09-20T19:34:34.349 INFO:tasks.rados.rados.0.smithi042.stdout:2628: done (0 left) 2024-09-20T19:34:34.349 INFO:tasks.rados.rados.0.smithi042.stdout:2629: copy_from oid 44 from oid 25 current snap is 303 2024-09-20T19:34:34.349 INFO:tasks.rados.rados.0.smithi042.stdout:2630: delete oid 33 current snap is 303 2024-09-20T19:34:34.364 INFO:tasks.rados.rados.0.smithi042.stdout:2629: finishing copy_from racing read to smithi04268048-44 2024-09-20T19:34:34.370 INFO:tasks.rados.rados.0.smithi042.stdout:2630: done (1 left) 2024-09-20T19:34:34.370 INFO:tasks.rados.rados.0.smithi042.stdout:2631: read oid 43 snap 300 2024-09-20T19:34:34.370 INFO:tasks.rados.rados.0.smithi042.stdout:2631: expect deleted 2024-09-20T19:34:34.370 INFO:tasks.rados.rados.0.smithi042.stdout:2632: delete oid 35 current snap is 303 2024-09-20T19:34:34.378 INFO:tasks.rados.rados.0.smithi042.stdout:2631: done (2 left) 2024-09-20T19:34:34.378 INFO:tasks.rados.rados.0.smithi042.stdout:2632: done (1 left) 2024-09-20T19:34:34.378 INFO:tasks.rados.rados.0.smithi042.stdout:2633: snap_create 2024-09-20T19:34:34.379 INFO:tasks.rados.rados.0.smithi042.stdout:2629: finishing copy_from to smithi04268048-44 2024-09-20T19:34:34.379 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 918 (ObjNum 916 snap 302 seq_num 916) dirty exists 2024-09-20T19:34:34.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:34.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:34.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:34.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:34.662 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:34.682 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:34.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:34 smithi162 bash[24557]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:34.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:34.926 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.926 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.926 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.926 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.926 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[22706]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:33.995141+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.42:0/1315599795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:34.284995+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.927 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: audit 2024-09-20T19:34:34.342871+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? v1:172.21.15.42:0/1493140757' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:34:34.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.928 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:34 smithi042 bash[32075]: cluster 2024-09-20T19:34:34.346850+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2024-09-20T19:34:34.940 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:34.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:35.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:35.072 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:35.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T19:34:35.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:35.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:35.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:35.369 INFO:tasks.rados.rados.0.smithi042.stdout:2629: done (1 left) 2024-09-20T19:34:35.369 INFO:tasks.rados.rados.0.smithi042.stdout:2633: done (0 left) 2024-09-20T19:34:35.369 INFO:tasks.rados.rados.0.smithi042.stdout:2634: snap_create 2024-09-20T19:34:35.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:35.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:35.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:35.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:35 smithi162 bash[24557]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:35 smithi162 bash[24557]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:35 smithi162 bash[24557]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:35 smithi162 bash[24557]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:35.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[22706]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[22706]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[22706]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[22706]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[32075]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[32075]: cluster 2024-09-20T19:34:34.373535+0000 mgr.y (mgr.24491) 624 : cluster [DBG] pgmap v1147: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 176 MiB data, 766 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s; 6.8 MiB/s, 27 objects/s recovering 2024-09-20T19:34:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[32075]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:35 smithi042 bash[32075]: cluster 2024-09-20T19:34:35.360107+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2024-09-20T19:34:35.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:35.973 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:35.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:36.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:36.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:36.161 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T19:34:36.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:36.274 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:36.450 INFO:tasks.rados.rados.0.smithi042.stdout:2634: done (0 left) 2024-09-20T19:34:36.450 INFO:tasks.rados.rados.0.smithi042.stdout:2635: write oid 42 current snap is 305 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2635: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2635: writing smithi04268048-42 from 645633 to 1361824 tid 1 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2635: writing smithi04268048-42 from 1927893 to 2534208 tid 2 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2635: writing smithi04268048-42 from 3137495 to 3445633 tid 3 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2636: write (excl) oid 18 current snap is 305 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2636: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-20T19:34:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2636: writing smithi04268048-18 from 662440 to 1170233 tid 1 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2636: writing smithi04268048-18 from 1702728 to 2452701 tid 2 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2636: writing smithi04268048-18 from 2881271 to 3438464 tid 3 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2637: write (excl) oid 48 current snap is 305 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2635: finishing write tid 1 to smithi04268048-42 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2637: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-20T19:34:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2637: writing smithi04268048-48 from 679247 to 1262289 tid 1 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2637: writing smithi04268048-48 from 2044857 to 2654841 tid 2 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2637: writing smithi04268048-48 from 3192341 to 3479247 tid 3 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2638: write (excl) oid 14 current snap is 305 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2635: finishing write tid 2 to smithi04268048-42 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2635: finishing write tid 3 to smithi04268048-42 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2638: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-20T19:34:36.453 INFO:tasks.rados.rados.0.smithi042.stdout:2638: writing smithi04268048-14 from 696054 to 1354345 tid 1 2024-09-20T19:34:36.455 INFO:tasks.rados.rados.0.smithi042.stdout:2638: writing smithi04268048-14 from 2103339 to 2856981 tid 2 2024-09-20T19:34:36.455 INFO:tasks.rados.rados.0.smithi042.stdout:2639: read oid 30 snap -1 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2639: expect (ObjNum 876 snap 291 seq_num 876) 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2640: copy_from oid 2 from oid 8 current snap is 305 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2635: finishing write tid 4 to smithi04268048-42 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2635: finishing write tid 5 to smithi04268048-42 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2635: oid 42 updating version 0 to 1114 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2635: oid 42 version 1114 is already newer than 1112 2024-09-20T19:34:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2635: oid 42 version 1114 is already newer than 1111 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2635: oid 42 version 1114 is already newer than 1113 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1114 (ObjNum 919 snap 305 seq_num 919) dirty exists 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2635: left oid 42 (ObjNum 919 snap 305 seq_num 919) 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2635: done (5 left) 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2641: snap_create 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2636: finishing write tid 1 to smithi04268048-18 2024-09-20T19:34:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:2636: finishing write tid 2 to smithi04268048-18 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: finishing write tid 3 to smithi04268048-18 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: finishing write tid 4 to smithi04268048-18 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: finishing write tid 5 to smithi04268048-18 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: oid 18 updating version 0 to 1096 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: oid 18 updating version 1096 to 1098 2024-09-20T19:34:36.458 INFO:tasks.rados.rados.0.smithi042.stdout:2636: oid 18 version 1098 is already newer than 1095 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2636: oid 18 version 1098 is already newer than 1097 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1098 (ObjNum 920 snap 305 seq_num 920) dirty exists 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2636: left oid 18 (ObjNum 920 snap 305 seq_num 920) 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2637: finishing write tid 1 to smithi04268048-48 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2637: finishing write tid 2 to smithi04268048-48 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2637: finishing write tid 3 to smithi04268048-48 2024-09-20T19:34:36.459 INFO:tasks.rados.rados.0.smithi042.stdout:2637: finishing write tid 4 to smithi04268048-48 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: finishing write tid 5 to smithi04268048-48 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: oid 48 updating version 0 to 925 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: oid 48 version 925 is already newer than 924 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: oid 48 updating version 925 to 927 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: oid 48 version 927 is already newer than 926 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 927 (ObjNum 921 snap 305 seq_num 921) dirty exists 2024-09-20T19:34:36.460 INFO:tasks.rados.rados.0.smithi042.stdout:2637: left oid 48 (ObjNum 921 snap 305 seq_num 921) 2024-09-20T19:34:36.461 INFO:tasks.rados.rados.0.smithi042.stdout:2639: expect (ObjNum 876 snap 291 seq_num 876) 2024-09-20T19:34:36.494 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:36.549 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:36.559 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:36.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: finishing write tid 1 to smithi04268048-14 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: finishing write tid 2 to smithi04268048-14 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: finishing write tid 3 to smithi04268048-14 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: finishing write tid 4 to smithi04268048-14 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: oid 14 updating version 0 to 817 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: oid 14 updating version 817 to 819 2024-09-20T19:34:36.577 INFO:tasks.rados.rados.0.smithi042.stdout:2638: oid 14 version 819 is already newer than 818 2024-09-20T19:34:36.578 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 819 (ObjNum 922 snap 305 seq_num 922) dirty exists 2024-09-20T19:34:36.578 INFO:tasks.rados.rados.0.smithi042.stdout:2638: left oid 14 (ObjNum 922 snap 305 seq_num 922) 2024-09-20T19:34:36.578 INFO:tasks.rados.rados.0.smithi042.stdout:2640: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:34:36.578 INFO:tasks.rados.rados.0.smithi042.stdout:2640: finishing copy_from to smithi04268048-2 2024-09-20T19:34:36.578 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:34:36.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:36.706 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:36.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:36.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:36.966 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:36.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:37.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:37.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:37.227 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:37.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:37.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2636: done (5 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2637: done (4 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2638: done (3 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2639: done (2 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2640: done (1 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2641: done (0 left) 2024-09-20T19:34:37.378 INFO:tasks.rados.rados.0.smithi042.stdout:2642: snap_create 2024-09-20T19:34:37.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:37.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:37.591 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:37.617 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[22706]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[22706]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[22706]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[22706]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[32075]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[32075]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[32075]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:37 smithi042 bash[32075]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:37 smithi162 bash[24557]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:37 smithi162 bash[24557]: cluster 2024-09-20T19:34:36.362765+0000 mon.a (mon.0) 2279 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2024-09-20T19:34:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:37 smithi162 bash[24557]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:37 smithi162 bash[24557]: cluster 2024-09-20T19:34:36.374256+0000 mgr.y (mgr.24491) 625 : cluster [DBG] pgmap v1150: 178 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 167 active+clean; 175 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 259 B/s rd, 423 KiB/s wr, 1 op/s 2024-09-20T19:34:37.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:37.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:37.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:38.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:38.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:38.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:34:38.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:38.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2642: done (0 left) 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2643: read oid 21 snap 305 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2643: expect (ObjNum 881 snap 291 seq_num 881) 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2644: read oid 29 snap -1 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2644: expect (ObjNum 877 snap 291 seq_num 877) 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2645: read oid 37 snap -1 2024-09-20T19:34:38.382 INFO:tasks.rados.rados.0.smithi042.stdout:2645: expect (ObjNum 907 snap 299 seq_num 907) 2024-09-20T19:34:38.383 INFO:tasks.rados.rados.0.smithi042.stdout:2646: write (excl) oid 26 current snap is 307 2024-09-20T19:34:38.383 INFO:tasks.rados.rados.0.smithi042.stdout:2646: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-20T19:34:38.395 INFO:tasks.rados.rados.0.smithi042.stdout:2646: writing smithi04268048-26 from 712861 to 1446401 tid 1 2024-09-20T19:34:38.407 INFO:tasks.rados.rados.0.smithi042.stdout:2646: writing smithi04268048-26 from 2161821 to 2775474 tid 2 2024-09-20T19:34:38.407 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:38.408 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:38 smithi042 bash[32075]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.409 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:38 smithi042 bash[32075]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:38 smithi042 bash[22706]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:38 smithi042 bash[22706]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:38.411 INFO:tasks.rados.rados.0.smithi042.stdout:2646: writing smithi04268048-26 from 3247187 to 3512861 tid 3 2024-09-20T19:34:38.411 INFO:tasks.rados.rados.0.smithi042.stdout:2647: write (excl) oid 15 current snap is 307 2024-09-20T19:34:38.412 INFO:tasks.rados.rados.0.smithi042.stdout:2647: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-20T19:34:38.417 INFO:tasks.rados.rados.0.smithi042.stdout:2647: writing smithi04268048-15 from 729668 to 1138457 tid 1 2024-09-20T19:34:38.426 INFO:tasks.rados.rados.0.smithi042.stdout:2647: writing smithi04268048-15 from 1820303 to 2577614 tid 2 2024-09-20T19:34:38.431 INFO:tasks.rados.rados.0.smithi042.stdout:2647: writing smithi04268048-15 from 3274610 to 3529668 tid 3 2024-09-20T19:34:38.431 INFO:tasks.rados.rados.0.smithi042.stdout:2648: rollback oid 10 current snap is 307 2024-09-20T19:34:38.431 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 306 2024-09-20T19:34:38.431 INFO:tasks.rados.rados.0.smithi042.stdout:2644: expect (ObjNum 877 snap 291 seq_num 877) 2024-09-20T19:34:38.461 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:38.490 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:38.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:38.519 INFO:tasks.rados.rados.0.smithi042.stdout:2645: expect (ObjNum 907 snap 299 seq_num 907) 2024-09-20T19:34:38.528 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1084525 2024-09-20T19:34:38.534 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:38.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:38 smithi162 bash[24557]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:38 smithi162 bash[24557]: cluster 2024-09-20T19:34:37.376371+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2024-09-20T19:34:38.560 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:38.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:38.638 INFO:tasks.rados.rados.0.smithi042.stdout:2643: expect (ObjNum 881 snap 291 seq_num 881) 2024-09-20T19:34:38.638 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:38.728 INFO:tasks.rados.rados.0.smithi042.stdout:2646: finishing write tid 1 to smithi04268048-26 2024-09-20T19:34:38.728 INFO:tasks.rados.rados.0.smithi042.stdout:2646: finishing write tid 2 to smithi04268048-26 2024-09-20T19:34:38.728 INFO:tasks.rados.rados.0.smithi042.stdout:2646: finishing write tid 3 to smithi04268048-26 2024-09-20T19:34:38.728 INFO:tasks.rados.rados.0.smithi042.stdout:2646: finishing write tid 4 to smithi04268048-26 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: finishing write tid 5 to smithi04268048-26 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: oid 26 updating version 0 to 1212 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: oid 26 version 1212 is already newer than 1211 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: oid 26 updating version 1212 to 1213 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: oid 26 updating version 1213 to 1214 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1214 (ObjNum 923 snap 307 seq_num 923) dirty exists 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2646: left oid 26 (ObjNum 923 snap 307 seq_num 923) 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2647: finishing write tid 1 to smithi04268048-15 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2647: finishing write tid 2 to smithi04268048-15 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2648: finishing rollback tid 0 to smithi04268048-10 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2648: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2648: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 844 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2647: finishing write tid 3 to smithi04268048-15 2024-09-20T19:34:38.729 INFO:tasks.rados.rados.0.smithi042.stdout:2643: done (5 left) 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2644: done (4 left) 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2645: done (3 left) 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2646: done (2 left) 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2648: done (1 left) 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2649: write (excl) oid 18 current snap is 307 2024-09-20T19:34:38.730 INFO:tasks.rados.rados.0.smithi042.stdout:2649: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-20T19:34:38.734 INFO:tasks.rados.rados.0.smithi042.stdout:2649: writing smithi04268048-18 from 746475 to 1230513 tid 1 2024-09-20T19:34:38.741 INFO:tasks.rados.rados.0.smithi042.stdout:2649: writing smithi04268048-18 from 1762432 to 2379754 tid 2 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2649: writing smithi04268048-18 from 2785680 to 3315586 tid 3 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2650: write oid 42 current snap is 307 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: finishing write tid 4 to smithi04268048-15 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: finishing write tid 5 to smithi04268048-15 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: oid 15 updating version 0 to 724 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: oid 15 updating version 724 to 726 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: oid 15 version 726 is already newer than 725 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: oid 15 updating version 726 to 727 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 727 (ObjNum 924 snap 307 seq_num 924) dirty exists 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2647: left oid 15 (ObjNum 924 snap 307 seq_num 924) 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2649: finishing write tid 1 to smithi04268048-18 2024-09-20T19:34:38.748 INFO:tasks.rados.rados.0.smithi042.stdout:2649: finishing write tid 2 to smithi04268048-18 2024-09-20T19:34:38.749 INFO:tasks.rados.rados.0.smithi042.stdout:2650: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-20T19:34:38.757 INFO:tasks.rados.rados.0.smithi042.stdout:2650: writing smithi04268048-42 from 763282 to 1322569 tid 1 2024-09-20T19:34:38.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.767 INFO:tasks.rados.rados.0.smithi042.stdout:2650: writing smithi04268048-42 from 1820914 to 2581894 tid 2 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2650: writing smithi04268048-42 from 3213103 to 3563282 tid 3 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2647: done (2 left) 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2651: read oid 14 snap -1 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2651: expect (ObjNum 922 snap 305 seq_num 922) 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2652: copy_from oid 30 from oid 23 current snap is 307 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2649: finishing write tid 3 to smithi04268048-18 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2649: finishing write tid 4 to smithi04268048-18 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2649: finishing write tid 5 to smithi04268048-18 2024-09-20T19:34:38.774 INFO:tasks.rados.rados.0.smithi042.stdout:2649: oid 18 updating version 0 to 1103 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2649: oid 18 version 1103 is already newer than 1101 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2649: oid 18 version 1103 is already newer than 1102 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2649: oid 18 version 1103 is already newer than 1100 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1103 (ObjNum 925 snap 307 seq_num 925) dirty exists 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2649: left oid 18 (ObjNum 925 snap 307 seq_num 925) 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2650: finishing write tid 1 to smithi04268048-42 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2650: finishing write tid 2 to smithi04268048-42 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2649: done (3 left) 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2653: copy_from oid 18 from oid 2 current snap is 307 2024-09-20T19:34:38.775 INFO:tasks.rados.rados.0.smithi042.stdout:2654: snap_create 2024-09-20T19:34:38.779 INFO:tasks.rados.rados.0.smithi042.stdout:2650: finishing write tid 3 to smithi04268048-42 2024-09-20T19:34:38.780 INFO:tasks.rados.rados.0.smithi042.stdout:2650: finishing write tid 4 to smithi04268048-42 2024-09-20T19:34:38.780 INFO:tasks.rados.rados.0.smithi042.stdout:2650: finishing write tid 5 to smithi04268048-42 2024-09-20T19:34:38.780 INFO:tasks.rados.rados.0.smithi042.stdout:2650: oid 42 updating version 0 to 1118 2024-09-20T19:34:38.781 INFO:tasks.rados.rados.0.smithi042.stdout:2650: oid 42 version 1118 is already newer than 1117 2024-09-20T19:34:38.781 INFO:tasks.rados.rados.0.smithi042.stdout:2650: oid 42 updating version 1118 to 1119 2024-09-20T19:34:38.781 INFO:tasks.rados.rados.0.smithi042.stdout:2650: oid 42 version 1119 is already newer than 1116 2024-09-20T19:34:38.781 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1119 (ObjNum 926 snap 307 seq_num 926) dirty exists 2024-09-20T19:34:38.781 INFO:tasks.rados.rados.0.smithi042.stdout:2650: left oid 42 (ObjNum 926 snap 307 seq_num 926) 2024-09-20T19:34:38.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:38.793 INFO:tasks.rados.rados.0.smithi042.stdout:2651: expect (ObjNum 922 snap 305 seq_num 922) 2024-09-20T19:34:38.793 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:38.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:38.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:38.955 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:39.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:39.036 INFO:tasks.rados.rados.0.smithi042.stdout:2652: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:34:39.036 INFO:tasks.rados.rados.0.smithi042.stdout:2653: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:34:39.036 INFO:tasks.rados.rados.0.smithi042.stdout:2653: finishing copy_from to smithi04268048-18 2024-09-20T19:34:39.036 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:34:39.036 INFO:tasks.rados.rados.0.smithi042.stdout:2652: finishing copy_from to smithi04268048-30 2024-09-20T19:34:39.037 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 958 (ObjNum 915 snap 300 seq_num 915) dirty exists 2024-09-20T19:34:39.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:39.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:39.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:39.405 INFO:tasks.rados.rados.0.smithi042.stdout:2650: done (4 left) 2024-09-20T19:34:39.405 INFO:tasks.rados.rados.0.smithi042.stdout:2651: done (3 left) 2024-09-20T19:34:39.405 INFO:tasks.rados.rados.0.smithi042.stdout:2652: done (2 left) 2024-09-20T19:34:39.405 INFO:tasks.rados.rados.0.smithi042.stdout:2653: done (1 left) 2024-09-20T19:34:39.405 INFO:tasks.rados.rados.0.smithi042.stdout:2654: done (0 left) 2024-09-20T19:34:39.406 INFO:tasks.rados.rados.0.smithi042.stdout:2655: snap_create 2024-09-20T19:34:39.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:39.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:39.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:39.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[32075]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[32075]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[32075]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.551 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[32075]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.551 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[22706]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.551 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[22706]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.551 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[22706]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.551 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:39 smithi042 bash[22706]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:39 smithi162 bash[24557]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:39 smithi162 bash[24557]: cluster 2024-09-20T19:34:38.374798+0000 mgr.y (mgr.24491) 626 : cluster [DBG] pgmap v1153: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 189 MiB data, 793 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 12 op/s 2024-09-20T19:34:39.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:39 smithi162 bash[24557]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:39 smithi162 bash[24557]: cluster 2024-09-20T19:34:38.376228+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2024-09-20T19:34:39.567 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:39.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:39.698 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:39.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:39.896 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:34:39.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:39.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:39.993 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:40.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:40.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:40.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:40.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:40.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:40.408 INFO:tasks.rados.rados.0.smithi042.stdout:2655: done (0 left) 2024-09-20T19:34:40.408 INFO:tasks.rados.rados.0.smithi042.stdout:2656: write (excl) oid 47 current snap is 309 2024-09-20T19:34:40.409 INFO:tasks.rados.rados.0.smithi042.stdout:2656: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-09-20T19:34:40.416 INFO:tasks.rados.rados.0.smithi042.stdout:2656: writing smithi04268048-47 from 780089 to 1414625 tid 1 2024-09-20T19:34:40.422 INFO:tasks.rados.rados.0.smithi042.stdout:2656: writing smithi04268048-47 from 1879396 to 2500387 tid 2 2024-09-20T19:34:40.429 INFO:tasks.rados.rados.0.smithi042.stdout:2656: writing smithi04268048-47 from 3240526 to 3580089 tid 3 2024-09-20T19:34:40.429 INFO:tasks.rados.rados.0.smithi042.stdout:2657: copy_from oid 1 from oid 42 current snap is 309 2024-09-20T19:34:40.429 INFO:tasks.rados.rados.0.smithi042.stdout:2658: write (excl) oid 17 current snap is 309 2024-09-20T19:34:40.430 INFO:tasks.rados.rados.0.smithi042.stdout:2658: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2024-09-20T19:34:40.435 INFO:tasks.rados.rados.0.smithi042.stdout:2658: writing smithi04268048-17 from 796896 to 1223034 tid 1 2024-09-20T19:34:40.444 INFO:tasks.rados.rados.0.smithi042.stdout:2658: writing smithi04268048-17 from 1654231 to 2418880 tid 2 2024-09-20T19:34:40.451 INFO:tasks.rados.rados.0.smithi042.stdout:2658: writing smithi04268048-17 from 2984302 to 3464377 tid 3 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2659: snap_create 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: finishing write tid 1 to smithi04268048-47 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: finishing write tid 2 to smithi04268048-47 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: finishing write tid 3 to smithi04268048-47 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2658: finishing write tid 1 to smithi04268048-17 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: finishing write tid 4 to smithi04268048-47 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: finishing write tid 5 to smithi04268048-47 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: oid 47 updating version 0 to 1122 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: oid 47 updating version 1122 to 1124 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: oid 47 version 1124 is already newer than 1123 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: oid 47 version 1124 is already newer than 1121 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1124 (ObjNum 927 snap 309 seq_num 927) dirty exists 2024-09-20T19:34:40.452 INFO:tasks.rados.rados.0.smithi042.stdout:2656: left oid 47 (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:34:40.453 INFO:tasks.rados.rados.0.smithi042.stdout:2658: finishing write tid 2 to smithi04268048-17 2024-09-20T19:34:40.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.455 INFO:tasks.rados.rados.0.smithi042.stdout:2658: finishing write tid 3 to smithi04268048-17 2024-09-20T19:34:40.457 INFO:tasks.rados.rados.0.smithi042.stdout:2658: finishing write tid 4 to smithi04268048-17 2024-09-20T19:34:40.457 INFO:tasks.rados.rados.0.smithi042.stdout:2658: finishing write tid 5 to smithi04268048-17 2024-09-20T19:34:40.457 INFO:tasks.rados.rados.0.smithi042.stdout:2658: oid 17 updating version 0 to 961 2024-09-20T19:34:40.457 INFO:tasks.rados.rados.0.smithi042.stdout:2658: oid 17 version 961 is already newer than 960 2024-09-20T19:34:40.457 INFO:tasks.rados.rados.0.smithi042.stdout:2658: oid 17 version 961 is already newer than 959 2024-09-20T19:34:40.458 INFO:tasks.rados.rados.0.smithi042.stdout:2658: oid 17 updating version 961 to 962 2024-09-20T19:34:40.458 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 962 (ObjNum 928 snap 309 seq_num 928) dirty exists 2024-09-20T19:34:40.458 INFO:tasks.rados.rados.0.smithi042.stdout:2658: left oid 17 (ObjNum 928 snap 309 seq_num 928) 2024-09-20T19:34:40.458 INFO:tasks.rados.rados.0.smithi042.stdout:2657: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:34:40.462 INFO:tasks.rados.rados.0.smithi042.stdout:2657: finishing copy_from to smithi04268048-1 2024-09-20T19:34:40.462 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1119 (ObjNum 926 snap 307 seq_num 926) dirty exists 2024-09-20T19:34:40.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:40.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:40.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:40 smithi042 bash[32075]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:40 smithi042 bash[32075]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:40 smithi042 bash[22706]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:40 smithi042 bash[22706]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:40.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:40.719 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:40 smithi162 bash[24557]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.719 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:40 smithi162 bash[24557]: cluster 2024-09-20T19:34:39.397745+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2024-09-20T19:34:40.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:40.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:40.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:40.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:41.096 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:41.124 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:41.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:41.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:41.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:41.384 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:41.412 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:34:41.417 INFO:tasks.rados.rados.0.smithi042.stdout:2656: done (3 left) 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2657: done (2 left) 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2658: done (1 left) 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2659: done (0 left) 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2660: rollback oid 33 current snap is 310 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 307 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2661: copy_from oid 39 from oid 45 current snap is 310 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2662: read oid 36 snap 306 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2662: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2663: copy_from oid 49 from oid 47 current snap is 310 2024-09-20T19:34:41.418 INFO:tasks.rados.rados.0.smithi042.stdout:2664: snap_create 2024-09-20T19:34:41.427 INFO:tasks.rados.rados.0.smithi042.stdout:2660: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:34:41.430 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 734 (ObjNum 225 snap 0 seq_num 1414780128) dirty dne 2024-09-20T19:34:41.432 INFO:tasks.rados.rados.0.smithi042.stdout:2662: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:34:41.477 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 108962 2024-09-20T19:34:41.487 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:41 smithi162 bash[24557]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.558 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2042463 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:2663: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:2661: finishing copy_from to smithi04268048-39 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:2661: got expected ENOENT (src dne) 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:2661: finishing copy_from racing read to smithi04268048-39 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:2663: finishing copy_from to smithi04268048-49 2024-09-20T19:34:41.559 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1124 (ObjNum 927 snap 309 seq_num 927) dirty exists 2024-09-20T19:34:41.585 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:34:41.586 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:34:41.586 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:34:41.586 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:34:41.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[22706]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: audit 2024-09-20T19:34:39.717919+0000 mgr.y (mgr.24491) 627 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: cluster 2024-09-20T19:34:40.375396+0000 mgr.y (mgr.24491) 628 : cluster [DBG] pgmap v1155: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 191 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 14 op/s 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: cluster 2024-09-20T19:34:40.405023+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:41 smithi042 bash[32075]: audit 2024-09-20T19:34:40.778296+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:34:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:41.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:41.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:41.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:41.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:41.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:42.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:42.039 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:42.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:42.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:42.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:42.424 INFO:tasks.rados.rados.0.smithi042.stdout:2660: done (4 left) 2024-09-20T19:34:42.424 INFO:tasks.rados.rados.0.smithi042.stdout:2661: done (3 left) 2024-09-20T19:34:42.424 INFO:tasks.rados.rados.0.smithi042.stdout:2662: done (2 left) 2024-09-20T19:34:42.425 INFO:tasks.rados.rados.0.smithi042.stdout:2663: done (1 left) 2024-09-20T19:34:42.425 INFO:tasks.rados.rados.0.smithi042.stdout:2664: done (0 left) 2024-09-20T19:34:42.425 INFO:tasks.rados.rados.0.smithi042.stdout:2665: delete oid 24 current snap is 311 2024-09-20T19:34:42.429 INFO:tasks.rados.rados.0.smithi042.stdout:2665: done (0 left) 2024-09-20T19:34:42.429 INFO:tasks.rados.rados.0.smithi042.stdout:2666: delete oid 3 current snap is 311 2024-09-20T19:34:42.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T19:34:42.436 INFO:tasks.rados.rados.0.smithi042.stdout:2666: done (0 left) 2024-09-20T19:34:42.436 INFO:tasks.rados.rados.0.smithi042.stdout:2667: snap_create 2024-09-20T19:34:42.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:42.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:42.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[22706]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.553 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.554 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:42 smithi042 bash[32075]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:42.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:42.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:42.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.401059+0000 mon.a (mon.0) 2285 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: audit 2024-09-20T19:34:41.411888+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.42:0/1497606073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.416374+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.514085+0000 osd.6 (osd.6) 162 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.516346+0000 osd.6 (osd.6) 163 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:34:42.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.548042+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:34:42.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:42 smithi162 bash[24557]: cluster 2024-09-20T19:34:41.550262+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:34:42.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:42.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:42.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:42.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:43.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:43.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:43.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:43.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:43.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2667: done (0 left) 2024-09-20T19:34:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2668: write oid 23 current snap is 312 2024-09-20T19:34:43.354 INFO:tasks.rados.rados.0.smithi042.stdout:2668: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2024-09-20T19:34:43.354 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:43.359 INFO:tasks.rados.rados.0.smithi042.stdout:2668: writing smithi04268048-23 from 447317 to 1099202 tid 1 2024-09-20T19:34:43.367 INFO:tasks.rados.rados.0.smithi042.stdout:2668: writing smithi04268048-23 from 1713324 to 2341653 tid 2 2024-09-20T19:34:43.377 INFO:tasks.rados.rados.0.smithi042.stdout:2668: writing smithi04268048-23 from 3066571 to 3647317 tid 3 2024-09-20T19:34:43.377 INFO:tasks.rados.rados.0.smithi042.stdout:2669: write (excl) oid 11 current snap is 312 2024-09-20T19:34:43.377 INFO:tasks.rados.rados.0.smithi042.stdout:2669: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-09-20T19:34:43.384 INFO:tasks.rados.rados.0.smithi042.stdout:2669: writing smithi04268048-11 from 464124 to 1191258 tid 1 2024-09-20T19:34:43.393 INFO:tasks.rados.rados.0.smithi042.stdout:2669: writing smithi04268048-11 from 1771806 to 2543793 tid 2 2024-09-20T19:34:43.400 INFO:tasks.rados.rados.0.smithi042.stdout:2669: writing smithi04268048-11 from 2977641 to 3619157 tid 3 2024-09-20T19:34:43.400 INFO:tasks.rados.rados.0.smithi042.stdout:2670: copy_from oid 17 from oid 4 current snap is 312 2024-09-20T19:34:43.400 INFO:tasks.rados.rados.0.smithi042.stdout:2668: finishing write tid 1 to smithi04268048-23 2024-09-20T19:34:43.400 INFO:tasks.rados.rados.0.smithi042.stdout:2671: write oid 14 current snap is 312 2024-09-20T19:34:43.400 INFO:tasks.rados.rados.0.smithi042.stdout:2671: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-09-20T19:34:43.405 INFO:tasks.rados.rados.0.smithi042.stdout:2671: writing smithi04268048-14 from 480931 to 883314 tid 1 2024-09-20T19:34:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:2671: writing smithi04268048-14 from 1313935 to 1829580 tid 2 2024-09-20T19:34:43.418 INFO:tasks.rados.rados.0.smithi042.stdout:2671: writing smithi04268048-14 from 2488711 to 2941499 tid 3 2024-09-20T19:34:43.423 INFO:tasks.rados.rados.0.smithi042.stdout:2671: writing smithi04268048-14 from 3443230 to 3680931 tid 4 2024-09-20T19:34:43.423 INFO:tasks.rados.rados.0.smithi042.stdout:2672: read oid 40 snap -1 2024-09-20T19:34:43.423 INFO:tasks.rados.rados.0.smithi042.stdout:2672: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:34:43.423 INFO:tasks.rados.rados.0.smithi042.stdout:2673: snap_create 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: finishing write tid 2 to smithi04268048-23 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: finishing write tid 3 to smithi04268048-23 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: finishing write tid 4 to smithi04268048-23 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: finishing write tid 5 to smithi04268048-23 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: oid 23 updating version 0 to 963 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: oid 23 updating version 963 to 965 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: oid 23 updating version 965 to 966 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: oid 23 version 966 is already newer than 964 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 966 (ObjNum 931 snap 312 seq_num 931) dirty exists 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2668: left oid 23 (ObjNum 931 snap 312 seq_num 931) 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2669: finishing write tid 1 to smithi04268048-11 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2669: finishing write tid 2 to smithi04268048-11 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2669: finishing write tid 3 to smithi04268048-11 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2669: finishing write tid 4 to smithi04268048-11 2024-09-20T19:34:43.424 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 1 to smithi04268048-14 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: finishing write tid 5 to smithi04268048-11 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: oid 11 updating version 0 to 961 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: oid 11 updating version 961 to 963 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: oid 11 updating version 963 to 964 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: oid 11 version 964 is already newer than 962 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 964 (ObjNum 932 snap 312 seq_num 932) dirty exists 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2669: left oid 11 (ObjNum 932 snap 312 seq_num 932) 2024-09-20T19:34:43.425 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 2 to smithi04268048-14 2024-09-20T19:34:43.428 INFO:tasks.rados.rados.0.smithi042.stdout:2672: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:34:43.429 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:43.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2627607 2024-09-20T19:34:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:43.513 INFO:tasks.rados.rados.0.smithi042.stdout:2670: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:34:43.513 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 3 to smithi04268048-14 2024-09-20T19:34:43.513 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 4 to smithi04268048-14 2024-09-20T19:34:43.513 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 5 to smithi04268048-14 2024-09-20T19:34:43.513 INFO:tasks.rados.rados.0.smithi042.stdout:2671: finishing write tid 6 to smithi04268048-14 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: oid 14 updating version 0 to 820 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: oid 14 updating version 820 to 824 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: oid 14 version 824 is already newer than 822 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: oid 14 version 824 is already newer than 821 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: oid 14 version 824 is already newer than 823 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 824 (ObjNum 933 snap 312 seq_num 933) dirty exists 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2671: left oid 14 (ObjNum 933 snap 312 seq_num 933) 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:2670: finishing copy_from to smithi04268048-17 2024-09-20T19:34:43.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1091 (ObjNum 884 snap 291 seq_num 884) dirty exists 2024-09-20T19:34:43.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:43.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.595 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.596 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.597 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:43 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:43.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:43.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.377020+0000 mgr.y (mgr.24491) 629 : cluster [DBG] pgmap v1158: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 217 MiB data, 850 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.421463+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.547898+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:42.553108+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: audit 2024-09-20T19:34:43.177654+0000 mon.a (mon.0) 2289 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:43 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.348692+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2024-09-20T19:34:43.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:43.857 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:43.960 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:43.989 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:43.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:44.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:44.118 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:44.221 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:44.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:44.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T19:34:44.437 INFO:tasks.rados.rados.0.smithi042.stdout:2668: done (5 left) 2024-09-20T19:34:44.437 INFO:tasks.rados.rados.0.smithi042.stdout:2669: done (4 left) 2024-09-20T19:34:44.438 INFO:tasks.rados.rados.0.smithi042.stdout:2670: done (3 left) 2024-09-20T19:34:44.438 INFO:tasks.rados.rados.0.smithi042.stdout:2671: done (2 left) 2024-09-20T19:34:44.438 INFO:tasks.rados.rados.0.smithi042.stdout:2672: done (1 left) 2024-09-20T19:34:44.438 INFO:tasks.rados.rados.0.smithi042.stdout:2673: done (0 left) 2024-09-20T19:34:44.438 INFO:tasks.rados.rados.0.smithi042.stdout:2674: delete oid 2 current snap is 313 2024-09-20T19:34:44.442 INFO:tasks.rados.rados.0.smithi042.stdout:2674: done (0 left) 2024-09-20T19:34:44.442 INFO:tasks.rados.rados.0.smithi042.stdout:2675: read oid 36 snap -1 2024-09-20T19:34:44.442 INFO:tasks.rados.rados.0.smithi042.stdout:2675: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:34:44.442 INFO:tasks.rados.rados.0.smithi042.stdout:2676: snap_remove snap 278 2024-09-20T19:34:44.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:44.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:44.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:44 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:44.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.269733+0000 osd.1 (osd.1) 263 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.272541+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.487559+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.489657+0000 osd.6 (osd.6) 165 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: audit 2024-09-20T19:34:43.706810+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:34:44.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.064102+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:34:44.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:44 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.066432+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:34:44.734 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:44.763 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:44.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:44.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:44.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:44.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:45.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:45.156 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:45.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:45.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:34:45.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:45.436 INFO:tasks.rados.rados.0.smithi042.stdout:2676: done (1 left) 2024-09-20T19:34:45.436 INFO:tasks.rados.rados.0.smithi042.stdout:2677: read oid 39 snap 290 2024-09-20T19:34:45.436 INFO:tasks.rados.rados.0.smithi042.stdout:2677: expect (ObjNum 853 snap 281 seq_num 853) 2024-09-20T19:34:45.442 INFO:tasks.rados.rados.0.smithi042.stdout:2678: read oid 20 snap -1 2024-09-20T19:34:45.442 INFO:tasks.rados.rados.0.smithi042.stdout:2678: expect (ObjNum 858 snap 284 seq_num 858) 2024-09-20T19:34:45.442 INFO:tasks.rados.rados.0.smithi042.stdout:2679: write (excl) oid 15 current snap is 313 2024-09-20T19:34:45.442 INFO:tasks.rados.rados.0.smithi042.stdout:2679: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-20T19:34:45.443 INFO:tasks.rados.rados.0.smithi042.stdout:2679: writing smithi04268048-15 from 514545 to 1067426 tid 1 2024-09-20T19:34:45.451 INFO:tasks.rados.rados.0.smithi042.stdout:2679: writing smithi04268048-15 from 1830899 to 2350213 tid 2 2024-09-20T19:34:45.461 INFO:tasks.rados.rados.0.smithi042.stdout:2679: writing smithi04268048-15 from 2943557 to 3418889 tid 3 2024-09-20T19:34:45.461 INFO:tasks.rados.rados.0.smithi042.stdout:2680: copy_from oid 23 from oid 14 current snap is 313 2024-09-20T19:34:45.461 INFO:tasks.rados.rados.0.smithi042.stdout:2675: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:34:45.461 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:45.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:45.543 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:45.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:45 smithi162 bash[24557]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.606 INFO:tasks.rados.rados.0.smithi042.stdout:2678: expect (ObjNum 858 snap 284 seq_num 858) 2024-09-20T19:34:45.627 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2263235 2024-09-20T19:34:45.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.649 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2209203 2024-09-20T19:34:45.651 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 60076 2024-09-20T19:34:45.674 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[32075]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.940825+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:43.943648+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.068127+0000 osd.5 (osd.5) 243 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.070315+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.293896+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.295693+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.377805+0000 mgr.y (mgr.24491) 630 : cluster [DBG] pgmap v1161: 178 pgs: 178 active+clean; 217 MiB data, 859 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: audit 2024-09-20T19:34:44.426123+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.42:0/2924951988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.433974+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.508460+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:45 smithi042 bash[22706]: cluster 2024-09-20T19:34:44.512448+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:34:45.685 INFO:tasks.rados.rados.0.smithi042.stdout:2679: finishing write tid 1 to smithi04268048-15 2024-09-20T19:34:45.685 INFO:tasks.rados.rados.0.smithi042.stdout:2679: finishing write tid 2 to smithi04268048-15 2024-09-20T19:34:45.685 INFO:tasks.rados.rados.0.smithi042.stdout:2677: expect (ObjNum 853 snap 281 seq_num 853) 2024-09-20T19:34:45.727 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1150837 2024-09-20T19:34:45.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T19:34:45.776 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:45.787 INFO:tasks.rados.rados.0.smithi042.stdout:2679: finishing write tid 3 to smithi04268048-15 2024-09-20T19:34:45.788 INFO:tasks.rados.rados.0.smithi042.stdout:2675: done (4 left) 2024-09-20T19:34:45.788 INFO:tasks.rados.rados.0.smithi042.stdout:2677: done (3 left) 2024-09-20T19:34:45.788 INFO:tasks.rados.rados.0.smithi042.stdout:2678: done (2 left) 2024-09-20T19:34:45.788 INFO:tasks.rados.rados.0.smithi042.stdout:2681: snap_remove snap 300 2024-09-20T19:34:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:45.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.933 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:45.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:46.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:46.073 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:34:46.074 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:34:46.093 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:34:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:34:46.110 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:34:46.111 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:34:46.128 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:34:46.128 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:34:46.149 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:34:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:34:46.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.170 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:34:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:34:46.190 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:34:46.190 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:34:46.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:46.208 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:34:46.209 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:34:46.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:46.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.452 INFO:tasks.rados.rados.0.smithi042.stdout:2681: done (2 left) 2024-09-20T19:34:46.453 INFO:tasks.rados.rados.0.smithi042.stdout:2682: write oid 30 current snap is 313 2024-09-20T19:34:46.453 INFO:tasks.rados.rados.0.smithi042.stdout:2682: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-20T19:34:46.459 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.460 INFO:tasks.rados.rados.0.smithi042.stdout:2682: writing smithi04268048-30 from 531352 to 1275835 tid 1 2024-09-20T19:34:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:46.469 INFO:tasks.rados.rados.0.smithi042.stdout:2682: writing smithi04268048-30 from 2005734 to 2785059 tid 2 2024-09-20T19:34:46.475 INFO:tasks.rados.rados.0.smithi042.stdout:2682: writing smithi04268048-30 from 3487333 to 3731352 tid 3 2024-09-20T19:34:46.475 INFO:tasks.rados.rados.0.smithi042.stdout:2683: read oid 27 snap -1 2024-09-20T19:34:46.475 INFO:tasks.rados.rados.0.smithi042.stdout:2683: expect deleted 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2684: rollback oid 34 current snap is 313 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 302 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: finishing write tid 4 to smithi04268048-15 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: finishing write tid 5 to smithi04268048-15 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: oid 15 updating version 0 to 731 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: oid 15 version 731 is already newer than 728 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: oid 15 version 731 is already newer than 730 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: oid 15 version 731 is already newer than 729 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 731 (ObjNum 935 snap 313 seq_num 935) dirty exists 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: left oid 15 (ObjNum 935 snap 313 seq_num 935) 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2680: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2680: finishing copy_from to smithi04268048-23 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 824 (ObjNum 933 snap 312 seq_num 933) dirty exists 2024-09-20T19:34:46.476 INFO:tasks.rados.rados.0.smithi042.stdout:2679: done (4 left) 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2680: done (3 left) 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2685: read oid 41 snap -1 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2685: expect deleted 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2686: rollback oid 50 current snap is 313 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 302 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2682: finishing write tid 1 to smithi04268048-30 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2682: finishing write tid 2 to smithi04268048-30 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2687: rollback oid 44 current snap is 313 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 295 2024-09-20T19:34:46.477 INFO:tasks.rados.rados.0.smithi042.stdout:2688: snap_create 2024-09-20T19:34:46.481 INFO:tasks.rados.rados.0.smithi042.stdout:2686: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:34:46.481 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 939 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:34:46.481 INFO:tasks.rados.rados.0.smithi042.stdout:2684: finishing rollback tid 0 to smithi04268048-34 2024-09-20T19:34:46.482 INFO:tasks.rados.rados.0.smithi042.stdout:2687: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:34:46.482 INFO:tasks.rados.rados.0.smithi042.stdout:2682: finishing write tid 3 to smithi04268048-30 2024-09-20T19:34:46.482 INFO:tasks.rados.rados.0.smithi042.stdout:2682: finishing write tid 4 to smithi04268048-30 2024-09-20T19:34:46.482 INFO:tasks.rados.rados.0.smithi042.stdout:2682: finishing write tid 5 to smithi04268048-30 2024-09-20T19:34:46.482 INFO:tasks.rados.rados.0.smithi042.stdout:2682: oid 30 updating version 0 to 1120 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2682: oid 30 version 1120 is already newer than 1119 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2682: oid 30 updating version 1120 to 1121 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2682: oid 30 version 1121 is already newer than 1118 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1121 (ObjNum 936 snap 313 seq_num 936) dirty exists 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2682: left oid 30 (ObjNum 936 snap 313 seq_num 936) 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2684: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:2684: finishing rollback tid 2 to smithi04268048-34 2024-09-20T19:34:46.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1072 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:34:46.484 INFO:tasks.rados.rados.0.smithi042.stdout:2687: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:34:46.484 INFO:tasks.rados.rados.0.smithi042.stdout:2687: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:34:46.484 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1129 (ObjNum 868 snap 288 seq_num 868) dirty exists 2024-09-20T19:34:46.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.574 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:46.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[32075]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:46 smithi042 bash[22706]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:46.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:46.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.431304+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.546321+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e scrub starts 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: cluster 2024-09-20T19:34:45.548226+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e scrub ok 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:46 smithi162 bash[24557]: audit 2024-09-20T19:34:46.055236+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.42:0/2330368566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:46.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:46.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:46.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:46.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:46.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:47.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:47.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:47.209 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:34:47.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:47.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:47.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:47.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:47.464 INFO:tasks.rados.rados.0.smithi042.stdout:2682: done (6 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2683: done (5 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2684: done (4 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2685: done (3 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2686: done (2 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2687: done (1 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2688: done (0 left) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2689: read oid 49 snap -1 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2689: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2690: read oid 34 snap -1 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2690: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2691: copy_from oid 12 from oid 2 current snap is 314 2024-09-20T19:34:47.465 INFO:tasks.rados.rados.0.smithi042.stdout:2692: copy_from oid 19 from oid 8 current snap is 314 2024-09-20T19:34:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:2693: read oid 16 snap -1 2024-09-20T19:34:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:2693: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:2694: write (excl) oid 11 current snap is 314 2024-09-20T19:34:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:2694: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-20T19:34:47.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:47.477 INFO:tasks.rados.rados.0.smithi042.stdout:2694: writing smithi04268048-11 from 548159 to 967891 tid 1 2024-09-20T19:34:47.491 INFO:tasks.rados.rados.0.smithi042.stdout:2694: writing smithi04268048-11 from 1664216 to 2187199 tid 2 2024-09-20T19:34:47.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:47.500 INFO:tasks.rados.rados.0.smithi042.stdout:2694: writing smithi04268048-11 from 2714756 to 3328985 tid 3 2024-09-20T19:34:47.500 INFO:tasks.rados.rados.0.smithi042.stdout:2695: delete oid 24 current snap is 314 2024-09-20T19:34:47.500 INFO:tasks.rados.rados.0.smithi042.stdout:2689: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:34:47.500 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:47.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:47.617 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2872609 2024-09-20T19:34:47.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:47.705 INFO:tasks.rados.rados.0.smithi042.stdout:2692: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:34:47.706 INFO:tasks.rados.rados.0.smithi042.stdout:2692: finishing copy_from to smithi04268048-19 2024-09-20T19:34:47.706 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:34:47.706 INFO:tasks.rados.rados.0.smithi042.stdout:2690: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:34:47.706 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:47.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:47.739 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.740 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[32075]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:47 smithi042 bash[22706]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.753 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:47.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:47.783 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1281284 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.260667+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.261765+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.378596+0000 mgr.y (mgr.24491) 631 : cluster [DBG] pgmap v1164: 178 pgs: 178 active+clean; 217 MiB data, 864 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 2 op/s 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.449514+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.521636+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:47 smithi162 bash[24557]: cluster 2024-09-20T19:34:46.528116+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:34:47.806 INFO:tasks.rados.rados.0.smithi042.stdout:2691: finishing copy_from to smithi04268048-12 2024-09-20T19:34:47.806 INFO:tasks.rados.rados.0.smithi042.stdout:2691: got expected ENOENT (src dne) 2024-09-20T19:34:47.806 INFO:tasks.rados.rados.0.smithi042.stdout:2691: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:34:47.806 INFO:tasks.rados.rados.0.smithi042.stdout:2693: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:47.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:47.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:47.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:47.908 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2694: finishing write tid 1 to smithi04268048-11 2024-09-20T19:34:47.967 INFO:tasks.rados.rados.0.smithi042.stdout:2694: finishing write tid 2 to smithi04268048-11 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: finishing write tid 3 to smithi04268048-11 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: finishing write tid 4 to smithi04268048-11 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: finishing write tid 5 to smithi04268048-11 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: oid 11 updating version 0 to 972 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: oid 11 version 972 is already newer than 970 2024-09-20T19:34:47.968 INFO:tasks.rados.rados.0.smithi042.stdout:2694: oid 11 version 972 is already newer than 971 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2694: oid 11 version 972 is already newer than 969 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 972 (ObjNum 937 snap 314 seq_num 937) dirty exists 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2694: left oid 11 (ObjNum 937 snap 314 seq_num 937) 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2689: done (6 left) 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2690: done (5 left) 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2691: done (4 left) 2024-09-20T19:34:47.969 INFO:tasks.rados.rados.0.smithi042.stdout:2692: done (3 left) 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2693: done (2 left) 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2694: done (1 left) 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2695: done (0 left) 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2696: rollback oid 35 current snap is 314 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 296 2024-09-20T19:34:47.970 INFO:tasks.rados.rados.0.smithi042.stdout:2697: delete oid 45 current snap is 314 2024-09-20T19:34:47.971 INFO:tasks.rados.rados.0.smithi042.stdout:2696: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:34:47.971 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1065 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T19:34:47.971 INFO:tasks.rados.rados.0.smithi042.stdout:2696: done (1 left) 2024-09-20T19:34:47.972 INFO:tasks.rados.rados.0.smithi042.stdout:2697: done (0 left) 2024-09-20T19:34:47.972 INFO:tasks.rados.rados.0.smithi042.stdout:2698: snap_remove snap 305 2024-09-20T19:34:47.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:48.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:48.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:48.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:48.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:48.360 INFO:tasks.rados.rados.0.smithi042.stdout:2698: done (0 left) 2024-09-20T19:34:48.361 INFO:tasks.rados.rados.0.smithi042.stdout:2699: read oid 48 snap -1 2024-09-20T19:34:48.361 INFO:tasks.rados.rados.0.smithi042.stdout:2699: expect (ObjNum 921 snap 305 seq_num 921) 2024-09-20T19:34:48.361 INFO:tasks.rados.rados.0.smithi042.stdout:2700: write (excl) oid 26 current snap is 314 2024-09-20T19:34:48.361 INFO:tasks.rados.rados.0.smithi042.stdout:2700: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-20T19:34:48.367 INFO:tasks.rados.rados.0.smithi042.stdout:2700: writing smithi04268048-26 from 598580 to 1244059 tid 1 2024-09-20T19:34:48.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:48.374 INFO:tasks.rados.rados.0.smithi042.stdout:2700: writing smithi04268048-26 from 1723309 to 2509972 tid 2 2024-09-20T19:34:48.381 INFO:tasks.rados.rados.0.smithi042.stdout:2700: writing smithi04268048-26 from 3080672 to 3528717 tid 3 2024-09-20T19:34:48.381 INFO:tasks.rados.rados.0.smithi042.stdout:2701: write oid 33 current snap is 314 2024-09-20T19:34:48.381 INFO:tasks.rados.rados.0.smithi042.stdout:2701: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-20T19:34:48.388 INFO:tasks.rados.rados.0.smithi042.stdout:2701: writing smithi04268048-33 from 615387 to 1336115 tid 1 2024-09-20T19:34:48.395 INFO:tasks.rados.rados.0.smithi042.stdout:2701: writing smithi04268048-33 from 1781791 to 2312112 tid 2 2024-09-20T19:34:48.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:48.404 INFO:tasks.rados.rados.0.smithi042.stdout:2701: writing smithi04268048-33 from 3108095 to 3815387 tid 3 2024-09-20T19:34:48.404 INFO:tasks.rados.rados.0.smithi042.stdout:2702: read oid 17 snap -1 2024-09-20T19:34:48.404 INFO:tasks.rados.rados.0.smithi042.stdout:2702: expect (ObjNum 884 snap 291 seq_num 884) 2024-09-20T19:34:48.404 INFO:tasks.rados.rados.0.smithi042.stdout:2703: snap_remove snap 311 2024-09-20T19:34:48.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:48.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:48.660 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.757 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.757 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.757 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:48 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:48.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:48.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:48.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:48.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.080992+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.083528+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.293008+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.c scrub starts 2024-09-20T19:34:48.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.295402+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.c scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.462969+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.534751+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:47.537608+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.150918+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.152570+0000 osd.7 (osd.7) 192 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.234691+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:34:48.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.236970+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:34:48.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:48 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.354707+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2024-09-20T19:34:48.934 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:34:48.934 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:34:49.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:49.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:49.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.293 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:49.383 INFO:tasks.rados.rados.0.smithi042.stdout:2703: done (4 left) 2024-09-20T19:34:49.383 INFO:tasks.rados.rados.0.smithi042.stdout:2704: write (excl) oid 7 current snap is 314 2024-09-20T19:34:49.383 INFO:tasks.rados.rados.0.smithi042.stdout:2704: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-20T19:34:49.393 INFO:tasks.rados.rados.0.smithi042.stdout:2704: writing smithi04268048-7 from 632194 to 1428171 tid 1 2024-09-20T19:34:49.396 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.404 INFO:tasks.rados.rados.0.smithi042.stdout:2704: writing smithi04268048-7 from 2123920 to 2914252 tid 2 2024-09-20T19:34:49.411 INFO:tasks.rados.rados.0.smithi042.stdout:2704: writing smithi04268048-7 from 3419165 to 3832194 tid 3 2024-09-20T19:34:49.411 INFO:tasks.rados.rados.0.smithi042.stdout:2705: rollback oid 22 current snap is 314 2024-09-20T19:34:49.411 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 306 2024-09-20T19:34:49.411 INFO:tasks.rados.rados.0.smithi042.stdout:2699: expect (ObjNum 921 snap 305 seq_num 921) 2024-09-20T19:34:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:49.456 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:34:49.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:49.547 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2700: finishing write tid 1 to smithi04268048-26 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2700: finishing write tid 2 to smithi04268048-26 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2699: done (5 left) 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2706: write (excl) oid 20 current snap is 314 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2700: finishing write tid 3 to smithi04268048-26 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2700: finishing write tid 4 to smithi04268048-26 2024-09-20T19:34:49.623 INFO:tasks.rados.rados.0.smithi042.stdout:2706: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2024-09-20T19:34:49.629 INFO:tasks.rados.rados.0.smithi042.stdout:2706: writing smithi04268048-20 from 649001 to 1236580 tid 1 2024-09-20T19:34:49.636 INFO:tasks.rados.rados.0.smithi042.stdout:2706: writing smithi04268048-20 from 1898755 to 2432745 tid 2 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2706: writing smithi04268048-20 from 3162941 to 3849001 tid 3 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2707: delete oid 15 current snap is 314 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2700: finishing write tid 5 to smithi04268048-26 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2700: oid 26 updating version 0 to 1216 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2700: oid 26 updating version 1216 to 1218 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2700: oid 26 version 1218 is already newer than 1217 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:2700: oid 26 version 1218 is already newer than 1215 2024-09-20T19:34:49.646 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1218 (ObjNum 940 snap 314 seq_num 940) dirty exists 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2700: left oid 26 (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: finishing write tid 1 to smithi04268048-33 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: finishing write tid 2 to smithi04268048-33 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: finishing write tid 3 to smithi04268048-33 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: finishing write tid 4 to smithi04268048-33 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: finishing write tid 5 to smithi04268048-33 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: oid 33 updating version 0 to 1286 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: oid 33 updating version 1286 to 1289 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: oid 33 version 1289 is already newer than 1287 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: oid 33 version 1289 is already newer than 1288 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1289 (ObjNum 941 snap 314 seq_num 941) dirty exists 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2701: left oid 33 (ObjNum 941 snap 314 seq_num 941) 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:2702: expect (ObjNum 884 snap 291 seq_num 884) 2024-09-20T19:34:49.647 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:49.651 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.659 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[32075]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:49 smithi042 bash[22706]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:49.730 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 50194 2024-09-20T19:34:49.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:49.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:49.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.068207+0000 osd.5 (osd.5) 247 : cluster [DBG] 5.f scrub starts 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.070330+0000 osd.5 (osd.5) 248 : cluster [DBG] 5.f scrub ok 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.379546+0000 mgr.y (mgr.24491) 632 : cluster [DBG] pgmap v1168: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 226 MiB data, 892 MiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.183943+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub starts 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.184955+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.b scrub ok 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:49 smithi162 bash[24557]: cluster 2024-09-20T19:34:49.376129+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2024-09-20T19:34:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:49.813 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:49.897 INFO:tasks.rados.rados.0.smithi042.stdout:2704: finishing write tid 1 to smithi04268048-7 2024-09-20T19:34:49.897 INFO:tasks.rados.rados.0.smithi042.stdout:2704: finishing write tid 2 to smithi04268048-7 2024-09-20T19:34:49.897 INFO:tasks.rados.rados.0.smithi042.stdout:2704: finishing write tid 3 to smithi04268048-7 2024-09-20T19:34:49.897 INFO:tasks.rados.rados.0.smithi042.stdout:2705: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2705: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2705: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1027 (ObjNum 890 snap 294 seq_num 890) dirty exists 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: finishing write tid 4 to smithi04268048-7 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: finishing write tid 5 to smithi04268048-7 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: oid 7 updating version 0 to 1219 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: oid 7 updating version 1219 to 1221 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: oid 7 updating version 1221 to 1222 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: oid 7 version 1222 is already newer than 1220 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1222 (ObjNum 942 snap 314 seq_num 942) dirty exists 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2704: left oid 7 (ObjNum 942 snap 314 seq_num 942) 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2706: finishing write tid 1 to smithi04268048-20 2024-09-20T19:34:49.898 INFO:tasks.rados.rados.0.smithi042.stdout:2706: finishing write tid 2 to smithi04268048-20 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: finishing write tid 3 to smithi04268048-20 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: finishing write tid 4 to smithi04268048-20 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: finishing write tid 5 to smithi04268048-20 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: oid 20 updating version 0 to 998 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: oid 20 updating version 998 to 999 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: oid 20 version 999 is already newer than 997 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: oid 20 version 999 is already newer than 996 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 999 (ObjNum 943 snap 314 seq_num 943) dirty exists 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2706: left oid 20 (ObjNum 943 snap 314 seq_num 943) 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2700: done (6 left) 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2701: done (5 left) 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2702: done (4 left) 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2704: done (3 left) 2024-09-20T19:34:49.899 INFO:tasks.rados.rados.0.smithi042.stdout:2705: done (2 left) 2024-09-20T19:34:49.900 INFO:tasks.rados.rados.0.smithi042.stdout:2706: done (1 left) 2024-09-20T19:34:49.900 INFO:tasks.rados.rados.0.smithi042.stdout:2707: done (0 left) 2024-09-20T19:34:49.900 INFO:tasks.rados.rados.0.smithi042.stdout:2708: delete oid 44 current snap is 314 2024-09-20T19:34:49.901 INFO:tasks.rados.rados.0.smithi042.stdout:2708: done (0 left) 2024-09-20T19:34:49.901 INFO:tasks.rados.rados.0.smithi042.stdout:2709: write oid 19 current snap is 314 2024-09-20T19:34:49.901 INFO:tasks.rados.rados.0.smithi042.stdout:2709: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-20T19:34:49.905 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.918 INFO:tasks.rados.rados.0.smithi042.stdout:2709: writing smithi04268048-19 from 699422 to 1112748 tid 1 2024-09-20T19:34:49.927 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:49.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:49.929 INFO:tasks.rados.rados.0.smithi042.stdout:2709: writing smithi04268048-19 from 1557848 to 2239165 tid 2 2024-09-20T19:34:49.938 INFO:tasks.rados.rados.0.smithi042.stdout:2709: writing smithi04268048-19 from 3012504 to 3760887 tid 3 2024-09-20T19:34:49.938 INFO:tasks.rados.rados.0.smithi042.stdout:2710: write oid 33 current snap is 314 2024-09-20T19:34:49.938 INFO:tasks.rados.rados.0.smithi042.stdout:2709: finishing write tid 1 to smithi04268048-19 2024-09-20T19:34:49.938 INFO:tasks.rados.rados.0.smithi042.stdout:2709: finishing write tid 2 to smithi04268048-19 2024-09-20T19:34:49.938 INFO:tasks.rados.rados.0.smithi042.stdout:2710: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-20T19:34:49.944 INFO:tasks.rados.rados.0.smithi042.stdout:2710: writing smithi04268048-33 from 716229 to 1204804 tid 1 2024-09-20T19:34:49.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T19:34:49.952 INFO:tasks.rados.rados.0.smithi042.stdout:2710: writing smithi04268048-33 from 1616330 to 2157658 tid 2 2024-09-20T19:34:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:2710: writing smithi04268048-33 from 2756280 to 3315935 tid 3 2024-09-20T19:34:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:2711: rollback oid 13 current snap is 314 2024-09-20T19:34:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 290 2024-09-20T19:34:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:2709: finishing write tid 3 to smithi04268048-19 2024-09-20T19:34:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:2709: finishing write tid 4 to smithi04268048-19 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: finishing write tid 5 to smithi04268048-19 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: oid 19 updating version 0 to 857 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: oid 19 updating version 857 to 860 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: oid 19 version 860 is already newer than 858 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: oid 19 version 860 is already newer than 859 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 860 (ObjNum 946 snap 314 seq_num 946) dirty exists 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: left oid 19 (ObjNum 946 snap 314 seq_num 946) 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2710: finishing write tid 1 to smithi04268048-33 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2710: finishing write tid 2 to smithi04268048-33 2024-09-20T19:34:49.965 INFO:tasks.rados.rados.0.smithi042.stdout:2709: done (2 left) 2024-09-20T19:34:49.966 INFO:tasks.rados.rados.0.smithi042.stdout:2712: snap_create 2024-09-20T19:34:49.967 INFO:tasks.rados.rados.0.smithi042.stdout:2711: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:34:49.967 INFO:tasks.rados.rados.0.smithi042.stdout:2711: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:34:49.967 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 593 (ObjNum 860 snap 284 seq_num 860) dirty exists 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: finishing write tid 3 to smithi04268048-33 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: finishing write tid 4 to smithi04268048-33 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: finishing write tid 5 to smithi04268048-33 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: oid 33 updating version 0 to 1291 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: oid 33 updating version 1291 to 1292 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: oid 33 updating version 1292 to 1293 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: oid 33 version 1293 is already newer than 1290 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1293 (ObjNum 947 snap 314 seq_num 947) dirty exists 2024-09-20T19:34:49.970 INFO:tasks.rados.rados.0.smithi042.stdout:2710: left oid 33 (ObjNum 947 snap 314 seq_num 947) 2024-09-20T19:34:50.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:50.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:50.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:50.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:50.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:50.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:50.403 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:50.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:50.462 INFO:tasks.rados.rados.0.smithi042.stdout:2710: done (2 left) 2024-09-20T19:34:50.462 INFO:tasks.rados.rados.0.smithi042.stdout:2711: done (1 left) 2024-09-20T19:34:50.462 INFO:tasks.rados.rados.0.smithi042.stdout:2712: done (0 left) 2024-09-20T19:34:50.462 INFO:tasks.rados.rados.0.smithi042.stdout:2713: delete oid 6 current snap is 315 2024-09-20T19:34:50.474 INFO:tasks.rados.rados.0.smithi042.stdout:2713: done (0 left) 2024-09-20T19:34:50.474 INFO:tasks.rados.rados.0.smithi042.stdout:2714: rollback oid 35 current snap is 315 2024-09-20T19:34:50.474 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 307 2024-09-20T19:34:50.474 INFO:tasks.rados.rados.0.smithi042.stdout:2715: snap_remove snap 314 2024-09-20T19:34:50.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:50.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:50 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:50.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.938092+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:48.939090+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.139173+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:34:50.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:50 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.140491+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:34:50.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:50.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:50.950 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:50.951 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:50.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:50.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:51.053 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:51.180 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:51.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:51.444 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.478 INFO:tasks.rados.rados.0.smithi042.stdout:2715: done (1 left) 2024-09-20T19:34:51.478 INFO:tasks.rados.rados.0.smithi042.stdout:2716: snap_remove snap 313 2024-09-20T19:34:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:51.592 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:51.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:51 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:51.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:51.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:51.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:51.995 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: audit 2024-09-20T19:34:49.725870+0000 mgr.y (mgr.24491) 633 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.380505+0000 mgr.y (mgr.24491) 634 : cluster [DBG] pgmap v1170: 178 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 159 active+clean; 230 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 6.0 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:50.461729+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.146209+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:51.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:51 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.148485+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:34:52.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:52.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:52.137 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:52.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:52.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:52.490 INFO:tasks.rados.rados.0.smithi042.stdout:2716: done (1 left) 2024-09-20T19:34:52.490 INFO:tasks.rados.rados.0.smithi042.stdout:2717: rollback oid 14 current snap is 315 2024-09-20T19:34:52.490 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 295 2024-09-20T19:34:52.491 INFO:tasks.rados.rados.0.smithi042.stdout:2718: write oid 3 current snap is 315 2024-09-20T19:34:52.491 INFO:tasks.rados.rados.0.smithi042.stdout:2718: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-09-20T19:34:52.499 INFO:tasks.rados.rados.0.smithi042.stdout:2718: writing smithi04268048-3 from 749843 to 1388916 tid 1 2024-09-20T19:34:52.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:52.512 INFO:tasks.rados.rados.0.smithi042.stdout:2718: writing smithi04268048-3 from 2133294 to 2678291 tid 2 2024-09-20T19:34:52.525 INFO:tasks.rados.rados.0.smithi042.stdout:2718: writing smithi04268048-3 from 3211126 to 3793325 tid 3 2024-09-20T19:34:52.525 INFO:tasks.rados.rados.0.smithi042.stdout:2719: snap_create 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2714: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1065 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2718: finishing write tid 1 to smithi04268048-3 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2718: finishing write tid 2 to smithi04268048-3 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2717: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2717: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:2717: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:34:52.526 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 827 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:34:52.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:52.530 INFO:tasks.rados.rados.0.smithi042.stdout:2718: finishing write tid 3 to smithi04268048-3 2024-09-20T19:34:52.530 INFO:tasks.rados.rados.0.smithi042.stdout:2718: finishing write tid 4 to smithi04268048-3 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: finishing write tid 5 to smithi04268048-3 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: oid 3 updating version 0 to 953 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: oid 3 updating version 953 to 954 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: oid 3 updating version 954 to 955 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: oid 3 updating version 955 to 956 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 956 (ObjNum 949 snap 315 seq_num 949) dirty exists 2024-09-20T19:34:52.531 INFO:tasks.rados.rados.0.smithi042.stdout:2718: left oid 3 (ObjNum 949 snap 315 seq_num 949) 2024-09-20T19:34:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:52.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:52.659 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:52.762 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:52 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:52.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:52.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:52.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.474730+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.519447+0000 osd.6 (osd.6) 166 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.522913+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.295459+0000 osd.2 (osd.2) 241 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:34:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:52 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.301445+0000 osd.2 (osd.2) 242 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:34:53.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:53.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:53.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:53.177 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:53.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:53.305 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:53.360 INFO:tasks.rados.rados.0.smithi042.stdout:2714: done (3 left) 2024-09-20T19:34:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:2717: done (2 left) 2024-09-20T19:34:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:2718: done (1 left) 2024-09-20T19:34:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:2719: done (0 left) 2024-09-20T19:34:53.361 INFO:tasks.rados.rados.0.smithi042.stdout:2720: delete oid 1 current snap is 316 2024-09-20T19:34:53.376 INFO:tasks.rados.rados.0.smithi042.stdout:2720: done (0 left) 2024-09-20T19:34:53.376 INFO:tasks.rados.rados.0.smithi042.stdout:2721: write (excl) oid 47 current snap is 316 2024-09-20T19:34:53.376 INFO:tasks.rados.rados.0.smithi042.stdout:2721: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-20T19:34:53.381 INFO:tasks.rados.rados.0.smithi042.stdout:2721: writing smithi04268048-47 from 783457 to 1289381 tid 1 2024-09-20T19:34:53.390 INFO:tasks.rados.rados.0.smithi042.stdout:2721: writing smithi04268048-47 from 1850258 to 2398924 tid 2 2024-09-20T19:34:53.399 INFO:tasks.rados.rados.0.smithi042.stdout:2721: writing smithi04268048-47 from 2865972 to 3587068 tid 3 2024-09-20T19:34:53.400 INFO:tasks.rados.rados.0.smithi042.stdout:2722: write (excl) oid 32 current snap is 316 2024-09-20T19:34:53.400 INFO:tasks.rados.rados.0.smithi042.stdout:2721: finishing write tid 1 to smithi04268048-47 2024-09-20T19:34:53.400 INFO:tasks.rados.rados.0.smithi042.stdout:2722: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-20T19:34:53.407 INFO:tasks.rados.rados.0.smithi042.stdout:2722: writing smithi04268048-32 from 400264 to 981437 tid 1 2024-09-20T19:34:53.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:53.415 INFO:tasks.rados.rados.0.smithi042.stdout:2722: writing smithi04268048-32 from 1508740 to 2000264 tid 2 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2723: snap_create 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: finishing write tid 2 to smithi04268048-47 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: finishing write tid 3 to smithi04268048-47 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: finishing write tid 4 to smithi04268048-47 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: finishing write tid 5 to smithi04268048-47 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: oid 47 updating version 0 to 1133 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: oid 47 version 1133 is already newer than 1132 2024-09-20T19:34:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:2721: oid 47 updating version 1133 to 1134 2024-09-20T19:34:53.417 INFO:tasks.rados.rados.0.smithi042.stdout:2721: oid 47 updating version 1134 to 1135 2024-09-20T19:34:53.417 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1135 (ObjNum 951 snap 316 seq_num 951) dirty exists 2024-09-20T19:34:53.417 INFO:tasks.rados.rados.0.smithi042.stdout:2721: left oid 47 (ObjNum 951 snap 316 seq_num 951) 2024-09-20T19:34:53.417 INFO:tasks.rados.rados.0.smithi042.stdout:2722: finishing write tid 1 to smithi04268048-32 2024-09-20T19:34:53.420 INFO:tasks.rados.rados.0.smithi042.stdout:2722: finishing write tid 2 to smithi04268048-32 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: finishing write tid 3 to smithi04268048-32 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: finishing write tid 4 to smithi04268048-32 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: oid 32 updating version 0 to 1326 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: oid 32 version 1326 is already newer than 1325 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: oid 32 version 1326 is already newer than 1324 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1326 (ObjNum 952 snap 316 seq_num 952) dirty exists 2024-09-20T19:34:53.421 INFO:tasks.rados.rados.0.smithi042.stdout:2722: left oid 32 (ObjNum 952 snap 316 seq_num 952) 2024-09-20T19:34:53.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:53.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:53.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.545 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.546 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:53 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:53.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.677 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.899384+0000 osd.0 (osd.0) 61 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:51.901983+0000 osd.0 (osd.0) 62 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.382415+0000 mgr.y (mgr.24491) 635 : cluster [DBG] pgmap v1173: 178 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 165 active+clean; 224 MiB data, 901 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T19:34:53.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:52.478058+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2024-09-20T19:34:53.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:53 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.357433+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2024-09-20T19:34:53.690 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:53.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:53.819 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:53.922 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:53.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:54.019 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:34:54.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:54.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:54.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:54.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T19:34:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:34:54.361 INFO:tasks.rados.rados.0.smithi042.stdout:2721: done (2 left) 2024-09-20T19:34:54.361 INFO:tasks.rados.rados.0.smithi042.stdout:2722: done (1 left) 2024-09-20T19:34:54.362 INFO:tasks.rados.rados.0.smithi042.stdout:2723: done (0 left) 2024-09-20T19:34:54.362 INFO:tasks.rados.rados.0.smithi042.stdout:2724: rollback oid 15 current snap is 317 2024-09-20T19:34:54.362 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 306 2024-09-20T19:34:54.362 INFO:tasks.rados.rados.0.smithi042.stdout:2725: snap_remove snap 307 2024-09-20T19:34:54.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:54.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:54.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:54.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:54 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:54.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:54.737 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:34:54.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.008246+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.a scrub starts 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.009929+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.a scrub ok 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.190252+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.244657+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.281885+0000 osd.1 (osd.1) 269 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.283686+0000 osd.1 (osd.1) 270 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.350544+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:34:54.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.350579+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.352279+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.42:0/3247751374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: audit 2024-09-20T19:34:54.352354+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? v1:172.21.15.42:0/3327575055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.357179+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.362494+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:54 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.365564+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:34:54.815 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:54.841 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:54.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:54.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:54.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:55.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:55.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:55.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:55.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:55.357 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:55.371 INFO:tasks.rados.rados.0.smithi042.stdout:2725: done (1 left) 2024-09-20T19:34:55.371 INFO:tasks.rados.rados.0.smithi042.stdout:2726: snap_create 2024-09-20T19:34:55.371 INFO:tasks.rados.rados.0.smithi042.stdout:2724: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:34:55.371 INFO:tasks.rados.rados.0.smithi042.stdout:2724: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:34:55.371 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 734 (ObjNum 811 snap 268 seq_num 811) dirty exists 2024-09-20T19:34:55.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:55.486 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:55.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:55 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:55.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.899436+0000 osd.0 (osd.0) 63 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:53.902994+0000 osd.0 (osd.0) 64 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.382870+0000 mgr.y (mgr.24491) 636 : cluster [DBG] pgmap v1177: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 224 MiB data, 906 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.172804+0000 osd.2 (osd.2) 245 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:55.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.174577+0000 osd.2 (osd.2) 246 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.323330+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.325559+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:55 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.364996+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2024-09-20T19:34:55.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:55.745 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:55.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:55.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:55.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:56.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:56.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:56.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:56.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:56.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:56.374 INFO:tasks.rados.rados.0.smithi042.stdout:2724: done (1 left) 2024-09-20T19:34:56.375 INFO:tasks.rados.rados.0.smithi042.stdout:2726: done (0 left) 2024-09-20T19:34:56.375 INFO:tasks.rados.rados.0.smithi042.stdout:2727: write (excl) oid 40 current snap is 318 2024-09-20T19:34:56.375 INFO:tasks.rados.rados.0.smithi042.stdout:2727: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-20T19:34:56.383 INFO:tasks.rados.rados.0.smithi042.stdout:2727: writing smithi04268048-40 from 417071 to 1073493 tid 1 2024-09-20T19:34:56.391 INFO:tasks.rados.rados.0.smithi042.stdout:2727: writing smithi04268048-40 from 1567222 to 2017071 tid 2 2024-09-20T19:34:56.391 INFO:tasks.rados.rados.0.smithi042.stdout:2728: delete oid 47 current snap is 318 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: finishing write tid 1 to smithi04268048-40 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: finishing write tid 2 to smithi04268048-40 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: finishing write tid 3 to smithi04268048-40 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: finishing write tid 4 to smithi04268048-40 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: oid 40 updating version 0 to 1052 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: oid 40 updating version 1052 to 1054 2024-09-20T19:34:56.397 INFO:tasks.rados.rados.0.smithi042.stdout:2727: oid 40 version 1054 is already newer than 1053 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1054 (ObjNum 953 snap 318 seq_num 953) dirty exists 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:2727: left oid 40 (ObjNum 953 snap 318 seq_num 953) 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:2727: done (1 left) 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:2728: done (0 left) 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:2729: read oid 16 snap -1 2024-09-20T19:34:56.398 INFO:tasks.rados.rados.0.smithi042.stdout:2729: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:56.399 INFO:tasks.rados.rados.0.smithi042.stdout:2730: write (excl) oid 40 current snap is 318 2024-09-20T19:34:56.399 INFO:tasks.rados.rados.0.smithi042.stdout:2730: seq_num 955 ranges {450685=406920,1567833=482852} 2024-09-20T19:34:56.403 INFO:tasks.rados.rados.0.smithi042.stdout:2730: writing smithi04268048-40 from 450685 to 857605 tid 1 2024-09-20T19:34:56.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:56.410 INFO:tasks.rados.rados.0.smithi042.stdout:2730: writing smithi04268048-40 from 1567833 to 2050685 tid 2 2024-09-20T19:34:56.411 INFO:tasks.rados.rados.0.smithi042.stdout:2731: write oid 28 current snap is 318 2024-09-20T19:34:56.411 INFO:tasks.rados.rados.0.smithi042.stdout:2731: seq_num 956 ranges {467492=482169,1626315=441177} 2024-09-20T19:34:56.415 INFO:tasks.rados.rados.0.smithi042.stdout:2731: writing smithi04268048-28 from 467492 to 949661 tid 1 2024-09-20T19:34:56.422 INFO:tasks.rados.rados.0.smithi042.stdout:2731: writing smithi04268048-28 from 1626315 to 2067492 tid 2 2024-09-20T19:34:56.422 INFO:tasks.rados.rados.0.smithi042.stdout:2732: write oid 1 current snap is 318 2024-09-20T19:34:56.422 INFO:tasks.rados.rados.0.smithi042.stdout:2730: finishing write tid 1 to smithi04268048-40 2024-09-20T19:34:56.422 INFO:tasks.rados.rados.0.smithi042.stdout:2729: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:34:56.459 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3173354 2024-09-20T19:34:56.465 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 321524 2024-09-20T19:34:56.497 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.498 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.499 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:34:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:34:56.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.519 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.520 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:56 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.523 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 93202 2024-09-20T19:34:56.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:56.582 INFO:tasks.rados.rados.0.smithi042.stdout:2730: finishing write tid 2 to smithi04268048-40 2024-09-20T19:34:56.582 INFO:tasks.rados.rados.0.smithi042.stdout:2730: finishing write tid 3 to smithi04268048-40 2024-09-20T19:34:56.582 INFO:tasks.rados.rados.0.smithi042.stdout:2732: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-20T19:34:56.588 INFO:tasks.rados.rados.0.smithi042.stdout:2732: writing smithi04268048-1 from 484299 to 1041717 tid 1 2024-09-20T19:34:56.594 INFO:tasks.rados.rados.0.smithi042.stdout:2732: writing smithi04268048-1 from 1684797 to 2084299 tid 2 2024-09-20T19:34:56.594 INFO:tasks.rados.rados.0.smithi042.stdout:2729: done (3 left) 2024-09-20T19:34:56.594 INFO:tasks.rados.rados.0.smithi042.stdout:2733: write oid 2 current snap is 318 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2730: finishing write tid 4 to smithi04268048-40 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2730: oid 40 updating version 0 to 1055 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2730: oid 40 updating version 1055 to 1056 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2730: oid 40 updating version 1056 to 1057 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1057 (ObjNum 955 snap 318 seq_num 955) dirty exists 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2730: left oid 40 (ObjNum 955 snap 318 seq_num 955) 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2731: finishing write tid 1 to smithi04268048-28 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2731: finishing write tid 2 to smithi04268048-28 2024-09-20T19:34:56.595 INFO:tasks.rados.rados.0.smithi042.stdout:2733: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-20T19:34:56.601 INFO:tasks.rados.rados.0.smithi042.stdout:2733: writing smithi04268048-2 from 501106 to 1250126 tid 1 2024-09-20T19:34:56.606 INFO:tasks.rados.rados.0.smithi042.stdout:2733: writing smithi04268048-2 from 1743279 to 2101106 tid 2 2024-09-20T19:34:56.606 INFO:tasks.rados.rados.0.smithi042.stdout:2730: done (3 left) 2024-09-20T19:34:56.606 INFO:tasks.rados.rados.0.smithi042.stdout:2734: snap_remove snap 306 2024-09-20T19:34:56.620 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:34:56.620 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:34:56.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:56.634 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T19:34:56.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:56.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:56.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.940232+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:34:56.766 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:54.942422+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.455888+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.460578+0000 osd.6 (osd.6) 169 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.535780+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:55.537213+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.361336+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.f scrub starts 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.365394+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.f scrub ok 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.767 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:56 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.371342+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2024-09-20T19:34:56.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:56.883 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:56.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:57.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:57.015 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:34:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:34:57.032 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:34:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:34:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:57.051 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:34:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:34:57.067 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:34:57.068 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:34:57.087 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:34:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:34:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:34:57.106 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:34:57.127 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:34:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:34:57.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:57.149 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:34:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:34:57.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:57.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:57.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:57.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:57.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T19:34:57.520 INFO:tasks.rados.rados.0.smithi042.stdout:2734: done (3 left) 2024-09-20T19:34:57.520 INFO:tasks.rados.rados.0.smithi042.stdout:2735: delete oid 23 current snap is 318 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: finishing write tid 3 to smithi04268048-28 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: finishing write tid 4 to smithi04268048-28 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: oid 28 updating version 0 to 863 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: oid 28 version 863 is already newer than 861 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: oid 28 version 863 is already newer than 862 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 863 (ObjNum 956 snap 318 seq_num 956) dirty exists 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2731: left oid 28 (ObjNum 956 snap 318 seq_num 956) 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: finishing write tid 1 to smithi04268048-1 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: finishing write tid 2 to smithi04268048-1 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: finishing write tid 3 to smithi04268048-1 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: finishing write tid 4 to smithi04268048-1 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: oid 1 updating version 0 to 1122 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: oid 1 updating version 1122 to 1123 2024-09-20T19:34:57.521 INFO:tasks.rados.rados.0.smithi042.stdout:2732: oid 1 version 1123 is already newer than 1121 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1123 (ObjNum 957 snap 318 seq_num 957) dirty exists 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2732: left oid 1 (ObjNum 957 snap 318 seq_num 957) 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: finishing write tid 1 to smithi04268048-2 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: finishing write tid 2 to smithi04268048-2 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: finishing write tid 3 to smithi04268048-2 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: finishing write tid 4 to smithi04268048-2 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: oid 2 updating version 0 to 1296 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: oid 2 version 1296 is already newer than 1294 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: oid 2 version 1296 is already newer than 1295 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1296 (ObjNum 958 snap 318 seq_num 958) dirty exists 2024-09-20T19:34:57.522 INFO:tasks.rados.rados.0.smithi042.stdout:2733: left oid 2 (ObjNum 958 snap 318 seq_num 958) 2024-09-20T19:34:57.529 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2731: done (3 left) 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2732: done (2 left) 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2733: done (1 left) 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2735: done (0 left) 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2736: read oid 1 snap -1 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2736: expect (ObjNum 957 snap 318 seq_num 957) 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2737: rollback oid 35 current snap is 318 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 315 2024-09-20T19:34:57.536 INFO:tasks.rados.rados.0.smithi042.stdout:2738: snap_create 2024-09-20T19:34:57.539 INFO:tasks.rados.rados.0.smithi042.stdout:2737: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:34:57.539 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1065 (ObjNum 225 snap 0 seq_num 1415473232) dirty dne 2024-09-20T19:34:57.542 INFO:tasks.rados.rados.0.smithi042.stdout:2736: expect (ObjNum 957 snap 318 seq_num 957) 2024-09-20T19:34:57.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:57 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:57.657 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:56.383569+0000 mgr.y (mgr.24491) 637 : cluster [DBG] pgmap v1180: 178 pgs: 4 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 231 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.499568+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.499795+0000 mon.a (mon.0) 2313 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: audit 2024-09-20T19:34:56.995570+0000 mon.a (mon.0) 2314 : audit [DBG] from='client.? v1:172.21.15.42:0/2342326083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.134658+0000 osd.6 (osd.6) 170 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.135892+0000 osd.6 (osd.6) 171 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.366879+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.a scrub starts 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.672 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:57 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.369100+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.a scrub ok 2024-09-20T19:34:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:57.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:57.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:34:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:34:57.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:57.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:57.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:58.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:34:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:34:58.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:58.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:58.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:58.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:58.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:58.514 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[51183]: logger=sqlstore.transactions t=2024-09-20T19:34:58.207335788Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T19:34:58.526 INFO:tasks.rados.rados.0.smithi042.stdout:2736: done (2 left) 2024-09-20T19:34:58.526 INFO:tasks.rados.rados.0.smithi042.stdout:2737: done (1 left) 2024-09-20T19:34:58.526 INFO:tasks.rados.rados.0.smithi042.stdout:2738: done (0 left) 2024-09-20T19:34:58.526 INFO:tasks.rados.rados.0.smithi042.stdout:2739: delete oid 12 current snap is 319 2024-09-20T19:34:58.532 INFO:tasks.rados.rados.0.smithi042.stdout:2739: done (0 left) 2024-09-20T19:34:58.533 INFO:tasks.rados.rados.0.smithi042.stdout:2740: write oid 17 current snap is 319 2024-09-20T19:34:58.533 INFO:tasks.rados.rados.0.smithi042.stdout:2740: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-20T19:34:58.539 INFO:tasks.rados.rados.0.smithi042.stdout:2740: writing smithi04268048-17 from 551527 to 1126294 tid 1 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2740: writing smithi04268048-17 from 1918725 to 2151527 tid 2 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2741: read oid 24 snap -1 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2741: expect deleted 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2742: read oid 34 snap -1 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2742: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2743: write (excl) oid 48 current snap is 319 2024-09-20T19:34:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:2743: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-20T19:34:58.552 INFO:tasks.rados.rados.0.smithi042.stdout:2743: writing smithi04268048-48 from 568334 to 1218350 tid 1 2024-09-20T19:34:58.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:58.558 INFO:tasks.rados.rados.0.smithi042.stdout:2743: writing smithi04268048-48 from 1860854 to 2168334 tid 2 2024-09-20T19:34:58.558 INFO:tasks.rados.rados.0.smithi042.stdout:2744: rollback oid 39 current snap is 319 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 295 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: finishing write tid 1 to smithi04268048-17 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: finishing write tid 2 to smithi04268048-17 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: finishing write tid 3 to smithi04268048-17 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2745: delete oid 19 current snap is 319 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: finishing write tid 4 to smithi04268048-17 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: oid 17 updating version 0 to 1095 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: oid 17 version 1095 is already newer than 1093 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: oid 17 version 1095 is already newer than 1094 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1095 (ObjNum 961 snap 319 seq_num 961) dirty exists 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2740: left oid 17 (ObjNum 961 snap 319 seq_num 961) 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:2742: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:34:58.559 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:34:58.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:34:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:34:58.598 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1904667 2024-09-20T19:34:58.633 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2331720 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: finishing write tid 1 to smithi04268048-48 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: finishing write tid 2 to smithi04268048-48 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: finishing write tid 3 to smithi04268048-48 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: finishing write tid 4 to smithi04268048-48 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: oid 48 updating version 0 to 936 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: oid 48 updating version 936 to 937 2024-09-20T19:34:58.644 INFO:tasks.rados.rados.0.smithi042.stdout:2743: oid 48 version 937 is already newer than 935 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 937 (ObjNum 962 snap 319 seq_num 962) dirty exists 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2743: left oid 48 (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2744: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2744: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2744: finishing rollback tid 2 to smithi04268048-39 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 737 (ObjNum 853 snap 281 seq_num 853) dirty exists 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2740: done (5 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2741: done (4 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2742: done (3 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2743: done (2 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2744: done (1 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2745: done (0 left) 2024-09-20T19:34:58.645 INFO:tasks.rados.rados.0.smithi042.stdout:2746: snap_create 2024-09-20T19:34:58.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:34:58.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[22706]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.676 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:58 smithi042 bash[32075]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:58.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.463919+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.465377+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:57.509503+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 1]}]': finished 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:57.509716+0000 mon.a (mon.0) 2316 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: cluster 2024-09-20T19:34:57.517833+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:58.214402+0000 mon.a (mon.0) 2318 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:34:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:58 smithi162 bash[24557]: audit 2024-09-20T19:34:58.216599+0000 mon.a (mon.0) 2319 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:34:58.813 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:58.940 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:58.961 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:34:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:58.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:34:59.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:59.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:59.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:59.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:59.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:34:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:34:59.452 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:2746: done (0 left) 2024-09-20T19:34:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:2747: copy_from oid 15 from oid 48 current snap is 320 2024-09-20T19:34:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:2748: delete oid 1 current snap is 320 2024-09-20T19:34:59.555 INFO:tasks.rados.rados.0.smithi042.stdout:2748: done (1 left) 2024-09-20T19:34:59.555 INFO:tasks.rados.rados.0.smithi042.stdout:2749: read oid 4 snap -1 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2749: expect (ObjNum 884 snap 291 seq_num 884) 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2750: copy_from oid 9 from oid 14 current snap is 320 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2751: copy_from oid 39 from oid 24 current snap is 320 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2752: rollback oid 45 current snap is 320 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 310 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2753: write (excl) oid 42 current snap is 320 2024-09-20T19:34:59.556 INFO:tasks.rados.rados.0.smithi042.stdout:2753: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-20T19:34:59.571 INFO:tasks.rados.rados.0.smithi042.stdout:2753: writing smithi04268048-42 from 618755 to 1094518 tid 1 2024-09-20T19:34:59.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:59.589 INFO:tasks.rados.rados.0.smithi042.stdout:2753: writing smithi04268048-42 from 1636300 to 2094296 tid 2 2024-09-20T19:34:59.589 INFO:tasks.rados.rados.0.smithi042.stdout:2754: write (excl) oid 12 current snap is 320 2024-09-20T19:34:59.589 INFO:tasks.rados.rados.0.smithi042.stdout:2754: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-20T19:34:59.603 INFO:tasks.rados.rados.0.smithi042.stdout:2754: writing smithi04268048-12 from 635562 to 1302927 tid 1 2024-09-20T19:34:59.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:34:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2754: writing smithi04268048-12 from 2094782 to 2235562 tid 2 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2755: read oid 49 snap -1 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2755: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2756: write oid 44 current snap is 320 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2752: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1310 (ObjNum 49 snap 0 seq_num 1415952944) dirty dne 2024-09-20T19:34:59.611 INFO:tasks.rados.rados.0.smithi042.stdout:2756: seq_num 967 ranges {652369=742614,2153264=99105} 2024-09-20T19:34:59.621 INFO:tasks.rados.rados.0.smithi042.stdout:2756: writing smithi04268048-44 from 652369 to 1394983 tid 1 2024-09-20T19:34:59.627 INFO:tasks.rados.rados.0.smithi042.stdout:2756: writing smithi04268048-44 from 2153264 to 2252369 tid 2 2024-09-20T19:34:59.627 INFO:tasks.rados.rados.0.smithi042.stdout:2752: done (8 left) 2024-09-20T19:34:59.627 INFO:tasks.rados.rados.0.smithi042.stdout:2757: copy_from oid 40 from oid 8 current snap is 320 2024-09-20T19:34:59.627 INFO:tasks.rados.rados.0.smithi042.stdout:2749: expect (ObjNum 884 snap 291 seq_num 884) 2024-09-20T19:34:59.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.724 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.736 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:34:59.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:34:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:34:59.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:59.856 INFO:tasks.rados.rados.0.smithi042.stdout:2747: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2747: finishing copy_from to smithi04268048-15 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 937 (ObjNum 962 snap 319 seq_num 962) dirty exists 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2751: finishing copy_from to smithi04268048-39 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2751: got expected ENOENT (src dne) 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2751: finishing copy_from racing read to smithi04268048-39 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2753: finishing write tid 1 to smithi04268048-42 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2750: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2750: finishing copy_from to smithi04268048-9 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 827 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2753: finishing write tid 2 to smithi04268048-42 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2753: finishing write tid 3 to smithi04268048-42 2024-09-20T19:34:59.857 INFO:tasks.rados.rados.0.smithi042.stdout:2753: finishing write tid 4 to smithi04268048-42 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2753: oid 42 updating version 0 to 1145 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2753: oid 42 version 1145 is already newer than 1143 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2753: oid 42 version 1145 is already newer than 1144 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1145 (ObjNum 965 snap 320 seq_num 965) dirty exists 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2753: left oid 42 (ObjNum 965 snap 320 seq_num 965) 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: finishing write tid 1 to smithi04268048-12 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: finishing write tid 2 to smithi04268048-12 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: finishing write tid 3 to smithi04268048-12 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: finishing write tid 4 to smithi04268048-12 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: oid 12 updating version 0 to 738 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: oid 12 version 738 is already newer than 736 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: oid 12 version 738 is already newer than 737 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 738 (ObjNum 966 snap 320 seq_num 966) dirty exists 2024-09-20T19:34:59.858 INFO:tasks.rados.rados.0.smithi042.stdout:2754: left oid 12 (ObjNum 966 snap 320 seq_num 966) 2024-09-20T19:34:59.859 INFO:tasks.rados.rados.0.smithi042.stdout:2755: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:34:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:34:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.384227+0000 mgr.y (mgr.24491) 638 : cluster [DBG] pgmap v1182: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 249 MiB data, 926 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.524029+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.622329+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:58.623632+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.379405+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.381719+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.426110+0000 osd.2 (osd.2) 247 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:34:59 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.428788+0000 osd.2 (osd.2) 248 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:34:59.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:34:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:59.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: finishing write tid 1 to smithi04268048-44 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: finishing write tid 2 to smithi04268048-44 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: finishing write tid 3 to smithi04268048-44 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: finishing write tid 4 to smithi04268048-44 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: oid 44 updating version 0 to 1144 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: oid 44 version 1144 is already newer than 1142 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: oid 44 version 1144 is already newer than 1143 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1144 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2756: left oid 44 (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2757: finishing copy_from racing read to smithi04268048-40 2024-09-20T19:35:00.056 INFO:tasks.rados.rados.0.smithi042.stdout:2757: finishing copy_from to smithi04268048-40 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2747: done (8 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2749: done (7 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2750: done (6 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2751: done (5 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2753: done (4 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2754: done (3 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2755: done (2 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2756: done (1 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2757: done (0 left) 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2758: read oid 1 snap -1 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2758: expect deleted 2024-09-20T19:35:00.057 INFO:tasks.rados.rados.0.smithi042.stdout:2759: delete oid 45 current snap is 320 2024-09-20T19:35:00.058 INFO:tasks.rados.rados.0.smithi042.stdout:2758: done (1 left) 2024-09-20T19:35:00.059 INFO:tasks.rados.rados.0.smithi042.stdout:2759: done (0 left) 2024-09-20T19:35:00.059 INFO:tasks.rados.rados.0.smithi042.stdout:2760: read oid 47 snap 309 2024-09-20T19:35:00.059 INFO:tasks.rados.rados.0.smithi042.stdout:2760: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:35:00.059 INFO:tasks.rados.rados.0.smithi042.stdout:2761: snap_remove snap 315 2024-09-20T19:35:00.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:00.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:00.250 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:00.353 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:00.380 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:00.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:35:00.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:00.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:00.547 INFO:tasks.rados.rados.0.smithi042.stdout:2761: done (1 left) 2024-09-20T19:35:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2762: read oid 21 snap -1 2024-09-20T19:35:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2762: expect (ObjNum 881 snap 291 seq_num 881) 2024-09-20T19:35:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2763: read oid 1 snap -1 2024-09-20T19:35:00.548 INFO:tasks.rados.rados.0.smithi042.stdout:2763: expect deleted 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2764: read oid 38 snap -1 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2764: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2765: read oid 28 snap 301 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2765: expect (ObjNum 882 snap 291 seq_num 882) 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2766: snap_create 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:2760: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:35:00.549 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:00.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.623 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:00.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:00.748 INFO:tasks.rados.rados.0.smithi042.stdout:2764: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:35:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:00.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.535997+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.641434+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:34:59.643695+0000 osd.4 (osd.4) 100 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.252860+0000 osd.1 (osd.1) 279 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.253611+0000 osd.1 (osd.1) 280 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:00.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:00.881 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 599525 2024-09-20T19:35:00.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:00.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:00.944 INFO:tasks.rados.rados.0.smithi042.stdout:2762: expect (ObjNum 881 snap 291 seq_num 881) 2024-09-20T19:35:00.970 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:00.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:01.006 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1428212 2024-09-20T19:35:01.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:01.020 INFO:tasks.rados.rados.0.smithi042.stdout:2765: expect (ObjNum 882 snap 291 seq_num 882) 2024-09-20T19:35:01.020 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:01.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:01.157 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:01.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:01.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:01.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:01.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:01.565 INFO:tasks.rados.rados.0.smithi042.stdout:2760: done (5 left) 2024-09-20T19:35:01.565 INFO:tasks.rados.rados.0.smithi042.stdout:2762: done (4 left) 2024-09-20T19:35:01.565 INFO:tasks.rados.rados.0.smithi042.stdout:2763: done (3 left) 2024-09-20T19:35:01.565 INFO:tasks.rados.rados.0.smithi042.stdout:2764: done (2 left) 2024-09-20T19:35:01.566 INFO:tasks.rados.rados.0.smithi042.stdout:2765: done (1 left) 2024-09-20T19:35:01.566 INFO:tasks.rados.rados.0.smithi042.stdout:2766: done (0 left) 2024-09-20T19:35:01.566 INFO:tasks.rados.rados.0.smithi042.stdout:2767: copy_from oid 17 from oid 10 current snap is 321 2024-09-20T19:35:01.566 INFO:tasks.rados.rados.0.smithi042.stdout:2768: snap_remove snap 317 2024-09-20T19:35:01.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[22706]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:01 smithi042 bash[32075]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.765 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:01.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: audit 2024-09-20T19:34:59.733929+0000 mgr.y (mgr.24491) 639 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:01.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.384827+0000 mgr.y (mgr.24491) 640 : cluster [DBG] pgmap v1185: 178 pgs: 2 peering, 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 250 MiB data, 928 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.416387+0000 osd.6 (osd.6) 174 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.419205+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:01 smithi162 bash[24557]: cluster 2024-09-20T19:35:00.543586+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2024-09-20T19:35:01.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:01.919 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:01.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:02.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:02.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:02.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:02.282 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:02.311 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:02.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:02.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2768: done (1 left) 2024-09-20T19:35:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2769: write (excl) oid 32 current snap is 321 2024-09-20T19:35:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2767: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:35:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:2767: finishing copy_from to smithi04268048-17 2024-09-20T19:35:02.576 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 844 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:35:02.577 INFO:tasks.rados.rados.0.smithi042.stdout:2769: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-20T19:35:02.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:02.586 INFO:tasks.rados.rados.0.smithi042.stdout:2769: writing smithi04268048-32 from 685983 to 1179095 tid 1 2024-09-20T19:35:02.595 INFO:tasks.rados.rados.0.smithi042.stdout:2769: writing smithi04268048-32 from 1870228 to 2285983 tid 2 2024-09-20T19:35:02.595 INFO:tasks.rados.rados.0.smithi042.stdout:2767: done (1 left) 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2770: copy_from oid 16 from oid 44 current snap is 321 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2771: copy_from oid 42 from oid 48 current snap is 321 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2772: read oid 30 snap -1 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2772: expect (ObjNum 936 snap 313 seq_num 936) 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2773: write (excl) oid 18 current snap is 321 2024-09-20T19:35:02.596 INFO:tasks.rados.rados.0.smithi042.stdout:2773: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-20T19:35:02.605 INFO:tasks.rados.rados.0.smithi042.stdout:2773: writing smithi04268048-18 from 702790 to 1271151 tid 1 2024-09-20T19:35:02.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: writing smithi04268048-18 from 1812357 to 2302790 tid 2 2024-09-20T19:35:02.614 INFO:tasks.rados.rados.0.smithi042.stdout:2774: snap_remove snap 302 2024-09-20T19:35:02.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:02 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:02.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:01.558057+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.412474+0000 osd.1 (osd.1) 281 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:02 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.415419+0000 osd.1 (osd.1) 282 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:35:02.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:02.954 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:02.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:02.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:03.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:03.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:03.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2774: done (5 left) 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2775: read oid 28 snap -1 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2775: expect (ObjNum 956 snap 318 seq_num 956) 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2776: rollback oid 9 current snap is 321 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 319 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2777: delete oid 4 current snap is 321 2024-09-20T19:35:03.382 INFO:tasks.rados.rados.0.smithi042.stdout:2772: expect (ObjNum 936 snap 313 seq_num 936) 2024-09-20T19:35:03.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:03.465 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:03.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:03.539 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:03.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:03.588 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:03 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:03.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:03.613 INFO:tasks.rados.rados.0.smithi042.stdout:2773: finishing write tid 1 to smithi04268048-18 2024-09-20T19:35:03.613 INFO:tasks.rados.rados.0.smithi042.stdout:2769: finishing write tid 1 to smithi04268048-32 2024-09-20T19:35:03.613 INFO:tasks.rados.rados.0.smithi042.stdout:2771: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: finishing write tid 2 to smithi04268048-18 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: finishing write tid 2 to smithi04268048-32 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: finishing write tid 3 to smithi04268048-18 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: finishing write tid 3 to smithi04268048-32 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: finishing write tid 4 to smithi04268048-32 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: oid 32 updating version 0 to 1336 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: oid 32 updating version 1336 to 1338 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: oid 32 version 1338 is already newer than 1337 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1338 (ObjNum 969 snap 321 seq_num 969) dirty exists 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2769: left oid 32 (ObjNum 969 snap 321 seq_num 969) 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: finishing write tid 4 to smithi04268048-18 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: oid 18 updating version 0 to 1136 2024-09-20T19:35:03.614 INFO:tasks.rados.rados.0.smithi042.stdout:2773: oid 18 version 1136 is already newer than 1135 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2773: oid 18 updating version 1136 to 1137 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1137 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2773: left oid 18 (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2769: done (7 left) 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2772: done (6 left) 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2773: done (5 left) 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2777: done (4 left) 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2778: write (excl) oid 31 current snap is 321 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2770: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2771: finishing copy_from to smithi04268048-42 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 937 (ObjNum 962 snap 319 seq_num 962) dirty exists 2024-09-20T19:35:03.615 INFO:tasks.rados.rados.0.smithi042.stdout:2770: finishing copy_from to smithi04268048-16 2024-09-20T19:35:03.616 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1144 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:03.616 INFO:tasks.rados.rados.0.smithi042.stdout:2775: expect (ObjNum 956 snap 318 seq_num 956) 2024-09-20T19:35:03.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:03.711 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:03.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:03.763 INFO:tasks.rados.rados.0.smithi042.stdout:2776: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:35:03.763 INFO:tasks.rados.rados.0.smithi042.stdout:2776: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:35:03.763 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 982 (ObjNum 129 snap 0 seq_num 1415228176) dirty dne 2024-09-20T19:35:03.763 INFO:tasks.rados.rados.0.smithi042.stdout:2778: seq_num 972 ranges {736404=718859,1929321=407083} 2024-09-20T19:35:03.783 INFO:tasks.rados.rados.0.smithi042.stdout:2778: writing smithi04268048-31 from 736404 to 1455263 tid 1 2024-09-20T19:35:03.794 INFO:tasks.rados.rados.0.smithi042.stdout:2778: writing smithi04268048-31 from 1929321 to 2336404 tid 2 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2770: done (4 left) 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2771: done (3 left) 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2775: done (2 left) 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2776: done (1 left) 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2779: write (excl) oid 42 current snap is 321 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2778: finishing write tid 1 to smithi04268048-31 2024-09-20T19:35:03.795 INFO:tasks.rados.rados.0.smithi042.stdout:2779: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-20T19:35:03.811 INFO:tasks.rados.rados.0.smithi042.stdout:2779: writing smithi04268048-42 from 753211 to 1547319 tid 1 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2779: writing smithi04268048-42 from 1987803 to 2353211 tid 2 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2780: copy_from oid 21 from oid 36 current snap is 321 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: finishing write tid 2 to smithi04268048-31 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: finishing write tid 3 to smithi04268048-31 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: finishing write tid 4 to smithi04268048-31 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: oid 31 updating version 0 to 865 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: oid 31 updating version 865 to 866 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:2778: oid 31 version 866 is already newer than 864 2024-09-20T19:35:03.817 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 866 (ObjNum 972 snap 321 seq_num 972) dirty exists 2024-09-20T19:35:03.818 INFO:tasks.rados.rados.0.smithi042.stdout:2778: left oid 31 (ObjNum 972 snap 321 seq_num 972) 2024-09-20T19:35:03.818 INFO:tasks.rados.rados.0.smithi042.stdout:2778: done (2 left) 2024-09-20T19:35:03.818 INFO:tasks.rados.rados.0.smithi042.stdout:2781: delete oid 14 current snap is 321 2024-09-20T19:35:03.820 INFO:tasks.rados.rados.0.smithi042.stdout:2781: done (2 left) 2024-09-20T19:35:03.820 INFO:tasks.rados.rados.0.smithi042.stdout:2782: snap_create 2024-09-20T19:35:03.820 INFO:tasks.rados.rados.0.smithi042.stdout:2779: finishing write tid 1 to smithi04268048-42 2024-09-20T19:35:03.821 INFO:tasks.rados.rados.0.smithi042.stdout:2779: finishing write tid 2 to smithi04268048-42 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: finishing write tid 3 to smithi04268048-42 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: finishing write tid 4 to smithi04268048-42 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: oid 42 updating version 0 to 1154 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: oid 42 updating version 1154 to 1155 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: oid 42 updating version 1155 to 1156 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1156 (ObjNum 973 snap 321 seq_num 973) dirty exists 2024-09-20T19:35:03.822 INFO:tasks.rados.rados.0.smithi042.stdout:2779: left oid 42 (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:03.826 INFO:tasks.rados.rados.0.smithi042.stdout:2780: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:35:03.828 INFO:tasks.rados.rados.0.smithi042.stdout:2780: finishing copy_from to smithi04268048-21 2024-09-20T19:35:03.829 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 584 (ObjNum 838 snap 276 seq_num 838) dirty exists 2024-09-20T19:35:03.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:03.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:03.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:03.960 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:35:03.960 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:35:03.960 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:35:03.960 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:35:03.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:03.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:04.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.386482+0000 mgr.y (mgr.24491) 641 : cluster [DBG] pgmap v1188: 178 pgs: 2 peering, 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 168 active+clean; 260 MiB data, 959 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:02.569843+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:03 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.369549+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2024-09-20T19:35:04.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:04.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:04.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:04.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:04.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:04.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:04.578 INFO:tasks.rados.rados.0.smithi042.stdout:2779: done (2 left) 2024-09-20T19:35:04.579 INFO:tasks.rados.rados.0.smithi042.stdout:2780: done (1 left) 2024-09-20T19:35:04.579 INFO:tasks.rados.rados.0.smithi042.stdout:2782: done (0 left) 2024-09-20T19:35:04.579 INFO:tasks.rados.rados.0.smithi042.stdout:2783: delete oid 24 current snap is 322 2024-09-20T19:35:04.592 INFO:tasks.rados.rados.0.smithi042.stdout:2783: done (0 left) 2024-09-20T19:35:04.592 INFO:tasks.rados.rados.0.smithi042.stdout:2784: read oid 17 snap -1 2024-09-20T19:35:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:2784: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:2785: read oid 11 snap 316 2024-09-20T19:35:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:2785: expect (ObjNum 937 snap 314 seq_num 937) 2024-09-20T19:35:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:2786: copy_from oid 5 from oid 30 current snap is 322 2024-09-20T19:35:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:2787: snap_create 2024-09-20T19:35:04.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:04.618 INFO:tasks.rados.rados.0.smithi042.stdout:2785: expect (ObjNum 937 snap 314 seq_num 937) 2024-09-20T19:35:04.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:04.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:04.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T19:35:04.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:04.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:04.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:04.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:04 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:04.883 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:04.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:04.922 INFO:tasks.rados.rados.0.smithi042.stdout:2784: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:04.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:04.990 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:04.992 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:05.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:04 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:05.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:04 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.418588+0000 osd.6 (osd.6) 176 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:35:05.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:04 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:05.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:04 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.419624+0000 osd.6 (osd.6) 177 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:35:05.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:05.128 INFO:tasks.rados.rados.0.smithi042.stdout:2786: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:35:05.128 INFO:tasks.rados.rados.0.smithi042.stdout:2786: finishing copy_from to smithi04268048-5 2024-09-20T19:35:05.128 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1121 (ObjNum 936 snap 313 seq_num 936) dirty exists 2024-09-20T19:35:05.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:05.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.268 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:05.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:05.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:05.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T19:35:05.581 INFO:tasks.rados.rados.0.smithi042.stdout:2784: done (3 left) 2024-09-20T19:35:05.581 INFO:tasks.rados.rados.0.smithi042.stdout:2785: done (2 left) 2024-09-20T19:35:05.581 INFO:tasks.rados.rados.0.smithi042.stdout:2786: done (1 left) 2024-09-20T19:35:05.582 INFO:tasks.rados.rados.0.smithi042.stdout:2787: done (0 left) 2024-09-20T19:35:05.582 INFO:tasks.rados.rados.0.smithi042.stdout:2788: delete oid 49 current snap is 323 2024-09-20T19:35:05.590 INFO:tasks.rados.rados.0.smithi042.stdout:2788: done (0 left) 2024-09-20T19:35:05.590 INFO:tasks.rados.rados.0.smithi042.stdout:2789: snap_remove snap 321 2024-09-20T19:35:05.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.643 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:05 smithi162 bash[24557]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.657 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:05.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:05.787 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:05.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:05.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:05.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:05.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[22706]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.929008+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:03.930285+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.387276+0000 mgr.y (mgr.24491) 642 : cluster [DBG] pgmap v1191: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 25 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.498714+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.500307+0000 osd.1 (osd.1) 284 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.577628+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.622572+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: cluster 2024-09-20T19:35:04.624367+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: audit 2024-09-20T19:35:05.511863+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.42:0/2515658496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:05.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:05 smithi042 bash[32075]: audit 2024-09-20T19:35:05.512299+0000 mon.a (mon.0) 2327 : audit [INF] from='client.25805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:35:06.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:06.047 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:06.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:06.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:06.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:06.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:06.436 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:06.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:06.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:2789: done (0 left) 2024-09-20T19:35:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:2790: snap_create 2024-09-20T19:35:06.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:06.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:06 smithi162 bash[24557]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:06.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:06.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[32075]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: audit 2024-09-20T19:35:05.572972+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.579517+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.732461+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:06 smithi042 bash[22706]: cluster 2024-09-20T19:35:05.739931+0000 osd.0 (osd.0) 68 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:35:06.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:06.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:07.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:07.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:07.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:07.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:07.550 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:07.597 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:35:07.607 INFO:tasks.rados.rados.0.smithi042.stdout:2790: done (0 left) 2024-09-20T19:35:07.607 INFO:tasks.rados.rados.0.smithi042.stdout:2791: write oid 39 current snap is 324 2024-09-20T19:35:07.607 INFO:tasks.rados.rados.0.smithi042.stdout:2791: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-20T19:35:07.612 INFO:tasks.rados.rados.0.smithi042.stdout:2791: writing smithi04268048-39 from 420439 to 831896 tid 1 2024-09-20T19:35:07.619 INFO:tasks.rados.rados.0.smithi042.stdout:2791: writing smithi04268048-39 from 1421731 to 1901741 tid 2 2024-09-20T19:35:07.619 INFO:tasks.rados.rados.0.smithi042.stdout:2792: write (excl) oid 31 current snap is 324 2024-09-20T19:35:07.619 INFO:tasks.rados.rados.0.smithi042.stdout:2792: seq_num 978 ranges {437246=486706,1363860=623668} 2024-09-20T19:35:07.624 INFO:tasks.rados.rados.0.smithi042.stdout:2792: writing smithi04268048-31 from 437246 to 923952 tid 1 2024-09-20T19:35:07.631 INFO:tasks.rados.rados.0.smithi042.stdout:2792: writing smithi04268048-31 from 1363860 to 1987528 tid 2 2024-09-20T19:35:07.631 INFO:tasks.rados.rados.0.smithi042.stdout:2793: read oid 21 snap -1 2024-09-20T19:35:07.631 INFO:tasks.rados.rados.0.smithi042.stdout:2793: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2794: read oid 10 snap -1 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2794: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2795: snap_create 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: finishing write tid 1 to smithi04268048-39 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: finishing write tid 2 to smithi04268048-39 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: finishing write tid 3 to smithi04268048-39 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: finishing write tid 4 to smithi04268048-39 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: oid 39 updating version 0 to 738 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: oid 39 updating version 738 to 740 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: oid 39 version 740 is already newer than 739 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 740 (ObjNum 977 snap 324 seq_num 977) dirty exists 2024-09-20T19:35:07.632 INFO:tasks.rados.rados.0.smithi042.stdout:2791: left oid 39 (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:35:07.634 INFO:tasks.rados.rados.0.smithi042.stdout:2793: expect (ObjNum 838 snap 276 seq_num 838) 2024-09-20T19:35:07.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.662 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1498185 2024-09-20T19:35:07.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:07.766 INFO:tasks.rados.rados.0.smithi042.stdout:2794: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:07.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:07.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:07 smithi162 bash[24557]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:07.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:07.832 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:07.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:07.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[32075]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.388065+0000 mgr.y (mgr.24491) 643 : cluster [DBG] pgmap v1194: 178 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 170 active+clean; 265 MiB data, 967 MiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 637 KiB/s wr, 8 op/s; 27 B/s, 0 objects/s recovering 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.596798+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.708598+0000 osd.0 (osd.0) 69 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: cluster 2024-09-20T19:35:06.711003+0000 osd.0 (osd.0) 70 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:07 smithi042 bash[22706]: audit 2024-09-20T19:35:07.526277+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:07.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:07.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:07.971 INFO:tasks.rados.rados.0.smithi042.stdout:2792: finishing write tid 1 to smithi04268048-31 2024-09-20T19:35:07.971 INFO:tasks.rados.rados.0.smithi042.stdout:2792: finishing write tid 2 to smithi04268048-31 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: finishing write tid 3 to smithi04268048-31 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: finishing write tid 4 to smithi04268048-31 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: oid 31 updating version 0 to 868 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: oid 31 updating version 868 to 869 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: oid 31 updating version 869 to 870 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 870 (ObjNum 978 snap 324 seq_num 978) dirty exists 2024-09-20T19:35:07.972 INFO:tasks.rados.rados.0.smithi042.stdout:2792: left oid 31 (ObjNum 978 snap 324 seq_num 978) 2024-09-20T19:35:08.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:08.161 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T19:35:08.162 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:35:08.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.198 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:08.301 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:08.327 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:08.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:08.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:08.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:08.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:08.620 INFO:tasks.rados.rados.0.smithi042.stdout:2791: done (4 left) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2792: done (3 left) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2793: done (2 left) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2794: done (1 left) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2795: done (0 left) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2796: copy_from oid 30 from oid 11 current snap is 325 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2797: rollback oid 49 current snap is 325 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 49 to 312 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2798: read oid 33 snap -1 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2798: expect (ObjNum 947 snap 314 seq_num 947) 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2799: write (excl) oid 31 current snap is 325 2024-09-20T19:35:08.621 INFO:tasks.rados.rados.0.smithi042.stdout:2799: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-20T19:35:08.637 INFO:tasks.rados.rados.0.smithi042.stdout:2799: writing smithi04268048-31 from 454053 to 1016008 tid 1 2024-09-20T19:35:08.646 INFO:tasks.rados.rados.0.smithi042.stdout:2799: writing smithi04268048-31 from 1422342 to 1906021 tid 2 2024-09-20T19:35:08.653 INFO:tasks.rados.rados.0.smithi042.stdout:2799: writing smithi04268048-31 from 2368404 to 2454053 tid 3 2024-09-20T19:35:08.653 INFO:tasks.rados.rados.0.smithi042.stdout:2800: copy_from oid 20 from oid 43 current snap is 325 2024-09-20T19:35:08.653 INFO:tasks.rados.rados.0.smithi042.stdout:2801: delete oid 37 current snap is 325 2024-09-20T19:35:08.653 INFO:tasks.rados.rados.0.smithi042.stdout:2798: expect (ObjNum 947 snap 314 seq_num 947) 2024-09-20T19:35:08.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.704 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:08 smithi162 bash[24557]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:08.812 INFO:tasks.rados.rados.0.smithi042.stdout:2797: finishing rollback tid 1 to smithi04268048-49 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2797: finishing rollback tid 2 to smithi04268048-49 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1346 (ObjNum 927 snap 309 seq_num 927) dirty exists 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: finishing write tid 1 to smithi04268048-31 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2800: finishing copy_from to smithi04268048-20 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2800: got expected ENOENT (src dne) 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2800: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: finishing write tid 2 to smithi04268048-31 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: finishing write tid 3 to smithi04268048-31 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: finishing write tid 4 to smithi04268048-31 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: finishing write tid 5 to smithi04268048-31 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: oid 31 updating version 0 to 872 2024-09-20T19:35:08.813 INFO:tasks.rados.rados.0.smithi042.stdout:2799: oid 31 updating version 872 to 875 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2799: oid 31 version 875 is already newer than 873 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2799: oid 31 version 875 is already newer than 874 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 875 (ObjNum 979 snap 325 seq_num 979) dirty exists 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2799: left oid 31 (ObjNum 979 snap 325 seq_num 979) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2796: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2796: finishing copy_from to smithi04268048-30 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 972 (ObjNum 937 snap 314 seq_num 937) dirty exists 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2796: done (5 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2797: done (4 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2798: done (3 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2799: done (2 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2800: done (1 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2801: done (0 left) 2024-09-20T19:35:08.814 INFO:tasks.rados.rados.0.smithi042.stdout:2802: read oid 47 snap -1 2024-09-20T19:35:08.815 INFO:tasks.rados.rados.0.smithi042.stdout:2802: expect deleted 2024-09-20T19:35:08.815 INFO:tasks.rados.rados.0.smithi042.stdout:2803: write (excl) oid 28 current snap is 325 2024-09-20T19:35:08.815 INFO:tasks.rados.rados.0.smithi042.stdout:2803: seq_num 981 ranges {487667=428806,1655659=770995} 2024-09-20T19:35:08.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:08.824 INFO:tasks.rados.rados.0.smithi042.stdout:2803: writing smithi04268048-28 from 487667 to 916473 tid 1 2024-09-20T19:35:08.840 INFO:tasks.rados.rados.0.smithi042.stdout:2803: writing smithi04268048-28 from 1655659 to 2426654 tid 2 2024-09-20T19:35:08.840 INFO:tasks.rados.rados.0.smithi042.stdout:2804: snap_remove snap 320 2024-09-20T19:35:08.845 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[22706]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.475150+0000 osd.2 (osd.2) 249 : cluster [DBG] 7.a scrub starts 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.477251+0000 osd.2 (osd.2) 250 : cluster [DBG] 7.a scrub ok 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.562627+0000 osd.4 (osd.4) 103 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.564186+0000 osd.4 (osd.4) 104 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.593399+0000 mon.a (mon.0) 2332 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: audit 2024-09-20T19:35:07.595371+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.42:0/1365792945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:08 smithi042 bash[32075]: cluster 2024-09-20T19:35:07.605595+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2024-09-20T19:35:08.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:08.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:09.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:09.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:09.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:09.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:09.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:09.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:09.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:09.495 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:35:09.495 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:35:09.514 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:35:09.514 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:35:09.522 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:09.530 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:35:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:35:09.547 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:35:09.547 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:35:09.565 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:35:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:35:09.584 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:35:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:35:09.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:09.603 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:35:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:35:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:09.625 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:35:09.625 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:35:09.637 INFO:tasks.rados.rados.0.smithi042.stdout:2804: done (2 left) 2024-09-20T19:35:09.637 INFO:tasks.rados.rados.0.smithi042.stdout:2805: snap_remove snap 301 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[32075]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:09 smithi042 bash[22706]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:09.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:09.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:09.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: cluster 2024-09-20T19:35:08.388662+0000 mgr.y (mgr.24491) 644 : cluster [DBG] pgmap v1197: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 278 MiB data, 999 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T19:35:09.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: cluster 2024-09-20T19:35:08.616833+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2024-09-20T19:35:09.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[24557]: audit 2024-09-20T19:35:09.474408+0000 mon.a (mon.0) 2336 : audit [DBG] from='client.? v1:172.21.15.42:0/2935161592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:09.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:35:09.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:09.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:09.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.001 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:10.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:10.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:10.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:10.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:10.361 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:10.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:10.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:10.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2805: done (2 left) 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2806: read oid 41 snap -1 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2806: expect deleted 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2807: delete oid 10 current snap is 325 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2803: finishing write tid 1 to smithi04268048-28 2024-09-20T19:35:10.649 INFO:tasks.rados.rados.0.smithi042.stdout:2803: finishing write tid 2 to smithi04268048-28 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: finishing write tid 3 to smithi04268048-28 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: finishing write tid 4 to smithi04268048-28 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: oid 28 updating version 0 to 866 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: oid 28 version 866 is already newer than 865 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: oid 28 updating version 866 to 867 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 867 (ObjNum 981 snap 325 seq_num 981) dirty exists 2024-09-20T19:35:10.650 INFO:tasks.rados.rados.0.smithi042.stdout:2803: left oid 28 (ObjNum 981 snap 325 seq_num 981) 2024-09-20T19:35:10.655 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2802: done (3 left) 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2803: done (2 left) 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2806: done (1 left) 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2807: done (0 left) 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2808: read oid 46 snap -1 2024-09-20T19:35:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:2808: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:2809: read oid 4 snap -1 2024-09-20T19:35:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:2809: expect deleted 2024-09-20T19:35:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:2810: copy_from oid 5 from oid 39 current snap is 325 2024-09-20T19:35:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:2811: write (excl) oid 3 current snap is 325 2024-09-20T19:35:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:2811: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-20T19:35:10.669 INFO:tasks.rados.rados.0.smithi042.stdout:2811: writing smithi04268048-3 from 521281 to 1100585 tid 1 2024-09-20T19:35:10.678 INFO:tasks.rados.rados.0.smithi042.stdout:2811: writing smithi04268048-3 from 1656270 to 2430934 tid 2 2024-09-20T19:35:10.678 INFO:tasks.rados.rados.0.smithi042.stdout:2812: rollback oid 17 current snap is 325 2024-09-20T19:35:10.678 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 323 2024-09-20T19:35:10.678 INFO:tasks.rados.rados.0.smithi042.stdout:2808: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:10.719 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 626169 2024-09-20T19:35:10.759 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.760 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2480899 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: finishing write tid 1 to smithi04268048-3 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2812: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2812: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2812: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1098 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2810: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: finishing write tid 2 to smithi04268048-3 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: finishing write tid 3 to smithi04268048-3 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: finishing write tid 4 to smithi04268048-3 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: oid 3 updating version 0 to 958 2024-09-20T19:35:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:2811: oid 3 version 958 is already newer than 957 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2811: oid 3 updating version 958 to 959 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 959 (ObjNum 983 snap 325 seq_num 983) dirty exists 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2811: left oid 3 (ObjNum 983 snap 325 seq_num 983) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2810: finishing copy_from to smithi04268048-5 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 740 (ObjNum 977 snap 324 seq_num 977) dirty exists 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2808: done (4 left) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2809: done (3 left) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2810: done (2 left) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2811: done (1 left) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2812: done (0 left) 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2813: write (excl) oid 28 current snap is 325 2024-09-20T19:35:10.771 INFO:tasks.rados.rados.0.smithi042.stdout:2813: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-20T19:35:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:10.785 INFO:tasks.rados.rados.0.smithi042.stdout:2813: writing smithi04268048-28 from 538088 to 1192641 tid 1 2024-09-20T19:35:10.793 INFO:tasks.rados.rados.0.smithi042.stdout:2813: writing smithi04268048-28 from 1714752 to 2349427 tid 2 2024-09-20T19:35:10.793 INFO:tasks.rados.rados.0.smithi042.stdout:2814: snap_remove snap 312 2024-09-20T19:35:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:10 smithi162 bash[24557]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:10.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:10 smithi162 bash[24557]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:10.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:10.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:10.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:10.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:10 smithi042 bash[32075]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:10.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:10 smithi042 bash[32075]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:10 smithi042 bash[22706]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:10.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:10 smithi042 bash[22706]: cluster 2024-09-20T19:35:09.634007+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2024-09-20T19:35:11.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:11.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:11.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:11.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:11.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:35:11.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:11.419 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:35:11.419 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:35:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:11.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:11.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:11.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2814: done (1 left) 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2815: rollback oid 10 current snap is 325 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 303 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: finishing write tid 1 to smithi04268048-28 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: finishing write tid 2 to smithi04268048-28 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: finishing write tid 3 to smithi04268048-28 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: finishing write tid 4 to smithi04268048-28 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: oid 28 updating version 0 to 868 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: oid 28 updating version 868 to 870 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: oid 28 version 870 is already newer than 869 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 870 (ObjNum 984 snap 325 seq_num 984) dirty exists 2024-09-20T19:35:11.651 INFO:tasks.rados.rados.0.smithi042.stdout:2813: left oid 28 (ObjNum 984 snap 325 seq_num 984) 2024-09-20T19:35:11.652 INFO:tasks.rados.rados.0.smithi042.stdout:2813: done (1 left) 2024-09-20T19:35:11.652 INFO:tasks.rados.rados.0.smithi042.stdout:2816: read oid 28 snap -1 2024-09-20T19:35:11.652 INFO:tasks.rados.rados.0.smithi042.stdout:2816: expect (ObjNum 984 snap 325 seq_num 984) 2024-09-20T19:35:11.652 INFO:tasks.rados.rados.0.smithi042.stdout:2817: snap_remove snap 308 2024-09-20T19:35:11.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.656 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[22706]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:11 smithi042 bash[32075]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.672 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:11.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:11.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: audit 2024-09-20T19:35:09.742018+0000 mgr.y (mgr.24491) 645 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: cluster 2024-09-20T19:35:10.389278+0000 mgr.y (mgr.24491) 646 : cluster [DBG] pgmap v1200: 178 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 172 active+clean; 279 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: cluster 2024-09-20T19:35:10.645146+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2024-09-20T19:35:11.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:11 smithi162 bash[24557]: audit 2024-09-20T19:35:11.629252+0000 mon.a (mon.0) 2339 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:35:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:11.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:11.927 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:11.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:12.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:12.054 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:12.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:12.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T19:35:12.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:12.301 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:12.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:12.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:12.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:12.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:35:12.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.655 INFO:tasks.rados.rados.0.smithi042.stdout:2817: done (2 left) 2024-09-20T19:35:12.655 INFO:tasks.rados.rados.0.smithi042.stdout:2818: snap_remove snap 304 2024-09-20T19:35:12.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:12 smithi162 bash[24557]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:12.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:12.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:12.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:12.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[22706]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: cluster 2024-09-20T19:35:11.649474+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.340711+0000 mon.a (mon.0) 2341 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.341474+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:12 smithi042 bash[32075]: audit 2024-09-20T19:35:12.346719+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:12.932 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:12.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:13.033 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:13.100 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:13.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:13.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:13.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:13.379 INFO:tasks.rados.rados.0.smithi042.stdout:2818: done (2 left) 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:2819: rollback oid 38 current snap is 325 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 290 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:2820: delete oid 29 current snap is 325 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:2815: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:2815: finishing rollback tid 2 to smithi04268048-10 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 858 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:35:13.380 INFO:tasks.rados.rados.0.smithi042.stdout:2816: expect (ObjNum 984 snap 325 seq_num 984) 2024-09-20T19:35:13.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.443 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2417957 2024-09-20T19:35:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:13.491 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1552890 2024-09-20T19:35:13.528 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:13.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:13.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:13.611 INFO:tasks.rados.rados.0.smithi042.stdout:2819: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2819: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2819: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 867 (ObjNum 844 snap 276 seq_num 844) dirty exists 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2815: done (3 left) 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2816: done (2 left) 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2819: done (1 left) 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2820: done (0 left) 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2821: write oid 7 current snap is 325 2024-09-20T19:35:13.612 INFO:tasks.rados.rados.0.smithi042.stdout:2821: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-20T19:35:13.620 INFO:tasks.rados.rados.0.smithi042.stdout:2821: writing smithi04268048-7 from 571702 to 976753 tid 1 2024-09-20T19:35:13.630 INFO:tasks.rados.rados.0.smithi042.stdout:2821: writing smithi04268048-7 from 1431716 to 2070060 tid 2 2024-09-20T19:35:13.635 INFO:tasks.rados.rados.0.smithi042.stdout:2821: writing smithi04268048-7 from 2560365 to 2571702 tid 3 2024-09-20T19:35:13.635 INFO:tasks.rados.rados.0.smithi042.stdout:2822: write oid 29 current snap is 325 2024-09-20T19:35:13.635 INFO:tasks.rados.rados.0.smithi042.stdout:2821: finishing write tid 1 to smithi04268048-7 2024-09-20T19:35:13.635 INFO:tasks.rados.rados.0.smithi042.stdout:2821: finishing write tid 2 to smithi04268048-7 2024-09-20T19:35:13.635 INFO:tasks.rados.rados.0.smithi042.stdout:2822: seq_num 987 ranges {588509=480300,1773845=782002} 2024-09-20T19:35:13.644 INFO:tasks.rados.rados.0.smithi042.stdout:2822: writing smithi04268048-29 from 588509 to 1068809 tid 1 2024-09-20T19:35:13.658 INFO:tasks.rados.rados.0.smithi042.stdout:2822: writing smithi04268048-29 from 1773845 to 2555847 tid 2 2024-09-20T19:35:13.658 INFO:tasks.rados.rados.0.smithi042.stdout:2823: read oid 47 snap 309 2024-09-20T19:35:13.658 INFO:tasks.rados.rados.0.smithi042.stdout:2823: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:35:13.658 INFO:tasks.rados.rados.0.smithi042.stdout:2824: snap_remove snap 296 2024-09-20T19:35:13.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:13 smithi162 bash[24557]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.698 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:13.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:13.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[22706]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:12.390668+0000 mgr.y (mgr.24491) 647 : cluster [DBG] pgmap v1203: 178 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 160 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:12.654063+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: audit 2024-09-20T19:35:13.252281+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: audit 2024-09-20T19:35:13.253722+0000 mon.a (mon.0) 2346 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:13 smithi042 bash[32075]: cluster 2024-09-20T19:35:13.374916+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2024-09-20T19:35:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:13.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:14.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:14.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:14.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:14.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:14.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:14.380 INFO:tasks.rados.rados.0.smithi042.stdout:2824: done (3 left) 2024-09-20T19:35:14.380 INFO:tasks.rados.rados.0.smithi042.stdout:2825: snap_remove snap 322 2024-09-20T19:35:14.450 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:14.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:14.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:14.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:14.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:14.862 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:14.965 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:14.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:15.040 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:35:15.043 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:35:15.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:15.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:15.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:15.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:15.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:15.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.380 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2825: done (3 left) 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2826: copy_from oid 38 from oid 16 current snap is 325 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: finishing write tid 3 to smithi04268048-7 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: finishing write tid 4 to smithi04268048-7 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: finishing write tid 5 to smithi04268048-7 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: oid 7 updating version 0 to 1224 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: oid 7 updating version 1224 to 1225 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: oid 7 updating version 1225 to 1226 2024-09-20T19:35:15.393 INFO:tasks.rados.rados.0.smithi042.stdout:2821: oid 7 version 1226 is already newer than 1223 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1226 (ObjNum 986 snap 325 seq_num 986) dirty exists 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2821: left oid 7 (ObjNum 986 snap 325 seq_num 986) 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: finishing write tid 1 to smithi04268048-29 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: finishing write tid 2 to smithi04268048-29 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: finishing write tid 3 to smithi04268048-29 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: finishing write tid 4 to smithi04268048-29 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: oid 29 updating version 0 to 863 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: oid 29 updating version 863 to 865 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: oid 29 version 865 is already newer than 864 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 865 (ObjNum 987 snap 325 seq_num 987) dirty exists 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2822: left oid 29 (ObjNum 987 snap 325 seq_num 987) 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:2823: expect (ObjNum 927 snap 309 seq_num 927) 2024-09-20T19:35:15.394 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:15.467 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:15.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.508 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:15.538 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:15.610 INFO:tasks.rados.rados.0.smithi042.stdout:2826: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2826: finishing copy_from to smithi04268048-38 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1144 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2821: done (3 left) 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2822: done (2 left) 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2823: done (1 left) 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2826: done (0 left) 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2827: rollback oid 31 current snap is 325 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 290 2024-09-20T19:35:15.611 INFO:tasks.rados.rados.0.smithi042.stdout:2828: copy_from oid 21 from oid 19 current snap is 325 2024-09-20T19:35:15.612 INFO:tasks.rados.rados.0.smithi042.stdout:2829: copy_from oid 33 from oid 44 current snap is 325 2024-09-20T19:35:15.612 INFO:tasks.rados.rados.0.smithi042.stdout:2830: snap_remove snap 310 2024-09-20T19:35:15.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:15.635 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[32075]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[32075]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[32075]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[32075]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[22706]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[22706]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[22706]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:15 smithi042 bash[22706]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.749 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:15 smithi162 bash[24557]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.749 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:15 smithi162 bash[24557]: cluster 2024-09-20T19:35:14.376702+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2024-09-20T19:35:15.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:15 smithi162 bash[24557]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:15 smithi162 bash[24557]: cluster 2024-09-20T19:35:14.391640+0000 mgr.y (mgr.24491) 648 : cluster [DBG] pgmap v1207: 178 pgs: 15 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 294 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T19:35:15.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:15.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:15.892 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:15.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:15.903 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1207,"stamp":"2024-09-20T19:35:14.390914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":308680713,"num_objects":343,"num_object_clones":99,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":11,"num_read":3472,"num_read_kb":957473,"num_write":1745,"num_write_kb":377239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23999,"ondisk_log_size":23999,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1059300,"kb_used_data":705064,"kb_used_omap":92,"kb_used_meta":352675,"kb_avail":655156764,"statfs":{"total":671965249536,"available":670880526336,"internally_reserved":0,"allocated":721985536,"data_stored":945738055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":361139897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15797778,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":30320,"num_write":25,"num_write_kb":10411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.750830"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":3526,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.673233+0000","last_change":"2024-09-20T19:35:11.673491+0000","last_active":"2024-09-20T19:35:11.673233+0000","last_peered":"2024-09-20T19:35:11.673233+0000","last_clean":"2024-09-20T19:35:11.673233+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:35:11.673233+0000","last_undegraded":"2024-09-20T19:35:11.673233+0000","last_fullsized":"2024-09-20T19:35:11.673233+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:34:02.755546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:02.755546+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:20.397312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018733883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"7.22","version":"795'247","reported_seq":3165,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615771+0000","last_change":"2024-09-20T19:35:06.605399+0000","last_active":"2024-09-20T19:35:08.615771+0000","last_peered":"2024-09-20T19:35:08.615771+0000","last_clean":"2024-09-20T19:35:08.615771+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:35:08.615771+0000","last_undegraded":"2024-09-20T19:35:08.615771+0000","last_fullsized":"2024-09-20T19:35:08.615771+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"721'206","last_scrub_stamp":"2024-09-20T19:33:52.586094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:52.586094+0000","objects_scrubbed":2,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:08.712255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040101219999999996,"stat_sum":{"num_bytes":3696466,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":6178,"num_write":9,"num_write_kb":1122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"7.21","version":"803'655","reported_seq":4059,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.672240+0000","last_change":"2024-09-20T19:35:11.672240+0000","last_active":"2024-09-20T19:35:11.672240+0000","last_peered":"2024-09-20T19:35:11.672240+0000","last_clean":"2024-09-20T19:35:11.672240+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:35:11.672240+0000","last_undegraded":"2024-09-20T19:35:11.672240+0000","last_fullsized":"2024-09-20T19:35:11.672240+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"722'630","last_scrub_stamp":"2024-09-20T19:34:05.048343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.048343+0000","objects_scrubbed":2,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.895874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0057260840000000002,"stat_sum":{"num_bytes":5933267,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":34234,"num_write":3,"num_write_kb":2200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"7.20","version":"777'863","reported_seq":4202,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661880+0000","last_change":"2024-09-20T19:35:11.662193+0000","last_active":"2024-09-20T19:35:11.661880+0000","last_peered":"2024-09-20T19:35:11.661880+0000","last_clean":"2024-09-20T19:35:11.661880+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:35:11.661880+0000","last_undegraded":"2024-09-20T19:35:11.661880+0000","last_fullsized":"2024-09-20T19:35:11.661880+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"721'837","last_scrub_stamp":"2024-09-20T19:33:59.352105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.352105+0000","objects_scrubbed":2,"log_size":863,"log_dups_size":0,"ondisk_log_size":863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.339564+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0087349799999999998,"stat_sum":{"num_bytes":5272743,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":27240,"num_write":10,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"7.1a","version":"800'561","reported_seq":3627,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.852398+0000","last_change":"2024-09-20T19:35:06.606402+0000","last_active":"2024-09-20T19:35:08.852398+0000","last_peered":"2024-09-20T19:35:08.852398+0000","last_clean":"2024-09-20T19:35:08.852398+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:35:08.852398+0000","last_undegraded":"2024-09-20T19:35:08.852398+0000","last_fullsized":"2024-09-20T19:35:08.852398+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'493","last_scrub_stamp":"2024-09-20T19:33:55.653122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.653122+0000","objects_scrubbed":4,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0036990959999999998,"stat_sum":{"num_bytes":13986279,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":27,"num_read_kb":20105,"num_write":25,"num_write_kb":12121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":1001,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663502+0000","last_change":"2024-09-20T19:33:43.387357+0000","last_active":"2024-09-20T19:35:11.663502+0000","last_peered":"2024-09-20T19:35:11.663502+0000","last_clean":"2024-09-20T19:35:11.663502+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:35:11.663502+0000","last_undegraded":"2024-09-20T19:35:11.663502+0000","last_fullsized":"2024-09-20T19:35:11.663502+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1323,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669548+0000","last_change":"2024-09-20T19:34:58.530958+0000","last_active":"2024-09-20T19:35:12.669548+0000","last_peered":"2024-09-20T19:35:12.669548+0000","last_clean":"2024-09-20T19:35:12.669548+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:35:12.669548+0000","last_undegraded":"2024-09-20T19:35:12.669548+0000","last_fullsized":"2024-09-20T19:35:12.669548+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1235,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654603+0000","last_change":"2024-09-20T19:33:43.389098+0000","last_active":"2024-09-20T19:35:11.654603+0000","last_peered":"2024-09-20T19:35:11.654603+0000","last_clean":"2024-09-20T19:35:11.654603+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:35:11.654603+0000","last_undegraded":"2024-09-20T19:35:11.654603+0000","last_fullsized":"2024-09-20T19:35:11.654603+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:32:58.834586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:58.834586+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.277653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1293,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615253+0000","last_change":"2024-09-20T19:35:04.624469+0000","last_active":"2024-09-20T19:35:08.615253+0000","last_peered":"2024-09-20T19:35:08.615253+0000","last_clean":"2024-09-20T19:35:08.615253+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:35:08.615253+0000","last_undegraded":"2024-09-20T19:35:08.615253+0000","last_fullsized":"2024-09-20T19:35:08.615253+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.624374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.624374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.762788+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":3814,"reported_epoch":804,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:12.670454+0000","last_change":"2024-09-20T19:35:12.658507+0000","last_active":"2024-09-20T19:35:12.670454+0000","last_peered":"2024-09-20T19:35:12.670454+0000","last_clean":"2024-09-20T19:35:12.670454+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:35:12.670454+0000","last_undegraded":"2024-09-20T19:35:12.670454+0000","last_fullsized":"2024-09-20T19:35:12.670454+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:33:53.566287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.566287+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:20.194028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.001598862,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.1e","version":"0'0","reported_seq":1058,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669697+0000","last_change":"2024-09-20T19:33:43.367928+0000","last_active":"2024-09-20T19:35:12.669697+0000","last_peered":"2024-09-20T19:35:12.669697+0000","last_clean":"2024-09-20T19:35:12.669697+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:35:12.669697+0000","last_undegraded":"2024-09-20T19:35:12.669697+0000","last_fullsized":"2024-09-20T19:35:12.669697+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1397,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615602+0000","last_change":"2024-09-20T19:34:47.537704+0000","last_active":"2024-09-20T19:35:08.615602+0000","last_peered":"2024-09-20T19:35:08.615602+0000","last_clean":"2024-09-20T19:35:08.615602+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:35:08.615602+0000","last_undegraded":"2024-09-20T19:35:08.615602+0000","last_fullsized":"2024-09-20T19:35:08.615602+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:47.537616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.537616+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.097439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1532,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651207+0000","last_change":"2024-09-20T19:34:54.283813+0000","last_active":"2024-09-20T19:35:11.651207+0000","last_peered":"2024-09-20T19:35:11.651207+0000","last_clean":"2024-09-20T19:35:11.651207+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:35:11.651207+0000","last_undegraded":"2024-09-20T19:35:11.651207+0000","last_fullsized":"2024-09-20T19:35:11.651207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.283695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.283695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.569450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1355,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.616288+0000","last_change":"2024-09-20T19:34:58.623699+0000","last_active":"2024-09-20T19:35:08.616288+0000","last_peered":"2024-09-20T19:35:08.616288+0000","last_clean":"2024-09-20T19:35:08.616288+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:35:08.616288+0000","last_undegraded":"2024-09-20T19:35:08.616288+0000","last_fullsized":"2024-09-20T19:35:08.616288+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:58.623637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:58.623637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:23.832554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"802'1148","reported_seq":4437,"reported_epoch":804,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:12.668159+0000","last_change":"2024-09-20T19:35:12.653850+0000","last_active":"2024-09-20T19:35:12.668159+0000","last_peered":"2024-09-20T19:35:12.668159+0000","last_clean":"2024-09-20T19:35:12.668159+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:35:12.668159+0000","last_undegraded":"2024-09-20T19:35:12.668159+0000","last_fullsized":"2024-09-20T19:35:12.668159+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99909787999999999,"stat_sum":{"num_bytes":14018061,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":63,"num_read_kb":46979,"num_write":61,"num_write_kb":27274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1529,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.669696+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:35:11.669696+0000","last_peered":"2024-09-20T19:35:11.669696+0000","last_clean":"2024-09-20T19:35:11.669696+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:35:11.669696+0000","last_undegraded":"2024-09-20T19:35:11.669696+0000","last_fullsized":"2024-09-20T19:35:11.669696+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1392,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615975+0000","last_change":"2024-09-20T19:34:42.553244+0000","last_active":"2024-09-20T19:35:08.615975+0000","last_peered":"2024-09-20T19:35:08.615975+0000","last_clean":"2024-09-20T19:35:08.615975+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:35:08.615975+0000","last_undegraded":"2024-09-20T19:35:08.615975+0000","last_fullsized":"2024-09-20T19:35:08.615975+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:34:42.553118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:42.553118+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.964837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1521,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.664273+0000","last_change":"2024-09-20T19:34:50.140557+0000","last_active":"2024-09-20T19:35:11.664273+0000","last_peered":"2024-09-20T19:35:11.664273+0000","last_clean":"2024-09-20T19:35:11.664273+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:35:11.664273+0000","last_undegraded":"2024-09-20T19:35:11.664273+0000","last_fullsized":"2024-09-20T19:35:11.664273+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:50.140498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:50.140498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:55.537342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1534,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662589+0000","last_change":"2024-09-20T19:33:43.447076+0000","last_active":"2024-09-20T19:35:11.662589+0000","last_peered":"2024-09-20T19:35:11.662589+0000","last_clean":"2024-09-20T19:35:11.662589+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:35:11.662589+0000","last_undegraded":"2024-09-20T19:35:11.662589+0000","last_fullsized":"2024-09-20T19:35:11.662589+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:19.094498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"743'406","reported_seq":3741,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.668555+0000","last_change":"2024-09-20T19:35:11.668619+0000","last_active":"2024-09-20T19:35:11.668555+0000","last_peered":"2024-09-20T19:35:11.668555+0000","last_clean":"2024-09-20T19:35:11.668555+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:35:11.668555+0000","last_undegraded":"2024-09-20T19:35:11.668555+0000","last_fullsized":"2024-09-20T19:35:11.668555+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'290","last_scrub_stamp":"2024-09-20T19:29:01.442771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:01.442771+0000","objects_scrubbed":4,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014058658,"stat_sum":{"num_bytes":2084266,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":36395,"num_write":28,"num_write_kb":7429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1518,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665360+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:35:11.665360+0000","last_peered":"2024-09-20T19:35:11.665360+0000","last_clean":"2024-09-20T19:35:11.665360+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:35:11.665360+0000","last_undegraded":"2024-09-20T19:35:11.665360+0000","last_fullsized":"2024-09-20T19:35:11.665360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1425,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615731+0000","last_change":"2024-09-20T19:34:26.418165+0000","last_active":"2024-09-20T19:35:08.615731+0000","last_peered":"2024-09-20T19:35:08.615731+0000","last_clean":"2024-09-20T19:35:08.615731+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:35:08.615731+0000","last_undegraded":"2024-09-20T19:35:08.615731+0000","last_fullsized":"2024-09-20T19:35:08.615731+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.451996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.451996+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.609262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1514,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661934+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:35:11.661934+0000","last_peered":"2024-09-20T19:35:11.661934+0000","last_clean":"2024-09-20T19:35:11.661934+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:35:11.661934+0000","last_undegraded":"2024-09-20T19:35:11.661934+0000","last_fullsized":"2024-09-20T19:35:11.661934+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.072932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":492,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.667892+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:35:12.667892+0000","last_peered":"2024-09-20T19:35:12.667892+0000","last_clean":"2024-09-20T19:35:12.667892+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:35:12.667892+0000","last_undegraded":"2024-09-20T19:35:12.667892+0000","last_fullsized":"2024-09-20T19:35:12.667892+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1150,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.660361+0000","last_change":"2024-09-20T19:34:07.057170+0000","last_active":"2024-09-20T19:35:11.660361+0000","last_peered":"2024-09-20T19:35:11.660361+0000","last_clean":"2024-09-20T19:35:11.660361+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:35:11.660361+0000","last_undegraded":"2024-09-20T19:35:11.660361+0000","last_fullsized":"2024-09-20T19:35:11.660361+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:07.057076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:07.057076+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.891767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"795'302","reported_seq":3626,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617004+0000","last_change":"2024-09-20T19:35:06.610962+0000","last_active":"2024-09-20T19:35:08.617004+0000","last_peered":"2024-09-20T19:35:08.617004+0000","last_clean":"2024-09-20T19:35:08.617004+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:35:08.617004+0000","last_undegraded":"2024-09-20T19:35:08.617004+0000","last_fullsized":"2024-09-20T19:35:08.617004+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"735'281","last_scrub_stamp":"2024-09-20T19:34:05.563300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.563300+0000","objects_scrubbed":3,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:19.292348+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.005635981,"stat_sum":{"num_bytes":4958231,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":5339,"num_write":5,"num_write_kb":1766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.d","version":"523'31","reported_seq":1376,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.616190+0000","last_change":"2024-09-20T19:34:13.521735+0000","last_active":"2024-09-20T19:35:08.616190+0000","last_peered":"2024-09-20T19:35:08.616190+0000","last_clean":"2024-09-20T19:35:08.616190+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:35:08.616190+0000","last_undegraded":"2024-09-20T19:35:08.616190+0000","last_fullsized":"2024-09-20T19:35:08.616190+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:34:13.521620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.521620+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.065586+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1522,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.664871+0000","last_change":"2024-09-20T19:34:49.185014+0000","last_active":"2024-09-20T19:35:11.664871+0000","last_peered":"2024-09-20T19:35:11.664871+0000","last_clean":"2024-09-20T19:35:11.664871+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:35:11.664871+0000","last_undegraded":"2024-09-20T19:35:11.664871+0000","last_fullsized":"2024-09-20T19:35:11.664871+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:49.184958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:49.184958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:49.964266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1534,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651842+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:35:11.651842+0000","last_peered":"2024-09-20T19:35:11.651842+0000","last_clean":"2024-09-20T19:35:11.651842+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:35:11.651842+0000","last_undegraded":"2024-09-20T19:35:11.651842+0000","last_fullsized":"2024-09-20T19:35:11.651842+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.211201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.658014+0000","last_change":"2024-09-20T19:34:26.402007+0000","last_active":"2024-09-20T19:35:11.658014+0000","last_peered":"2024-09-20T19:35:11.658014+0000","last_clean":"2024-09-20T19:35:11.658014+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:35:11.658014+0000","last_undegraded":"2024-09-20T19:35:11.658014+0000","last_fullsized":"2024-09-20T19:35:11.658014+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:06.544189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.544189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:14.091652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1615,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.666062+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:35:11.666062+0000","last_peered":"2024-09-20T19:35:11.666062+0000","last_clean":"2024-09-20T19:35:11.666062+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:35:11.666062+0000","last_undegraded":"2024-09-20T19:35:11.666062+0000","last_fullsized":"2024-09-20T19:35:11.666062+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"803'565","reported_seq":3688,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:12.669718+0000","last_change":"2024-09-20T19:35:12.658334+0000","last_active":"2024-09-20T19:35:12.669718+0000","last_peered":"2024-09-20T19:35:12.669718+0000","last_clean":"2024-09-20T19:35:12.669718+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:35:12.669718+0000","last_undegraded":"2024-09-20T19:35:12.669718+0000","last_fullsized":"2024-09-20T19:35:12.669718+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'544","last_scrub_stamp":"2024-09-20T19:33:54.617348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.617348+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:15.710306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.99692826599999995,"stat_sum":{"num_bytes":6252600,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":10988,"num_write":12,"num_write_kb":5302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.c","version":"523'20","reported_seq":1419,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.616387+0000","last_change":"2024-09-20T19:33:43.467110+0000","last_active":"2024-09-20T19:35:08.616387+0000","last_peered":"2024-09-20T19:35:08.616387+0000","last_clean":"2024-09-20T19:35:08.616387+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:35:08.616387+0000","last_undegraded":"2024-09-20T19:35:08.616387+0000","last_fullsized":"2024-09-20T19:35:08.616387+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:32:10.444013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.444013+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.713160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1558,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653703+0000","last_change":"2024-09-20T19:34:57.369277+0000","last_active":"2024-09-20T19:35:11.653703+0000","last_peered":"2024-09-20T19:35:11.653703+0000","last_clean":"2024-09-20T19:35:11.653703+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:35:11.653703+0000","last_undegraded":"2024-09-20T19:35:11.653703+0000","last_fullsized":"2024-09-20T19:35:11.653703+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.369117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.369117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:07.185078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1748,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657713+0000","last_change":"2024-09-20T19:34:17.102983+0000","last_active":"2024-09-20T19:35:11.657713+0000","last_peered":"2024-09-20T19:35:11.657713+0000","last_clean":"2024-09-20T19:35:11.657713+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:35:11.657713+0000","last_undegraded":"2024-09-20T19:35:11.657713+0000","last_fullsized":"2024-09-20T19:35:11.657713+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:34:17.102808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.102808+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:34.479538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1358,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617033+0000","last_change":"2024-09-20T19:34:45.548359+0000","last_active":"2024-09-20T19:35:08.617033+0000","last_peered":"2024-09-20T19:35:08.617033+0000","last_clean":"2024-09-20T19:35:08.617033+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:35:08.617033+0000","last_undegraded":"2024-09-20T19:35:08.617033+0000","last_fullsized":"2024-09-20T19:35:08.617033+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:45.548235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:45.548235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.055514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1121,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.660604+0000","last_change":"2024-09-20T19:34:10.011707+0000","last_active":"2024-09-20T19:35:11.660604+0000","last_peered":"2024-09-20T19:35:11.660604+0000","last_clean":"2024-09-20T19:35:11.660604+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:35:11.660604+0000","last_undegraded":"2024-09-20T19:35:11.660604+0000","last_fullsized":"2024-09-20T19:35:11.660604+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:10.011637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.011637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.773891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"803'874","reported_seq":4297,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.672745+0000","last_change":"2024-09-20T19:35:11.672745+0000","last_active":"2024-09-20T19:35:11.672745+0000","last_peered":"2024-09-20T19:35:11.672745+0000","last_clean":"2024-09-20T19:35:11.672745+0000","last_became_active":"2024-09-20T19:33:46.371630+0000","last_became_peered":"2024-09-20T19:33:46.371630+0000","last_unstale":"2024-09-20T19:35:11.672745+0000","last_undegraded":"2024-09-20T19:35:11.672745+0000","last_fullsized":"2024-09-20T19:35:11.672745+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'835","last_scrub_stamp":"2024-09-20T19:33:55.628320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.628320+0000","objects_scrubbed":2,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060711269999999999,"stat_sum":{"num_bytes":8034340,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":8533,"num_write":9,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.f","version":"523'27","reported_seq":1625,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653061+0000","last_change":"2024-09-20T19:34:56.365553+0000","last_active":"2024-09-20T19:35:11.653061+0000","last_peered":"2024-09-20T19:35:11.653061+0000","last_clean":"2024-09-20T19:35:11.653061+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:35:11.653061+0000","last_undegraded":"2024-09-20T19:35:11.653061+0000","last_fullsized":"2024-09-20T19:35:11.653061+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:56.365402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:56.365402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.638116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1558,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651432+0000","last_change":"2024-09-20T19:34:46.261824+0000","last_active":"2024-09-20T19:35:11.651432+0000","last_peered":"2024-09-20T19:35:11.651432+0000","last_clean":"2024-09-20T19:35:11.651432+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:35:11.651432+0000","last_undegraded":"2024-09-20T19:35:11.651432+0000","last_fullsized":"2024-09-20T19:35:11.651432+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:46.261770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.261770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:04.225781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1528,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617184+0000","last_change":"2024-09-20T19:33:59.507413+0000","last_active":"2024-09-20T19:35:08.617184+0000","last_peered":"2024-09-20T19:35:08.617184+0000","last_clean":"2024-09-20T19:35:08.617184+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:35:08.617184+0000","last_undegraded":"2024-09-20T19:35:08.617184+0000","last_fullsized":"2024-09-20T19:35:08.617184+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:33:59.507348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.507348+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:17.162783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1527,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662741+0000","last_change":"2024-09-20T19:34:17.113285+0000","last_active":"2024-09-20T19:35:11.662741+0000","last_peered":"2024-09-20T19:35:11.662741+0000","last_clean":"2024-09-20T19:35:11.662741+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:35:11.662741+0000","last_undegraded":"2024-09-20T19:35:11.662741+0000","last_fullsized":"2024-09-20T19:35:11.662741+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:17.113170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.113170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:44.530510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"802'570","reported_seq":3949,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.668987+0000","last_change":"2024-09-20T19:35:11.669049+0000","last_active":"2024-09-20T19:35:11.668987+0000","last_peered":"2024-09-20T19:35:11.668987+0000","last_clean":"2024-09-20T19:35:11.668987+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:35:11.668987+0000","last_undegraded":"2024-09-20T19:35:11.668987+0000","last_fullsized":"2024-09-20T19:35:11.668987+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"717'538","last_scrub_stamp":"2024-09-20T19:34:06.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.656444+0000","objects_scrubbed":1,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:28.527824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012326775999999999,"stat_sum":{"num_bytes":8530920,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":14,"num_write_kb":4347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.e","version":"523'19","reported_seq":1400,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617368+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:35:08.617368+0000","last_peered":"2024-09-20T19:35:08.617368+0000","last_clean":"2024-09-20T19:35:08.617368+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:35:08.617368+0000","last_undegraded":"2024-09-20T19:35:08.617368+0000","last_fullsized":"2024-09-20T19:35:08.617368+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.587592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":187,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663310+0000","last_change":"2024-09-20T19:33:43.387910+0000","last_active":"2024-09-20T19:35:11.663310+0000","last_peered":"2024-09-20T19:35:11.663310+0000","last_clean":"2024-09-20T19:35:11.663310+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:35:11.663310+0000","last_undegraded":"2024-09-20T19:35:11.663310+0000","last_fullsized":"2024-09-20T19:35:11.663310+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1531,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.673554+0000","last_change":"2024-09-20T19:34:48.070464+0000","last_active":"2024-09-20T19:35:11.673554+0000","last_peered":"2024-09-20T19:35:11.673554+0000","last_clean":"2024-09-20T19:35:11.673554+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:35:11.673554+0000","last_undegraded":"2024-09-20T19:35:11.673554+0000","last_fullsized":"2024-09-20T19:35:11.673554+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.070340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.070340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:14.947230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":997,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654773+0000","last_change":"2024-09-20T19:33:43.352629+0000","last_active":"2024-09-20T19:35:11.654773+0000","last_peered":"2024-09-20T19:35:11.654773+0000","last_clean":"2024-09-20T19:35:11.654773+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:35:11.654773+0000","last_undegraded":"2024-09-20T19:35:11.654773+0000","last_fullsized":"2024-09-20T19:35:11.654773+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":358,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662262+0000","last_change":"2024-09-20T19:35:03.930343+0000","last_active":"2024-09-20T19:35:11.662262+0000","last_peered":"2024-09-20T19:35:11.662262+0000","last_clean":"2024-09-20T19:35:11.662262+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:35:11.662262+0000","last_undegraded":"2024-09-20T19:35:11.662262+0000","last_fullsized":"2024-09-20T19:35:11.662262+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:03.930289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.930289+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.856845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":3086,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.652637+0000","last_change":"2024-09-20T19:35:11.652615+0000","last_active":"2024-09-20T19:35:11.652637+0000","last_peered":"2024-09-20T19:35:11.652637+0000","last_clean":"2024-09-20T19:35:11.652637+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:35:11.652637+0000","last_undegraded":"2024-09-20T19:35:11.652637+0000","last_fullsized":"2024-09-20T19:35:11.652637+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:33:57.391147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:57.391147+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030616329999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.1","version":"523'28","reported_seq":1369,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615642+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:35:08.615642+0000","last_peered":"2024-09-20T19:35:08.615642+0000","last_clean":"2024-09-20T19:35:08.615642+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:35:08.615642+0000","last_undegraded":"2024-09-20T19:35:08.615642+0000","last_fullsized":"2024-09-20T19:35:08.615642+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.650160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1520,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656444+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:35:11.656444+0000","last_peered":"2024-09-20T19:35:11.656444+0000","last_clean":"2024-09-20T19:35:11.656444+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:35:11.656444+0000","last_undegraded":"2024-09-20T19:35:11.656444+0000","last_fullsized":"2024-09-20T19:35:11.656444+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":426,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.648595+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:35:11.648595+0000","last_peered":"2024-09-20T19:35:11.648595+0000","last_clean":"2024-09-20T19:35:11.648595+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:35:11.648595+0000","last_undegraded":"2024-09-20T19:35:11.648595+0000","last_fullsized":"2024-09-20T19:35:11.648595+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:17.820899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1508,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663636+0000","last_change":"2024-09-20T19:34:16.144529+0000","last_active":"2024-09-20T19:35:11.663636+0000","last_peered":"2024-09-20T19:35:11.663636+0000","last_clean":"2024-09-20T19:35:11.663636+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:35:11.663636+0000","last_undegraded":"2024-09-20T19:35:11.663636+0000","last_fullsized":"2024-09-20T19:35:11.663636+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:16.144473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:16.144473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.559736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":931,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.616324+0000","last_change":"2024-09-20T19:35:07.564250+0000","last_active":"2024-09-20T19:35:08.616324+0000","last_peered":"2024-09-20T19:35:08.616324+0000","last_clean":"2024-09-20T19:35:08.616324+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:35:08.616324+0000","last_undegraded":"2024-09-20T19:35:08.616324+0000","last_fullsized":"2024-09-20T19:35:08.616324+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:07.564190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.564190+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:29.024831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"718'421","reported_seq":3610,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662824+0000","last_change":"2024-09-20T19:35:11.663097+0000","last_active":"2024-09-20T19:35:11.662824+0000","last_peered":"2024-09-20T19:35:11.662824+0000","last_clean":"2024-09-20T19:35:11.662824+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:35:11.662824+0000","last_undegraded":"2024-09-20T19:35:11.662824+0000","last_fullsized":"2024-09-20T19:35:11.662824+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'421","last_scrub_stamp":"2024-09-20T19:34:59.428803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.428803+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:00.888067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0070191890000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.0","version":"523'30","reported_seq":1246,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665477+0000","last_change":"2024-09-20T19:33:43.476313+0000","last_active":"2024-09-20T19:35:11.665477+0000","last_peered":"2024-09-20T19:35:11.665477+0000","last_clean":"2024-09-20T19:35:11.665477+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:35:11.665477+0000","last_undegraded":"2024-09-20T19:35:11.665477+0000","last_fullsized":"2024-09-20T19:35:11.665477+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1528,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.652325+0000","last_change":"2024-09-20T19:32:15.096541+0000","last_active":"2024-09-20T19:35:11.652325+0000","last_peered":"2024-09-20T19:35:11.652325+0000","last_clean":"2024-09-20T19:35:11.652325+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:35:11.652325+0000","last_undegraded":"2024-09-20T19:35:11.652325+0000","last_fullsized":"2024-09-20T19:35:11.652325+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:32:15.096450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.096450+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:37.278967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1328,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615237+0000","last_change":"2024-09-20T19:33:43.375389+0000","last_active":"2024-09-20T19:35:08.615237+0000","last_peered":"2024-09-20T19:35:08.615237+0000","last_clean":"2024-09-20T19:35:08.615237+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:35:08.615237+0000","last_undegraded":"2024-09-20T19:35:08.615237+0000","last_fullsized":"2024-09-20T19:35:08.615237+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:18.400304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.400304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.537825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1290,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615857+0000","last_change":"2024-09-20T19:34:59.643830+0000","last_active":"2024-09-20T19:35:08.615857+0000","last_peered":"2024-09-20T19:35:08.615857+0000","last_clean":"2024-09-20T19:35:08.615857+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:35:08.615857+0000","last_undegraded":"2024-09-20T19:35:08.615857+0000","last_fullsized":"2024-09-20T19:35:08.615857+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.643704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.643704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1241,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655025+0000","last_change":"2024-09-20T19:35:00.419430+0000","last_active":"2024-09-20T19:35:11.655025+0000","last_peered":"2024-09-20T19:35:11.655025+0000","last_clean":"2024-09-20T19:35:11.655025+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:35:11.655025+0000","last_undegraded":"2024-09-20T19:35:11.655025+0000","last_fullsized":"2024-09-20T19:35:11.655025+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:00.419230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:00.419230+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:26.608264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"803'860","reported_seq":4412,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.670563+0000","last_change":"2024-09-20T19:35:11.668171+0000","last_active":"2024-09-20T19:35:11.670563+0000","last_peered":"2024-09-20T19:35:11.670563+0000","last_clean":"2024-09-20T19:35:11.670563+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:35:11.670563+0000","last_undegraded":"2024-09-20T19:35:11.670563+0000","last_fullsized":"2024-09-20T19:35:11.670563+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'829","last_scrub_stamp":"2024-09-20T19:34:01.119894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:01.119894+0000","objects_scrubbed":1,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019969039999999999,"stat_sum":{"num_bytes":10223208,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":35560,"num_write":12,"num_write_kb":2467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.3","version":"523'29","reported_seq":1341,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669053+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:35:12.669053+0000","last_peered":"2024-09-20T19:35:12.669053+0000","last_clean":"2024-09-20T19:35:12.669053+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:35:12.669053+0000","last_undegraded":"2024-09-20T19:35:12.669053+0000","last_fullsized":"2024-09-20T19:35:12.669053+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1486,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657534+0000","last_change":"2024-09-20T19:34:05.253077+0000","last_active":"2024-09-20T19:35:11.657534+0000","last_peered":"2024-09-20T19:35:11.657534+0000","last_clean":"2024-09-20T19:35:11.657534+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:35:11.657534+0000","last_undegraded":"2024-09-20T19:35:11.657534+0000","last_fullsized":"2024-09-20T19:35:11.657534+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:05.253020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.253020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:27.698757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1524,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655371+0000","last_change":"2024-09-20T19:33:06.757095+0000","last_active":"2024-09-20T19:35:11.655371+0000","last_peered":"2024-09-20T19:35:11.655371+0000","last_clean":"2024-09-20T19:35:11.655371+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:35:11.655371+0000","last_undegraded":"2024-09-20T19:35:11.655371+0000","last_fullsized":"2024-09-20T19:35:11.655371+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.756963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.756963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1527,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651137+0000","last_change":"2024-09-20T19:34:55.325695+0000","last_active":"2024-09-20T19:35:11.651137+0000","last_peered":"2024-09-20T19:35:11.651137+0000","last_clean":"2024-09-20T19:35:11.651137+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:35:11.651137+0000","last_undegraded":"2024-09-20T19:35:11.651137+0000","last_fullsized":"2024-09-20T19:35:11.651137+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.325573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.325573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:58.162804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1234,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655580+0000","last_change":"2024-09-20T19:33:43.353150+0000","last_active":"2024-09-20T19:35:11.655580+0000","last_peered":"2024-09-20T19:35:11.655580+0000","last_clean":"2024-09-20T19:35:11.655580+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:35:11.655580+0000","last_undegraded":"2024-09-20T19:35:11.655580+0000","last_fullsized":"2024-09-20T19:35:11.655580+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":3852,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615439+0000","last_change":"2024-09-20T19:35:06.604114+0000","last_active":"2024-09-20T19:35:08.615439+0000","last_peered":"2024-09-20T19:35:08.615439+0000","last_clean":"2024-09-20T19:35:08.615439+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:35:08.615439+0000","last_undegraded":"2024-09-20T19:35:08.615439+0000","last_fullsized":"2024-09-20T19:35:08.615439+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:33:53.608544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.608544+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:19.490389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002839977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.2","version":"523'20","reported_seq":1581,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653554+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:35:11.653554+0000","last_peered":"2024-09-20T19:35:11.653554+0000","last_clean":"2024-09-20T19:35:11.653554+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:35:11.653554+0000","last_undegraded":"2024-09-20T19:35:11.653554+0000","last_fullsized":"2024-09-20T19:35:11.653554+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.034460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1536,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.652229+0000","last_change":"2024-09-20T19:34:48.237100+0000","last_active":"2024-09-20T19:35:11.652229+0000","last_peered":"2024-09-20T19:35:11.652229+0000","last_clean":"2024-09-20T19:35:11.652229+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:35:11.652229+0000","last_undegraded":"2024-09-20T19:35:11.652229+0000","last_fullsized":"2024-09-20T19:35:11.652229+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.236983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.236983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.560188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1463,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.670352+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:35:12.670352+0000","last_peered":"2024-09-20T19:35:12.670352+0000","last_clean":"2024-09-20T19:35:12.670352+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:35:12.670352+0000","last_undegraded":"2024-09-20T19:35:12.670352+0000","last_fullsized":"2024-09-20T19:35:12.670352+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1326,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.668272+0000","last_change":"2024-09-20T19:34:26.391734+0000","last_active":"2024-09-20T19:35:12.668272+0000","last_peered":"2024-09-20T19:35:12.668272+0000","last_clean":"2024-09-20T19:35:12.668272+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:35:12.668272+0000","last_undegraded":"2024-09-20T19:35:12.668272+0000","last_fullsized":"2024-09-20T19:35:12.668272+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.455584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.455584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1124,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662291+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:35:11.662291+0000","last_peered":"2024-09-20T19:35:11.662291+0000","last_clean":"2024-09-20T19:35:11.662291+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:35:11.662291+0000","last_undegraded":"2024-09-20T19:35:11.662291+0000","last_fullsized":"2024-09-20T19:35:11.662291+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:25.079668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"803'517","reported_seq":3966,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.673340+0000","last_change":"2024-09-20T19:35:11.673340+0000","last_active":"2024-09-20T19:35:11.673340+0000","last_peered":"2024-09-20T19:35:11.673340+0000","last_clean":"2024-09-20T19:35:11.673340+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:35:11.673340+0000","last_undegraded":"2024-09-20T19:35:11.673340+0000","last_fullsized":"2024-09-20T19:35:11.673340+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'400","last_scrub_stamp":"2024-09-20T19:31:36.808095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.808095+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011156008,"stat_sum":{"num_bytes":6150497,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":77,"num_read_kb":42900,"num_write":55,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.4","version":"715'45","reported_seq":1862,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.652129+0000","last_change":"2024-09-20T19:34:17.143462+0000","last_active":"2024-09-20T19:35:11.652129+0000","last_peered":"2024-09-20T19:35:11.652129+0000","last_clean":"2024-09-20T19:35:11.652129+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:35:11.652129+0000","last_undegraded":"2024-09-20T19:35:11.652129+0000","last_fullsized":"2024-09-20T19:35:11.652129+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'45","last_scrub_stamp":"2024-09-20T19:34:17.143380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.143380+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.481481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1537,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661347+0000","last_change":"2024-09-20T19:34:14.039743+0000","last_active":"2024-09-20T19:35:11.661347+0000","last_peered":"2024-09-20T19:35:11.661347+0000","last_clean":"2024-09-20T19:35:11.661347+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:35:11.661347+0000","last_undegraded":"2024-09-20T19:35:11.661347+0000","last_fullsized":"2024-09-20T19:35:11.661347+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:14.039620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:14.039620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.864667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1316,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669007+0000","last_change":"2024-09-20T19:32:17.478298+0000","last_active":"2024-09-20T19:35:12.669007+0000","last_peered":"2024-09-20T19:35:12.669007+0000","last_clean":"2024-09-20T19:35:12.669007+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:35:12.669007+0000","last_undegraded":"2024-09-20T19:35:12.669007+0000","last_fullsized":"2024-09-20T19:35:12.669007+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:17.478143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:17.478143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":971,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617075+0000","last_change":"2024-09-20T19:33:43.347560+0000","last_active":"2024-09-20T19:35:08.617075+0000","last_peered":"2024-09-20T19:35:08.617075+0000","last_clean":"2024-09-20T19:35:08.617075+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:35:08.617075+0000","last_undegraded":"2024-09-20T19:35:08.617075+0000","last_fullsized":"2024-09-20T19:35:08.617075+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:21.605528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1593,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665457+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:35:11.665457+0000","last_peered":"2024-09-20T19:35:11.665457+0000","last_clean":"2024-09-20T19:35:11.665457+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:35:11.665457+0000","last_undegraded":"2024-09-20T19:35:11.665457+0000","last_fullsized":"2024-09-20T19:35:11.665457+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"802'947","reported_seq":3974,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:12.668980+0000","last_change":"2024-09-20T19:35:11.645766+0000","last_active":"2024-09-20T19:35:12.668980+0000","last_peered":"2024-09-20T19:35:12.668980+0000","last_clean":"2024-09-20T19:35:12.668980+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:35:12.668980+0000","last_undegraded":"2024-09-20T19:35:12.668980+0000","last_fullsized":"2024-09-20T19:35:12.668980+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'779","last_scrub_stamp":"2024-09-20T19:31:22.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.595437+0000","objects_scrubbed":4,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011827770000000001,"stat_sum":{"num_bytes":15399993,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":83,"num_read_kb":81254,"num_write":63,"num_write_kb":22306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.7","version":"523'17","reported_seq":1497,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.658581+0000","last_change":"2024-09-20T19:33:04.118188+0000","last_active":"2024-09-20T19:35:11.658581+0000","last_peered":"2024-09-20T19:35:11.658581+0000","last_clean":"2024-09-20T19:35:11.658581+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:35:11.658581+0000","last_undegraded":"2024-09-20T19:35:11.658581+0000","last_fullsized":"2024-09-20T19:35:11.658581+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:33:04.117998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.117998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.179464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2547,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.648850+0000","last_change":"2024-09-20T19:35:02.415570+0000","last_active":"2024-09-20T19:35:11.648850+0000","last_peered":"2024-09-20T19:35:11.648850+0000","last_clean":"2024-09-20T19:35:11.648850+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:35:11.648850+0000","last_undegraded":"2024-09-20T19:35:11.648850+0000","last_fullsized":"2024-09-20T19:35:11.648850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:35:02.415432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:35:02.415432+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.696507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1006,"num_read_kb":1001,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1548,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654759+0000","last_change":"2024-09-20T19:33:43.443451+0000","last_active":"2024-09-20T19:35:11.654759+0000","last_peered":"2024-09-20T19:35:11.654759+0000","last_clean":"2024-09-20T19:35:11.654759+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:35:11.654759+0000","last_undegraded":"2024-09-20T19:35:11.654759+0000","last_fullsized":"2024-09-20T19:35:11.654759+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:13.137322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:13.137322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:33.197164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657794+0000","last_change":"2024-09-20T19:34:26.389239+0000","last_active":"2024-09-20T19:35:11.657794+0000","last_peered":"2024-09-20T19:35:11.657794+0000","last_clean":"2024-09-20T19:35:11.657794+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:35:11.657794+0000","last_undegraded":"2024-09-20T19:35:11.657794+0000","last_fullsized":"2024-09-20T19:35:11.657794+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:04.320119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.320119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.160769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1526,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.669180+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:35:11.669180+0000","last_peered":"2024-09-20T19:35:11.669180+0000","last_clean":"2024-09-20T19:35:11.669180+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:35:11.669180+0000","last_undegraded":"2024-09-20T19:35:11.669180+0000","last_fullsized":"2024-09-20T19:35:11.669180+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1126,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653290+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:35:11.653290+0000","last_peered":"2024-09-20T19:35:11.653290+0000","last_clean":"2024-09-20T19:35:11.653290+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:35:11.653290+0000","last_undegraded":"2024-09-20T19:35:11.653290+0000","last_fullsized":"2024-09-20T19:35:11.653290+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.253974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"802'988","reported_seq":4373,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.665538+0000","last_change":"2024-09-20T19:35:11.507303+0000","last_active":"2024-09-20T19:35:11.665538+0000","last_peered":"2024-09-20T19:35:11.665538+0000","last_clean":"2024-09-20T19:35:11.665538+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:35:11.665538+0000","last_undegraded":"2024-09-20T19:35:11.665538+0000","last_fullsized":"2024-09-20T19:35:11.665538+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.88259252799999999,"stat_sum":{"num_bytes":15312257,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":66,"num_read_kb":40205,"num_write":48,"num_write_kb":14898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":1356,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669712+0000","last_change":"2024-09-20T19:34:26.408655+0000","last_active":"2024-09-20T19:35:12.669712+0000","last_peered":"2024-09-20T19:35:12.669712+0000","last_clean":"2024-09-20T19:35:12.669712+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:35:12.669712+0000","last_undegraded":"2024-09-20T19:35:12.669712+0000","last_fullsized":"2024-09-20T19:35:12.669712+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:32:57.496961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.496961+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1270,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669569+0000","last_change":"2024-09-20T19:34:43.943794+0000","last_active":"2024-09-20T19:35:12.669569+0000","last_peered":"2024-09-20T19:35:12.669569+0000","last_clean":"2024-09-20T19:35:12.669569+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:35:12.669569+0000","last_undegraded":"2024-09-20T19:35:12.669569+0000","last_fullsized":"2024-09-20T19:35:12.669569+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:34:43.943661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.943661+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:50.854911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1531,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.666059+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:35:11.666059+0000","last_peered":"2024-09-20T19:35:11.666059+0000","last_clean":"2024-09-20T19:35:11.666059+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:35:11.666059+0000","last_undegraded":"2024-09-20T19:35:11.666059+0000","last_fullsized":"2024-09-20T19:35:11.666059+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1523,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.673646+0000","last_change":"2024-09-20T19:32:23.505615+0000","last_active":"2024-09-20T19:35:11.673646+0000","last_peered":"2024-09-20T19:35:11.673646+0000","last_clean":"2024-09-20T19:35:11.673646+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:35:11.673646+0000","last_undegraded":"2024-09-20T19:35:11.673646+0000","last_fullsized":"2024-09-20T19:35:11.673646+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.505475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.505475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.081565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1555,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651745+0000","last_change":"2024-09-20T19:35:04.500375+0000","last_active":"2024-09-20T19:35:11.651745+0000","last_peered":"2024-09-20T19:35:11.651745+0000","last_clean":"2024-09-20T19:35:11.651745+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:35:11.651745+0000","last_undegraded":"2024-09-20T19:35:11.651745+0000","last_fullsized":"2024-09-20T19:35:11.651745+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.500313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.786196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":3302,"reported_epoch":804,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:12.669946+0000","last_change":"2024-09-20T19:35:12.657522+0000","last_active":"2024-09-20T19:35:12.669946+0000","last_peered":"2024-09-20T19:35:12.669946+0000","last_clean":"2024-09-20T19:35:12.669946+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:35:12.669946+0000","last_undegraded":"2024-09-20T19:35:12.669946+0000","last_fullsized":"2024-09-20T19:35:12.669946+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:35:06.711015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:06.711015+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:31.635845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.000829647,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.5","version":"523'30","reported_seq":1774,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655509+0000","last_change":"2024-09-20T19:34:55.460785+0000","last_active":"2024-09-20T19:35:11.655509+0000","last_peered":"2024-09-20T19:35:11.655509+0000","last_clean":"2024-09-20T19:35:11.655509+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:35:11.655509+0000","last_undegraded":"2024-09-20T19:35:11.655509+0000","last_fullsized":"2024-09-20T19:35:11.655509+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:34:55.460588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.460588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.240558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":375,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661076+0000","last_change":"2024-09-20T19:33:43.357559+0000","last_active":"2024-09-20T19:35:11.661076+0000","last_peered":"2024-09-20T19:35:11.661076+0000","last_clean":"2024-09-20T19:35:11.661076+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:35:11.661076+0000","last_undegraded":"2024-09-20T19:35:11.661076+0000","last_fullsized":"2024-09-20T19:35:11.661076+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:32:57.875788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.875788+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:36.079972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1521,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662154+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:35:11.662154+0000","last_peered":"2024-09-20T19:35:11.662154+0000","last_clean":"2024-09-20T19:35:11.662154+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:35:11.662154+0000","last_undegraded":"2024-09-20T19:35:11.662154+0000","last_fullsized":"2024-09-20T19:35:11.662154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.324932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1580,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.666126+0000","last_change":"2024-09-20T19:34:48.152641+0000","last_active":"2024-09-20T19:35:11.666126+0000","last_peered":"2024-09-20T19:35:11.666126+0000","last_clean":"2024-09-20T19:35:11.666126+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:35:11.666126+0000","last_undegraded":"2024-09-20T19:35:11.666126+0000","last_fullsized":"2024-09-20T19:35:11.666126+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:48.152576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.152576+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.506442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1504,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665040+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:35:11.665040+0000","last_peered":"2024-09-20T19:35:11.665040+0000","last_clean":"2024-09-20T19:35:11.665040+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:35:11.665040+0000","last_undegraded":"2024-09-20T19:35:11.665040+0000","last_fullsized":"2024-09-20T19:35:11.665040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1533,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.673972+0000","last_change":"2024-09-20T19:34:15.087033+0000","last_active":"2024-09-20T19:35:11.673972+0000","last_peered":"2024-09-20T19:35:11.673972+0000","last_clean":"2024-09-20T19:35:11.673972+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:35:11.673972+0000","last_undegraded":"2024-09-20T19:35:11.673972+0000","last_fullsized":"2024-09-20T19:35:11.673972+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:15.086902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:15.086902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:28.923469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"781'375","reported_seq":3445,"reported_epoch":804,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:12.668751+0000","last_change":"2024-09-20T19:35:11.653841+0000","last_active":"2024-09-20T19:35:12.668751+0000","last_peered":"2024-09-20T19:35:12.668751+0000","last_clean":"2024-09-20T19:35:12.668751+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:35:12.668751+0000","last_undegraded":"2024-09-20T19:35:12.668751+0000","last_fullsized":"2024-09-20T19:35:12.668751+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'375","last_scrub_stamp":"2024-09-20T19:34:53.903004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.903004+0000","objects_scrubbed":2,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.092414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0004324309999999,"stat_sum":{"num_bytes":4908040,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":31,"num_write_kb":7400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.a","version":"523'35","reported_seq":1563,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655532+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:35:11.655532+0000","last_peered":"2024-09-20T19:35:11.655532+0000","last_clean":"2024-09-20T19:35:11.655532+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:35:11.655532+0000","last_undegraded":"2024-09-20T19:35:11.655532+0000","last_fullsized":"2024-09-20T19:35:11.655532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:11.545395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1549,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657760+0000","last_change":"2024-09-20T19:34:47.295550+0000","last_active":"2024-09-20T19:35:11.657760+0000","last_peered":"2024-09-20T19:35:11.657760+0000","last_clean":"2024-09-20T19:35:11.657760+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:35:11.657760+0000","last_undegraded":"2024-09-20T19:35:11.657760+0000","last_fullsized":"2024-09-20T19:35:11.657760+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:47.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.295412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:50.958066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":102,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656603+0000","last_change":"2024-09-20T19:34:26.390149+0000","last_active":"2024-09-20T19:35:11.656603+0000","last_peered":"2024-09-20T19:35:11.656603+0000","last_clean":"2024-09-20T19:35:11.656603+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:35:11.656603+0000","last_undegraded":"2024-09-20T19:35:11.656603+0000","last_fullsized":"2024-09-20T19:35:11.656603+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:29.295406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.666111+0000","last_change":"2024-09-20T19:33:43.369554+0000","last_active":"2024-09-20T19:35:11.666111+0000","last_peered":"2024-09-20T19:35:11.666111+0000","last_clean":"2024-09-20T19:35:11.666111+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:35:11.666111+0000","last_undegraded":"2024-09-20T19:35:11.666111+0000","last_fullsized":"2024-09-20T19:35:11.666111+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"803'1356","reported_seq":4685,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.669666+0000","last_change":"2024-09-20T19:35:11.669952+0000","last_active":"2024-09-20T19:35:11.669666+0000","last_peered":"2024-09-20T19:35:11.669666+0000","last_clean":"2024-09-20T19:35:11.669666+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:35:11.669666+0000","last_undegraded":"2024-09-20T19:35:11.669666+0000","last_fullsized":"2024-09-20T19:35:11.669666+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'1322","last_scrub_stamp":"2024-09-20T19:34:52.301450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:52.301450+0000","objects_scrubbed":6,"log_size":1356,"log_dups_size":0,"ondisk_log_size":1356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:58.973940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.016898699,"stat_sum":{"num_bytes":21092245,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":145,"num_read_kb":113319,"num_write":99,"num_write_kb":41236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.b","version":"523'15","reported_seq":1331,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.670281+0000","last_change":"2024-09-20T19:34:58.566556+0000","last_active":"2024-09-20T19:35:12.670281+0000","last_peered":"2024-09-20T19:35:12.670281+0000","last_clean":"2024-09-20T19:35:12.670281+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:35:12.670281+0000","last_undegraded":"2024-09-20T19:35:12.670281+0000","last_fullsized":"2024-09-20T19:35:12.670281+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.464807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.464807+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1556,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654183+0000","last_change":"2024-09-20T19:34:26.406213+0000","last_active":"2024-09-20T19:35:11.654183+0000","last_peered":"2024-09-20T19:35:11.654183+0000","last_clean":"2024-09-20T19:35:11.654183+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:35:11.654183+0000","last_undegraded":"2024-09-20T19:35:11.654183+0000","last_fullsized":"2024-09-20T19:35:11.654183+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.260232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1555,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655640+0000","last_change":"2024-09-20T19:34:44.295768+0000","last_active":"2024-09-20T19:35:11.655640+0000","last_peered":"2024-09-20T19:35:11.655640+0000","last_clean":"2024-09-20T19:35:11.655640+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:35:11.655640+0000","last_undegraded":"2024-09-20T19:35:11.655640+0000","last_fullsized":"2024-09-20T19:35:11.655640+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.295697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.295697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:44.355330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1332,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669851+0000","last_change":"2024-09-20T19:34:26.405297+0000","last_active":"2024-09-20T19:35:12.669851+0000","last_peered":"2024-09-20T19:35:12.669851+0000","last_clean":"2024-09-20T19:35:12.669851+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:35:12.669851+0000","last_undegraded":"2024-09-20T19:35:12.669851+0000","last_fullsized":"2024-09-20T19:35:12.669851+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:32:10.330783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.330783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"802'548","reported_seq":3877,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.652402+0000","last_change":"2024-09-20T19:35:10.650244+0000","last_active":"2024-09-20T19:35:11.652402+0000","last_peered":"2024-09-20T19:35:11.652402+0000","last_clean":"2024-09-20T19:35:11.652402+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:35:11.652402+0000","last_undegraded":"2024-09-20T19:35:11.652402+0000","last_fullsized":"2024-09-20T19:35:11.652402+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"724'501","last_scrub_stamp":"2024-09-20T19:33:52.436716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:52.436716+0000","objects_scrubbed":2,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010747090000000001,"stat_sum":{"num_bytes":5664355,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":32,"num_read_kb":30475,"num_write":14,"num_write_kb":2597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.8","version":"523'27","reported_seq":1575,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.660559+0000","last_change":"2024-09-20T19:34:47.083599+0000","last_active":"2024-09-20T19:35:11.660559+0000","last_peered":"2024-09-20T19:35:11.660559+0000","last_clean":"2024-09-20T19:35:11.660559+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:35:11.660559+0000","last_undegraded":"2024-09-20T19:35:11.660559+0000","last_fullsized":"2024-09-20T19:35:11.660559+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:47.083533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.083533+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.559644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":106,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657926+0000","last_change":"2024-09-20T19:34:26.404678+0000","last_active":"2024-09-20T19:35:11.657926+0000","last_peered":"2024-09-20T19:35:11.657926+0000","last_clean":"2024-09-20T19:35:11.657926+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:35:11.657926+0000","last_undegraded":"2024-09-20T19:35:11.657926+0000","last_fullsized":"2024-09-20T19:35:11.657926+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:35.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.568627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.252995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1740,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665709+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:35:11.665709+0000","last_peered":"2024-09-20T19:35:11.665709+0000","last_clean":"2024-09-20T19:35:11.665709+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:35:11.665709+0000","last_undegraded":"2024-09-20T19:35:11.665709+0000","last_fullsized":"2024-09-20T19:35:11.665709+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1527,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.660775+0000","last_change":"2024-09-20T19:34:53.009990+0000","last_active":"2024-09-20T19:35:11.660775+0000","last_peered":"2024-09-20T19:35:11.660775+0000","last_clean":"2024-09-20T19:35:11.660775+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:35:11.660775+0000","last_undegraded":"2024-09-20T19:35:11.660775+0000","last_fullsized":"2024-09-20T19:35:11.660775+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:53.009934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.009934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:06.466332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":3445,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662271+0000","last_change":"2024-09-20T19:35:11.662634+0000","last_active":"2024-09-20T19:35:11.662271+0000","last_peered":"2024-09-20T19:35:11.662271+0000","last_clean":"2024-09-20T19:35:11.662271+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:35:11.662271+0000","last_undegraded":"2024-09-20T19:35:11.662271+0000","last_fullsized":"2024-09-20T19:35:11.662271+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:35:07.477259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.477259+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:34.019142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007491358,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.9","version":"523'18","reported_seq":1396,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615541+0000","last_change":"2024-09-20T19:33:43.369566+0000","last_active":"2024-09-20T19:35:08.615541+0000","last_peered":"2024-09-20T19:35:08.615541+0000","last_clean":"2024-09-20T19:35:08.615541+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:35:08.615541+0000","last_undegraded":"2024-09-20T19:35:08.615541+0000","last_fullsized":"2024-09-20T19:35:08.615541+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:27.634845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1384,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617335+0000","last_change":"2024-09-20T19:34:08.546458+0000","last_active":"2024-09-20T19:35:08.617335+0000","last_peered":"2024-09-20T19:35:08.617335+0000","last_clean":"2024-09-20T19:35:08.617335+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:35:08.617335+0000","last_undegraded":"2024-09-20T19:35:08.617335+0000","last_fullsized":"2024-09-20T19:35:08.617335+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:34:08.546356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.546356+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.318383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1533,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654491+0000","last_change":"2024-09-20T19:33:06.111877+0000","last_active":"2024-09-20T19:35:11.654491+0000","last_peered":"2024-09-20T19:35:11.654491+0000","last_clean":"2024-09-20T19:35:11.654491+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:35:11.654491+0000","last_undegraded":"2024-09-20T19:35:11.654491+0000","last_fullsized":"2024-09-20T19:35:11.654491+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.111753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.111753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:19.125065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1014,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663500+0000","last_change":"2024-09-20T19:33:43.447359+0000","last_active":"2024-09-20T19:35:11.663500+0000","last_peered":"2024-09-20T19:35:11.663500+0000","last_clean":"2024-09-20T19:35:11.663500+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:35:11.663500+0000","last_undegraded":"2024-09-20T19:35:11.663500+0000","last_fullsized":"2024-09-20T19:35:11.663500+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"800'1162","reported_seq":4859,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.669577+0000","last_change":"2024-09-20T19:35:11.507270+0000","last_active":"2024-09-20T19:35:11.669577+0000","last_peered":"2024-09-20T19:35:11.669577+0000","last_clean":"2024-09-20T19:35:11.669577+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:35:11.669577+0000","last_undegraded":"2024-09-20T19:35:11.669577+0000","last_fullsized":"2024-09-20T19:35:11.669577+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'1072","last_scrub_stamp":"2024-09-20T19:34:09.034773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:09.034773+0000","objects_scrubbed":12,"log_size":1162,"log_dups_size":0,"ondisk_log_size":1162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.097313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.84956808399999995,"stat_sum":{"num_bytes":22354309,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":136,"num_read_kb":79475,"num_write":144,"num_write_kb":50886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"}]},{"pgid":"4.16","version":"523'15","reported_seq":1333,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669445+0000","last_change":"2024-09-20T19:33:43.390311+0000","last_active":"2024-09-20T19:35:12.669445+0000","last_peered":"2024-09-20T19:35:12.669445+0000","last_clean":"2024-09-20T19:35:12.669445+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:35:12.669445+0000","last_undegraded":"2024-09-20T19:35:12.669445+0000","last_fullsized":"2024-09-20T19:35:12.669445+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1565,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654659+0000","last_change":"2024-09-20T19:34:54.365631+0000","last_active":"2024-09-20T19:35:11.654659+0000","last_peered":"2024-09-20T19:35:11.654659+0000","last_clean":"2024-09-20T19:35:11.654659+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:35:11.654659+0000","last_undegraded":"2024-09-20T19:35:11.654659+0000","last_fullsized":"2024-09-20T19:35:11.654659+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.365568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.365568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:55.360414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1009,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.652611+0000","last_change":"2024-09-20T19:34:10.220726+0000","last_active":"2024-09-20T19:35:11.652611+0000","last_peered":"2024-09-20T19:35:11.652611+0000","last_clean":"2024-09-20T19:35:11.652611+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:35:11.652611+0000","last_undegraded":"2024-09-20T19:35:11.652611+0000","last_fullsized":"2024-09-20T19:35:11.652611+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:10.220603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.220603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.004978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1521,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.657150+0000","last_change":"2024-09-20T19:33:02.129851+0000","last_active":"2024-09-20T19:35:11.657150+0000","last_peered":"2024-09-20T19:35:11.657150+0000","last_clean":"2024-09-20T19:35:11.657150+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:35:11.657150+0000","last_undegraded":"2024-09-20T19:35:11.657150+0000","last_fullsized":"2024-09-20T19:35:11.657150+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.129712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.129712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.209559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"803'1163","reported_seq":4578,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.674506+0000","last_change":"2024-09-20T19:35:11.674844+0000","last_active":"2024-09-20T19:35:11.674506+0000","last_peered":"2024-09-20T19:35:11.674506+0000","last_clean":"2024-09-20T19:35:11.674506+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:35:11.674506+0000","last_undegraded":"2024-09-20T19:35:11.674506+0000","last_fullsized":"2024-09-20T19:35:11.674506+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1069","last_scrub_stamp":"2024-09-20T19:34:11.169527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.169527+0000","objects_scrubbed":7,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:24.808484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017608677,"stat_sum":{"num_bytes":26298638,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":198,"num_read_kb":134074,"num_write":116,"num_write_kb":47684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.11","version":"0'0","reported_seq":1531,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655327+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:35:11.655327+0000","last_peered":"2024-09-20T19:35:11.655327+0000","last_clean":"2024-09-20T19:35:11.655327+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:35:11.655327+0000","last_undegraded":"2024-09-20T19:35:11.655327+0000","last_fullsized":"2024-09-20T19:35:11.655327+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.149622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1182,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665139+0000","last_change":"2024-09-20T19:33:43.464473+0000","last_active":"2024-09-20T19:35:11.665139+0000","last_peered":"2024-09-20T19:35:11.665139+0000","last_clean":"2024-09-20T19:35:11.665139+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:35:11.665139+0000","last_undegraded":"2024-09-20T19:35:11.665139+0000","last_fullsized":"2024-09-20T19:35:11.665139+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1538,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663520+0000","last_change":"2024-09-20T19:34:08.991943+0000","last_active":"2024-09-20T19:35:11.663520+0000","last_peered":"2024-09-20T19:35:11.663520+0000","last_clean":"2024-09-20T19:35:11.663520+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:35:11.663520+0000","last_undegraded":"2024-09-20T19:35:11.663520+0000","last_fullsized":"2024-09-20T19:35:11.663520+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:08.991819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.991819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:24.368397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1510,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665795+0000","last_change":"2024-09-20T19:34:44.066655+0000","last_active":"2024-09-20T19:35:11.665795+0000","last_peered":"2024-09-20T19:35:11.665795+0000","last_clean":"2024-09-20T19:35:11.665795+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:35:11.665795+0000","last_undegraded":"2024-09-20T19:35:11.665795+0000","last_fullsized":"2024-09-20T19:35:11.665795+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:00.702253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"802'656","reported_seq":3911,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.668065+0000","last_change":"2024-09-20T19:35:11.513073+0000","last_active":"2024-09-20T19:35:11.668065+0000","last_peered":"2024-09-20T19:35:11.668065+0000","last_clean":"2024-09-20T19:35:11.668065+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:35:11.668065+0000","last_undegraded":"2024-09-20T19:35:11.668065+0000","last_fullsized":"2024-09-20T19:35:11.668065+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.87915046500000005,"stat_sum":{"num_bytes":15732540,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":24868,"num_write":40,"num_write_kb":16825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.14","version":"523'20","reported_seq":1199,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651957+0000","last_change":"2024-09-20T19:33:43.463851+0000","last_active":"2024-09-20T19:35:11.651957+0000","last_peered":"2024-09-20T19:35:11.651957+0000","last_clean":"2024-09-20T19:35:11.651957+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:35:11.651957+0000","last_undegraded":"2024-09-20T19:35:11.651957+0000","last_fullsized":"2024-09-20T19:35:11.651957+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.281258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1546,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661197+0000","last_change":"2024-09-20T19:34:11.058115+0000","last_active":"2024-09-20T19:35:11.661197+0000","last_peered":"2024-09-20T19:35:11.661197+0000","last_clean":"2024-09-20T19:35:11.661197+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:35:11.661197+0000","last_undegraded":"2024-09-20T19:35:11.661197+0000","last_fullsized":"2024-09-20T19:35:11.661197+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:11.058052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.058052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.593298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1741,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.662860+0000","last_change":"2024-09-20T19:32:11.594974+0000","last_active":"2024-09-20T19:35:11.662860+0000","last_peered":"2024-09-20T19:35:11.662860+0000","last_clean":"2024-09-20T19:35:11.662860+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:35:11.662860+0000","last_undegraded":"2024-09-20T19:35:11.662860+0000","last_fullsized":"2024-09-20T19:35:11.662860+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:11.594902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.594902+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.031023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1305,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.670172+0000","last_change":"2024-09-20T19:33:43.374566+0000","last_active":"2024-09-20T19:35:12.670172+0000","last_peered":"2024-09-20T19:35:12.670172+0000","last_clean":"2024-09-20T19:35:12.670172+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:35:12.670172+0000","last_undegraded":"2024-09-20T19:35:12.670172+0000","last_fullsized":"2024-09-20T19:35:12.670172+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"795'511","reported_seq":3569,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617254+0000","last_change":"2024-09-20T19:35:06.610324+0000","last_active":"2024-09-20T19:35:08.617254+0000","last_peered":"2024-09-20T19:35:08.617254+0000","last_clean":"2024-09-20T19:35:08.617254+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:35:08.617254+0000","last_undegraded":"2024-09-20T19:35:08.617254+0000","last_fullsized":"2024-09-20T19:35:08.617254+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'490","last_scrub_stamp":"2024-09-20T19:34:46.528129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.528129+0000","objects_scrubbed":6,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.439744+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0028316729999999998,"stat_sum":{"num_bytes":18367932,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":103,"num_read_kb":64664,"num_write":68,"num_write_kb":25480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.15","version":"523'15","reported_seq":1584,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663006+0000","last_change":"2024-09-20T19:34:08.010996+0000","last_active":"2024-09-20T19:35:11.663006+0000","last_peered":"2024-09-20T19:35:11.663006+0000","last_clean":"2024-09-20T19:35:11.663006+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:35:11.663006+0000","last_undegraded":"2024-09-20T19:35:11.663006+0000","last_fullsized":"2024-09-20T19:35:11.663006+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:08.010906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.010906+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:15.302000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1278,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.670414+0000","last_change":"2024-09-20T19:33:43.369959+0000","last_active":"2024-09-20T19:35:12.670414+0000","last_peered":"2024-09-20T19:35:12.670414+0000","last_clean":"2024-09-20T19:35:12.670414+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:35:12.670414+0000","last_undegraded":"2024-09-20T19:35:12.670414+0000","last_fullsized":"2024-09-20T19:35:12.670414+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":190,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654029+0000","last_change":"2024-09-20T19:34:26.406147+0000","last_active":"2024-09-20T19:35:11.654029+0000","last_peered":"2024-09-20T19:35:11.654029+0000","last_clean":"2024-09-20T19:35:11.654029+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:35:11.654029+0000","last_undegraded":"2024-09-20T19:35:11.654029+0000","last_fullsized":"2024-09-20T19:35:11.654029+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'15","last_scrub_stamp":"2024-09-20T19:33:53.222431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.222431+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:21.534717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1353,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617150+0000","last_change":"2024-09-20T19:34:55.537273+0000","last_active":"2024-09-20T19:35:08.617150+0000","last_peered":"2024-09-20T19:35:08.617150+0000","last_clean":"2024-09-20T19:35:08.617150+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:35:08.617150+0000","last_undegraded":"2024-09-20T19:35:08.617150+0000","last_fullsized":"2024-09-20T19:35:08.617150+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.537217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.537217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:13.844779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"800'518","reported_seq":3874,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.664983+0000","last_change":"2024-09-20T19:35:11.512892+0000","last_active":"2024-09-20T19:35:11.664983+0000","last_peered":"2024-09-20T19:35:11.664983+0000","last_clean":"2024-09-20T19:35:11.664983+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:35:11.664983+0000","last_undegraded":"2024-09-20T19:35:11.664983+0000","last_fullsized":"2024-09-20T19:35:11.664983+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.85845876399999999,"stat_sum":{"num_bytes":10850446,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":40651,"num_write":21,"num_write_kb":10100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"}]},{"pgid":"4.12","version":"523'15","reported_seq":1553,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653535+0000","last_change":"2024-09-20T19:32:31.080255+0000","last_active":"2024-09-20T19:35:11.653535+0000","last_peered":"2024-09-20T19:35:11.653535+0000","last_clean":"2024-09-20T19:35:11.653535+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:35:11.653535+0000","last_undegraded":"2024-09-20T19:35:11.653535+0000","last_fullsized":"2024-09-20T19:35:11.653535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:31.080102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.080102+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:38.313239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1530,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654886+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:35:11.654886+0000","last_peered":"2024-09-20T19:35:11.654886+0000","last_clean":"2024-09-20T19:35:11.654886+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:35:11.654886+0000","last_undegraded":"2024-09-20T19:35:11.654886+0000","last_fullsized":"2024-09-20T19:35:11.654886+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.355670+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1033,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.668483+0000","last_change":"2024-09-20T19:33:43.447329+0000","last_active":"2024-09-20T19:35:12.668483+0000","last_peered":"2024-09-20T19:35:12.668483+0000","last_clean":"2024-09-20T19:35:12.668483+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:35:12.668483+0000","last_undegraded":"2024-09-20T19:35:12.668483+0000","last_fullsized":"2024-09-20T19:35:12.668483+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1288,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.668415+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:35:12.668415+0000","last_peered":"2024-09-20T19:35:12.668415+0000","last_clean":"2024-09-20T19:35:12.668415+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:35:12.668415+0000","last_undegraded":"2024-09-20T19:35:12.668415+0000","last_fullsized":"2024-09-20T19:35:12.668415+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"803'700","reported_seq":4012,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.661581+0000","last_change":"2024-09-20T19:35:11.661817+0000","last_active":"2024-09-20T19:35:11.661581+0000","last_peered":"2024-09-20T19:35:11.661581+0000","last_clean":"2024-09-20T19:35:11.661581+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:35:11.661581+0000","last_undegraded":"2024-09-20T19:35:11.661581+0000","last_fullsized":"2024-09-20T19:35:11.661581+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"744'658","last_scrub_stamp":"2024-09-20T19:34:13.122172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.122172+0000","objects_scrubbed":3,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.093172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0091962450000000005,"stat_sum":{"num_bytes":8451524,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":37169,"num_write":46,"num_write_kb":13747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"5"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.13","version":"523'19","reported_seq":1361,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.615965+0000","last_change":"2024-09-20T19:34:41.550328+0000","last_active":"2024-09-20T19:35:08.615965+0000","last_peered":"2024-09-20T19:35:08.615965+0000","last_clean":"2024-09-20T19:35:08.615965+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:35:08.615965+0000","last_undegraded":"2024-09-20T19:35:08.615965+0000","last_fullsized":"2024-09-20T19:35:08.615965+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:34:41.550266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.550266+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.918942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1543,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654426+0000","last_change":"2024-09-20T19:31:20.343942+0000","last_active":"2024-09-20T19:35:11.654426+0000","last_peered":"2024-09-20T19:35:11.654426+0000","last_clean":"2024-09-20T19:35:11.654426+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:35:11.654426+0000","last_undegraded":"2024-09-20T19:35:11.654426+0000","last_fullsized":"2024-09-20T19:35:11.654426+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:20.343882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.343882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:34.329982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1545,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653441+0000","last_change":"2024-09-20T19:34:59.381856+0000","last_active":"2024-09-20T19:35:11.653441+0000","last_peered":"2024-09-20T19:35:11.653441+0000","last_clean":"2024-09-20T19:35:11.653441+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:35:11.653441+0000","last_undegraded":"2024-09-20T19:35:11.653441+0000","last_fullsized":"2024-09-20T19:35:11.653441+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.381728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.381728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:24.871997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1037,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669303+0000","last_change":"2024-09-20T19:34:54.942515+0000","last_active":"2024-09-20T19:35:12.669303+0000","last_peered":"2024-09-20T19:35:12.669303+0000","last_clean":"2024-09-20T19:35:12.669303+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:35:12.669303+0000","last_undegraded":"2024-09-20T19:35:12.669303+0000","last_fullsized":"2024-09-20T19:35:12.669303+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.942431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.942431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"791'482","reported_seq":3831,"reported_epoch":804,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:12.668820+0000","last_change":"2024-09-20T19:35:12.656295+0000","last_active":"2024-09-20T19:35:12.668820+0000","last_peered":"2024-09-20T19:35:12.668820+0000","last_clean":"2024-09-20T19:35:12.668820+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:35:12.668820+0000","last_undegraded":"2024-09-20T19:35:12.668820+0000","last_fullsized":"2024-09-20T19:35:12.668820+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"777'478","last_scrub_stamp":"2024-09-20T19:34:51.901991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.901991+0000","objects_scrubbed":2,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:56.729961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0001242800000001,"stat_sum":{"num_bytes":5277827,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":35,"num_write_kb":10121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.10","version":"523'8","reported_seq":1191,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.649049+0000","last_change":"2024-09-20T19:34:43.272725+0000","last_active":"2024-09-20T19:35:11.649049+0000","last_peered":"2024-09-20T19:35:11.649049+0000","last_clean":"2024-09-20T19:35:11.649049+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:35:11.649049+0000","last_undegraded":"2024-09-20T19:35:11.649049+0000","last_fullsized":"2024-09-20T19:35:11.649049+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:34:43.272551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.272551+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:51.459112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1526,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663661+0000","last_change":"2024-09-20T19:34:26.399743+0000","last_active":"2024-09-20T19:35:11.663661+0000","last_peered":"2024-09-20T19:35:11.663661+0000","last_clean":"2024-09-20T19:35:11.663661+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:35:11.663661+0000","last_undegraded":"2024-09-20T19:35:11.663661+0000","last_fullsized":"2024-09-20T19:35:11.663661+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:03.325428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:03.325428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.020789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1523,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654477+0000","last_change":"2024-09-20T19:35:03.419683+0000","last_active":"2024-09-20T19:35:11.654477+0000","last_peered":"2024-09-20T19:35:11.654477+0000","last_clean":"2024-09-20T19:35:11.654477+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:35:11.654477+0000","last_undegraded":"2024-09-20T19:35:11.654477+0000","last_fullsized":"2024-09-20T19:35:11.654477+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:03.419628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.419628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.847292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1502,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663981+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:35:11.663981+0000","last_peered":"2024-09-20T19:35:11.663981+0000","last_clean":"2024-09-20T19:35:11.663981+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:35:11.663981+0000","last_undegraded":"2024-09-20T19:35:11.663981+0000","last_fullsized":"2024-09-20T19:35:11.663981+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":3390,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.669158+0000","last_change":"2024-09-20T19:35:11.669209+0000","last_active":"2024-09-20T19:35:11.669158+0000","last_peered":"2024-09-20T19:35:11.669158+0000","last_clean":"2024-09-20T19:35:11.669158+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:35:11.669158+0000","last_undegraded":"2024-09-20T19:35:11.669158+0000","last_fullsized":"2024-09-20T19:35:11.669158+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:29:00.472909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.472909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014264565999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.11","version":"523'19","reported_seq":398,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.666161+0000","last_change":"2024-09-20T19:33:43.373453+0000","last_active":"2024-09-20T19:35:11.666161+0000","last_peered":"2024-09-20T19:35:11.666161+0000","last_clean":"2024-09-20T19:35:11.666161+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:35:11.666161+0000","last_undegraded":"2024-09-20T19:35:11.666161+0000","last_fullsized":"2024-09-20T19:35:11.666161+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1520,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656302+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:35:11.656302+0000","last_peered":"2024-09-20T19:35:11.656302+0000","last_clean":"2024-09-20T19:35:11.656302+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:35:11.656302+0000","last_undegraded":"2024-09-20T19:35:11.656302+0000","last_fullsized":"2024-09-20T19:35:11.656302+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1522,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.663906+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:35:11.663906+0000","last_peered":"2024-09-20T19:35:11.663906+0000","last_clean":"2024-09-20T19:35:11.663906+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:35:11.663906+0000","last_undegraded":"2024-09-20T19:35:11.663906+0000","last_fullsized":"2024-09-20T19:35:11.663906+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1045,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669177+0000","last_change":"2024-09-20T19:33:43.367812+0000","last_active":"2024-09-20T19:35:12.669177+0000","last_peered":"2024-09-20T19:35:12.669177+0000","last_clean":"2024-09-20T19:35:12.669177+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:35:12.669177+0000","last_undegraded":"2024-09-20T19:35:12.669177+0000","last_fullsized":"2024-09-20T19:35:12.669177+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.767876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.767876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"802'624","reported_seq":3969,"reported_epoch":803,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:11.652169+0000","last_change":"2024-09-20T19:35:11.642587+0000","last_active":"2024-09-20T19:35:11.652169+0000","last_peered":"2024-09-20T19:35:11.652169+0000","last_clean":"2024-09-20T19:35:11.652169+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:35:11.652169+0000","last_undegraded":"2024-09-20T19:35:11.652169+0000","last_fullsized":"2024-09-20T19:35:11.652169+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"721'544","last_scrub_stamp":"2024-09-20T19:34:00.387013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:00.387013+0000","objects_scrubbed":3,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.99103446100000003,"stat_sum":{"num_bytes":23511699,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":29,"num_read_kb":26746,"num_write":31,"num_write_kb":16313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1c","version":"77'1","reported_seq":1508,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.665098+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:35:11.665098+0000","last_peered":"2024-09-20T19:35:11.665098+0000","last_clean":"2024-09-20T19:35:11.665098+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:35:11.665098+0000","last_undegraded":"2024-09-20T19:35:11.665098+0000","last_fullsized":"2024-09-20T19:35:11.665098+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1547,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.660613+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:35:11.660613+0000","last_peered":"2024-09-20T19:35:11.660613+0000","last_clean":"2024-09-20T19:35:11.660613+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:35:11.660613+0000","last_undegraded":"2024-09-20T19:35:11.660613+0000","last_fullsized":"2024-09-20T19:35:11.660613+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.363084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":1332,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669574+0000","last_change":"2024-09-20T19:35:05.740151+0000","last_active":"2024-09-20T19:35:12.669574+0000","last_peered":"2024-09-20T19:35:12.669574+0000","last_clean":"2024-09-20T19:35:12.669574+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:35:12.669574+0000","last_undegraded":"2024-09-20T19:35:12.669574+0000","last_fullsized":"2024-09-20T19:35:12.669574+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:35:05.739968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:05.739968+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:18.343465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1745,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656180+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:35:11.656180+0000","last_peered":"2024-09-20T19:35:11.656180+0000","last_clean":"2024-09-20T19:35:11.656180+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:35:11.656180+0000","last_undegraded":"2024-09-20T19:35:11.656180+0000","last_fullsized":"2024-09-20T19:35:11.656180+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.435375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"801'877","reported_seq":4401,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.672136+0000","last_change":"2024-09-20T19:35:11.672188+0000","last_active":"2024-09-20T19:35:11.672136+0000","last_peered":"2024-09-20T19:35:11.672136+0000","last_clean":"2024-09-20T19:35:11.672136+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:35:11.672136+0000","last_undegraded":"2024-09-20T19:35:11.672136+0000","last_fullsized":"2024-09-20T19:35:11.672136+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"722'858","last_scrub_stamp":"2024-09-20T19:34:06.076547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.076547+0000","objects_scrubbed":3,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.156736+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0045942530000000004,"stat_sum":{"num_bytes":7989887,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":8232,"num_write":13,"num_write_kb":3293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":1522,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.651949+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:35:11.651949+0000","last_peered":"2024-09-20T19:35:11.651949+0000","last_clean":"2024-09-20T19:35:11.651949+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:35:11.651949+0000","last_undegraded":"2024-09-20T19:35:11.651949+0000","last_fullsized":"2024-09-20T19:35:11.651949+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:18.112487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":360,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655186+0000","last_change":"2024-09-20T19:34:43.489833+0000","last_active":"2024-09-20T19:35:11.655186+0000","last_peered":"2024-09-20T19:35:11.655186+0000","last_clean":"2024-09-20T19:35:11.655186+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:35:11.655186+0000","last_undegraded":"2024-09-20T19:35:11.655186+0000","last_fullsized":"2024-09-20T19:35:11.655186+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:34:43.489670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.489670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.767313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1570,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656167+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:35:11.656167+0000","last_peered":"2024-09-20T19:35:11.656167+0000","last_clean":"2024-09-20T19:35:11.656167+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:35:11.656167+0000","last_undegraded":"2024-09-20T19:35:11.656167+0000","last_fullsized":"2024-09-20T19:35:11.656167+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:09.474818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1317,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669989+0000","last_change":"2024-09-20T19:34:26.392917+0000","last_active":"2024-09-20T19:35:12.669989+0000","last_peered":"2024-09-20T19:35:12.669989+0000","last_clean":"2024-09-20T19:35:12.669989+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:35:12.669989+0000","last_undegraded":"2024-09-20T19:35:12.669989+0000","last_fullsized":"2024-09-20T19:35:12.669989+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.455533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.455533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"802'614","reported_seq":3759,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.668886+0000","last_change":"2024-09-20T19:35:11.668949+0000","last_active":"2024-09-20T19:35:11.668886+0000","last_peered":"2024-09-20T19:35:11.668886+0000","last_clean":"2024-09-20T19:35:11.668886+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:35:11.668886+0000","last_undegraded":"2024-09-20T19:35:11.668886+0000","last_fullsized":"2024-09-20T19:35:11.668886+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'580","last_scrub_stamp":"2024-09-20T19:34:41.516352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.516352+0000","objects_scrubbed":4,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:46.303278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012865783,"stat_sum":{"num_bytes":17731636,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":7581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"130","length":"1"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":1335,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.617126+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:35:08.617126+0000","last_peered":"2024-09-20T19:35:08.617126+0000","last_clean":"2024-09-20T19:35:08.617126+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:35:08.617126+0000","last_undegraded":"2024-09-20T19:35:08.617126+0000","last_fullsized":"2024-09-20T19:35:08.617126+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:24.610221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1530,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.655684+0000","last_change":"2024-09-20T19:32:15.866641+0000","last_active":"2024-09-20T19:35:11.655684+0000","last_peered":"2024-09-20T19:35:11.655684+0000","last_clean":"2024-09-20T19:35:11.655684+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:35:11.655684+0000","last_undegraded":"2024-09-20T19:35:11.655684+0000","last_fullsized":"2024-09-20T19:35:11.655684+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.866572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.866572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":118,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.656947+0000","last_change":"2024-09-20T19:34:26.411567+0000","last_active":"2024-09-20T19:35:11.656947+0000","last_peered":"2024-09-20T19:35:11.656947+0000","last_clean":"2024-09-20T19:35:11.656947+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:35:11.656947+0000","last_undegraded":"2024-09-20T19:35:11.656947+0000","last_fullsized":"2024-09-20T19:35:11.656947+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:33:05.356320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:05.356320+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:37.327595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1533,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.653907+0000","last_change":"2024-09-20T19:32:16.075391+0000","last_active":"2024-09-20T19:35:11.653907+0000","last_peered":"2024-09-20T19:35:11.653907+0000","last_clean":"2024-09-20T19:35:11.653907+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:35:11.653907+0000","last_undegraded":"2024-09-20T19:35:11.653907+0000","last_fullsized":"2024-09-20T19:35:11.653907+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.075336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.075336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.344567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":3589,"reported_epoch":803,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:11.651784+0000","last_change":"2024-09-20T19:35:11.651757+0000","last_active":"2024-09-20T19:35:11.651784+0000","last_peered":"2024-09-20T19:35:11.651784+0000","last_clean":"2024-09-20T19:35:11.651784+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:35:11.651784+0000","last_undegraded":"2024-09-20T19:35:11.651784+0000","last_fullsized":"2024-09-20T19:35:11.651784+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:33:59.395301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.395301+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.99170169100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12f","length":"2"},{"start":"134","length":"2"},{"start":"139","length":"1"},{"start":"13b","length":"3"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1f","version":"0'0","reported_seq":170,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-09-20T19:35:12.669304+0000","last_change":"2024-09-20T19:33:58.560877+0000","last_active":"2024-09-20T19:35:12.669304+0000","last_peered":"2024-09-20T19:35:12.669304+0000","last_clean":"2024-09-20T19:35:12.669304+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:35:12.669304+0000","last_undegraded":"2024-09-20T19:35:12.669304+0000","last_fullsized":"2024-09-20T19:35:12.669304+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:58.560821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:58.560821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1015,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.664012+0000","last_change":"2024-09-20T19:34:51.148618+0000","last_active":"2024-09-20T19:35:11.664012+0000","last_peered":"2024-09-20T19:35:11.664012+0000","last_clean":"2024-09-20T19:35:11.664012+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:35:11.664012+0000","last_undegraded":"2024-09-20T19:35:11.664012+0000","last_fullsized":"2024-09-20T19:35:11.664012+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.148494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.148494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.914652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":403,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-20T19:35:11.654034+0000","last_change":"2024-09-20T19:33:43.461467+0000","last_active":"2024-09-20T19:35:11.654034+0000","last_peered":"2024-09-20T19:35:11.654034+0000","last_clean":"2024-09-20T19:35:11.654034+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:35:11.654034+0000","last_undegraded":"2024-09-20T19:35:11.654034+0000","last_fullsized":"2024-09-20T19:35:11.654034+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:10.800347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.800347+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.311263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1348,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-20T19:35:08.616268+0000","last_change":"2024-09-20T19:34:44.512596+0000","last_active":"2024-09-20T19:35:08.616268+0000","last_peered":"2024-09-20T19:35:08.616268+0000","last_clean":"2024-09-20T19:35:08.616268+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:35:08.616268+0000","last_undegraded":"2024-09-20T19:35:08.616268+0000","last_fullsized":"2024-09-20T19:35:08.616268+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.512457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.512457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:51.089920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":308084206,"num_objects":144,"num_object_clones":99,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":11,"num_read":1350,"num_read_kb":955599,"num_write":1023,"num_write_kb":377194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":777482240,"data_stored":1044757276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":22317,"ondisk_log_size":22317,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1006,"num_read_kb":1001,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907158,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175584,"kb_used_data":126348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93569568,"statfs":{"total":95995035648,"available":95815237632,"internally_reserved":0,"allocated":129380352,"data_stored":167339737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234209,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153676,"kb_used_data":90976,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93591476,"statfs":{"total":95995035648,"available":95837671424,"internally_reserved":0,"allocated":93159424,"data_stored":123155622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561259,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144428,"kb_used_data":86872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93600724,"statfs":{"total":95995035648,"available":95847141376,"internally_reserved":0,"allocated":88956928,"data_stored":118687472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928569,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119972,"kb_used_data":75288,"kb_used_omap":35,"kb_used_meta":44444,"kb_avail":93625180,"statfs":{"total":95995035648,"available":95872184320,"internally_reserved":0,"allocated":77094912,"data_stored":101141886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":45511601},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378895,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168744,"kb_used_data":111188,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93576408,"statfs":{"total":95995035648,"available":95822241792,"internally_reserved":0,"allocated":113856512,"data_stored":137721357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145096,"kb_used_data":95708,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93600056,"statfs":{"total":95995035648,"available":95846457344,"internally_reserved":0,"allocated":98004992,"data_stored":143511103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648062,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":151800,"kb_used_data":118684,"kb_used_omap":49,"kb_used_meta":33038,"kb_avail":93593352,"statfs":{"total":95995035648,"available":95839592448,"internally_reserved":0,"allocated":121532416,"data_stored":154180878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":33831465},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120659968,"data_stored":153694041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97144832,"data_stored":145921358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112726016,"data_stored":139869175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77336576,"data_stored":106602947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87498752,"data_stored":117609996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91688960,"data_stored":122085832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":128188416,"data_stored":166525596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:35:15.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:35:15.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:16.125 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:16.149 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:16.252 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:16.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:16.360 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:16.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:16.388 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[32075]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[32075]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[32075]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[32075]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[22706]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[22706]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[22706]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.389 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:16 smithi042 bash[22706]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.399 INFO:tasks.rados.rados.0.smithi042.stdout:2830: done (3 left) 2024-09-20T19:35:16.399 INFO:tasks.rados.rados.0.smithi042.stdout:2831: read oid 46 snap 286 2024-09-20T19:35:16.399 INFO:tasks.rados.rados.0.smithi042.stdout:2831: expect deleted 2024-09-20T19:35:16.399 INFO:tasks.rados.rados.0.smithi042.stdout:2832: snap_remove snap 323 2024-09-20T19:35:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:16.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:16.540 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:16.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:16.671 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:16.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:16.801 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:16 smithi162 bash[24557]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:16 smithi162 bash[24557]: audit 2024-09-20T19:35:15.024381+0000 mgr.y (mgr.24491) 649 : audit [DBG] from='client.25823 v1:172.21.15.42:0/79232012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:16.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:16 smithi162 bash[24557]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:16 smithi162 bash[24557]: cluster 2024-09-20T19:35:15.380027+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2024-09-20T19:35:16.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:16.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:16.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:17.036 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:17.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:17.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:17.296 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:17.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:17.377 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:17.400 INFO:tasks.rados.rados.0.smithi042.stdout:2832: done (4 left) 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2833: delete oid 49 current snap is 325 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2828: finishing copy_from to smithi04268048-21 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2828: got expected ENOENT (src dne) 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2828: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2827: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2827: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2827: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 882 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2829: finishing copy_from racing read to smithi04268048-33 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:2829: finishing copy_from to smithi04268048-33 2024-09-20T19:35:17.401 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1144 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2827: done (4 left) 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2828: done (3 left) 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2829: done (2 left) 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2831: done (1 left) 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2833: done (0 left) 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2834: write (excl) oid 46 current snap is 325 2024-09-20T19:35:17.425 INFO:tasks.rados.rados.0.smithi042.stdout:2834: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-20T19:35:17.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:17.433 INFO:tasks.rados.rados.0.smithi042.stdout:2834: writing smithi04268048-46 from 622123 to 1369274 tid 1 2024-09-20T19:35:17.441 INFO:tasks.rados.rados.0.smithi042.stdout:2834: writing smithi04268048-46 from 2007162 to 2622123 tid 2 2024-09-20T19:35:17.441 INFO:tasks.rados.rados.0.smithi042.stdout:2835: delete oid 34 current snap is 325 2024-09-20T19:35:17.441 INFO:tasks.rados.rados.0.smithi042.stdout:2834: finishing write tid 1 to smithi04268048-46 2024-09-20T19:35:17.444 INFO:tasks.rados.rados.0.smithi042.stdout:2835: done (1 left) 2024-09-20T19:35:17.445 INFO:tasks.rados.rados.0.smithi042.stdout:2836: write (excl) oid 39 current snap is 325 2024-09-20T19:35:17.445 INFO:tasks.rados.rados.0.smithi042.stdout:2836: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-20T19:35:17.449 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:17.452 INFO:tasks.rados.rados.0.smithi042.stdout:2836: writing smithi04268048-39 from 655737 to 1153386 tid 1 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2836: writing smithi04268048-39 from 1607773 to 2397113 tid 2 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2837: snap_create 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: finishing write tid 2 to smithi04268048-46 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: finishing write tid 3 to smithi04268048-46 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: finishing write tid 4 to smithi04268048-46 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: oid 46 updating version 0 to 1156 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: oid 46 version 1156 is already newer than 1154 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: oid 46 version 1156 is already newer than 1155 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1156 (ObjNum 989 snap 325 seq_num 989) dirty exists 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2834: left oid 46 (ObjNum 989 snap 325 seq_num 989) 2024-09-20T19:35:17.459 INFO:tasks.rados.rados.0.smithi042.stdout:2836: finishing write tid 1 to smithi04268048-39 2024-09-20T19:35:17.465 INFO:tasks.rados.rados.0.smithi042.stdout:2836: finishing write tid 2 to smithi04268048-39 2024-09-20T19:35:17.465 INFO:tasks.rados.rados.0.smithi042.stdout:2836: finishing write tid 3 to smithi04268048-39 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:2836: finishing write tid 4 to smithi04268048-39 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:2836: oid 39 updating version 0 to 741 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:2836: oid 39 updating version 741 to 743 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:2836: oid 39 version 743 is already newer than 742 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 743 (ObjNum 991 snap 325 seq_num 991) dirty exists 2024-09-20T19:35:17.466 INFO:tasks.rados.rados.0.smithi042.stdout:2836: left oid 39 (ObjNum 991 snap 325 seq_num 991) 2024-09-20T19:35:17.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:17 smithi162 bash[24557]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:17 smithi162 bash[24557]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:17 smithi162 bash[24557]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:17 smithi162 bash[24557]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[22706]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[22706]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[22706]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[22706]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[32075]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.564 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[32075]: cluster 2024-09-20T19:35:16.392209+0000 mgr.y (mgr.24491) 650 : cluster [DBG] pgmap v1209: 178 pgs: 15 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 151 active+clean; 296 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 573 KiB/s wr, 8 op/s 2024-09-20T19:35:17.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[32075]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.565 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:17 smithi042 bash[32075]: cluster 2024-09-20T19:35:16.397952+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2024-09-20T19:35:17.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:17.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:17.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:17.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:17.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:17.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:17.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:17.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:18.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:18.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:18.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:18.418 INFO:tasks.rados.rados.0.smithi042.stdout:2834: done (2 left) 2024-09-20T19:35:18.418 INFO:tasks.rados.rados.0.smithi042.stdout:2836: done (1 left) 2024-09-20T19:35:18.419 INFO:tasks.rados.rados.0.smithi042.stdout:2837: done (0 left) 2024-09-20T19:35:18.419 INFO:tasks.rados.rados.0.smithi042.stdout:2838: write (excl) oid 34 current snap is 326 2024-09-20T19:35:18.419 INFO:tasks.rados.rados.0.smithi042.stdout:2838: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-20T19:35:18.427 INFO:tasks.rados.rados.0.smithi042.stdout:2838: writing smithi04268048-34 from 672544 to 1245442 tid 1 2024-09-20T19:35:18.434 INFO:tasks.rados.rados.0.smithi042.stdout:2838: writing smithi04268048-34 from 1666255 to 2199253 tid 2 2024-09-20T19:35:18.434 INFO:tasks.rados.rados.0.smithi042.stdout:2839: read oid 31 snap 309 2024-09-20T19:35:18.435 INFO:tasks.rados.rados.0.smithi042.stdout:2839: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:35:18.435 INFO:tasks.rados.rados.0.smithi042.stdout:2840: delete oid 41 current snap is 326 2024-09-20T19:35:18.435 INFO:tasks.rados.rados.0.smithi042.stdout:2838: finishing write tid 1 to smithi04268048-34 2024-09-20T19:35:18.436 INFO:tasks.rados.rados.0.smithi042.stdout:2840: done (2 left) 2024-09-20T19:35:18.437 INFO:tasks.rados.rados.0.smithi042.stdout:2841: copy_from oid 28 from oid 42 current snap is 326 2024-09-20T19:35:18.437 INFO:tasks.rados.rados.0.smithi042.stdout:2842: delete oid 35 current snap is 326 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: finishing write tid 2 to smithi04268048-34 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: finishing write tid 3 to smithi04268048-34 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: finishing write tid 4 to smithi04268048-34 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: oid 34 updating version 0 to 1075 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: oid 34 updating version 1075 to 1076 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:2838: oid 34 version 1076 is already newer than 1074 2024-09-20T19:35:18.439 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1076 (ObjNum 992 snap 326 seq_num 992) dirty exists 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2838: left oid 34 (ObjNum 992 snap 326 seq_num 992) 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2838: done (3 left) 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2842: done (2 left) 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2843: copy_from oid 25 from oid 8 current snap is 326 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2844: rollback oid 14 current snap is 326 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 303 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2845: write (excl) oid 36 current snap is 326 2024-09-20T19:35:18.440 INFO:tasks.rados.rados.0.smithi042.stdout:2845: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-20T19:35:18.448 INFO:tasks.rados.rados.0.smithi042.stdout:2845: writing smithi04268048-36 from 722965 to 1521610 tid 1 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2845: writing smithi04268048-36 from 2125348 to 2722965 tid 2 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2846: read oid 10 snap -1 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2846: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2847: read oid 33 snap -1 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2847: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2848: rollback oid 41 current snap is 326 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 290 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2844: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2849: read oid 17 snap -1 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2849: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:18.457 INFO:tasks.rados.rados.0.smithi042.stdout:2850: write oid 3 current snap is 326 2024-09-20T19:35:18.458 INFO:tasks.rados.rados.0.smithi042.stdout:2844: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:35:18.458 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1123 (ObjNum 827 snap 273 seq_num 827) dirty exists 2024-09-20T19:35:18.458 INFO:tasks.rados.rados.0.smithi042.stdout:2839: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:35:18.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:18.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:18.567 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 488141 2024-09-20T19:35:18.589 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:18.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:18 smithi162 bash[24557]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:18 smithi162 bash[24557]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:18 smithi042 bash[32075]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.601 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:18 smithi042 bash[32075]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:18 smithi042 bash[22706]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:18 smithi042 bash[22706]: cluster 2024-09-20T19:35:17.399425+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-09-20T19:35:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:18.615 INFO:tasks.rados.rados.0.smithi042.stdout:2845: finishing write tid 1 to smithi04268048-36 2024-09-20T19:35:18.615 INFO:tasks.rados.rados.0.smithi042.stdout:2843: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:35:18.615 INFO:tasks.rados.rados.0.smithi042.stdout:2841: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:35:18.615 INFO:tasks.rados.rados.0.smithi042.stdout:2850: seq_num 996 ranges {739772=590247,1900183=540336} 2024-09-20T19:35:18.622 INFO:tasks.rados.rados.0.smithi042.stdout:2850: writing smithi04268048-3 from 739772 to 1330019 tid 1 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2850: writing smithi04268048-3 from 1900183 to 2440519 tid 2 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2839: done (9 left) 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2844: done (8 left) 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2851: copy_from oid 45 from oid 13 current snap is 326 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2848: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 1415000160) dirty dne 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:2849: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:18.629 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:18.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:18.724 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:35:18.724 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:35:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:18.831 INFO:tasks.rados.rados.0.smithi042.stdout:2847: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:35:18.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:18.874 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2138093 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: finishing write tid 2 to smithi04268048-36 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: finishing write tid 3 to smithi04268048-36 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2843: finishing copy_from to smithi04268048-25 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 856 (ObjNum 847 snap 278 seq_num 847) dirty exists 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: finishing write tid 4 to smithi04268048-36 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: oid 36 updating version 0 to 586 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: oid 36 version 586 is already newer than 585 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: oid 36 updating version 586 to 587 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 587 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2845: left oid 36 (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:35:18.883 INFO:tasks.rados.rados.0.smithi042.stdout:2841: finishing copy_from to smithi04268048-28 2024-09-20T19:35:18.884 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1156 (ObjNum 973 snap 321 seq_num 973) dirty exists 2024-09-20T19:35:18.884 INFO:tasks.rados.rados.0.smithi042.stdout:2846: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:18.918 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2047367 2024-09-20T19:35:18.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:18.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:19.080 INFO:tasks.rados.rados.0.smithi042.stdout:2850: finishing write tid 1 to smithi04268048-3 2024-09-20T19:35:19.080 INFO:tasks.rados.rados.0.smithi042.stdout:2850: finishing write tid 2 to smithi04268048-3 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: finishing write tid 3 to smithi04268048-3 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: finishing write tid 4 to smithi04268048-3 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: oid 3 updating version 0 to 961 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: oid 3 updating version 961 to 962 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: oid 3 version 962 is already newer than 960 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 962 (ObjNum 996 snap 326 seq_num 996) dirty exists 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2850: left oid 3 (ObjNum 996 snap 326 seq_num 996) 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2851: finishing copy_from racing read to smithi04268048-45 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2851: finishing copy_from to smithi04268048-45 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 593 (ObjNum 860 snap 284 seq_num 860) dirty exists 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2841: done (8 left) 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2843: done (7 left) 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2845: done (6 left) 2024-09-20T19:35:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:2846: done (5 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2847: done (4 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2848: done (3 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2849: done (2 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2850: done (1 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2851: done (0 left) 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2852: rollback oid 14 current snap is 326 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 325 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2853: write oid 10 current snap is 326 2024-09-20T19:35:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:2853: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-20T19:35:19.088 INFO:tasks.rados.rados.0.smithi042.stdout:2853: writing smithi04268048-10 from 756579 to 1422075 tid 1 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2853: writing smithi04268048-10 from 1958665 to 2359012 tid 2 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2854: rollback oid 39 current snap is 326 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 303 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2852: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2852: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1125 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2852: done (2 left) 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2855: read oid 28 snap -1 2024-09-20T19:35:19.095 INFO:tasks.rados.rados.0.smithi042.stdout:2855: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:19.096 INFO:tasks.rados.rados.0.smithi042.stdout:2856: delete oid 29 current snap is 326 2024-09-20T19:35:19.096 INFO:tasks.rados.rados.0.smithi042.stdout:2853: finishing write tid 1 to smithi04268048-10 2024-09-20T19:35:19.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:19.101 INFO:tasks.rados.rados.0.smithi042.stdout:2854: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:35:19.102 INFO:tasks.rados.rados.0.smithi042.stdout:2854: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:35:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:2854: finishing rollback tid 2 to smithi04268048-39 2024-09-20T19:35:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 746 (ObjNum 900 snap 296 seq_num 900) dirty exists 2024-09-20T19:35:19.103 INFO:tasks.rados.rados.0.smithi042.stdout:2853: finishing write tid 2 to smithi04268048-10 2024-09-20T19:35:19.106 INFO:tasks.rados.rados.0.smithi042.stdout:2853: finishing write tid 3 to smithi04268048-10 2024-09-20T19:35:19.106 INFO:tasks.rados.rados.0.smithi042.stdout:2853: finishing write tid 4 to smithi04268048-10 2024-09-20T19:35:19.106 INFO:tasks.rados.rados.0.smithi042.stdout:2853: oid 10 updating version 0 to 875 2024-09-20T19:35:19.106 INFO:tasks.rados.rados.0.smithi042.stdout:2853: oid 10 version 875 is already newer than 874 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2853: oid 10 version 875 is already newer than 873 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 875 (ObjNum 997 snap 326 seq_num 997) dirty exists 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2853: left oid 10 (ObjNum 997 snap 326 seq_num 997) 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2853: done (3 left) 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2854: done (2 left) 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2856: done (1 left) 2024-09-20T19:35:19.107 INFO:tasks.rados.rados.0.smithi042.stdout:2857: snap_create 2024-09-20T19:35:19.110 INFO:tasks.rados.rados.0.smithi042.stdout:2855: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:19.110 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 256998 2024-09-20T19:35:19.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:19.190 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:19.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:19.248 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:19.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:19.422 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:35:19.425 INFO:tasks.rados.rados.0.smithi042.stdout:2855: done (1 left) 2024-09-20T19:35:19.425 INFO:tasks.rados.rados.0.smithi042.stdout:2857: done (0 left) 2024-09-20T19:35:19.425 INFO:tasks.rados.rados.0.smithi042.stdout:2858: rollback oid 11 current snap is 327 2024-09-20T19:35:19.425 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 11 to 326 2024-09-20T19:35:19.426 INFO:tasks.rados.rados.0.smithi042.stdout:2859: write oid 37 current snap is 327 2024-09-20T19:35:19.426 INFO:tasks.rados.rados.0.smithi042.stdout:2859: seq_num 999 ranges {790193=415994,1959276=404016} 2024-09-20T19:35:19.432 INFO:tasks.rados.rados.0.smithi042.stdout:2859: writing smithi04268048-37 from 790193 to 1206187 tid 1 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2859: writing smithi04268048-37 from 1959276 to 2363292 tid 2 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2860: read oid 50 snap -1 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2860: expect deleted 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2861: read oid 10 snap 316 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2861: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2862: delete oid 4 current snap is 327 2024-09-20T19:35:19.440 INFO:tasks.rados.rados.0.smithi042.stdout:2859: finishing write tid 1 to smithi04268048-37 2024-09-20T19:35:19.443 INFO:tasks.rados.rados.0.smithi042.stdout:2858: finishing rollback tid 0 to smithi04268048-11 2024-09-20T19:35:19.444 INFO:tasks.rados.rados.0.smithi042.stdout:2858: finishing rollback tid 1 to smithi04268048-11 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: finishing write tid 2 to smithi04268048-37 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: finishing write tid 3 to smithi04268048-37 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: finishing write tid 4 to smithi04268048-37 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: oid 37 updating version 0 to 1001 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: oid 37 updating version 1001 to 1003 2024-09-20T19:35:19.445 INFO:tasks.rados.rados.0.smithi042.stdout:2859: oid 37 version 1003 is already newer than 1002 2024-09-20T19:35:19.446 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1003 (ObjNum 999 snap 327 seq_num 999) dirty exists 2024-09-20T19:35:19.446 INFO:tasks.rados.rados.0.smithi042.stdout:2859: left oid 37 (ObjNum 999 snap 327 seq_num 999) 2024-09-20T19:35:19.449 INFO:tasks.rados.rados.0.smithi042.stdout:2858: finishing rollback tid 2 to smithi04268048-11 2024-09-20T19:35:19.449 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 994 (ObjNum 937 snap 314 seq_num 937) dirty exists 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2858: done (4 left) 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2859: done (3 left) 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2860: done (2 left) 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2862: done (1 left) 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2863: copy_from oid 8 from oid 29 current snap is 327 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2864: copy_from oid 34 from oid 7 current snap is 327 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2865: write oid 2 current snap is 327 2024-09-20T19:35:19.452 INFO:tasks.rados.rados.0.smithi042.stdout:2865: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-20T19:35:19.458 INFO:tasks.rados.rados.0.smithi042.stdout:2865: writing smithi04268048-2 from 423807 to 990299 tid 1 2024-09-20T19:35:19.465 INFO:tasks.rados.rados.0.smithi042.stdout:2865: writing smithi04268048-2 from 1676240 to 2083925 tid 2 2024-09-20T19:35:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:2865: writing smithi04268048-2 from 2739004 to 2823807 tid 3 2024-09-20T19:35:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:2866: read oid 42 snap -1 2024-09-20T19:35:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:2866: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:2867: write (excl) oid 30 current snap is 327 2024-09-20T19:35:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:2867: seq_num 1002 ranges {440614=641741,1734722=551343} 2024-09-20T19:35:19.475 INFO:tasks.rados.rados.0.smithi042.stdout:2867: writing smithi04268048-30 from 440614 to 1082355 tid 1 2024-09-20T19:35:19.480 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:19.483 INFO:tasks.rados.rados.0.smithi042.stdout:2867: writing smithi04268048-30 from 1734722 to 2286065 tid 2 2024-09-20T19:35:19.483 INFO:tasks.rados.rados.0.smithi042.stdout:2868: delete oid 49 current snap is 327 2024-09-20T19:35:19.483 INFO:tasks.rados.rados.0.smithi042.stdout:2861: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:35:19.483 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.490 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[22706]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:19 smithi042 bash[32075]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:19.556 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:19.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:35:19.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:19.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:19.655 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1898352 2024-09-20T19:35:19.688 INFO:tasks.rados.rados.0.smithi042.stdout:2863: finishing copy_from to smithi04268048-8 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2863: got expected ENOENT (src dne) 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2863: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2864: finishing copy_from racing read to smithi04268048-34 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2865: finishing write tid 1 to smithi04268048-2 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2864: finishing copy_from to smithi04268048-34 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1226 (ObjNum 986 snap 325 seq_num 986) dirty exists 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2867: finishing write tid 1 to smithi04268048-30 2024-09-20T19:35:19.689 INFO:tasks.rados.rados.0.smithi042.stdout:2866: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:19.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: cluster 2024-09-20T19:35:18.393238+0000 mgr.y (mgr.24491) 651 : cluster [DBG] pgmap v1212: 178 pgs: 13 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 153 active+clean; 304 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: cluster 2024-09-20T19:35:18.417617+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[24557]: audit 2024-09-20T19:35:19.297605+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:19.759 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:19.845 INFO:tasks.rados.rados.0.smithi042.stdout:2865: finishing write tid 2 to smithi04268048-2 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: finishing write tid 3 to smithi04268048-2 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: finishing write tid 4 to smithi04268048-2 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: finishing write tid 5 to smithi04268048-2 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: oid 2 updating version 0 to 1298 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: oid 2 updating version 1298 to 1299 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: oid 2 updating version 1299 to 1300 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: oid 2 version 1300 is already newer than 1297 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1300 (ObjNum 1001 snap 327 seq_num 1001) dirty exists 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2865: left oid 2 (ObjNum 1001 snap 327 seq_num 1001) 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2867: finishing write tid 2 to smithi04268048-30 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2867: finishing write tid 3 to smithi04268048-30 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2867: finishing write tid 4 to smithi04268048-30 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2867: oid 30 updating version 0 to 1194 2024-09-20T19:35:19.846 INFO:tasks.rados.rados.0.smithi042.stdout:2867: oid 30 updating version 1194 to 1195 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2867: oid 30 version 1195 is already newer than 1193 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1195 (ObjNum 1002 snap 327 seq_num 1002) dirty exists 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2867: left oid 30 (ObjNum 1002 snap 327 seq_num 1002) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2861: done (6 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2863: done (5 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2864: done (4 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2865: done (3 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2866: done (2 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2867: done (1 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2868: done (0 left) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2869: read oid 26 snap -1 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2869: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2870: copy_from oid 23 from oid 16 current snap is 327 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2871: write oid 12 current snap is 327 2024-09-20T19:35:19.847 INFO:tasks.rados.rados.0.smithi042.stdout:2871: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-20T19:35:19.853 INFO:tasks.rados.rados.0.smithi042.stdout:2871: writing smithi04268048-12 from 474228 to 982820 tid 1 2024-09-20T19:35:19.861 INFO:tasks.rados.rados.0.smithi042.stdout:2871: writing smithi04268048-12 from 1451686 to 2006698 tid 2 2024-09-20T19:35:19.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:19.868 INFO:tasks.rados.rados.0.smithi042.stdout:2871: writing smithi04268048-12 from 2704920 to 2874228 tid 3 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2872: read oid 27 snap -1 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2872: expect deleted 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2873: read oid 2 snap -1 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2873: expect (ObjNum 1001 snap 327 seq_num 1001) 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2874: read oid 22 snap -1 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2874: expect (ObjNum 890 snap 294 seq_num 890) 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2875: rollback oid 46 current snap is 327 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 295 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2870: finishing copy_from racing read to smithi04268048-23 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2870: finishing copy_from to smithi04268048-23 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1144 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:19.869 INFO:tasks.rados.rados.0.smithi042.stdout:2869: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:19.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:19.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.001 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:35:20.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:20.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:20.126 INFO:tasks.rados.rados.0.smithi042.stdout:2871: finishing write tid 1 to smithi04268048-12 2024-09-20T19:35:20.126 INFO:tasks.rados.rados.0.smithi042.stdout:2871: finishing write tid 2 to smithi04268048-12 2024-09-20T19:35:20.126 INFO:tasks.rados.rados.0.smithi042.stdout:2871: finishing write tid 3 to smithi04268048-12 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: finishing write tid 4 to smithi04268048-12 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2875: finishing rollback tid 0 to smithi04268048-46 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: finishing write tid 5 to smithi04268048-12 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: oid 12 updating version 0 to 740 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: oid 12 updating version 740 to 742 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: oid 12 version 742 is already newer than 741 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: oid 12 version 742 is already newer than 739 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 742 (ObjNum 1004 snap 327 seq_num 1004) dirty exists 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2871: left oid 12 (ObjNum 1004 snap 327 seq_num 1004) 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2875: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1159 (ObjNum 33 snap 0 seq_num 174353490) dirty dne 2024-09-20T19:35:20.127 INFO:tasks.rados.rados.0.smithi042.stdout:2873: expect (ObjNum 1001 snap 327 seq_num 1001) 2024-09-20T19:35:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:20.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.248 INFO:tasks.rados.rados.0.smithi042.stdout:2874: expect (ObjNum 890 snap 294 seq_num 890) 2024-09-20T19:35:20.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:20.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2869: done (6 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2870: done (5 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2871: done (4 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2872: done (3 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2873: done (2 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2874: done (1 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2875: done (0 left) 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2876: copy_from oid 49 from oid 31 current snap is 327 2024-09-20T19:35:20.399 INFO:tasks.rados.rados.0.smithi042.stdout:2877: snap_remove snap 286 2024-09-20T19:35:20.400 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:20.435 INFO:tasks.rados.rados.0.smithi042.stdout:2877: done (1 left) 2024-09-20T19:35:20.435 INFO:tasks.rados.rados.0.smithi042.stdout:2878: rollback oid 45 current snap is 327 2024-09-20T19:35:20.435 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 299 2024-09-20T19:35:20.435 INFO:tasks.rados.rados.0.smithi042.stdout:2876: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:35:20.435 INFO:tasks.rados.rados.0.smithi042.stdout:2876: finishing copy_from to smithi04268048-49 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 882 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2876: done (1 left) 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2879: read oid 46 snap 326 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2879: expect (ObjNum 989 snap 325 seq_num 989) 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2880: read oid 16 snap 303 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2880: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2881: read oid 34 snap 303 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2881: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2882: read oid 26 snap 318 2024-09-20T19:35:20.436 INFO:tasks.rados.rados.0.smithi042.stdout:2882: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:20.437 INFO:tasks.rados.rados.0.smithi042.stdout:2883: snap_create 2024-09-20T19:35:20.443 INFO:tasks.rados.rados.0.smithi042.stdout:2881: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:20.443 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:20.481 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 363719 2024-09-20T19:35:20.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:20 smithi162 bash[24557]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:20.546 INFO:tasks.rados.rados.0.smithi042.stdout:2879: expect (ObjNum 989 snap 325 seq_num 989) 2024-09-20T19:35:20.590 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1967947 2024-09-20T19:35:20.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.648 INFO:tasks.rados.rados.0.smithi042.stdout:2878: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:35:20.648 INFO:tasks.rados.rados.0.smithi042.stdout:2878: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:35:20.648 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1379 (ObjNum 129 snap 0 seq_num 1415363216) dirty dne 2024-09-20T19:35:20.648 INFO:tasks.rados.rados.0.smithi042.stdout:2882: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:20.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:20.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[32075]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: cluster 2024-09-20T19:35:19.413641+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: audit 2024-09-20T19:35:19.421108+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.42:0/3509708223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:20 smithi042 bash[22706]: cluster 2024-09-20T19:35:19.424825+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-09-20T19:35:20.711 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3103891 2024-09-20T19:35:20.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:20.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.898 INFO:tasks.rados.rados.0.smithi042.stdout:2880: expect (ObjNum 814 snap 268 seq_num 814) 2024-09-20T19:35:20.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:20.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:20.959 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:35:20.959 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:35:20.976 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:35:20.976 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:35:20.995 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3605747 2024-09-20T19:35:20.996 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:21.003 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:35:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:35:21.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:21.019 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:35:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:35:21.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:21.049 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:35:21.049 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:35:21.066 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:35:21.067 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:35:21.087 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:35:21.108 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:35:21.108 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:35:21.147 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:21.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:21.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:21.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:21.401 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:21.437 INFO:tasks.rados.rados.0.smithi042.stdout:2878: done (5 left) 2024-09-20T19:35:21.437 INFO:tasks.rados.rados.0.smithi042.stdout:2879: done (4 left) 2024-09-20T19:35:21.437 INFO:tasks.rados.rados.0.smithi042.stdout:2880: done (3 left) 2024-09-20T19:35:21.437 INFO:tasks.rados.rados.0.smithi042.stdout:2881: done (2 left) 2024-09-20T19:35:21.438 INFO:tasks.rados.rados.0.smithi042.stdout:2882: done (1 left) 2024-09-20T19:35:21.438 INFO:tasks.rados.rados.0.smithi042.stdout:2883: done (0 left) 2024-09-20T19:35:21.438 INFO:tasks.rados.rados.0.smithi042.stdout:2884: read oid 16 snap -1 2024-09-20T19:35:21.438 INFO:tasks.rados.rados.0.smithi042.stdout:2884: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:35:21.438 INFO:tasks.rados.rados.0.smithi042.stdout:2885: snap_remove snap 316 2024-09-20T19:35:21.438 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:21.531 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:21 smithi162 bash[24557]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:21.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[32075]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: audit 2024-09-20T19:35:19.750133+0000 mgr.y (mgr.24491) 652 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: cluster 2024-09-20T19:35:20.393683+0000 mgr.y (mgr.24491) 653 : cluster [DBG] pgmap v1215: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 307 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: cluster 2024-09-20T19:35:20.431775+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:21 smithi042 bash[22706]: audit 2024-09-20T19:35:20.939984+0000 mon.c (mon.2) 107 : audit [DBG] from='client.? v1:172.21.15.42:0/70975410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:21.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:21.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:21.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:21.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:22.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:22.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:22.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:22.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:22.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:22.442 INFO:tasks.rados.rados.0.smithi042.stdout:2885: done (1 left) 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:2886: read oid 31 snap -1 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:2886: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:2887: rollback oid 44 current snap is 328 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 319 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:2888: snap_create 2024-09-20T19:35:22.443 INFO:tasks.rados.rados.0.smithi042.stdout:2884: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:35:22.446 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:22.473 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:22.476 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:22.499 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:22.524 INFO:tasks.rados.rados.0.smithi042.stdout:2887: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:35:22.524 INFO:tasks.rados.rados.0.smithi042.stdout:2887: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:35:22.524 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1164 (ObjNum 289 snap 0 seq_num 173205935) dirty dne 2024-09-20T19:35:22.524 INFO:tasks.rados.rados.0.smithi042.stdout:2886: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:35:22.577 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:22.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[32075]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[32075]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[32075]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[32075]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[22706]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[22706]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[22706]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.592 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:22 smithi042 bash[22706]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.606 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:22.630 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:22.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:22.735 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:22.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:22 smithi162 bash[24557]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:22 smithi162 bash[24557]: cluster 2024-09-20T19:35:21.434452+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-09-20T19:35:22.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:22 smithi162 bash[24557]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:22 smithi162 bash[24557]: cluster 2024-09-20T19:35:22.436778+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2024-09-20T19:35:22.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:22.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:22.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:22.964 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:35:22.968 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:22.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:23.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:23.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:23.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:23.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:2884: done (3 left) 2024-09-20T19:35:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:2886: done (2 left) 2024-09-20T19:35:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:2887: done (1 left) 2024-09-20T19:35:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:2888: done (0 left) 2024-09-20T19:35:23.383 INFO:tasks.rados.rados.0.smithi042.stdout:2889: snap_create 2024-09-20T19:35:23.393 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:23.496 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:23.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:23.630 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[32075]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[32075]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[32075]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[32075]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[22706]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[22706]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[22706]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:23 smithi042 bash[22706]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:23.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:23.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:23 smithi162 bash[24557]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:23 smithi162 bash[24557]: cluster 2024-09-20T19:35:22.395654+0000 mgr.y (mgr.24491) 654 : cluster [DBG] pgmap v1218: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 312 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-09-20T19:35:23.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:23 smithi162 bash[24557]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:23 smithi162 bash[24557]: cluster 2024-09-20T19:35:23.380017+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2024-09-20T19:35:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:23.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:23.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:23.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:23.967 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:35:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:35:24.015 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:24.137 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.164 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:24.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:24.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:24.355 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:24.388 INFO:tasks.rados.rados.0.smithi042.stdout:2889: done (0 left) 2024-09-20T19:35:24.388 INFO:tasks.rados.rados.0.smithi042.stdout:2890: snap_remove snap 329 2024-09-20T19:35:24.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:24.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:35:24.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:24.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:24.657 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:24.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.805 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:24.883 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1220,"stamp":"2024-09-20T19:35:23.375624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334308301,"num_objects":359,"num_object_clones":116,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":11,"num_read":3634,"num_read_kb":1090822,"num_write":1829,"num_write_kb":410379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24365,"ondisk_log_size":24365,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1084700,"kb_used_data":760892,"kb_used_omap":92,"kb_used_meta":323555,"kb_avail":655131364,"statfs":{"total":671965249536,"available":670854516736,"internally_reserved":0,"allocated":779153408,"data_stored":1041791047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":331321017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12720309,"num_objects":10,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":91934,"num_write":59,"num_write_kb":21584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.954080"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":3580,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.537391+0000","last_change":"2024-09-20T19:35:19.707795+0000","last_active":"2024-09-20T19:35:21.537391+0000","last_peered":"2024-09-20T19:35:21.537391+0000","last_clean":"2024-09-20T19:35:21.537391+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:35:21.537391+0000","last_undegraded":"2024-09-20T19:35:21.537391+0000","last_fullsized":"2024-09-20T19:35:21.537391+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:34:02.755546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:02.755546+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.22809516599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"3"}]},{"pgid":"7.22","version":"808'259","reported_seq":3243,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418740+0000","last_change":"2024-09-20T19:35:17.406437+0000","last_active":"2024-09-20T19:35:18.418740+0000","last_peered":"2024-09-20T19:35:18.418740+0000","last_clean":"2024-09-20T19:35:18.418740+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:35:18.418740+0000","last_undegraded":"2024-09-20T19:35:18.418740+0000","last_fullsized":"2024-09-20T19:35:18.418740+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"721'206","last_scrub_stamp":"2024-09-20T19:33:52.586094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:52.586094+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027649559999999998,"stat_sum":{"num_bytes":3696466,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":6178,"num_write":9,"num_write_kb":1122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"7.21","version":"813'661","reported_seq":4119,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.691289+0000","last_change":"2024-09-20T19:35:21.691289+0000","last_active":"2024-09-20T19:35:21.691289+0000","last_peered":"2024-09-20T19:35:21.691289+0000","last_clean":"2024-09-20T19:35:21.691289+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:35:21.691289+0000","last_undegraded":"2024-09-20T19:35:21.691289+0000","last_fullsized":"2024-09-20T19:35:21.691289+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"722'630","last_scrub_stamp":"2024-09-20T19:34:05.048343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.048343+0000","objects_scrubbed":2,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.24293078600000001,"stat_sum":{"num_bytes":5933267,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":56175,"num_write":3,"num_write_kb":2200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"7.20","version":"812'878","reported_seq":4275,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435435+0000","last_change":"2024-09-20T19:35:20.462174+0000","last_active":"2024-09-20T19:35:21.435435+0000","last_peered":"2024-09-20T19:35:21.435435+0000","last_clean":"2024-09-20T19:35:21.435435+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:35:21.435435+0000","last_undegraded":"2024-09-20T19:35:21.435435+0000","last_fullsized":"2024-09-20T19:35:21.435435+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"721'837","last_scrub_stamp":"2024-09-20T19:33:59.352105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.352105+0000","objects_scrubbed":2,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:25.051362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.023385449999999999,"stat_sum":{"num_bytes":7525112,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":27240,"num_write":13,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"7.1a","version":"810'596","reported_seq":3729,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:19.102767+0000","last_change":"2024-09-20T19:35:17.429012+0000","last_active":"2024-09-20T19:35:19.102767+0000","last_peered":"2024-09-20T19:35:19.102767+0000","last_clean":"2024-09-20T19:35:19.102767+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:35:19.102767+0000","last_undegraded":"2024-09-20T19:35:19.102767+0000","last_fullsized":"2024-09-20T19:35:19.102767+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'493","last_scrub_stamp":"2024-09-20T19:33:55.653122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.653122+0000","objects_scrubbed":4,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.025241696000000001,"stat_sum":{"num_bytes":16666079,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":50,"num_read_kb":40526,"num_write":30,"num_write_kb":15680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"6.1b","version":"0'0","reported_seq":1021,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.531942+0000","last_change":"2024-09-20T19:33:43.387357+0000","last_active":"2024-09-20T19:35:21.531942+0000","last_peered":"2024-09-20T19:35:21.531942+0000","last_clean":"2024-09-20T19:35:21.531942+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:35:21.531942+0000","last_undegraded":"2024-09-20T19:35:21.531942+0000","last_fullsized":"2024-09-20T19:35:21.531942+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1343,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438787+0000","last_change":"2024-09-20T19:34:58.530958+0000","last_active":"2024-09-20T19:35:22.438787+0000","last_peered":"2024-09-20T19:35:22.438787+0000","last_clean":"2024-09-20T19:35:22.438787+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:35:22.438787+0000","last_undegraded":"2024-09-20T19:35:22.438787+0000","last_fullsized":"2024-09-20T19:35:22.438787+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1255,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433308+0000","last_change":"2024-09-20T19:33:43.389098+0000","last_active":"2024-09-20T19:35:21.433308+0000","last_peered":"2024-09-20T19:35:21.433308+0000","last_clean":"2024-09-20T19:35:21.433308+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:35:21.433308+0000","last_undegraded":"2024-09-20T19:35:21.433308+0000","last_fullsized":"2024-09-20T19:35:21.433308+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:32:58.834586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:58.834586+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.277653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1314,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418475+0000","last_change":"2024-09-20T19:35:04.624469+0000","last_active":"2024-09-20T19:35:18.418475+0000","last_peered":"2024-09-20T19:35:18.418475+0000","last_clean":"2024-09-20T19:35:18.418475+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:35:18.418475+0000","last_undegraded":"2024-09-20T19:35:18.418475+0000","last_fullsized":"2024-09-20T19:35:18.418475+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.624374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.624374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.762788+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":3864,"reported_epoch":814,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:22.694437+0000","last_change":"2024-09-20T19:35:22.694485+0000","last_active":"2024-09-20T19:35:22.694437+0000","last_peered":"2024-09-20T19:35:22.694437+0000","last_clean":"2024-09-20T19:35:22.694437+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:35:22.694437+0000","last_undegraded":"2024-09-20T19:35:22.694437+0000","last_fullsized":"2024-09-20T19:35:22.694437+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:33:53.566287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.566287+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.25514418100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.1e","version":"0'0","reported_seq":1078,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.441437+0000","last_change":"2024-09-20T19:33:43.367928+0000","last_active":"2024-09-20T19:35:22.441437+0000","last_peered":"2024-09-20T19:35:22.441437+0000","last_clean":"2024-09-20T19:35:22.441437+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:35:22.441437+0000","last_undegraded":"2024-09-20T19:35:22.441437+0000","last_fullsized":"2024-09-20T19:35:22.441437+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1418,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418879+0000","last_change":"2024-09-20T19:34:47.537704+0000","last_active":"2024-09-20T19:35:18.418879+0000","last_peered":"2024-09-20T19:35:18.418879+0000","last_clean":"2024-09-20T19:35:18.418879+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:35:18.418879+0000","last_undegraded":"2024-09-20T19:35:18.418879+0000","last_fullsized":"2024-09-20T19:35:18.418879+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:47.537616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.537616+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.097439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1552,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433278+0000","last_change":"2024-09-20T19:34:54.283813+0000","last_active":"2024-09-20T19:35:21.433278+0000","last_peered":"2024-09-20T19:35:21.433278+0000","last_clean":"2024-09-20T19:35:21.433278+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:35:21.433278+0000","last_undegraded":"2024-09-20T19:35:21.433278+0000","last_fullsized":"2024-09-20T19:35:21.433278+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.283695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.283695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.569450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1375,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419689+0000","last_change":"2024-09-20T19:34:58.623699+0000","last_active":"2024-09-20T19:35:18.419689+0000","last_peered":"2024-09-20T19:35:18.419689+0000","last_clean":"2024-09-20T19:35:18.419689+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:35:18.419689+0000","last_undegraded":"2024-09-20T19:35:18.419689+0000","last_fullsized":"2024-09-20T19:35:18.419689+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:58.623637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:58.623637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:23.832554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"814'1164","reported_seq":4498,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:22.451479+0000","last_change":"2024-09-20T19:35:21.688508+0000","last_active":"2024-09-20T19:35:22.451479+0000","last_peered":"2024-09-20T19:35:22.451479+0000","last_clean":"2024-09-20T19:35:22.451479+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:35:22.451479+0000","last_undegraded":"2024-09-20T19:35:22.451479+0000","last_fullsized":"2024-09-20T19:35:22.451479+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.24592773000000001,"stat_sum":{"num_bytes":15380173,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":73,"num_read_kb":53384,"num_write":69,"num_write_kb":28605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1549,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.534358+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:35:21.534358+0000","last_peered":"2024-09-20T19:35:21.534358+0000","last_clean":"2024-09-20T19:35:21.534358+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:35:21.534358+0000","last_undegraded":"2024-09-20T19:35:21.534358+0000","last_fullsized":"2024-09-20T19:35:21.534358+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1412,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418972+0000","last_change":"2024-09-20T19:34:42.553244+0000","last_active":"2024-09-20T19:35:18.418972+0000","last_peered":"2024-09-20T19:35:18.418972+0000","last_clean":"2024-09-20T19:35:18.418972+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:35:18.418972+0000","last_undegraded":"2024-09-20T19:35:18.418972+0000","last_fullsized":"2024-09-20T19:35:18.418972+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:34:42.553118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:42.553118+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.964837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1541,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532013+0000","last_change":"2024-09-20T19:34:50.140557+0000","last_active":"2024-09-20T19:35:21.532013+0000","last_peered":"2024-09-20T19:35:21.532013+0000","last_clean":"2024-09-20T19:35:21.532013+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:35:21.532013+0000","last_undegraded":"2024-09-20T19:35:21.532013+0000","last_fullsized":"2024-09-20T19:35:21.532013+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:50.140498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:50.140498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:55.537342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1554,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451501+0000","last_change":"2024-09-20T19:33:43.447076+0000","last_active":"2024-09-20T19:35:21.451501+0000","last_peered":"2024-09-20T19:35:21.451501+0000","last_clean":"2024-09-20T19:35:21.451501+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:35:21.451501+0000","last_undegraded":"2024-09-20T19:35:21.451501+0000","last_fullsized":"2024-09-20T19:35:21.451501+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.802489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"813'412","reported_seq":3802,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.699189+0000","last_change":"2024-09-20T19:35:21.699189+0000","last_active":"2024-09-20T19:35:21.699189+0000","last_peered":"2024-09-20T19:35:21.699189+0000","last_clean":"2024-09-20T19:35:21.699189+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:35:21.699189+0000","last_undegraded":"2024-09-20T19:35:21.699189+0000","last_fullsized":"2024-09-20T19:35:21.699189+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"481'290","last_scrub_stamp":"2024-09-20T19:29:01.442771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:01.442771+0000","objects_scrubbed":4,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.23373734900000001,"stat_sum":{"num_bytes":3521610,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":36396,"num_write":32,"num_write_kb":8793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"3"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1538,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.534727+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:35:21.534727+0000","last_peered":"2024-09-20T19:35:21.534727+0000","last_clean":"2024-09-20T19:35:21.534727+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:35:21.534727+0000","last_undegraded":"2024-09-20T19:35:21.534727+0000","last_fullsized":"2024-09-20T19:35:21.534727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1445,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419087+0000","last_change":"2024-09-20T19:34:26.418165+0000","last_active":"2024-09-20T19:35:18.419087+0000","last_peered":"2024-09-20T19:35:18.419087+0000","last_clean":"2024-09-20T19:35:18.419087+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:35:18.419087+0000","last_undegraded":"2024-09-20T19:35:18.419087+0000","last_fullsized":"2024-09-20T19:35:18.419087+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.451996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.451996+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.609262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1534,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451727+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:35:21.451727+0000","last_peered":"2024-09-20T19:35:21.451727+0000","last_clean":"2024-09-20T19:35:21.451727+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:35:21.451727+0000","last_undegraded":"2024-09-20T19:35:21.451727+0000","last_fullsized":"2024-09-20T19:35:21.451727+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.072932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":512,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.435961+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:35:22.435961+0000","last_peered":"2024-09-20T19:35:22.435961+0000","last_clean":"2024-09-20T19:35:22.435961+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:35:22.435961+0000","last_undegraded":"2024-09-20T19:35:22.435961+0000","last_fullsized":"2024-09-20T19:35:22.435961+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1170,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.449974+0000","last_change":"2024-09-20T19:34:07.057170+0000","last_active":"2024-09-20T19:35:21.449974+0000","last_peered":"2024-09-20T19:35:21.449974+0000","last_clean":"2024-09-20T19:35:21.449974+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:35:21.449974+0000","last_undegraded":"2024-09-20T19:35:21.449974+0000","last_fullsized":"2024-09-20T19:35:21.449974+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:07.057076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:07.057076+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.891767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"808'314","reported_seq":3704,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419662+0000","last_change":"2024-09-20T19:35:17.421407+0000","last_active":"2024-09-20T19:35:18.419662+0000","last_peered":"2024-09-20T19:35:18.419662+0000","last_clean":"2024-09-20T19:35:18.419662+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:35:18.419662+0000","last_undegraded":"2024-09-20T19:35:18.419662+0000","last_fullsized":"2024-09-20T19:35:18.419662+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"735'281","last_scrub_stamp":"2024-09-20T19:34:05.563300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.563300+0000","objects_scrubbed":3,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.258892+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.016336369,"stat_sum":{"num_bytes":4958231,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":5339,"num_write":5,"num_write_kb":1766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.d","version":"523'31","reported_seq":1396,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419506+0000","last_change":"2024-09-20T19:34:13.521735+0000","last_active":"2024-09-20T19:35:18.419506+0000","last_peered":"2024-09-20T19:35:18.419506+0000","last_clean":"2024-09-20T19:35:18.419506+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:35:18.419506+0000","last_undegraded":"2024-09-20T19:35:18.419506+0000","last_fullsized":"2024-09-20T19:35:18.419506+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:34:13.521620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.521620+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.065586+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1542,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.535533+0000","last_change":"2024-09-20T19:34:49.185014+0000","last_active":"2024-09-20T19:35:21.535533+0000","last_peered":"2024-09-20T19:35:21.535533+0000","last_clean":"2024-09-20T19:35:21.535533+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:35:21.535533+0000","last_undegraded":"2024-09-20T19:35:21.535533+0000","last_fullsized":"2024-09-20T19:35:21.535533+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:49.184958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:49.184958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:49.964266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1554,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435442+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:35:21.435442+0000","last_peered":"2024-09-20T19:35:21.435442+0000","last_clean":"2024-09-20T19:35:21.435442+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:35:21.435442+0000","last_undegraded":"2024-09-20T19:35:21.435442+0000","last_fullsized":"2024-09-20T19:35:21.435442+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.211201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":122,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433235+0000","last_change":"2024-09-20T19:34:26.402007+0000","last_active":"2024-09-20T19:35:21.433235+0000","last_peered":"2024-09-20T19:35:21.433235+0000","last_clean":"2024-09-20T19:35:21.433235+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:35:21.433235+0000","last_undegraded":"2024-09-20T19:35:21.433235+0000","last_fullsized":"2024-09-20T19:35:21.433235+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:06.544189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.544189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:14.318932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1635,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.534105+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:35:21.534105+0000","last_peered":"2024-09-20T19:35:21.534105+0000","last_clean":"2024-09-20T19:35:21.534105+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:35:21.534105+0000","last_undegraded":"2024-09-20T19:35:21.534105+0000","last_fullsized":"2024-09-20T19:35:21.534105+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"811'572","reported_seq":3741,"reported_epoch":814,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:22.694971+0000","last_change":"2024-09-20T19:35:22.695024+0000","last_active":"2024-09-20T19:35:22.694971+0000","last_peered":"2024-09-20T19:35:22.694971+0000","last_clean":"2024-09-20T19:35:22.694971+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:35:22.694971+0000","last_undegraded":"2024-09-20T19:35:22.694971+0000","last_fullsized":"2024-09-20T19:35:22.694971+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'544","last_scrub_stamp":"2024-09-20T19:33:54.617348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.617348+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.25693345499999998,"stat_sum":{"num_bytes":9042793,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":10989,"num_write":16,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.c","version":"523'20","reported_seq":1439,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419066+0000","last_change":"2024-09-20T19:33:43.467110+0000","last_active":"2024-09-20T19:35:18.419066+0000","last_peered":"2024-09-20T19:35:18.419066+0000","last_clean":"2024-09-20T19:35:18.419066+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:35:18.419066+0000","last_undegraded":"2024-09-20T19:35:18.419066+0000","last_fullsized":"2024-09-20T19:35:18.419066+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:32:10.444013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.444013+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.713160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1578,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434778+0000","last_change":"2024-09-20T19:34:57.369277+0000","last_active":"2024-09-20T19:35:21.434778+0000","last_peered":"2024-09-20T19:35:21.434778+0000","last_clean":"2024-09-20T19:35:21.434778+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:35:21.434778+0000","last_undegraded":"2024-09-20T19:35:21.434778+0000","last_fullsized":"2024-09-20T19:35:21.434778+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.369117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.369117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:07.185078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1770,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434559+0000","last_change":"2024-09-20T19:34:17.102983+0000","last_active":"2024-09-20T19:35:21.434559+0000","last_peered":"2024-09-20T19:35:21.434559+0000","last_clean":"2024-09-20T19:35:21.434559+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:35:21.434559+0000","last_undegraded":"2024-09-20T19:35:21.434559+0000","last_fullsized":"2024-09-20T19:35:21.434559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:34:17.102808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.102808+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:34.479538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1378,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420007+0000","last_change":"2024-09-20T19:34:45.548359+0000","last_active":"2024-09-20T19:35:18.420007+0000","last_peered":"2024-09-20T19:35:18.420007+0000","last_clean":"2024-09-20T19:35:18.420007+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:35:18.420007+0000","last_undegraded":"2024-09-20T19:35:18.420007+0000","last_fullsized":"2024-09-20T19:35:18.420007+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:45.548235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:45.548235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.055514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1141,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.452502+0000","last_change":"2024-09-20T19:34:10.011707+0000","last_active":"2024-09-20T19:35:21.452502+0000","last_peered":"2024-09-20T19:35:21.452502+0000","last_clean":"2024-09-20T19:35:21.452502+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:35:21.452502+0000","last_undegraded":"2024-09-20T19:35:21.452502+0000","last_fullsized":"2024-09-20T19:35:21.452502+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:10.011637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.011637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.773891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"811'878","reported_seq":4350,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.699063+0000","last_change":"2024-09-20T19:35:21.699123+0000","last_active":"2024-09-20T19:35:21.699063+0000","last_peered":"2024-09-20T19:35:21.699063+0000","last_clean":"2024-09-20T19:35:21.699063+0000","last_became_active":"2024-09-20T19:33:46.371630+0000","last_became_peered":"2024-09-20T19:33:46.371630+0000","last_unstale":"2024-09-20T19:35:21.699063+0000","last_undegraded":"2024-09-20T19:35:21.699063+0000","last_fullsized":"2024-09-20T19:35:21.699063+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'835","last_scrub_stamp":"2024-09-20T19:33:55.628320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.628320+0000","objects_scrubbed":2,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.16136679500000001,"stat_sum":{"num_bytes":8034340,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":8533,"num_write":9,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.f","version":"523'27","reported_seq":1645,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436421+0000","last_change":"2024-09-20T19:34:56.365553+0000","last_active":"2024-09-20T19:35:21.436421+0000","last_peered":"2024-09-20T19:35:21.436421+0000","last_clean":"2024-09-20T19:35:21.436421+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:35:21.436421+0000","last_undegraded":"2024-09-20T19:35:21.436421+0000","last_fullsized":"2024-09-20T19:35:21.436421+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:56.365402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:56.365402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.638116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1578,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433497+0000","last_change":"2024-09-20T19:34:46.261824+0000","last_active":"2024-09-20T19:35:21.433497+0000","last_peered":"2024-09-20T19:35:21.433497+0000","last_clean":"2024-09-20T19:35:21.433497+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:35:21.433497+0000","last_undegraded":"2024-09-20T19:35:21.433497+0000","last_fullsized":"2024-09-20T19:35:21.433497+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:46.261770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.261770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:04.225781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1550,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420233+0000","last_change":"2024-09-20T19:33:59.507413+0000","last_active":"2024-09-20T19:35:18.420233+0000","last_peered":"2024-09-20T19:35:18.420233+0000","last_clean":"2024-09-20T19:35:18.420233+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:35:18.420233+0000","last_undegraded":"2024-09-20T19:35:18.420233+0000","last_fullsized":"2024-09-20T19:35:18.420233+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:33:59.507348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.507348+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1547,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451144+0000","last_change":"2024-09-20T19:34:17.113285+0000","last_active":"2024-09-20T19:35:21.451144+0000","last_peered":"2024-09-20T19:35:21.451144+0000","last_clean":"2024-09-20T19:35:21.451144+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:35:21.451144+0000","last_undegraded":"2024-09-20T19:35:21.451144+0000","last_fullsized":"2024-09-20T19:35:21.451144+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:17.113170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.113170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:44.530510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"813'576","reported_seq":4010,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.710498+0000","last_change":"2024-09-20T19:35:21.710498+0000","last_active":"2024-09-20T19:35:21.710498+0000","last_peered":"2024-09-20T19:35:21.710498+0000","last_clean":"2024-09-20T19:35:21.710498+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:35:21.710498+0000","last_undegraded":"2024-09-20T19:35:21.710498+0000","last_fullsized":"2024-09-20T19:35:21.710498+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"717'538","last_scrub_stamp":"2024-09-20T19:34:06.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.656444+0000","objects_scrubbed":1,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.22636294200000001,"stat_sum":{"num_bytes":9879994,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":18,"num_write_kb":5452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.e","version":"523'19","reported_seq":1420,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420405+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:35:18.420405+0000","last_peered":"2024-09-20T19:35:18.420405+0000","last_clean":"2024-09-20T19:35:18.420405+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:35:18.420405+0000","last_undegraded":"2024-09-20T19:35:18.420405+0000","last_fullsized":"2024-09-20T19:35:18.420405+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.587592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":207,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.530973+0000","last_change":"2024-09-20T19:33:43.387910+0000","last_active":"2024-09-20T19:35:21.530973+0000","last_peered":"2024-09-20T19:35:21.530973+0000","last_clean":"2024-09-20T19:35:21.530973+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:35:21.530973+0000","last_undegraded":"2024-09-20T19:35:21.530973+0000","last_fullsized":"2024-09-20T19:35:21.530973+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1551,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.453696+0000","last_change":"2024-09-20T19:34:48.070464+0000","last_active":"2024-09-20T19:35:21.453696+0000","last_peered":"2024-09-20T19:35:21.453696+0000","last_clean":"2024-09-20T19:35:21.453696+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:35:21.453696+0000","last_undegraded":"2024-09-20T19:35:21.453696+0000","last_fullsized":"2024-09-20T19:35:21.453696+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.070340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.070340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:14.947230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1017,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.537926+0000","last_change":"2024-09-20T19:33:43.352629+0000","last_active":"2024-09-20T19:35:21.537926+0000","last_peered":"2024-09-20T19:35:21.537926+0000","last_clean":"2024-09-20T19:35:21.537926+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:35:21.537926+0000","last_undegraded":"2024-09-20T19:35:21.537926+0000","last_fullsized":"2024-09-20T19:35:21.537926+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.525029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":378,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451409+0000","last_change":"2024-09-20T19:35:03.930343+0000","last_active":"2024-09-20T19:35:21.451409+0000","last_peered":"2024-09-20T19:35:21.451409+0000","last_clean":"2024-09-20T19:35:21.451409+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:35:21.451409+0000","last_undegraded":"2024-09-20T19:35:21.451409+0000","last_fullsized":"2024-09-20T19:35:21.451409+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:03.930289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.930289+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.856845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":3147,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433916+0000","last_change":"2024-09-20T19:35:20.432000+0000","last_active":"2024-09-20T19:35:21.433916+0000","last_peered":"2024-09-20T19:35:21.433916+0000","last_clean":"2024-09-20T19:35:21.433916+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:35:21.433916+0000","last_undegraded":"2024-09-20T19:35:21.433916+0000","last_fullsized":"2024-09-20T19:35:21.433916+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:33:57.391147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:57.391147+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001065992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.1","version":"523'28","reported_seq":1389,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420175+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:35:18.420175+0000","last_peered":"2024-09-20T19:35:18.420175+0000","last_clean":"2024-09-20T19:35:18.420175+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:35:18.420175+0000","last_undegraded":"2024-09-20T19:35:18.420175+0000","last_fullsized":"2024-09-20T19:35:18.420175+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.650160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1540,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.539655+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:35:21.539655+0000","last_peered":"2024-09-20T19:35:21.539655+0000","last_clean":"2024-09-20T19:35:21.539655+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:35:21.539655+0000","last_undegraded":"2024-09-20T19:35:21.539655+0000","last_fullsized":"2024-09-20T19:35:21.539655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:13.574522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":448,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436110+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:35:21.436110+0000","last_peered":"2024-09-20T19:35:21.436110+0000","last_clean":"2024-09-20T19:35:21.436110+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:35:21.436110+0000","last_undegraded":"2024-09-20T19:35:21.436110+0000","last_fullsized":"2024-09-20T19:35:21.436110+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1528,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532462+0000","last_change":"2024-09-20T19:34:16.144529+0000","last_active":"2024-09-20T19:35:21.532462+0000","last_peered":"2024-09-20T19:35:21.532462+0000","last_clean":"2024-09-20T19:35:21.532462+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:35:21.532462+0000","last_undegraded":"2024-09-20T19:35:21.532462+0000","last_fullsized":"2024-09-20T19:35:21.532462+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:16.144473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:16.144473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.559736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":951,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419688+0000","last_change":"2024-09-20T19:35:07.564250+0000","last_active":"2024-09-20T19:35:18.419688+0000","last_peered":"2024-09-20T19:35:18.419688+0000","last_clean":"2024-09-20T19:35:18.419688+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:35:18.419688+0000","last_undegraded":"2024-09-20T19:35:18.419688+0000","last_fullsized":"2024-09-20T19:35:18.419688+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:07.564190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.564190+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:29.024831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"718'421","reported_seq":3668,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433128+0000","last_change":"2024-09-20T19:35:20.462703+0000","last_active":"2024-09-20T19:35:21.433128+0000","last_peered":"2024-09-20T19:35:21.433128+0000","last_clean":"2024-09-20T19:35:21.433128+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:35:21.433128+0000","last_undegraded":"2024-09-20T19:35:21.433128+0000","last_fullsized":"2024-09-20T19:35:21.433128+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'421","last_scrub_stamp":"2024-09-20T19:34:59.428803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.428803+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:29.156137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023653271,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.0","version":"523'30","reported_seq":1266,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.531649+0000","last_change":"2024-09-20T19:33:43.476313+0000","last_active":"2024-09-20T19:35:21.531649+0000","last_peered":"2024-09-20T19:35:21.531649+0000","last_clean":"2024-09-20T19:35:21.531649+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:35:21.531649+0000","last_undegraded":"2024-09-20T19:35:21.531649+0000","last_fullsized":"2024-09-20T19:35:21.531649+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1548,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436552+0000","last_change":"2024-09-20T19:32:15.096541+0000","last_active":"2024-09-20T19:35:21.436552+0000","last_peered":"2024-09-20T19:35:21.436552+0000","last_clean":"2024-09-20T19:35:21.436552+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:35:21.436552+0000","last_undegraded":"2024-09-20T19:35:21.436552+0000","last_fullsized":"2024-09-20T19:35:21.436552+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:32:15.096450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.096450+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:37.278967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1348,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418403+0000","last_change":"2024-09-20T19:33:43.375389+0000","last_active":"2024-09-20T19:35:18.418403+0000","last_peered":"2024-09-20T19:35:18.418403+0000","last_clean":"2024-09-20T19:35:18.418403+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:35:18.418403+0000","last_undegraded":"2024-09-20T19:35:18.418403+0000","last_fullsized":"2024-09-20T19:35:18.418403+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:18.400304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:18.400304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.537825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1310,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419199+0000","last_change":"2024-09-20T19:34:59.643830+0000","last_active":"2024-09-20T19:35:18.419199+0000","last_peered":"2024-09-20T19:35:18.419199+0000","last_clean":"2024-09-20T19:35:18.419199+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:35:18.419199+0000","last_undegraded":"2024-09-20T19:35:18.419199+0000","last_fullsized":"2024-09-20T19:35:18.419199+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.643704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.643704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1261,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.538585+0000","last_change":"2024-09-20T19:35:00.419430+0000","last_active":"2024-09-20T19:35:21.538585+0000","last_peered":"2024-09-20T19:35:21.538585+0000","last_clean":"2024-09-20T19:35:21.538585+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:35:21.538585+0000","last_undegraded":"2024-09-20T19:35:21.538585+0000","last_fullsized":"2024-09-20T19:35:21.538585+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:00.419230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:00.419230+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:26.608264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"813'881","reported_seq":4483,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.694400+0000","last_change":"2024-09-20T19:35:21.694400+0000","last_active":"2024-09-20T19:35:21.694400+0000","last_peered":"2024-09-20T19:35:21.694400+0000","last_clean":"2024-09-20T19:35:21.694400+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:35:21.694400+0000","last_undegraded":"2024-09-20T19:35:21.694400+0000","last_fullsized":"2024-09-20T19:35:21.694400+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'829","last_scrub_stamp":"2024-09-20T19:34:01.119894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:01.119894+0000","objects_scrubbed":1,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.240948102,"stat_sum":{"num_bytes":13877560,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":50,"num_read_kb":56717,"num_write":22,"num_write_kb":4742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.3","version":"523'29","reported_seq":1361,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438825+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:35:22.438825+0000","last_peered":"2024-09-20T19:35:22.438825+0000","last_clean":"2024-09-20T19:35:22.438825+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:35:22.438825+0000","last_undegraded":"2024-09-20T19:35:22.438825+0000","last_fullsized":"2024-09-20T19:35:22.438825+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1506,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433406+0000","last_change":"2024-09-20T19:34:05.253077+0000","last_active":"2024-09-20T19:35:21.433406+0000","last_peered":"2024-09-20T19:35:21.433406+0000","last_clean":"2024-09-20T19:35:21.433406+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:35:21.433406+0000","last_undegraded":"2024-09-20T19:35:21.433406+0000","last_fullsized":"2024-09-20T19:35:21.433406+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:05.253020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.253020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:27.698757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1544,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.535992+0000","last_change":"2024-09-20T19:33:06.757095+0000","last_active":"2024-09-20T19:35:21.535992+0000","last_peered":"2024-09-20T19:35:21.535992+0000","last_clean":"2024-09-20T19:35:21.535992+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:35:21.535992+0000","last_undegraded":"2024-09-20T19:35:21.535992+0000","last_fullsized":"2024-09-20T19:35:21.535992+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.756963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.756963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1547,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433174+0000","last_change":"2024-09-20T19:34:55.325695+0000","last_active":"2024-09-20T19:35:21.433174+0000","last_peered":"2024-09-20T19:35:21.433174+0000","last_clean":"2024-09-20T19:35:21.433174+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:35:21.433174+0000","last_undegraded":"2024-09-20T19:35:21.433174+0000","last_fullsized":"2024-09-20T19:35:21.433174+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.325573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.325573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:58.162804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1254,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.539071+0000","last_change":"2024-09-20T19:33:43.353150+0000","last_active":"2024-09-20T19:35:21.539071+0000","last_peered":"2024-09-20T19:35:21.539071+0000","last_clean":"2024-09-20T19:35:21.539071+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:35:21.539071+0000","last_undegraded":"2024-09-20T19:35:21.539071+0000","last_fullsized":"2024-09-20T19:35:21.539071+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:15.514065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":3918,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419372+0000","last_change":"2024-09-20T19:35:17.405259+0000","last_active":"2024-09-20T19:35:18.419372+0000","last_peered":"2024-09-20T19:35:18.419372+0000","last_clean":"2024-09-20T19:35:18.419372+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:35:18.419372+0000","last_undegraded":"2024-09-20T19:35:18.419372+0000","last_fullsized":"2024-09-20T19:35:18.419372+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:33:53.608544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.608544+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026684159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.2","version":"523'20","reported_seq":1601,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434517+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:35:21.434517+0000","last_peered":"2024-09-20T19:35:21.434517+0000","last_clean":"2024-09-20T19:35:21.434517+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:35:21.434517+0000","last_undegraded":"2024-09-20T19:35:21.434517+0000","last_fullsized":"2024-09-20T19:35:21.434517+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.034460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1556,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436235+0000","last_change":"2024-09-20T19:34:48.237100+0000","last_active":"2024-09-20T19:35:21.436235+0000","last_peered":"2024-09-20T19:35:21.436235+0000","last_clean":"2024-09-20T19:35:21.436235+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:35:21.436235+0000","last_undegraded":"2024-09-20T19:35:21.436235+0000","last_fullsized":"2024-09-20T19:35:21.436235+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.236983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.236983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.560188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1485,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.439636+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:35:22.439636+0000","last_peered":"2024-09-20T19:35:22.439636+0000","last_clean":"2024-09-20T19:35:22.439636+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:35:22.439636+0000","last_undegraded":"2024-09-20T19:35:22.439636+0000","last_fullsized":"2024-09-20T19:35:22.439636+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1346,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.436775+0000","last_change":"2024-09-20T19:34:26.391734+0000","last_active":"2024-09-20T19:35:22.436775+0000","last_peered":"2024-09-20T19:35:22.436775+0000","last_clean":"2024-09-20T19:35:22.436775+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:35:22.436775+0000","last_undegraded":"2024-09-20T19:35:22.436775+0000","last_fullsized":"2024-09-20T19:35:22.436775+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.455584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.455584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1144,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.450507+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:35:21.450507+0000","last_peered":"2024-09-20T19:35:21.450507+0000","last_clean":"2024-09-20T19:35:21.450507+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:35:21.450507+0000","last_undegraded":"2024-09-20T19:35:21.450507+0000","last_fullsized":"2024-09-20T19:35:21.450507+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:25.079668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"810'532","reported_seq":4034,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.539277+0000","last_change":"2024-09-20T19:35:19.708224+0000","last_active":"2024-09-20T19:35:21.539277+0000","last_peered":"2024-09-20T19:35:21.539277+0000","last_clean":"2024-09-20T19:35:21.539277+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:35:21.539277+0000","last_undegraded":"2024-09-20T19:35:21.539277+0000","last_fullsized":"2024-09-20T19:35:21.539277+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'400","last_scrub_stamp":"2024-09-20T19:31:36.808095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:36.808095+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.22580118099999999,"stat_sum":{"num_bytes":10336968,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":79,"num_read_kb":42901,"num_write":61,"num_write_kb":17237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.4","version":"715'45","reported_seq":1884,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435833+0000","last_change":"2024-09-20T19:34:17.143462+0000","last_active":"2024-09-20T19:35:21.435833+0000","last_peered":"2024-09-20T19:35:21.435833+0000","last_clean":"2024-09-20T19:35:21.435833+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:35:21.435833+0000","last_undegraded":"2024-09-20T19:35:21.435833+0000","last_fullsized":"2024-09-20T19:35:21.435833+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'45","last_scrub_stamp":"2024-09-20T19:34:17.143380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.143380+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.481481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1557,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.453699+0000","last_change":"2024-09-20T19:34:14.039743+0000","last_active":"2024-09-20T19:35:21.453699+0000","last_peered":"2024-09-20T19:35:21.453699+0000","last_clean":"2024-09-20T19:35:21.453699+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:35:21.453699+0000","last_undegraded":"2024-09-20T19:35:21.453699+0000","last_fullsized":"2024-09-20T19:35:21.453699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:14.039620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:14.039620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:31.864667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1336,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.437482+0000","last_change":"2024-09-20T19:32:17.478298+0000","last_active":"2024-09-20T19:35:22.437482+0000","last_peered":"2024-09-20T19:35:22.437482+0000","last_clean":"2024-09-20T19:35:22.437482+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:35:22.437482+0000","last_undegraded":"2024-09-20T19:35:22.437482+0000","last_fullsized":"2024-09-20T19:35:22.437482+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:17.478143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:17.478143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":991,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420084+0000","last_change":"2024-09-20T19:33:43.347560+0000","last_active":"2024-09-20T19:35:18.420084+0000","last_peered":"2024-09-20T19:35:18.420084+0000","last_clean":"2024-09-20T19:35:18.420084+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:35:18.420084+0000","last_undegraded":"2024-09-20T19:35:18.420084+0000","last_fullsized":"2024-09-20T19:35:18.420084+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:21.605528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1613,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532760+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:35:21.532760+0000","last_peered":"2024-09-20T19:35:21.532760+0000","last_clean":"2024-09-20T19:35:21.532760+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:35:21.532760+0000","last_undegraded":"2024-09-20T19:35:21.532760+0000","last_fullsized":"2024-09-20T19:35:21.532760+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"814'959","reported_seq":4031,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:22.695030+0000","last_change":"2024-09-20T19:35:22.695030+0000","last_active":"2024-09-20T19:35:22.695030+0000","last_peered":"2024-09-20T19:35:22.695030+0000","last_clean":"2024-09-20T19:35:22.695030+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:35:22.695030+0000","last_undegraded":"2024-09-20T19:35:22.695030+0000","last_fullsized":"2024-09-20T19:35:22.695030+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'779","last_scrub_stamp":"2024-09-20T19:31:22.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.595437+0000","objects_scrubbed":4,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.137456368,"stat_sum":{"num_bytes":17635522,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":84,"num_read_kb":81254,"num_write":64,"num_write_kb":24490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.7","version":"523'17","reported_seq":1517,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433939+0000","last_change":"2024-09-20T19:33:04.118188+0000","last_active":"2024-09-20T19:35:21.433939+0000","last_peered":"2024-09-20T19:35:21.433939+0000","last_clean":"2024-09-20T19:35:21.433939+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:35:21.433939+0000","last_undegraded":"2024-09-20T19:35:21.433939+0000","last_fullsized":"2024-09-20T19:35:21.433939+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:33:04.117998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.117998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.179464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2577,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436423+0000","last_change":"2024-09-20T19:35:02.415570+0000","last_active":"2024-09-20T19:35:21.436423+0000","last_peered":"2024-09-20T19:35:21.436423+0000","last_clean":"2024-09-20T19:35:21.436423+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:35:21.436423+0000","last_undegraded":"2024-09-20T19:35:21.436423+0000","last_fullsized":"2024-09-20T19:35:21.436423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:35:02.415432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:35:02.415432+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.696507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1016,"num_read_kb":1011,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1568,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433434+0000","last_change":"2024-09-20T19:33:43.443451+0000","last_active":"2024-09-20T19:35:21.433434+0000","last_peered":"2024-09-20T19:35:21.433434+0000","last_clean":"2024-09-20T19:35:21.433434+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:35:21.433434+0000","last_undegraded":"2024-09-20T19:35:21.433434+0000","last_fullsized":"2024-09-20T19:35:21.433434+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:13.137322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:13.137322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:33.197164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":122,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433693+0000","last_change":"2024-09-20T19:34:26.389239+0000","last_active":"2024-09-20T19:35:21.433693+0000","last_peered":"2024-09-20T19:35:21.433693+0000","last_clean":"2024-09-20T19:35:21.433693+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:35:21.433693+0000","last_undegraded":"2024-09-20T19:35:21.433693+0000","last_fullsized":"2024-09-20T19:35:21.433693+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:04.320119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.320119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.160769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1546,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.533902+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:35:21.533902+0000","last_peered":"2024-09-20T19:35:21.533902+0000","last_clean":"2024-09-20T19:35:21.533902+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:35:21.533902+0000","last_undegraded":"2024-09-20T19:35:21.533902+0000","last_fullsized":"2024-09-20T19:35:21.533902+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1146,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436718+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:35:21.436718+0000","last_peered":"2024-09-20T19:35:21.436718+0000","last_clean":"2024-09-20T19:35:21.436718+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:35:21.436718+0000","last_undegraded":"2024-09-20T19:35:21.436718+0000","last_fullsized":"2024-09-20T19:35:21.436718+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.253974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"812'998","reported_seq":4427,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.534759+0000","last_change":"2024-09-20T19:35:21.527873+0000","last_active":"2024-09-20T19:35:21.534759+0000","last_peered":"2024-09-20T19:35:21.534759+0000","last_clean":"2024-09-20T19:35:21.534759+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:35:21.534759+0000","last_undegraded":"2024-09-20T19:35:21.534759+0000","last_fullsized":"2024-09-20T19:35:21.534759+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.089453275999999998,"stat_sum":{"num_bytes":12102054,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":66,"num_read_kb":40205,"num_write":50,"num_write_kb":14898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":1376,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.441239+0000","last_change":"2024-09-20T19:34:26.408655+0000","last_active":"2024-09-20T19:35:22.441239+0000","last_peered":"2024-09-20T19:35:22.441239+0000","last_clean":"2024-09-20T19:35:22.441239+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:35:22.441239+0000","last_undegraded":"2024-09-20T19:35:22.441239+0000","last_fullsized":"2024-09-20T19:35:22.441239+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:32:57.496961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.496961+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1290,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438573+0000","last_change":"2024-09-20T19:34:43.943794+0000","last_active":"2024-09-20T19:35:22.438573+0000","last_peered":"2024-09-20T19:35:22.438573+0000","last_clean":"2024-09-20T19:35:22.438573+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:35:22.438573+0000","last_undegraded":"2024-09-20T19:35:22.438573+0000","last_fullsized":"2024-09-20T19:35:22.438573+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:34:43.943661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.943661+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:50.854911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1551,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532457+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:35:21.532457+0000","last_peered":"2024-09-20T19:35:21.532457+0000","last_clean":"2024-09-20T19:35:21.532457+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:35:21.532457+0000","last_undegraded":"2024-09-20T19:35:21.532457+0000","last_fullsized":"2024-09-20T19:35:21.532457+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1543,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.453845+0000","last_change":"2024-09-20T19:32:23.505615+0000","last_active":"2024-09-20T19:35:21.453845+0000","last_peered":"2024-09-20T19:35:21.453845+0000","last_clean":"2024-09-20T19:35:21.453845+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:35:21.453845+0000","last_undegraded":"2024-09-20T19:35:21.453845+0000","last_fullsized":"2024-09-20T19:35:21.453845+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.505475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.505475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.081565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1575,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435323+0000","last_change":"2024-09-20T19:35:04.500375+0000","last_active":"2024-09-20T19:35:21.435323+0000","last_peered":"2024-09-20T19:35:21.435323+0000","last_clean":"2024-09-20T19:35:21.435323+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:35:21.435323+0000","last_undegraded":"2024-09-20T19:35:21.435323+0000","last_fullsized":"2024-09-20T19:35:21.435323+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.500313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.786196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":3352,"reported_epoch":814,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:22.693945+0000","last_change":"2024-09-20T19:35:22.694027+0000","last_active":"2024-09-20T19:35:22.693945+0000","last_peered":"2024-09-20T19:35:22.693945+0000","last_clean":"2024-09-20T19:35:22.693945+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:35:22.693945+0000","last_undegraded":"2024-09-20T19:35:22.693945+0000","last_fullsized":"2024-09-20T19:35:22.693945+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:35:06.711015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:06.711015+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:32.584748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.25146180800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.5","version":"523'30","reported_seq":1796,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.538953+0000","last_change":"2024-09-20T19:34:55.460785+0000","last_active":"2024-09-20T19:35:21.538953+0000","last_peered":"2024-09-20T19:35:21.538953+0000","last_clean":"2024-09-20T19:35:21.538953+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:35:21.538953+0000","last_undegraded":"2024-09-20T19:35:21.538953+0000","last_fullsized":"2024-09-20T19:35:21.538953+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:34:55.460588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.460588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.240558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":395,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.453366+0000","last_change":"2024-09-20T19:33:43.357559+0000","last_active":"2024-09-20T19:35:21.453366+0000","last_peered":"2024-09-20T19:35:21.453366+0000","last_clean":"2024-09-20T19:35:21.453366+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:35:21.453366+0000","last_undegraded":"2024-09-20T19:35:21.453366+0000","last_fullsized":"2024-09-20T19:35:21.453366+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:32:57.875788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.875788+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:36.079972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1541,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.452615+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:35:21.452615+0000","last_peered":"2024-09-20T19:35:21.452615+0000","last_clean":"2024-09-20T19:35:21.452615+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:35:21.452615+0000","last_undegraded":"2024-09-20T19:35:21.452615+0000","last_fullsized":"2024-09-20T19:35:21.452615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.324932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1600,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532980+0000","last_change":"2024-09-20T19:34:48.152641+0000","last_active":"2024-09-20T19:35:21.532980+0000","last_peered":"2024-09-20T19:35:21.532980+0000","last_clean":"2024-09-20T19:35:21.532980+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:35:21.532980+0000","last_undegraded":"2024-09-20T19:35:21.532980+0000","last_fullsized":"2024-09-20T19:35:21.532980+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:48.152576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.152576+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.506442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1524,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.534400+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:35:21.534400+0000","last_peered":"2024-09-20T19:35:21.534400+0000","last_clean":"2024-09-20T19:35:21.534400+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:35:21.534400+0000","last_undegraded":"2024-09-20T19:35:21.534400+0000","last_fullsized":"2024-09-20T19:35:21.534400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1553,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.454175+0000","last_change":"2024-09-20T19:34:15.087033+0000","last_active":"2024-09-20T19:35:21.454175+0000","last_peered":"2024-09-20T19:35:21.454175+0000","last_clean":"2024-09-20T19:35:21.454175+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:35:21.454175+0000","last_undegraded":"2024-09-20T19:35:21.454175+0000","last_fullsized":"2024-09-20T19:35:21.454175+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:15.086902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:15.086902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:28.923469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"812'377","reported_seq":3498,"reported_epoch":814,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:22.692308+0000","last_change":"2024-09-20T19:35:22.692356+0000","last_active":"2024-09-20T19:35:22.692308+0000","last_peered":"2024-09-20T19:35:22.692308+0000","last_clean":"2024-09-20T19:35:22.692308+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:35:22.692308+0000","last_undegraded":"2024-09-20T19:35:22.692308+0000","last_fullsized":"2024-09-20T19:35:22.692308+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'375","last_scrub_stamp":"2024-09-20T19:34:53.903004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.903004+0000","objects_scrubbed":2,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:23.645556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.25462579800000001,"stat_sum":{"num_bytes":4908040,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":31,"num_write_kb":7400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.a","version":"523'35","reported_seq":1583,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.536493+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:35:21.536493+0000","last_peered":"2024-09-20T19:35:21.536493+0000","last_clean":"2024-09-20T19:35:21.536493+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:35:21.536493+0000","last_undegraded":"2024-09-20T19:35:21.536493+0000","last_fullsized":"2024-09-20T19:35:21.536493+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:11.545395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1569,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435109+0000","last_change":"2024-09-20T19:34:47.295550+0000","last_active":"2024-09-20T19:35:21.435109+0000","last_peered":"2024-09-20T19:35:21.435109+0000","last_clean":"2024-09-20T19:35:21.435109+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:35:21.435109+0000","last_undegraded":"2024-09-20T19:35:21.435109+0000","last_fullsized":"2024-09-20T19:35:21.435109+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:47.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.295412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:50.958066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":122,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433509+0000","last_change":"2024-09-20T19:34:26.390149+0000","last_active":"2024-09-20T19:35:21.433509+0000","last_peered":"2024-09-20T19:35:21.433509+0000","last_clean":"2024-09-20T19:35:21.433509+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:35:21.433509+0000","last_undegraded":"2024-09-20T19:35:21.433509+0000","last_fullsized":"2024-09-20T19:35:21.433509+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:29.295406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1529,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532697+0000","last_change":"2024-09-20T19:33:43.369554+0000","last_active":"2024-09-20T19:35:21.532697+0000","last_peered":"2024-09-20T19:35:21.532697+0000","last_clean":"2024-09-20T19:35:21.532697+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:35:21.532697+0000","last_undegraded":"2024-09-20T19:35:21.532697+0000","last_fullsized":"2024-09-20T19:35:21.532697+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"812'1385","reported_seq":4770,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435190+0000","last_change":"2024-09-20T19:35:20.465146+0000","last_active":"2024-09-20T19:35:21.435190+0000","last_peered":"2024-09-20T19:35:21.435190+0000","last_clean":"2024-09-20T19:35:21.435190+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:35:21.435190+0000","last_undegraded":"2024-09-20T19:35:21.435190+0000","last_fullsized":"2024-09-20T19:35:21.435190+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'1322","last_scrub_stamp":"2024-09-20T19:34:52.301450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:52.301450+0000","objects_scrubbed":6,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.532813+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.026549637000000001,"stat_sum":{"num_bytes":22252093,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":147,"num_read_kb":113319,"num_write":104,"num_write_kb":45866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.b","version":"523'15","reported_seq":1351,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.439512+0000","last_change":"2024-09-20T19:34:58.566556+0000","last_active":"2024-09-20T19:35:22.439512+0000","last_peered":"2024-09-20T19:35:22.439512+0000","last_clean":"2024-09-20T19:35:22.439512+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:35:22.439512+0000","last_undegraded":"2024-09-20T19:35:22.439512+0000","last_fullsized":"2024-09-20T19:35:22.439512+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:15.464807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.464807+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1576,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436993+0000","last_change":"2024-09-20T19:34:26.406213+0000","last_active":"2024-09-20T19:35:21.436993+0000","last_peered":"2024-09-20T19:35:21.436993+0000","last_clean":"2024-09-20T19:35:21.436993+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:35:21.436993+0000","last_undegraded":"2024-09-20T19:35:21.436993+0000","last_fullsized":"2024-09-20T19:35:21.436993+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.260232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1575,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434430+0000","last_change":"2024-09-20T19:34:44.295768+0000","last_active":"2024-09-20T19:35:21.434430+0000","last_peered":"2024-09-20T19:35:21.434430+0000","last_clean":"2024-09-20T19:35:21.434430+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:35:21.434430+0000","last_undegraded":"2024-09-20T19:35:21.434430+0000","last_fullsized":"2024-09-20T19:35:21.434430+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.295697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.295697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:44.355330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1352,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438853+0000","last_change":"2024-09-20T19:34:26.405297+0000","last_active":"2024-09-20T19:35:22.438853+0000","last_peered":"2024-09-20T19:35:22.438853+0000","last_clean":"2024-09-20T19:35:22.438853+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:35:22.438853+0000","last_undegraded":"2024-09-20T19:35:22.438853+0000","last_fullsized":"2024-09-20T19:35:22.438853+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:32:10.330783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.330783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"812'564","reported_seq":3958,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.435864+0000","last_change":"2024-09-20T19:35:20.432720+0000","last_active":"2024-09-20T19:35:21.435864+0000","last_peered":"2024-09-20T19:35:21.435864+0000","last_clean":"2024-09-20T19:35:21.435864+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:35:21.435864+0000","last_undegraded":"2024-09-20T19:35:21.435864+0000","last_fullsized":"2024-09-20T19:35:21.435864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"724'501","last_scrub_stamp":"2024-09-20T19:33:52.436716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:52.436716+0000","objects_scrubbed":2,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.25866083600000001,"stat_sum":{"num_bytes":10908601,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":44,"num_read_kb":44334,"num_write":20,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.8","version":"523'27","reported_seq":1595,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.450610+0000","last_change":"2024-09-20T19:34:47.083599+0000","last_active":"2024-09-20T19:35:21.450610+0000","last_peered":"2024-09-20T19:35:21.450610+0000","last_clean":"2024-09-20T19:35:21.450610+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:35:21.450610+0000","last_undegraded":"2024-09-20T19:35:21.450610+0000","last_fullsized":"2024-09-20T19:35:21.450610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:47.083533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.083533+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.559644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":126,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433794+0000","last_change":"2024-09-20T19:34:26.404678+0000","last_active":"2024-09-20T19:35:21.433794+0000","last_peered":"2024-09-20T19:35:21.433794+0000","last_clean":"2024-09-20T19:35:21.433794+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:35:21.433794+0000","last_undegraded":"2024-09-20T19:35:21.433794+0000","last_fullsized":"2024-09-20T19:35:21.433794+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:35.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.568627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:30.252995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1762,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.533399+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:35:21.533399+0000","last_peered":"2024-09-20T19:35:21.533399+0000","last_clean":"2024-09-20T19:35:21.533399+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:35:21.533399+0000","last_undegraded":"2024-09-20T19:35:21.533399+0000","last_fullsized":"2024-09-20T19:35:21.533399+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1547,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.452956+0000","last_change":"2024-09-20T19:34:53.009990+0000","last_active":"2024-09-20T19:35:21.452956+0000","last_peered":"2024-09-20T19:35:21.452956+0000","last_clean":"2024-09-20T19:35:21.452956+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:35:21.452956+0000","last_undegraded":"2024-09-20T19:35:21.452956+0000","last_fullsized":"2024-09-20T19:35:21.452956+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:53.009934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.009934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:06.466332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":3503,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.433878+0000","last_change":"2024-09-20T19:35:20.457755+0000","last_active":"2024-09-20T19:35:21.433878+0000","last_peered":"2024-09-20T19:35:21.433878+0000","last_clean":"2024-09-20T19:35:21.433878+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:35:21.433878+0000","last_undegraded":"2024-09-20T19:35:21.433878+0000","last_fullsized":"2024-09-20T19:35:21.433878+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:35:07.477259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.477259+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:23.033267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018967422000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.9","version":"523'18","reported_seq":1416,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420047+0000","last_change":"2024-09-20T19:33:43.369566+0000","last_active":"2024-09-20T19:35:18.420047+0000","last_peered":"2024-09-20T19:35:18.420047+0000","last_clean":"2024-09-20T19:35:18.420047+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:35:18.420047+0000","last_undegraded":"2024-09-20T19:35:18.420047+0000","last_fullsized":"2024-09-20T19:35:18.420047+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:27.634845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1404,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419854+0000","last_change":"2024-09-20T19:34:08.546458+0000","last_active":"2024-09-20T19:35:18.419854+0000","last_peered":"2024-09-20T19:35:18.419854+0000","last_clean":"2024-09-20T19:35:18.419854+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:35:18.419854+0000","last_undegraded":"2024-09-20T19:35:18.419854+0000","last_fullsized":"2024-09-20T19:35:18.419854+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:34:08.546356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.546356+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.318383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1553,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435519+0000","last_change":"2024-09-20T19:33:06.111877+0000","last_active":"2024-09-20T19:35:21.435519+0000","last_peered":"2024-09-20T19:35:21.435519+0000","last_clean":"2024-09-20T19:35:21.435519+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:35:21.435519+0000","last_undegraded":"2024-09-20T19:35:21.435519+0000","last_fullsized":"2024-09-20T19:35:21.435519+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.111753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.111753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:19.369782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1034,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.532200+0000","last_change":"2024-09-20T19:33:43.447359+0000","last_active":"2024-09-20T19:35:21.532200+0000","last_peered":"2024-09-20T19:35:21.532200+0000","last_clean":"2024-09-20T19:35:21.532200+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:35:21.532200+0000","last_undegraded":"2024-09-20T19:35:21.532200+0000","last_fullsized":"2024-09-20T19:35:21.532200+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"811'1173","reported_seq":4921,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.534311+0000","last_change":"2024-09-20T19:35:21.530091+0000","last_active":"2024-09-20T19:35:21.534311+0000","last_peered":"2024-09-20T19:35:21.534311+0000","last_clean":"2024-09-20T19:35:21.534311+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:35:21.534311+0000","last_undegraded":"2024-09-20T19:35:21.534311+0000","last_fullsized":"2024-09-20T19:35:21.534311+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'1072","last_scrub_stamp":"2024-09-20T19:34:09.034773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:09.034773+0000","objects_scrubbed":12,"log_size":1173,"log_dups_size":0,"ondisk_log_size":1173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:36.204090+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.089212847,"stat_sum":{"num_bytes":22354309,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":153,"num_read_kb":105574,"num_write":144,"num_write_kb":50886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"}]},{"pgid":"4.16","version":"523'15","reported_seq":1353,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.440502+0000","last_change":"2024-09-20T19:33:43.390311+0000","last_active":"2024-09-20T19:35:22.440502+0000","last_peered":"2024-09-20T19:35:22.440502+0000","last_clean":"2024-09-20T19:35:22.440502+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:35:22.440502+0000","last_undegraded":"2024-09-20T19:35:22.440502+0000","last_fullsized":"2024-09-20T19:35:22.440502+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1585,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435665+0000","last_change":"2024-09-20T19:34:54.365631+0000","last_active":"2024-09-20T19:35:21.435665+0000","last_peered":"2024-09-20T19:35:21.435665+0000","last_clean":"2024-09-20T19:35:21.435665+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:35:21.435665+0000","last_undegraded":"2024-09-20T19:35:21.435665+0000","last_fullsized":"2024-09-20T19:35:21.435665+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.365568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.365568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:55.360414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1029,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436238+0000","last_change":"2024-09-20T19:34:10.220726+0000","last_active":"2024-09-20T19:35:21.436238+0000","last_peered":"2024-09-20T19:35:21.436238+0000","last_clean":"2024-09-20T19:35:21.436238+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:35:21.436238+0000","last_undegraded":"2024-09-20T19:35:21.436238+0000","last_fullsized":"2024-09-20T19:35:21.436238+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:10.220603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.220603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.004978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1541,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434762+0000","last_change":"2024-09-20T19:33:02.129851+0000","last_active":"2024-09-20T19:35:21.434762+0000","last_peered":"2024-09-20T19:35:21.434762+0000","last_clean":"2024-09-20T19:35:21.434762+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:35:21.434762+0000","last_undegraded":"2024-09-20T19:35:21.434762+0000","last_fullsized":"2024-09-20T19:35:21.434762+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.129712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.129712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:23.209559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"812'1201","reported_seq":4670,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434631+0000","last_change":"2024-09-20T19:35:20.471675+0000","last_active":"2024-09-20T19:35:21.434631+0000","last_peered":"2024-09-20T19:35:21.434631+0000","last_clean":"2024-09-20T19:35:21.434631+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:35:21.434631+0000","last_undegraded":"2024-09-20T19:35:21.434631+0000","last_fullsized":"2024-09-20T19:35:21.434631+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1069","last_scrub_stamp":"2024-09-20T19:34:11.169527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.169527+0000","objects_scrubbed":7,"log_size":1201,"log_dups_size":0,"ondisk_log_size":1201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:38.523746+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.032380658999999999,"stat_sum":{"num_bytes":23827631,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":201,"num_read_kb":134075,"num_write":120,"num_write_kb":48850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"2.11","version":"0'0","reported_seq":1551,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.536327+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:35:21.536327+0000","last_peered":"2024-09-20T19:35:21.536327+0000","last_clean":"2024-09-20T19:35:21.536327+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:35:21.536327+0000","last_undegraded":"2024-09-20T19:35:21.536327+0000","last_fullsized":"2024-09-20T19:35:21.536327+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.149622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1202,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.537008+0000","last_change":"2024-09-20T19:33:43.464473+0000","last_active":"2024-09-20T19:35:21.537008+0000","last_peered":"2024-09-20T19:35:21.537008+0000","last_clean":"2024-09-20T19:35:21.537008+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:35:21.537008+0000","last_undegraded":"2024-09-20T19:35:21.537008+0000","last_fullsized":"2024-09-20T19:35:21.537008+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1558,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.452422+0000","last_change":"2024-09-20T19:34:08.991943+0000","last_active":"2024-09-20T19:35:21.452422+0000","last_peered":"2024-09-20T19:35:21.452422+0000","last_clean":"2024-09-20T19:35:21.452422+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:35:21.452422+0000","last_undegraded":"2024-09-20T19:35:21.452422+0000","last_fullsized":"2024-09-20T19:35:21.452422+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:08.991819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.991819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:24.368397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1530,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.533796+0000","last_change":"2024-09-20T19:34:44.066655+0000","last_active":"2024-09-20T19:35:21.533796+0000","last_peered":"2024-09-20T19:35:21.533796+0000","last_clean":"2024-09-20T19:35:21.533796+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:35:21.533796+0000","last_undegraded":"2024-09-20T19:35:21.533796+0000","last_fullsized":"2024-09-20T19:35:21.533796+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:00.702253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"811'667","reported_seq":3969,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.537755+0000","last_change":"2024-09-20T19:35:21.527824+0000","last_active":"2024-09-20T19:35:21.537755+0000","last_peered":"2024-09-20T19:35:21.537755+0000","last_clean":"2024-09-20T19:35:21.537755+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:35:21.537755+0000","last_undegraded":"2024-09-20T19:35:21.537755+0000","last_fullsized":"2024-09-20T19:35:21.537755+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.084449491,"stat_sum":{"num_bytes":19681787,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":63,"num_read_kb":31633,"num_write":46,"num_write_kb":20061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"13f","length":"1"},{"start":"143","length":"1"}]},{"pgid":"4.14","version":"523'20","reported_seq":1219,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435647+0000","last_change":"2024-09-20T19:33:43.463851+0000","last_active":"2024-09-20T19:35:21.435647+0000","last_peered":"2024-09-20T19:35:21.435647+0000","last_clean":"2024-09-20T19:35:21.435647+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:35:21.435647+0000","last_undegraded":"2024-09-20T19:35:21.435647+0000","last_fullsized":"2024-09-20T19:35:21.435647+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:26.281258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1566,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.453543+0000","last_change":"2024-09-20T19:34:11.058115+0000","last_active":"2024-09-20T19:35:21.453543+0000","last_peered":"2024-09-20T19:35:21.453543+0000","last_clean":"2024-09-20T19:35:21.453543+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:35:21.453543+0000","last_undegraded":"2024-09-20T19:35:21.453543+0000","last_fullsized":"2024-09-20T19:35:21.453543+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:11.058052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.058052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:32.593298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1763,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451549+0000","last_change":"2024-09-20T19:32:11.594974+0000","last_active":"2024-09-20T19:35:21.451549+0000","last_peered":"2024-09-20T19:35:21.451549+0000","last_clean":"2024-09-20T19:35:21.451549+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:35:21.451549+0000","last_undegraded":"2024-09-20T19:35:21.451549+0000","last_fullsized":"2024-09-20T19:35:21.451549+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:11.594902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.594902+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.031023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1325,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.442865+0000","last_change":"2024-09-20T19:33:43.374566+0000","last_active":"2024-09-20T19:35:22.442865+0000","last_peered":"2024-09-20T19:35:22.442865+0000","last_clean":"2024-09-20T19:35:22.442865+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:35:22.442865+0000","last_undegraded":"2024-09-20T19:35:22.442865+0000","last_fullsized":"2024-09-20T19:35:22.442865+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"809'534","reported_seq":3657,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420334+0000","last_change":"2024-09-20T19:35:17.429905+0000","last_active":"2024-09-20T19:35:18.420334+0000","last_peered":"2024-09-20T19:35:18.420334+0000","last_clean":"2024-09-20T19:35:18.420334+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:35:18.420334+0000","last_undegraded":"2024-09-20T19:35:18.420334+0000","last_fullsized":"2024-09-20T19:35:18.420334+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'490","last_scrub_stamp":"2024-09-20T19:34:46.528129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.528129+0000","objects_scrubbed":6,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.004288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.022838926999999998,"stat_sum":{"num_bytes":19422664,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":104,"num_read_kb":64665,"num_write":73,"num_write_kb":26512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.15","version":"523'15","reported_seq":1604,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.451849+0000","last_change":"2024-09-20T19:34:08.010996+0000","last_active":"2024-09-20T19:35:21.451849+0000","last_peered":"2024-09-20T19:35:21.451849+0000","last_clean":"2024-09-20T19:35:21.451849+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:35:21.451849+0000","last_undegraded":"2024-09-20T19:35:21.451849+0000","last_fullsized":"2024-09-20T19:35:21.451849+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:08.010906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.010906+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1298,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.439753+0000","last_change":"2024-09-20T19:33:43.369959+0000","last_active":"2024-09-20T19:35:22.439753+0000","last_peered":"2024-09-20T19:35:22.439753+0000","last_clean":"2024-09-20T19:35:22.439753+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:35:22.439753+0000","last_undegraded":"2024-09-20T19:35:22.439753+0000","last_fullsized":"2024-09-20T19:35:22.439753+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":212,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.537084+0000","last_change":"2024-09-20T19:34:26.406147+0000","last_active":"2024-09-20T19:35:21.537084+0000","last_peered":"2024-09-20T19:35:21.537084+0000","last_clean":"2024-09-20T19:35:21.537084+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:35:21.537084+0000","last_undegraded":"2024-09-20T19:35:21.537084+0000","last_fullsized":"2024-09-20T19:35:21.537084+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'15","last_scrub_stamp":"2024-09-20T19:33:53.222431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.222431+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1374,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.419823+0000","last_change":"2024-09-20T19:34:55.537273+0000","last_active":"2024-09-20T19:35:18.419823+0000","last_peered":"2024-09-20T19:35:18.419823+0000","last_clean":"2024-09-20T19:35:18.419823+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:35:18.419823+0000","last_undegraded":"2024-09-20T19:35:18.419823+0000","last_fullsized":"2024-09-20T19:35:18.419823+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.537217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.537217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:13.844779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"812'524","reported_seq":3923,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.534396+0000","last_change":"2024-09-20T19:35:21.530167+0000","last_active":"2024-09-20T19:35:21.534396+0000","last_peered":"2024-09-20T19:35:21.534396+0000","last_clean":"2024-09-20T19:35:21.534396+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:35:21.534396+0000","last_undegraded":"2024-09-20T19:35:21.534396+0000","last_fullsized":"2024-09-20T19:35:21.534396+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.093476750999999997,"stat_sum":{"num_bytes":10850446,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":40651,"num_write":21,"num_write_kb":10100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"13f","length":"1"}]},{"pgid":"4.12","version":"523'15","reported_seq":1573,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434406+0000","last_change":"2024-09-20T19:32:31.080255+0000","last_active":"2024-09-20T19:35:21.434406+0000","last_peered":"2024-09-20T19:35:21.434406+0000","last_clean":"2024-09-20T19:35:21.434406+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:35:21.434406+0000","last_undegraded":"2024-09-20T19:35:21.434406+0000","last_fullsized":"2024-09-20T19:35:21.434406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:31.080102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.080102+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:38.313239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1550,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.538141+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:35:21.538141+0000","last_peered":"2024-09-20T19:35:21.538141+0000","last_clean":"2024-09-20T19:35:21.538141+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:35:21.538141+0000","last_undegraded":"2024-09-20T19:35:21.538141+0000","last_fullsized":"2024-09-20T19:35:21.538141+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.355670+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1053,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438296+0000","last_change":"2024-09-20T19:33:43.447329+0000","last_active":"2024-09-20T19:35:22.438296+0000","last_peered":"2024-09-20T19:35:22.438296+0000","last_clean":"2024-09-20T19:35:22.438296+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:35:22.438296+0000","last_undegraded":"2024-09-20T19:35:22.438296+0000","last_fullsized":"2024-09-20T19:35:22.438296+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:15.558468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1308,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.436908+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:35:22.436908+0000","last_peered":"2024-09-20T19:35:22.436908+0000","last_clean":"2024-09-20T19:35:22.436908+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:35:22.436908+0000","last_undegraded":"2024-09-20T19:35:22.436908+0000","last_fullsized":"2024-09-20T19:35:22.436908+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"812'710","reported_seq":4080,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434894+0000","last_change":"2024-09-20T19:35:20.457027+0000","last_active":"2024-09-20T19:35:21.434894+0000","last_peered":"2024-09-20T19:35:21.434894+0000","last_clean":"2024-09-20T19:35:21.434894+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:35:21.434894+0000","last_undegraded":"2024-09-20T19:35:21.434894+0000","last_fullsized":"2024-09-20T19:35:21.434894+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"744'658","last_scrub_stamp":"2024-09-20T19:34:13.122172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.122172+0000","objects_scrubbed":3,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:28.092282+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018642749,"stat_sum":{"num_bytes":6451293,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":37169,"num_write":46,"num_write_kb":13747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"4.13","version":"523'19","reported_seq":1381,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418896+0000","last_change":"2024-09-20T19:34:41.550328+0000","last_active":"2024-09-20T19:35:18.418896+0000","last_peered":"2024-09-20T19:35:18.418896+0000","last_clean":"2024-09-20T19:35:18.418896+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:35:18.418896+0000","last_undegraded":"2024-09-20T19:35:18.418896+0000","last_fullsized":"2024-09-20T19:35:18.418896+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:34:41.550266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.550266+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.918942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1563,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436960+0000","last_change":"2024-09-20T19:31:20.343942+0000","last_active":"2024-09-20T19:35:21.436960+0000","last_peered":"2024-09-20T19:35:21.436960+0000","last_clean":"2024-09-20T19:35:21.436960+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:35:21.436960+0000","last_undegraded":"2024-09-20T19:35:21.436960+0000","last_fullsized":"2024-09-20T19:35:21.436960+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:31:20.343882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:20.343882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:34.329982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1565,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434223+0000","last_change":"2024-09-20T19:34:59.381856+0000","last_active":"2024-09-20T19:35:21.434223+0000","last_peered":"2024-09-20T19:35:21.434223+0000","last_clean":"2024-09-20T19:35:21.434223+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:35:21.434223+0000","last_undegraded":"2024-09-20T19:35:21.434223+0000","last_fullsized":"2024-09-20T19:35:21.434223+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.381728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.381728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:24.871997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1057,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.438295+0000","last_change":"2024-09-20T19:34:54.942515+0000","last_active":"2024-09-20T19:35:22.438295+0000","last_peered":"2024-09-20T19:35:22.438295+0000","last_clean":"2024-09-20T19:35:22.438295+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:35:22.438295+0000","last_undegraded":"2024-09-20T19:35:22.438295+0000","last_fullsized":"2024-09-20T19:35:22.438295+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.942431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.942431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"811'487","reported_seq":3886,"reported_epoch":814,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:22.692796+0000","last_change":"2024-09-20T19:35:22.692836+0000","last_active":"2024-09-20T19:35:22.692796+0000","last_peered":"2024-09-20T19:35:22.692796+0000","last_clean":"2024-09-20T19:35:22.692796+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:35:22.692796+0000","last_undegraded":"2024-09-20T19:35:22.692796+0000","last_fullsized":"2024-09-20T19:35:22.692796+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"777'478","last_scrub_stamp":"2024-09-20T19:34:51.901991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.901991+0000","objects_scrubbed":2,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:58.013687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.25418495099999999,"stat_sum":{"num_bytes":6980097,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":40,"num_write_kb":11327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"4.10","version":"523'8","reported_seq":1211,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436762+0000","last_change":"2024-09-20T19:34:43.272725+0000","last_active":"2024-09-20T19:35:21.436762+0000","last_peered":"2024-09-20T19:35:21.436762+0000","last_clean":"2024-09-20T19:35:21.436762+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:35:21.436762+0000","last_undegraded":"2024-09-20T19:35:21.436762+0000","last_fullsized":"2024-09-20T19:35:21.436762+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:34:43.272551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.272551+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:51.459112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1546,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.452790+0000","last_change":"2024-09-20T19:34:26.399743+0000","last_active":"2024-09-20T19:35:21.452790+0000","last_peered":"2024-09-20T19:35:21.452790+0000","last_clean":"2024-09-20T19:35:21.452790+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:35:21.452790+0000","last_undegraded":"2024-09-20T19:35:21.452790+0000","last_fullsized":"2024-09-20T19:35:21.452790+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:03.325428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:33:03.325428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:41.020789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1543,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.535754+0000","last_change":"2024-09-20T19:35:03.419683+0000","last_active":"2024-09-20T19:35:21.535754+0000","last_peered":"2024-09-20T19:35:21.535754+0000","last_clean":"2024-09-20T19:35:21.535754+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:35:21.535754+0000","last_undegraded":"2024-09-20T19:35:21.535754+0000","last_fullsized":"2024-09-20T19:35:21.535754+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:03.419628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.419628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.847292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1522,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.531347+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:35:21.531347+0000","last_peered":"2024-09-20T19:35:21.531347+0000","last_clean":"2024-09-20T19:35:21.531347+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:35:21.531347+0000","last_undegraded":"2024-09-20T19:35:21.531347+0000","last_fullsized":"2024-09-20T19:35:21.531347+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":3444,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.535928+0000","last_change":"2024-09-20T19:35:19.707438+0000","last_active":"2024-09-20T19:35:21.535928+0000","last_peered":"2024-09-20T19:35:21.535928+0000","last_clean":"2024-09-20T19:35:21.535928+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:35:21.535928+0000","last_undegraded":"2024-09-20T19:35:21.535928+0000","last_fullsized":"2024-09-20T19:35:21.535928+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:29:00.472909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:29:00.472909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.22982865999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"3"}]},{"pgid":"4.11","version":"523'19","reported_seq":418,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.534412+0000","last_change":"2024-09-20T19:33:43.373453+0000","last_active":"2024-09-20T19:35:21.534412+0000","last_peered":"2024-09-20T19:35:21.534412+0000","last_clean":"2024-09-20T19:35:21.534412+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:35:21.534412+0000","last_undegraded":"2024-09-20T19:35:21.534412+0000","last_fullsized":"2024-09-20T19:35:21.534412+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1540,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.539589+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:35:21.539589+0000","last_peered":"2024-09-20T19:35:21.539589+0000","last_clean":"2024-09-20T19:35:21.539589+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:35:21.539589+0000","last_undegraded":"2024-09-20T19:35:21.539589+0000","last_fullsized":"2024-09-20T19:35:21.539589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.449775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1542,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.531372+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:35:21.531372+0000","last_peered":"2024-09-20T19:35:21.531372+0000","last_clean":"2024-09-20T19:35:21.531372+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:35:21.531372+0000","last_undegraded":"2024-09-20T19:35:21.531372+0000","last_fullsized":"2024-09-20T19:35:21.531372+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1065,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.439030+0000","last_change":"2024-09-20T19:33:43.367812+0000","last_active":"2024-09-20T19:35:22.439030+0000","last_peered":"2024-09-20T19:35:22.439030+0000","last_clean":"2024-09-20T19:35:22.439030+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:35:22.439030+0000","last_undegraded":"2024-09-20T19:35:22.439030+0000","last_fullsized":"2024-09-20T19:35:22.439030+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.767876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.767876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"813'639","reported_seq":4040,"reported_epoch":813,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:21.665473+0000","last_change":"2024-09-20T19:35:21.665473+0000","last_active":"2024-09-20T19:35:21.665473+0000","last_peered":"2024-09-20T19:35:21.665473+0000","last_clean":"2024-09-20T19:35:21.665473+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:35:21.665473+0000","last_undegraded":"2024-09-20T19:35:21.665473+0000","last_fullsized":"2024-09-20T19:35:21.665473+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"721'544","last_scrub_stamp":"2024-09-20T19:34:00.387013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:00.387013+0000","objects_scrubbed":3,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":1,"snaptrim_duration":0.22860302800000001,"stat_sum":{"num_bytes":21045753,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":35475,"num_write":32,"num_write_kb":18513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1c","version":"77'1","reported_seq":1528,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.533826+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:35:21.533826+0000","last_peered":"2024-09-20T19:35:21.533826+0000","last_clean":"2024-09-20T19:35:21.533826+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:35:21.533826+0000","last_undegraded":"2024-09-20T19:35:21.533826+0000","last_fullsized":"2024-09-20T19:35:21.533826+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1567,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.450742+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:35:21.450742+0000","last_peered":"2024-09-20T19:35:21.450742+0000","last_clean":"2024-09-20T19:35:21.450742+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:35:21.450742+0000","last_undegraded":"2024-09-20T19:35:21.450742+0000","last_fullsized":"2024-09-20T19:35:21.450742+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.363084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":1352,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.440956+0000","last_change":"2024-09-20T19:35:05.740151+0000","last_active":"2024-09-20T19:35:22.440956+0000","last_peered":"2024-09-20T19:35:22.440956+0000","last_clean":"2024-09-20T19:35:22.440956+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:35:22.440956+0000","last_undegraded":"2024-09-20T19:35:22.440956+0000","last_fullsized":"2024-09-20T19:35:22.440956+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:35:05.739968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:05.739968+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:18.343465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1767,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.539441+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:35:21.539441+0000","last_peered":"2024-09-20T19:35:21.539441+0000","last_clean":"2024-09-20T19:35:21.539441+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:35:21.539441+0000","last_undegraded":"2024-09-20T19:35:21.539441+0000","last_fullsized":"2024-09-20T19:35:21.539441+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.435375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"811'886","reported_seq":4466,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.694320+0000","last_change":"2024-09-20T19:35:21.694380+0000","last_active":"2024-09-20T19:35:21.694320+0000","last_peered":"2024-09-20T19:35:21.694320+0000","last_clean":"2024-09-20T19:35:21.694320+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:35:21.694320+0000","last_undegraded":"2024-09-20T19:35:21.694320+0000","last_fullsized":"2024-09-20T19:35:21.694320+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"722'858","last_scrub_stamp":"2024-09-20T19:34:06.076547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.076547+0000","objects_scrubbed":3,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:22.041706+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.24159629499999999,"stat_sum":{"num_bytes":8707275,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":16188,"num_write":15,"num_write_kb":3293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":1542,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.435171+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:35:21.435171+0000","last_peered":"2024-09-20T19:35:21.435171+0000","last_clean":"2024-09-20T19:35:21.435171+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:35:21.435171+0000","last_undegraded":"2024-09-20T19:35:21.435171+0000","last_fullsized":"2024-09-20T19:35:21.435171+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.466563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":380,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.536091+0000","last_change":"2024-09-20T19:34:43.489833+0000","last_active":"2024-09-20T19:35:21.536091+0000","last_peered":"2024-09-20T19:35:21.536091+0000","last_clean":"2024-09-20T19:35:21.536091+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:35:21.536091+0000","last_undegraded":"2024-09-20T19:35:21.536091+0000","last_fullsized":"2024-09-20T19:35:21.536091+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:34:43.489670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.489670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.767313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1590,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.539519+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:35:21.539519+0000","last_peered":"2024-09-20T19:35:21.539519+0000","last_clean":"2024-09-20T19:35:21.539519+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:35:21.539519+0000","last_undegraded":"2024-09-20T19:35:21.539519+0000","last_fullsized":"2024-09-20T19:35:21.539519+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:09.474818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1337,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.442487+0000","last_change":"2024-09-20T19:34:26.392917+0000","last_active":"2024-09-20T19:35:22.442487+0000","last_peered":"2024-09-20T19:35:22.442487+0000","last_clean":"2024-09-20T19:35:22.442487+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:35:22.442487+0000","last_undegraded":"2024-09-20T19:35:22.442487+0000","last_fullsized":"2024-09-20T19:35:22.442487+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.455533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.455533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"811'628","reported_seq":3817,"reported_epoch":813,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:21.710387+0000","last_change":"2024-09-20T19:35:21.710434+0000","last_active":"2024-09-20T19:35:21.710387+0000","last_peered":"2024-09-20T19:35:21.710387+0000","last_clean":"2024-09-20T19:35:21.710387+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:35:21.710387+0000","last_undegraded":"2024-09-20T19:35:21.710387+0000","last_fullsized":"2024-09-20T19:35:21.710387+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'580","last_scrub_stamp":"2024-09-20T19:34:41.516352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.516352+0000","objects_scrubbed":4,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:04.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17264470400000001,"stat_sum":{"num_bytes":17731636,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":14,"num_write_kb":7581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":1355,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.420117+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:35:18.420117+0000","last_peered":"2024-09-20T19:35:18.420117+0000","last_clean":"2024-09-20T19:35:18.420117+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:35:18.420117+0000","last_undegraded":"2024-09-20T19:35:18.420117+0000","last_fullsized":"2024-09-20T19:35:18.420117+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:24.610221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1550,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.536780+0000","last_change":"2024-09-20T19:32:15.866641+0000","last_active":"2024-09-20T19:35:21.536780+0000","last_peered":"2024-09-20T19:35:21.536780+0000","last_clean":"2024-09-20T19:35:21.536780+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:35:21.536780+0000","last_undegraded":"2024-09-20T19:35:21.536780+0000","last_fullsized":"2024-09-20T19:35:21.536780+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.866572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.866572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":138,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.434225+0000","last_change":"2024-09-20T19:34:26.411567+0000","last_active":"2024-09-20T19:35:21.434225+0000","last_peered":"2024-09-20T19:35:21.434225+0000","last_clean":"2024-09-20T19:35:21.434225+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:35:21.434225+0000","last_undegraded":"2024-09-20T19:35:21.434225+0000","last_fullsized":"2024-09-20T19:35:21.434225+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:33:05.356320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:05.356320+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:37.327595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1553,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436687+0000","last_change":"2024-09-20T19:32:16.075391+0000","last_active":"2024-09-20T19:35:21.436687+0000","last_peered":"2024-09-20T19:35:21.436687+0000","last_clean":"2024-09-20T19:35:21.436687+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:35:21.436687+0000","last_undegraded":"2024-09-20T19:35:21.436687+0000","last_fullsized":"2024-09-20T19:35:21.436687+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.075336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.075336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:39.344567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":3650,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.664282+0000","last_change":"2024-09-20T19:35:21.665277+0000","last_active":"2024-09-20T19:35:21.664282+0000","last_peered":"2024-09-20T19:35:21.664282+0000","last_clean":"2024-09-20T19:35:21.664282+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:35:21.664282+0000","last_undegraded":"2024-09-20T19:35:21.664282+0000","last_fullsized":"2024-09-20T19:35:21.664282+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:33:59.395301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.395301+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.22915549099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"120","length":"1"},{"start":"12a","length":"1"},{"start":"12f","length":"1"},{"start":"132","length":"1"},{"start":"136","length":"1"},{"start":"138","length":"1"},{"start":"13a","length":"1"},{"start":"13f","length":"1"},{"start":"142","length":"4"}]},{"pgid":"6.1f","version":"0'0","reported_seq":190,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-20T19:35:22.437141+0000","last_change":"2024-09-20T19:33:58.560877+0000","last_active":"2024-09-20T19:35:22.437141+0000","last_peered":"2024-09-20T19:35:22.437141+0000","last_clean":"2024-09-20T19:35:22.437141+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:35:22.437141+0000","last_undegraded":"2024-09-20T19:35:22.437141+0000","last_fullsized":"2024-09-20T19:35:22.437141+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:58.560821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:58.560821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1035,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.535371+0000","last_change":"2024-09-20T19:34:51.148618+0000","last_active":"2024-09-20T19:35:21.535371+0000","last_peered":"2024-09-20T19:35:21.535371+0000","last_clean":"2024-09-20T19:35:21.535371+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:35:21.535371+0000","last_undegraded":"2024-09-20T19:35:21.535371+0000","last_fullsized":"2024-09-20T19:35:21.535371+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.148494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.148494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.914652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":423,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-20T19:35:21.436875+0000","last_change":"2024-09-20T19:33:43.461467+0000","last_active":"2024-09-20T19:35:21.436875+0000","last_peered":"2024-09-20T19:35:21.436875+0000","last_clean":"2024-09-20T19:35:21.436875+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:35:21.436875+0000","last_undegraded":"2024-09-20T19:35:21.436875+0000","last_fullsized":"2024-09-20T19:35:21.436875+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:10.800347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.800347+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:35.311263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1368,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-20T19:35:18.418948+0000","last_change":"2024-09-20T19:34:44.512596+0000","last_active":"2024-09-20T19:35:18.418948+0000","last_peered":"2024-09-20T19:35:18.418948+0000","last_clean":"2024-09-20T19:35:18.418948+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:35:18.418948+0000","last_undegraded":"2024-09-20T19:35:18.418948+0000","last_fullsized":"2024-09-20T19:35:18.418948+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.512457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.512457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:51.089920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":333711794,"num_objects":160,"num_object_clones":116,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":11,"num_read":1502,"num_read_kb":1088938,"num_write":1107,"num_write_kb":410334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826224640,"data_stored":1118944385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":22683,"ondisk_log_size":22683,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1016,"num_read_kb":1011,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907160,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184132,"kb_used_data":130732,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93561020,"statfs":{"total":95995035648,"available":95806484480,"internally_reserved":0,"allocated":133869568,"data_stored":174561889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234211,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149736,"kb_used_data":104656,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93595416,"statfs":{"total":95995035648,"available":95841705984,"internally_reserved":0,"allocated":107167744,"data_stored":141205231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561261,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153016,"kb_used_data":95456,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93592136,"statfs":{"total":95995035648,"available":95838347264,"internally_reserved":0,"allocated":97746944,"data_stored":132032301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928571,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112472,"kb_used_data":84612,"kb_used_omap":35,"kb_used_meta":27804,"kb_avail":93632680,"statfs":{"total":95995035648,"available":95879864320,"internally_reserved":0,"allocated":86642688,"data_stored":115654641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":28472241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378897,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180016,"kb_used_data":122456,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93565136,"statfs":{"total":95995035648,"available":95810699264,"internally_reserved":0,"allocated":125394944,"data_stored":156325329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155020,"kb_used_data":105780,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93590132,"statfs":{"total":95995035648,"available":95836295168,"internally_reserved":0,"allocated":108318720,"data_stored":161400739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648064,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150308,"kb_used_data":117200,"kb_used_omap":49,"kb_used_meta":33038,"kb_avail":93594844,"statfs":{"total":95995035648,"available":95841120256,"internally_reserved":0,"allocated":120012800,"data_stored":160610917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":33831465},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119005184,"data_stored":160039933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104464384,"data_stored":157979651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":120930304,"data_stored":152296573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":88150016,"data_stored":114428652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":96165888,"data_stored":130878351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":102715392,"data_stored":137201620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":132554752,"data_stored":173671274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:35:24.931 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T19:35:24.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:35:24.932 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-20T19:35:24.932 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.875299 2024-09-20T19:35:24.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 6 0.875298672369114 2024-09-20T19:35:24.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:24.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:24.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:25.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:25.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:25.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:25.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:25.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:25.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2890: done (0 left) 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2891: read oid 42 snap -1 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2891: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2892: read oid 32 snap -1 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2892: expect (ObjNum 969 snap 321 seq_num 969) 2024-09-20T19:35:25.392 INFO:tasks.rados.rados.0.smithi042.stdout:2893: delete oid 40 current snap is 330 2024-09-20T19:35:25.400 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.400 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.400 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.400 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.400 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.414 INFO:tasks.rados.rados.0.smithi042.stdout:2891: expect (ObjNum 973 snap 321 seq_num 973) 2024-09-20T19:35:25.449 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:25 smithi162 bash[24557]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.572 INFO:tasks.rados.rados.0.smithi042.stdout:2892: expect (ObjNum 969 snap 321 seq_num 969) 2024-09-20T19:35:25.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:25.603 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[22706]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: audit 2024-09-20T19:35:23.954552+0000 mgr.y (mgr.24491) 655 : audit [DBG] from='client.25841 v1:172.21.15.42:0/2848433143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.385195+0000 mon.a (mon.0) 2361 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.396213+0000 mgr.y (mgr.24491) 656 : cluster [DBG] pgmap v1222: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 319 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.3 MiB/s wr, 44 op/s 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.917837+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:25 smithi042 bash[32075]: cluster 2024-09-20T19:35:24.919526+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:25.707 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:25.729 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:25.729 INFO:tasks.rados.rados.0.smithi042.stdout:2891: done (2 left) 2024-09-20T19:35:25.729 INFO:tasks.rados.rados.0.smithi042.stdout:2892: done (1 left) 2024-09-20T19:35:25.730 INFO:tasks.rados.rados.0.smithi042.stdout:2893: done (0 left) 2024-09-20T19:35:25.730 INFO:tasks.rados.rados.0.smithi042.stdout:2894: delete oid 4 current snap is 330 2024-09-20T19:35:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:25.732 INFO:tasks.rados.rados.0.smithi042.stdout:2894: done (0 left) 2024-09-20T19:35:25.732 INFO:tasks.rados.rados.0.smithi042.stdout:2895: write oid 15 current snap is 330 2024-09-20T19:35:25.732 INFO:tasks.rados.rados.0.smithi042.stdout:2895: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-09-20T19:35:25.739 INFO:tasks.rados.rados.0.smithi042.stdout:2895: writing smithi04268048-15 from 524649 to 1258988 tid 1 2024-09-20T19:35:25.747 INFO:tasks.rados.rados.0.smithi042.stdout:2895: writing smithi04268048-15 from 1910779 to 2613118 tid 2 2024-09-20T19:35:25.747 INFO:tasks.rados.rados.0.smithi042.stdout:2896: read oid 47 snap 319 2024-09-20T19:35:25.747 INFO:tasks.rados.rados.0.smithi042.stdout:2896: expect deleted 2024-09-20T19:35:25.747 INFO:tasks.rados.rados.0.smithi042.stdout:2897: delete oid 7 current snap is 330 2024-09-20T19:35:25.747 INFO:tasks.rados.rados.0.smithi042.stdout:2895: finishing write tid 1 to smithi04268048-15 2024-09-20T19:35:25.752 INFO:tasks.rados.rados.0.smithi042.stdout:2896: done (2 left) 2024-09-20T19:35:25.752 INFO:tasks.rados.rados.0.smithi042.stdout:2897: done (1 left) 2024-09-20T19:35:25.752 INFO:tasks.rados.rados.0.smithi042.stdout:2898: read oid 39 snap -1 2024-09-20T19:35:25.752 INFO:tasks.rados.rados.0.smithi042.stdout:2898: expect (ObjNum 900 snap 296 seq_num 900) 2024-09-20T19:35:25.752 INFO:tasks.rados.rados.0.smithi042.stdout:2899: delete oid 28 current snap is 330 2024-09-20T19:35:25.753 INFO:tasks.rados.rados.0.smithi042.stdout:2895: finishing write tid 2 to smithi04268048-15 2024-09-20T19:35:25.756 INFO:tasks.rados.rados.0.smithi042.stdout:2895: finishing write tid 3 to smithi04268048-15 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2899: done (2 left) 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2900: read oid 48 snap 290 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2900: expect (ObjNum 868 snap 288 seq_num 868) 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2901: read oid 8 snap -1 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2901: expect (ObjNum 847 snap 278 seq_num 847) 2024-09-20T19:35:25.757 INFO:tasks.rados.rados.0.smithi042.stdout:2902: snap_create 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:2895: finishing write tid 4 to smithi04268048-15 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:2895: oid 15 updating version 0 to 938 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:2895: oid 15 updating version 938 to 939 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:2895: oid 15 updating version 939 to 940 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 940 (ObjNum 1007 snap 330 seq_num 1007) dirty exists 2024-09-20T19:35:25.758 INFO:tasks.rados.rados.0.smithi042.stdout:2895: left oid 15 (ObjNum 1007 snap 330 seq_num 1007) 2024-09-20T19:35:25.759 INFO:tasks.rados.rados.0.smithi042.stdout:2900: expect (ObjNum 868 snap 288 seq_num 868) 2024-09-20T19:35:25.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:25.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:25.880 INFO:tasks.rados.rados.0.smithi042.stdout:2901: expect (ObjNum 847 snap 278 seq_num 847) 2024-09-20T19:35:25.899 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:25.926 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1561613 2024-09-20T19:35:25.939 INFO:tasks.rados.rados.0.smithi042.stdout:2898: expect (ObjNum 900 snap 296 seq_num 900) 2024-09-20T19:35:25.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:25.976 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2964923 2024-09-20T19:35:25.982 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:25.983 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:25.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:26.023 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:35:26.023 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:35:26.023 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:35:26.023 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:35:26.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:26.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:26.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:26.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:26.400 INFO:tasks.rados.rados.0.smithi042.stdout:2895: done (4 left) 2024-09-20T19:35:26.401 INFO:tasks.rados.rados.0.smithi042.stdout:2898: done (3 left) 2024-09-20T19:35:26.401 INFO:tasks.rados.rados.0.smithi042.stdout:2900: done (2 left) 2024-09-20T19:35:26.401 INFO:tasks.rados.rados.0.smithi042.stdout:2901: done (1 left) 2024-09-20T19:35:26.401 INFO:tasks.rados.rados.0.smithi042.stdout:2902: done (0 left) 2024-09-20T19:35:26.401 INFO:tasks.rados.rados.0.smithi042.stdout:2903: delete oid 45 current snap is 331 2024-09-20T19:35:26.404 INFO:tasks.rados.rados.0.smithi042.stdout:2903: done (0 left) 2024-09-20T19:35:26.404 INFO:tasks.rados.rados.0.smithi042.stdout:2904: copy_from oid 15 from oid 18 current snap is 331 2024-09-20T19:35:26.404 INFO:tasks.rados.rados.0.smithi042.stdout:2905: copy_from oid 27 from oid 10 current snap is 331 2024-09-20T19:35:26.404 INFO:tasks.rados.rados.0.smithi042.stdout:2906: snap_remove snap 327 2024-09-20T19:35:26.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:26.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:26 smithi162 bash[24557]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.479 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:26 smithi162 bash[24557]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:26.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:26.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:26 smithi042 bash[32075]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:26 smithi042 bash[32075]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:26 smithi042 bash[22706]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:26 smithi042 bash[22706]: cluster 2024-09-20T19:35:25.388303+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2024-09-20T19:35:26.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:26.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:26.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:26.889 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:26.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:26.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:35:26.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:27.021 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:27.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:27.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:27.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:27.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:27.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:27.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:27.424 INFO:tasks.rados.rados.0.smithi042.stdout:2906: done (2 left) 2024-09-20T19:35:27.424 INFO:tasks.rados.rados.0.smithi042.stdout:2907: rollback oid 40 current snap is 331 2024-09-20T19:35:27.424 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 309 2024-09-20T19:35:27.425 INFO:tasks.rados.rados.0.smithi042.stdout:2908: write oid 43 current snap is 331 2024-09-20T19:35:27.425 INFO:tasks.rados.rados.0.smithi042.stdout:2908: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-09-20T19:35:27.427 INFO:tasks.rados.rados.0.smithi042.stdout:2908: writing smithi04268048-43 from 591877 to 1227212 tid 1 2024-09-20T19:35:27.434 INFO:tasks.rados.rados.0.smithi042.stdout:2908: writing smithi04268048-43 from 1628354 to 2338031 tid 2 2024-09-20T19:35:27.434 INFO:tasks.rados.rados.0.smithi042.stdout:2909: write oid 8 current snap is 331 2024-09-20T19:35:27.434 INFO:tasks.rados.rados.0.smithi042.stdout:2904: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:35:27.434 INFO:tasks.rados.rados.0.smithi042.stdout:2909: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-20T19:35:27.440 INFO:tasks.rados.rados.0.smithi042.stdout:2909: writing smithi04268048-8 from 608684 to 1035621 tid 1 2024-09-20T19:35:27.448 INFO:tasks.rados.rados.0.smithi042.stdout:2909: writing smithi04268048-8 from 1803189 to 2372877 tid 2 2024-09-20T19:35:27.452 INFO:tasks.rados.rados.0.smithi042.stdout:2909: writing smithi04268048-8 from 2924304 to 3008684 tid 3 2024-09-20T19:35:27.452 INFO:tasks.rados.rados.0.smithi042.stdout:2910: rollback oid 25 current snap is 331 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 319 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2904: finishing copy_from to smithi04268048-15 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1137 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2905: finishing copy_from racing read to smithi04268048-27 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2905: finishing copy_from to smithi04268048-27 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 875 (ObjNum 997 snap 326 seq_num 997) dirty exists 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2907: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2904: done (5 left) 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2905: done (4 left) 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2911: rollback oid 22 current snap is 331 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 309 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:2907: finishing rollback tid 2 to smithi04268048-40 2024-09-20T19:35:27.453 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1060 (ObjNum 897 snap 295 seq_num 897) dirty exists 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: finishing write tid 1 to smithi04268048-43 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: finishing write tid 2 to smithi04268048-43 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: finishing write tid 3 to smithi04268048-43 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: finishing write tid 4 to smithi04268048-43 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: oid 43 updating version 0 to 736 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: oid 43 updating version 736 to 737 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: oid 43 updating version 737 to 738 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 738 (ObjNum 1011 snap 331 seq_num 1011) dirty exists 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: left oid 43 (ObjNum 1011 snap 331 seq_num 1011) 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2909: finishing write tid 1 to smithi04268048-8 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2909: finishing write tid 2 to smithi04268048-8 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2907: done (4 left) 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2908: done (3 left) 2024-09-20T19:35:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:2912: delete oid 13 current snap is 331 2024-09-20T19:35:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:2912: done (3 left) 2024-09-20T19:35:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:2913: write oid 1 current snap is 331 2024-09-20T19:35:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:2910: finishing rollback tid 0 to smithi04268048-25 2024-09-20T19:35:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:2911: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:35:27.455 INFO:tasks.rados.rados.0.smithi042.stdout:2913: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-20T19:35:27.462 INFO:tasks.rados.rados.0.smithi042.stdout:2913: writing smithi04268048-1 from 642298 to 1219733 tid 1 2024-09-20T19:35:27.469 INFO:tasks.rados.rados.0.smithi042.stdout:2913: writing smithi04268048-1 from 1920153 to 2493510 tid 2 2024-09-20T19:35:27.473 INFO:tasks.rados.rados.0.smithi042.stdout:2913: writing smithi04268048-1 from 2979150 to 3042298 tid 3 2024-09-20T19:35:27.473 INFO:tasks.rados.rados.0.smithi042.stdout:2914: rollback oid 34 current snap is 331 2024-09-20T19:35:27.473 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 299 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: finishing write tid 3 to smithi04268048-8 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: finishing write tid 4 to smithi04268048-8 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2911: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: finishing write tid 5 to smithi04268048-8 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: oid 8 updating version 0 to 894 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: oid 8 version 894 is already newer than 893 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: oid 8 version 894 is already newer than 892 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: oid 8 updating version 894 to 895 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 895 (ObjNum 1012 snap 331 seq_num 1012) dirty exists 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2909: left oid 8 (ObjNum 1012 snap 331 seq_num 1012) 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2911: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1030 (ObjNum 890 snap 294 seq_num 890) dirty exists 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2910: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:35:27.474 INFO:tasks.rados.rados.0.smithi042.stdout:2910: finishing rollback tid 2 to smithi04268048-25 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 967 (ObjNum 916 snap 302 seq_num 916) dirty exists 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2909: done (4 left) 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2910: done (3 left) 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2911: done (2 left) 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2915: read oid 44 snap -1 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2915: expect deleted 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2916: rollback oid 22 current snap is 331 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 330 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2913: finishing write tid 1 to smithi04268048-1 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2913: finishing write tid 2 to smithi04268048-1 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2917: read oid 8 snap -1 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2917: expect (ObjNum 1012 snap 331 seq_num 1012) 2024-09-20T19:35:27.475 INFO:tasks.rados.rados.0.smithi042.stdout:2918: snap_create 2024-09-20T19:35:27.476 INFO:tasks.rados.rados.0.smithi042.stdout:2913: finishing write tid 3 to smithi04268048-1 2024-09-20T19:35:27.476 INFO:tasks.rados.rados.0.smithi042.stdout:2913: finishing write tid 4 to smithi04268048-1 2024-09-20T19:35:27.476 INFO:tasks.rados.rados.0.smithi042.stdout:2913: finishing write tid 5 to smithi04268048-1 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2913: oid 1 updating version 0 to 1126 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2913: oid 1 updating version 1126 to 1127 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2913: oid 1 version 1127 is already newer than 1125 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2913: oid 1 updating version 1127 to 1128 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1128 (ObjNum 1014 snap 331 seq_num 1014) dirty exists 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2913: left oid 1 (ObjNum 1014 snap 331 seq_num 1014) 2024-09-20T19:35:27.477 INFO:tasks.rados.rados.0.smithi042.stdout:2916: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:2916: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:2914: finishing rollback tid 0 to smithi04268048-34 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:2916: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1033 (ObjNum 890 snap 294 seq_num 890) dirty exists 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:2914: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:2914: finishing rollback tid 2 to smithi04268048-34 2024-09-20T19:35:27.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1229 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:35:27.483 INFO:tasks.rados.rados.0.smithi042.stdout:2917: expect (ObjNum 1012 snap 331 seq_num 1012) 2024-09-20T19:35:27.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:27.533 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:27.580 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1613035 2024-09-20T19:35:27.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:27.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:27 smithi162 bash[24557]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:27 smithi162 bash[24557]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:27 smithi162 bash[24557]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.647 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:27 smithi162 bash[24557]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[22706]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[22706]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[22706]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[22706]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[32075]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[32075]: cluster 2024-09-20T19:35:26.397770+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-09-20T19:35:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[32075]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:27 smithi042 bash[32075]: cluster 2024-09-20T19:35:26.397901+0000 mgr.y (mgr.24491) 657 : cluster [DBG] pgmap v1225: 178 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 155 active+clean; 314 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2024-09-20T19:35:27.765 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:27.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:27.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:28.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:28.153 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:28.282 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2913: done (5 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2914: done (4 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2915: done (3 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2916: done (2 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2917: done (1 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2918: done (0 left) 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2919: write oid 1 current snap is 332 2024-09-20T19:35:28.384 INFO:tasks.rados.rados.0.smithi042.stdout:2919: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-20T19:35:28.393 INFO:tasks.rados.rados.0.smithi042.stdout:2919: writing smithi04268048-1 from 659105 to 1311789 tid 1 2024-09-20T19:35:28.402 INFO:tasks.rados.rados.0.smithi042.stdout:2919: writing smithi04268048-1 from 1862282 to 2579297 tid 2 2024-09-20T19:35:28.402 INFO:tasks.rados.rados.0.smithi042.stdout:2920: copy_from oid 7 from oid 45 current snap is 332 2024-09-20T19:35:28.402 INFO:tasks.rados.rados.0.smithi042.stdout:2921: copy_from oid 41 from oid 26 current snap is 332 2024-09-20T19:35:28.402 INFO:tasks.rados.rados.0.smithi042.stdout:2922: snap_create 2024-09-20T19:35:28.411 INFO:tasks.rados.rados.0.smithi042.stdout:2920: finishing copy_from to smithi04268048-7 2024-09-20T19:35:28.411 INFO:tasks.rados.rados.0.smithi042.stdout:2920: got expected ENOENT (src dne) 2024-09-20T19:35:28.412 INFO:tasks.rados.rados.0.smithi042.stdout:2920: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:35:28.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.415 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:28.418 INFO:tasks.rados.rados.0.smithi042.stdout:2919: finishing write tid 1 to smithi04268048-1 2024-09-20T19:35:28.422 INFO:tasks.rados.rados.0.smithi042.stdout:2919: finishing write tid 2 to smithi04268048-1 2024-09-20T19:35:28.422 INFO:tasks.rados.rados.0.smithi042.stdout:2919: finishing write tid 3 to smithi04268048-1 2024-09-20T19:35:28.422 INFO:tasks.rados.rados.0.smithi042.stdout:2919: finishing write tid 4 to smithi04268048-1 2024-09-20T19:35:28.423 INFO:tasks.rados.rados.0.smithi042.stdout:2919: oid 1 updating version 0 to 1131 2024-09-20T19:35:28.423 INFO:tasks.rados.rados.0.smithi042.stdout:2919: oid 1 version 1131 is already newer than 1130 2024-09-20T19:35:28.423 INFO:tasks.rados.rados.0.smithi042.stdout:2919: oid 1 version 1131 is already newer than 1129 2024-09-20T19:35:28.423 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1131 (ObjNum 1015 snap 332 seq_num 1015) dirty exists 2024-09-20T19:35:28.423 INFO:tasks.rados.rados.0.smithi042.stdout:2919: left oid 1 (ObjNum 1015 snap 332 seq_num 1015) 2024-09-20T19:35:28.431 INFO:tasks.rados.rados.0.smithi042.stdout:2921: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:35:28.433 INFO:tasks.rados.rados.0.smithi042.stdout:2921: finishing copy_from to smithi04268048-41 2024-09-20T19:35:28.433 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1218 (ObjNum 940 snap 314 seq_num 940) dirty exists 2024-09-20T19:35:28.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:28.542 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:28.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:28 smithi162 bash[24557]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[32075]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: cluster 2024-09-20T19:35:27.414967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: audit 2024-09-20T19:35:28.269156+0000 mon.a (mon.0) 2365 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:28 smithi042 bash[22706]: cluster 2024-09-20T19:35:28.381399+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2024-09-20T19:35:28.702 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:28.805 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:28.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:28.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:29.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:29.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:29.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:29.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:2919: done (3 left) 2024-09-20T19:35:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:2920: done (2 left) 2024-09-20T19:35:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:2921: done (1 left) 2024-09-20T19:35:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:2922: done (0 left) 2024-09-20T19:35:29.383 INFO:tasks.rados.rados.0.smithi042.stdout:2923: write (excl) oid 38 current snap is 333 2024-09-20T19:35:29.384 INFO:tasks.rados.rados.0.smithi042.stdout:2923: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-20T19:35:29.394 INFO:tasks.rados.rados.0.smithi042.stdout:2923: writing smithi04268048-38 from 675912 to 1403845 tid 1 2024-09-20T19:35:29.402 INFO:tasks.rados.rados.0.smithi042.stdout:2923: writing smithi04268048-38 from 1920764 to 2381437 tid 2 2024-09-20T19:35:29.408 INFO:tasks.rados.rados.0.smithi042.stdout:2923: writing smithi04268048-38 from 2801290 to 3075912 tid 3 2024-09-20T19:35:29.409 INFO:tasks.rados.rados.0.smithi042.stdout:2924: snap_create 2024-09-20T19:35:29.409 INFO:tasks.rados.rados.0.smithi042.stdout:2923: finishing write tid 1 to smithi04268048-38 2024-09-20T19:35:29.409 INFO:tasks.rados.rados.0.smithi042.stdout:2923: finishing write tid 2 to smithi04268048-38 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: finishing write tid 3 to smithi04268048-38 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: finishing write tid 4 to smithi04268048-38 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: finishing write tid 5 to smithi04268048-38 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: oid 38 updating version 0 to 1146 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: oid 38 updating version 1146 to 1148 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: oid 38 version 1148 is already newer than 1147 2024-09-20T19:35:29.412 INFO:tasks.rados.rados.0.smithi042.stdout:2923: oid 38 version 1148 is already newer than 1145 2024-09-20T19:35:29.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1148 (ObjNum 1016 snap 333 seq_num 1016) dirty exists 2024-09-20T19:35:29.413 INFO:tasks.rados.rados.0.smithi042.stdout:2923: left oid 38 (ObjNum 1016 snap 333 seq_num 1016) 2024-09-20T19:35:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:29.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:29.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.462 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[32075]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.464 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:29 smithi042 bash[22706]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:29.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:29.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:28.398707+0000 mgr.y (mgr.24491) 658 : cluster [DBG] pgmap v1228: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.001268+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.722 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.002718+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:29.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[24557]: cluster 2024-09-20T19:35:29.379885+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2024-09-20T19:35:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:29.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:29.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:29.971 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:30.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:35:30.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:30.131 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:30.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:30.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:30.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:30.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:30.397 INFO:tasks.rados.rados.0.smithi042.stdout:2923: done (1 left) 2024-09-20T19:35:30.397 INFO:tasks.rados.rados.0.smithi042.stdout:2924: done (0 left) 2024-09-20T19:35:30.397 INFO:tasks.rados.rados.0.smithi042.stdout:2925: write (excl) oid 28 current snap is 334 2024-09-20T19:35:30.398 INFO:tasks.rados.rados.0.smithi042.stdout:2925: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-09-20T19:35:30.408 INFO:tasks.rados.rados.0.smithi042.stdout:2925: writing smithi04268048-28 from 692719 to 1095901 tid 1 2024-09-20T19:35:30.419 INFO:tasks.rados.rados.0.smithi042.stdout:2925: writing smithi04268048-28 from 1579246 to 2299930 tid 2 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2925: writing smithi04268048-28 from 2828713 to 3092719 tid 3 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2926: read oid 5 snap -1 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2926: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2927: read oid 19 snap -1 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2927: expect deleted 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:2928: rollback oid 15 current snap is 334 2024-09-20T19:35:30.427 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 318 2024-09-20T19:35:30.428 INFO:tasks.rados.rados.0.smithi042.stdout:2925: finishing write tid 1 to smithi04268048-28 2024-09-20T19:35:30.428 INFO:tasks.rados.rados.0.smithi042.stdout:2925: finishing write tid 2 to smithi04268048-28 2024-09-20T19:35:30.428 INFO:tasks.rados.rados.0.smithi042.stdout:2929: copy_from oid 22 from oid 27 current snap is 334 2024-09-20T19:35:30.428 INFO:tasks.rados.rados.0.smithi042.stdout:2930: delete oid 42 current snap is 334 2024-09-20T19:35:30.434 INFO:tasks.rados.rados.0.smithi042.stdout:2925: finishing write tid 3 to smithi04268048-28 2024-09-20T19:35:30.435 INFO:tasks.rados.rados.0.smithi042.stdout:2925: finishing write tid 4 to smithi04268048-28 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: finishing write tid 5 to smithi04268048-28 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: oid 28 updating version 0 to 1159 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: oid 28 version 1159 is already newer than 1158 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: oid 28 updating version 1159 to 1160 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: oid 28 updating version 1160 to 1161 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1161 (ObjNum 1017 snap 334 seq_num 1017) dirty exists 2024-09-20T19:35:30.436 INFO:tasks.rados.rados.0.smithi042.stdout:2925: left oid 28 (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:30.439 INFO:tasks.rados.rados.0.smithi042.stdout:2926: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:35:30.450 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 645933 2024-09-20T19:35:30.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:30.516 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:30.585 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:30.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2928: finishing rollback tid 0 to smithi04268048-15 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2928: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2928: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1140 (ObjNum 811 snap 268 seq_num 811) dirty exists 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2929: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2929: finishing copy_from to smithi04268048-22 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 875 (ObjNum 997 snap 326 seq_num 997) dirty exists 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2925: done (5 left) 2024-09-20T19:35:30.635 INFO:tasks.rados.rados.0.smithi042.stdout:2926: done (4 left) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2927: done (3 left) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2928: done (2 left) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2929: done (1 left) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2930: done (0 left) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2931: read oid 27 snap 333 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2931: expect (ObjNum 997 snap 326 seq_num 997) 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2932: rollback oid 16 current snap is 334 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 325 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2933: rollback oid 17 current snap is 334 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 333 2024-09-20T19:35:30.636 INFO:tasks.rados.rados.0.smithi042.stdout:2934: delete oid 3 current snap is 334 2024-09-20T19:35:30.642 INFO:tasks.rados.rados.0.smithi042.stdout:2933: finishing rollback tid 0 to smithi04268048-17 2024-09-20T19:35:30.642 INFO:tasks.rados.rados.0.smithi042.stdout:2932: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:35:30.643 INFO:tasks.rados.rados.0.smithi042.stdout:2933: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:35:30.643 INFO:tasks.rados.rados.0.smithi042.stdout:2933: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:35:30.643 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1147 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:35:30.643 INFO:tasks.rados.rados.0.smithi042.stdout:2933: done (3 left) 2024-09-20T19:35:30.643 INFO:tasks.rados.rados.0.smithi042.stdout:2934: done (2 left) 2024-09-20T19:35:30.644 INFO:tasks.rados.rados.0.smithi042.stdout:2935: snap_remove snap 303 2024-09-20T19:35:30.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:30.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:30.779 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:30.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:30.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:30.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:31.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:31.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:31.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:31.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.405 INFO:tasks.rados.rados.0.smithi042.stdout:2935: done (2 left) 2024-09-20T19:35:31.405 INFO:tasks.rados.rados.0.smithi042.stdout:2936: snap_create 2024-09-20T19:35:31.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.410 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[22706]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.411 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.412 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.412 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:31 smithi042 bash[32075]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.412 INFO:tasks.rados.rados.0.smithi042.stdout:2932: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:35:31.412 INFO:tasks.rados.rados.0.smithi042.stdout:2932: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:35:31.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1147 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:31.412 INFO:tasks.rados.rados.0.smithi042.stdout:2931: expect (ObjNum 997 snap 326 seq_num 997) 2024-09-20T19:35:31.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:31.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: audit 2024-09-20T19:35:29.758282+0000 mgr.y (mgr.24491) 659 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: cluster 2024-09-20T19:35:30.394881+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.549 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:31 smithi162 bash[24557]: cluster 2024-09-20T19:35:30.399666+0000 mgr.y (mgr.24491) 660 : cluster [DBG] pgmap v1231: 178 pgs: 8 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 162 active+clean; 317 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2024-09-20T19:35:31.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:31.571 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 955660 2024-09-20T19:35:31.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:31.741 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:31.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:31.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:31.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:31.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:32.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:32.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:32.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:32.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:32.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:35:32.401 INFO:tasks.rados.rados.0.smithi042.stdout:2931: done (2 left) 2024-09-20T19:35:32.401 INFO:tasks.rados.rados.0.smithi042.stdout:2932: done (1 left) 2024-09-20T19:35:32.401 INFO:tasks.rados.rados.0.smithi042.stdout:2936: done (0 left) 2024-09-20T19:35:32.401 INFO:tasks.rados.rados.0.smithi042.stdout:2937: write oid 45 current snap is 335 2024-09-20T19:35:32.401 INFO:tasks.rados.rados.0.smithi042.stdout:2937: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-20T19:35:32.410 INFO:tasks.rados.rados.0.smithi042.stdout:2937: writing smithi04268048-45 from 743140 to 1488422 tid 1 2024-09-20T19:35:32.417 INFO:tasks.rados.rados.0.smithi042.stdout:2937: writing smithi04268048-45 from 2154692 to 2622703 tid 2 2024-09-20T19:35:32.417 INFO:tasks.rados.rados.0.smithi042.stdout:2938: snap_remove snap 326 2024-09-20T19:35:32.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.465 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:32.529 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:35:32.529 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:35:32.547 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:35:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:32 smithi162 bash[24557]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.564 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:35:32.564 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:35:32.568 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:32.582 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:35:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:35:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:32.604 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:35:32.604 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:35:32.626 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:35:32.626 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:35:32.646 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:35:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:35:32.669 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:35:32.670 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:35:32.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[32075]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.715 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.716 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: cluster 2024-09-20T19:35:31.399185+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2024-09-20T19:35:32.716 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.716 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: audit 2024-09-20T19:35:31.739520+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.42:0/4287674486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.716 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.716 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:32 smithi042 bash[22706]: audit 2024-09-20T19:35:31.739998+0000 mon.a (mon.0) 2370 : audit [INF] from='client.25847 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:35:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:32.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:32.841 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:32.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:32.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:33.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:33.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:33.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:33.229 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:33.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:33.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:33.377 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.6 primary-affinity to 0.875299 (8573632) 2024-09-20T19:35:33.382 INFO:tasks.rados.rados.0.smithi042.stdout:2938: done (1 left) 2024-09-20T19:35:33.382 INFO:tasks.rados.rados.0.smithi042.stdout:2939: write oid 27 current snap is 335 2024-09-20T19:35:33.382 INFO:tasks.rados.rados.0.smithi042.stdout:2937: finishing write tid 1 to smithi04268048-45 2024-09-20T19:35:33.382 INFO:tasks.rados.rados.0.smithi042.stdout:2939: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-09-20T19:35:33.392 INFO:tasks.rados.rados.0.smithi042.stdout:2939: writing smithi04268048-27 from 759947 to 1180478 tid 1 2024-09-20T19:35:33.405 INFO:tasks.rados.rados.0.smithi042.stdout:2939: writing smithi04268048-27 from 1813174 to 2541196 tid 2 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2940: rollback oid 29 current snap is 335 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 332 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: finishing write tid 2 to smithi04268048-45 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: finishing write tid 3 to smithi04268048-45 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: finishing write tid 4 to smithi04268048-45 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: oid 45 updating version 0 to 1395 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: oid 45 version 1395 is already newer than 1393 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: oid 45 version 1395 is already newer than 1394 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1395 (ObjNum 1020 snap 335 seq_num 1020) dirty exists 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: left oid 45 (ObjNum 1020 snap 335 seq_num 1020) 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2939: finishing write tid 1 to smithi04268048-27 2024-09-20T19:35:33.407 INFO:tasks.rados.rados.0.smithi042.stdout:2937: done (2 left) 2024-09-20T19:35:33.408 INFO:tasks.rados.rados.0.smithi042.stdout:2941: read oid 25 snap -1 2024-09-20T19:35:33.408 INFO:tasks.rados.rados.0.smithi042.stdout:2941: expect (ObjNum 916 snap 302 seq_num 916) 2024-09-20T19:35:33.408 INFO:tasks.rados.rados.0.smithi042.stdout:2942: snap_create 2024-09-20T19:35:33.412 INFO:tasks.rados.rados.0.smithi042.stdout:2940: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:35:33.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 877 (ObjNum 129 snap 0 seq_num 1415596552) dirty dne 2024-09-20T19:35:33.414 INFO:tasks.rados.rados.0.smithi042.stdout:2939: finishing write tid 2 to smithi04268048-27 2024-09-20T19:35:33.414 INFO:tasks.rados.rados.0.smithi042.stdout:2939: finishing write tid 3 to smithi04268048-27 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:2939: finishing write tid 4 to smithi04268048-27 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:2939: oid 27 updating version 0 to 1193 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:2939: oid 27 version 1193 is already newer than 1191 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:2939: oid 27 version 1193 is already newer than 1192 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1193 (ObjNum 1021 snap 335 seq_num 1021) dirty exists 2024-09-20T19:35:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:2939: left oid 27 (ObjNum 1021 snap 335 seq_num 1021) 2024-09-20T19:35:33.417 INFO:tasks.rados.rados.0.smithi042.stdout:2941: expect (ObjNum 916 snap 302 seq_num 916) 2024-09-20T19:35:33.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:33.480 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:35:33.480 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:35:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:33.495 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3368574 2024-09-20T19:35:33.495 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:33.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:33 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.393801+0000 mon.a (mon.0) 2371 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.395275+0000 mon.a (mon.0) 2372 : audit [INF] from='client.25847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.399375+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.400237+0000 mgr.y (mgr.24491) 661 : cluster [DBG] pgmap v1234: 178 pgs: 8 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 161 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:35:33.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.506193+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.42:0/2031947248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:32.679589+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114} : dispatch 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.715942+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:32.719698+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: audit 2024-09-20T19:35:33.375972+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? v1:172.21.15.42:0/587654112' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.875298672369114}]': finished 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:33 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.381085+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2024-09-20T19:35:33.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:33.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:33.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:33.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:34.095 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:34.124 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:34.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:34.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:35:34.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:34.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:34.390 INFO:tasks.rados.rados.0.smithi042.stdout:2939: done (3 left) 2024-09-20T19:35:34.390 INFO:tasks.rados.rados.0.smithi042.stdout:2940: done (2 left) 2024-09-20T19:35:34.390 INFO:tasks.rados.rados.0.smithi042.stdout:2941: done (1 left) 2024-09-20T19:35:34.390 INFO:tasks.rados.rados.0.smithi042.stdout:2942: done (0 left) 2024-09-20T19:35:34.391 INFO:tasks.rados.rados.0.smithi042.stdout:2943: delete oid 45 current snap is 336 2024-09-20T19:35:34.398 INFO:tasks.rados.rados.0.smithi042.stdout:2943: done (0 left) 2024-09-20T19:35:34.398 INFO:tasks.rados.rados.0.smithi042.stdout:2944: write (excl) oid 31 current snap is 336 2024-09-20T19:35:34.398 INFO:tasks.rados.rados.0.smithi042.stdout:2944: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-20T19:35:34.412 INFO:tasks.rados.rados.0.smithi042.stdout:2944: writing smithi04268048-31 from 793561 to 1364590 tid 1 2024-09-20T19:35:34.426 INFO:tasks.rados.rados.0.smithi042.stdout:2944: writing smithi04268048-31 from 1813785 to 2545476 tid 2 2024-09-20T19:35:34.433 INFO:tasks.rados.rados.0.smithi042.stdout:2944: writing smithi04268048-31 from 2993251 to 3193561 tid 3 2024-09-20T19:35:34.433 INFO:tasks.rados.rados.0.smithi042.stdout:2945: read oid 25 snap 331 2024-09-20T19:35:34.433 INFO:tasks.rados.rados.0.smithi042.stdout:2945: expect (ObjNum 916 snap 302 seq_num 916) 2024-09-20T19:35:34.433 INFO:tasks.rados.rados.0.smithi042.stdout:2946: delete oid 39 current snap is 336 2024-09-20T19:35:34.433 INFO:tasks.rados.rados.0.smithi042.stdout:2944: finishing write tid 1 to smithi04268048-31 2024-09-20T19:35:34.434 INFO:tasks.rados.rados.0.smithi042.stdout:2944: finishing write tid 2 to smithi04268048-31 2024-09-20T19:35:34.436 INFO:tasks.rados.rados.0.smithi042.stdout:2944: finishing write tid 3 to smithi04268048-31 2024-09-20T19:35:34.436 INFO:tasks.rados.rados.0.smithi042.stdout:2944: finishing write tid 4 to smithi04268048-31 2024-09-20T19:35:34.437 INFO:tasks.rados.rados.0.smithi042.stdout:2944: finishing write tid 5 to smithi04268048-31 2024-09-20T19:35:34.437 INFO:tasks.rados.rados.0.smithi042.stdout:2944: oid 31 updating version 0 to 904 2024-09-20T19:35:34.437 INFO:tasks.rados.rados.0.smithi042.stdout:2944: oid 31 version 904 is already newer than 903 2024-09-20T19:35:34.437 INFO:tasks.rados.rados.0.smithi042.stdout:2944: oid 31 version 904 is already newer than 902 2024-09-20T19:35:34.437 INFO:tasks.rados.rados.0.smithi042.stdout:2944: oid 31 version 904 is already newer than 901 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 904 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2944: left oid 31 (ObjNum 1023 snap 336 seq_num 1023) 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2944: done (2 left) 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2946: done (1 left) 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2947: read oid 50 snap -1 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2947: expect deleted 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2948: rollback oid 9 current snap is 336 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 333 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2949: write (excl) oid 8 current snap is 336 2024-09-20T19:35:34.438 INFO:tasks.rados.rados.0.smithi042.stdout:2949: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-20T19:35:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:2949: writing smithi04268048-8 from 427175 to 1148702 tid 1 2024-09-20T19:35:34.467 INFO:tasks.rados.rados.0.smithi042.stdout:2949: writing smithi04268048-8 from 1930749 to 2666109 tid 2 2024-09-20T19:35:34.468 INFO:tasks.rados.rados.0.smithi042.stdout:2950: write oid 16 current snap is 336 2024-09-20T19:35:34.468 INFO:tasks.rados.rados.0.smithi042.stdout:2950: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-20T19:35:34.478 INFO:tasks.rados.rados.0.smithi042.stdout:2950: writing smithi04268048-16 from 443982 to 1240758 tid 1 2024-09-20T19:35:34.485 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:34.488 INFO:tasks.rados.rados.0.smithi042.stdout:2950: writing smithi04268048-16 from 1989231 to 2468249 tid 2 2024-09-20T19:35:34.493 INFO:tasks.rados.rados.0.smithi042.stdout:2950: writing smithi04268048-16 from 2959167 to 3243982 tid 3 2024-09-20T19:35:34.494 INFO:tasks.rados.rados.0.smithi042.stdout:2951: copy_from oid 21 from oid 12 current snap is 336 2024-09-20T19:35:34.494 INFO:tasks.rados.rados.0.smithi042.stdout:2948: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:35:34.494 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 982 (ObjNum 129 snap 0 seq_num 1415000160) dirty dne 2024-09-20T19:35:34.494 INFO:tasks.rados.rados.0.smithi042.stdout:2945: expect (ObjNum 916 snap 302 seq_num 916) 2024-09-20T19:35:34.510 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.511 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[32075]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.512 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:34 smithi042 bash[22706]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:34.536 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1416316 2024-09-20T19:35:34.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.273269+0000 osd.2 (osd.2) 251 : cluster [DBG] 2.e scrub starts 2024-09-20T19:35:34.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.276011+0000 osd.2 (osd.2) 252 : cluster [DBG] 2.e scrub ok 2024-09-20T19:35:34.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.302476+0000 osd.1 (osd.1) 285 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:35:34.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:33.305034+0000 osd.1 (osd.1) 286 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:35:34.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:34 smithi162 bash[24557]: cluster 2024-09-20T19:35:34.386014+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2024-09-20T19:35:34.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:34.622 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:34.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:34.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.767 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: finishing write tid 1 to smithi04268048-8 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: finishing write tid 2 to smithi04268048-8 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: finishing write tid 3 to smithi04268048-8 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: finishing write tid 4 to smithi04268048-8 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: oid 8 updating version 0 to 906 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: oid 8 updating version 906 to 908 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: oid 8 version 908 is already newer than 907 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 908 (ObjNum 1025 snap 336 seq_num 1025) dirty exists 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2949: left oid 8 (ObjNum 1025 snap 336 seq_num 1025) 2024-09-20T19:35:34.807 INFO:tasks.rados.rados.0.smithi042.stdout:2950: finishing write tid 1 to smithi04268048-16 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: finishing write tid 2 to smithi04268048-16 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: finishing write tid 3 to smithi04268048-16 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: finishing write tid 4 to smithi04268048-16 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: finishing write tid 5 to smithi04268048-16 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: oid 16 updating version 0 to 1151 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: oid 16 version 1151 is already newer than 1150 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: oid 16 version 1151 is already newer than 1148 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: oid 16 version 1151 is already newer than 1149 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1151 (ObjNum 1026 snap 336 seq_num 1026) dirty exists 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2950: left oid 16 (ObjNum 1026 snap 336 seq_num 1026) 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2951: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2951: finishing copy_from to smithi04268048-21 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 742 (ObjNum 1004 snap 327 seq_num 1004) dirty exists 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2945: done (5 left) 2024-09-20T19:35:34.808 INFO:tasks.rados.rados.0.smithi042.stdout:2947: done (4 left) 2024-09-20T19:35:34.809 INFO:tasks.rados.rados.0.smithi042.stdout:2948: done (3 left) 2024-09-20T19:35:34.809 INFO:tasks.rados.rados.0.smithi042.stdout:2949: done (2 left) 2024-09-20T19:35:34.809 INFO:tasks.rados.rados.0.smithi042.stdout:2950: done (1 left) 2024-09-20T19:35:34.809 INFO:tasks.rados.rados.0.smithi042.stdout:2951: done (0 left) 2024-09-20T19:35:34.809 INFO:tasks.rados.rados.0.smithi042.stdout:2952: snap_create 2024-09-20T19:35:34.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:34.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:34.998 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:35.024 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:35.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:35.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:35.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:35.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:35.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:35.438 INFO:tasks.rados.rados.0.smithi042.stdout:2952: done (0 left) 2024-09-20T19:35:35.438 INFO:tasks.rados.rados.0.smithi042.stdout:2953: write (excl) oid 15 current snap is 337 2024-09-20T19:35:35.438 INFO:tasks.rados.rados.0.smithi042.stdout:2953: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-09-20T19:35:35.446 INFO:tasks.rados.rados.0.smithi042.stdout:2953: writing smithi04268048-15 from 460789 to 1049167 tid 1 2024-09-20T19:35:35.460 INFO:tasks.rados.rados.0.smithi042.stdout:2953: writing smithi04268048-15 from 1764066 to 2503095 tid 2 2024-09-20T19:35:35.467 INFO:tasks.rados.rados.0.smithi042.stdout:2953: writing smithi04268048-15 from 3219296 to 3260789 tid 3 2024-09-20T19:35:35.467 INFO:tasks.rados.rados.0.smithi042.stdout:2954: rollback oid 43 current snap is 337 2024-09-20T19:35:35.467 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 335 2024-09-20T19:35:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:2953: finishing write tid 1 to smithi04268048-15 2024-09-20T19:35:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:2953: finishing write tid 2 to smithi04268048-15 2024-09-20T19:35:35.468 INFO:tasks.rados.rados.0.smithi042.stdout:2955: snap_remove snap 290 2024-09-20T19:35:35.487 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:35:35.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:35.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:35 smithi162 bash[24557]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:35 smithi162 bash[24557]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:35.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:35.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:35.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:35 smithi042 bash[32075]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:35 smithi042 bash[32075]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:35 smithi042 bash[22706]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:35 smithi042 bash[22706]: cluster 2024-09-20T19:35:34.401070+0000 mgr.y (mgr.24491) 662 : cluster [DBG] pgmap v1237: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 336 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:35:35.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:35.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:35.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:35.932 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:35.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:36.035 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:36.063 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:36.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:36.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:36.295 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:36.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:36.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2955: done (2 left) 2024-09-20T19:35:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2956: copy_from oid 12 from oid 9 current snap is 337 2024-09-20T19:35:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2953: finishing write tid 3 to smithi04268048-15 2024-09-20T19:35:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2953: finishing write tid 4 to smithi04268048-15 2024-09-20T19:35:36.451 INFO:tasks.rados.rados.0.smithi042.stdout:2954: finishing rollback tid 0 to smithi04268048-43 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2954: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2954: finishing rollback tid 2 to smithi04268048-43 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 741 (ObjNum 1011 snap 331 seq_num 1011) dirty exists 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: finishing write tid 5 to smithi04268048-15 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: oid 15 updating version 0 to 1144 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: oid 15 version 1144 is already newer than 1143 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: oid 15 version 1144 is already newer than 1141 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: oid 15 version 1144 is already newer than 1142 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1144 (ObjNum 1027 snap 337 seq_num 1027) dirty exists 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: left oid 15 (ObjNum 1027 snap 337 seq_num 1027) 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2953: done (2 left) 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2954: done (1 left) 2024-09-20T19:35:36.452 INFO:tasks.rados.rados.0.smithi042.stdout:2957: snap_create 2024-09-20T19:35:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2956: finishing copy_from to smithi04268048-12 2024-09-20T19:35:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2956: got expected ENOENT (src dne) 2024-09-20T19:35:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:2956: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:35:36.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:36.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:36.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:36.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:36 smithi042 bash[32075]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:36 smithi042 bash[32075]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:36 smithi042 bash[22706]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:36 smithi042 bash[22706]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:36.722 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:36.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:36 smithi162 bash[24557]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:36 smithi162 bash[24557]: cluster 2024-09-20T19:35:35.432645+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2024-09-20T19:35:36.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:36.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:36.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:35:36.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:37.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:37.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:37.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:37.268 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:37.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:37.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:2956: done (1 left) 2024-09-20T19:35:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:2957: done (0 left) 2024-09-20T19:35:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:2958: write (excl) oid 33 current snap is 338 2024-09-20T19:35:37.451 INFO:tasks.rados.rados.0.smithi042.stdout:2958: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-20T19:35:37.457 INFO:tasks.rados.rados.0.smithi042.stdout:2958: writing smithi04268048-33 from 477596 to 1141223 tid 1 2024-09-20T19:35:37.463 INFO:tasks.rados.rados.0.smithi042.stdout:2958: writing smithi04268048-33 from 1706195 to 2188882 tid 2 2024-09-20T19:35:37.470 INFO:tasks.rados.rados.0.smithi042.stdout:2958: writing smithi04268048-33 from 2614013 to 3181054 tid 3 2024-09-20T19:35:37.470 INFO:tasks.rados.rados.0.smithi042.stdout:2959: snap_create 2024-09-20T19:35:37.476 INFO:tasks.rados.rados.0.smithi042.stdout:2958: finishing write tid 1 to smithi04268048-33 2024-09-20T19:35:37.476 INFO:tasks.rados.rados.0.smithi042.stdout:2958: finishing write tid 2 to smithi04268048-33 2024-09-20T19:35:37.476 INFO:tasks.rados.rados.0.smithi042.stdout:2958: finishing write tid 3 to smithi04268048-33 2024-09-20T19:35:37.477 INFO:tasks.rados.rados.0.smithi042.stdout:2958: finishing write tid 4 to smithi04268048-33 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: finishing write tid 5 to smithi04268048-33 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: oid 33 updating version 0 to 1303 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: oid 33 version 1303 is already newer than 1301 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: oid 33 version 1303 is already newer than 1302 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: oid 33 updating version 1303 to 1304 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1304 (ObjNum 1028 snap 338 seq_num 1028) dirty exists 2024-09-20T19:35:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:2958: left oid 33 (ObjNum 1028 snap 338 seq_num 1028) 2024-09-20T19:35:37.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.512 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[32075]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.513 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.514 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:37 smithi042 bash[22706]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.525 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:37.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:37.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:37.787 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.265710+0000 osd.2 (osd.2) 253 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.267810+0000 osd.2 (osd.2) 254 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.401900+0000 mgr.y (mgr.24491) 663 : cluster [DBG] pgmap v1239: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 1011 KiB/s wr, 2 op/s 2024-09-20T19:35:37.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.447400+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2024-09-20T19:35:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.805891+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:35:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:37 smithi162 bash[24557]: cluster 2024-09-20T19:35:36.809936+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:35:37.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:37.919 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:37.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:38.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:38.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:38.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:38.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2958: done (1 left) 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2959: done (0 left) 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2960: copy_from oid 2 from oid 8 current snap is 339 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2961: read oid 43 snap -1 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2961: expect (ObjNum 1011 snap 331 seq_num 1011) 2024-09-20T19:35:38.392 INFO:tasks.rados.rados.0.smithi042.stdout:2962: delete oid 38 current snap is 339 2024-09-20T19:35:38.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:38.419 INFO:tasks.rados.rados.0.smithi042.stdout:2960: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:35:38.419 INFO:tasks.rados.rados.0.smithi042.stdout:2961: expect (ObjNum 1011 snap 331 seq_num 1011) 2024-09-20T19:35:38.419 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:38.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:38.474 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2084439 2024-09-20T19:35:38.506 INFO:tasks.rados.rados.0.smithi042.stdout:2960: finishing copy_from to smithi04268048-2 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 908 (ObjNum 1025 snap 336 seq_num 1025) dirty exists 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2960: done (2 left) 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2961: done (1 left) 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2962: done (0 left) 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2963: read oid 38 snap -1 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2963: expect deleted 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2964: rollback oid 21 current snap is 339 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 337 2024-09-20T19:35:38.515 INFO:tasks.rados.rados.0.smithi042.stdout:2965: read oid 40 snap -1 2024-09-20T19:35:38.516 INFO:tasks.rados.rados.0.smithi042.stdout:2965: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:35:38.516 INFO:tasks.rados.rados.0.smithi042.stdout:2966: delete oid 24 current snap is 339 2024-09-20T19:35:38.516 INFO:tasks.rados.rados.0.smithi042.stdout:2966: done (3 left) 2024-09-20T19:35:38.516 INFO:tasks.rados.rados.0.smithi042.stdout:2967: read oid 35 snap -1 2024-09-20T19:35:38.516 INFO:tasks.rados.rados.0.smithi042.stdout:2967: expect deleted 2024-09-20T19:35:38.517 INFO:tasks.rados.rados.0.smithi042.stdout:2968: write oid 50 current snap is 339 2024-09-20T19:35:38.517 INFO:tasks.rados.rados.0.smithi042.stdout:2968: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-09-20T19:35:38.517 INFO:tasks.rados.rados.0.smithi042.stdout:2968: writing smithi04268048-50 from 528017 to 1017391 tid 1 2024-09-20T19:35:38.523 INFO:tasks.rados.rados.0.smithi042.stdout:2968: writing smithi04268048-50 from 1481641 to 2111655 tid 2 2024-09-20T19:35:38.529 INFO:tasks.rados.rados.0.smithi042.stdout:2968: writing smithi04268048-50 from 2696282 to 3097139 tid 3 2024-09-20T19:35:38.530 INFO:tasks.rados.rados.0.smithi042.stdout:2969: snap_create 2024-09-20T19:35:38.530 INFO:tasks.rados.rados.0.smithi042.stdout:2965: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:35:38.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:38.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:38.583 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 217059 2024-09-20T19:35:38.614 INFO:tasks.rados.rados.0.smithi042.stdout:2964: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2964: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2964: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1225 (ObjNum 1004 snap 327 seq_num 1004) dirty exists 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: finishing write tid 1 to smithi04268048-50 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: finishing write tid 2 to smithi04268048-50 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: finishing write tid 3 to smithi04268048-50 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: finishing write tid 4 to smithi04268048-50 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: finishing write tid 5 to smithi04268048-50 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: oid 50 updating version 0 to 1008 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: oid 50 updating version 1008 to 1011 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: oid 50 version 1011 is already newer than 1010 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:2968: oid 50 version 1011 is already newer than 1009 2024-09-20T19:35:38.615 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1011 (ObjNum 1031 snap 339 seq_num 1031) dirty exists 2024-09-20T19:35:38.616 INFO:tasks.rados.rados.0.smithi042.stdout:2968: left oid 50 (ObjNum 1031 snap 339 seq_num 1031) 2024-09-20T19:35:38.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:38.687 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[22706]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.688 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.689 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:38 smithi042 bash[32075]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.326895+0000 osd.5 (osd.5) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.328335+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.369183+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.372635+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:37.443904+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:38 smithi162 bash[24557]: cluster 2024-09-20T19:35:38.386959+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2024-09-20T19:35:38.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:38.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:38.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:38.965 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:38.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:39.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:39.110 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:39.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:39.228 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:39.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:39.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:39.477 INFO:tasks.rados.rados.0.smithi042.stdout:2963: done (5 left) 2024-09-20T19:35:39.477 INFO:tasks.rados.rados.0.smithi042.stdout:2964: done (4 left) 2024-09-20T19:35:39.477 INFO:tasks.rados.rados.0.smithi042.stdout:2965: done (3 left) 2024-09-20T19:35:39.477 INFO:tasks.rados.rados.0.smithi042.stdout:2967: done (2 left) 2024-09-20T19:35:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:2968: done (1 left) 2024-09-20T19:35:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:2969: done (0 left) 2024-09-20T19:35:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:2970: read oid 13 snap -1 2024-09-20T19:35:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:2970: expect deleted 2024-09-20T19:35:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:2971: delete oid 34 current snap is 340 2024-09-20T19:35:39.486 INFO:tasks.rados.rados.0.smithi042.stdout:2970: done (1 left) 2024-09-20T19:35:39.486 INFO:tasks.rados.rados.0.smithi042.stdout:2971: done (0 left) 2024-09-20T19:35:39.486 INFO:tasks.rados.rados.0.smithi042.stdout:2972: delete oid 39 current snap is 340 2024-09-20T19:35:39.489 INFO:tasks.rados.rados.0.smithi042.stdout:2972: done (0 left) 2024-09-20T19:35:39.489 INFO:tasks.rados.rados.0.smithi042.stdout:2973: write oid 3 current snap is 340 2024-09-20T19:35:39.489 INFO:tasks.rados.rados.0.smithi042.stdout:2973: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-20T19:35:39.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:39.494 INFO:tasks.rados.rados.0.smithi042.stdout:2973: writing smithi04268048-3 from 578438 to 1009912 tid 1 2024-09-20T19:35:39.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:39 smithi162 bash[24557]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:39 smithi162 bash[24557]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:39.501 INFO:tasks.rados.rados.0.smithi042.stdout:2973: writing smithi04268048-3 from 1657087 to 2150781 tid 2 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2973: writing smithi04268048-3 from 2778551 to 3378438 tid 3 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2974: copy_from oid 50 from oid 36 current snap is 340 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2973: finishing write tid 1 to smithi04268048-3 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2973: finishing write tid 2 to smithi04268048-3 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2975: read oid 41 snap -1 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2975: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:39.513 INFO:tasks.rados.rados.0.smithi042.stdout:2976: write oid 9 current snap is 340 2024-09-20T19:35:39.514 INFO:tasks.rados.rados.0.smithi042.stdout:2976: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-20T19:35:39.524 INFO:tasks.rados.rados.0.smithi042.stdout:2976: writing smithi04268048-9 from 595245 to 1101968 tid 1 2024-09-20T19:35:39.535 INFO:tasks.rados.rados.0.smithi042.stdout:2976: writing smithi04268048-9 from 1715569 to 2352921 tid 2 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2976: writing smithi04268048-9 from 2805974 to 3368272 tid 3 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2977: read oid 28 snap -1 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2977: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2978: rollback oid 19 current snap is 340 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 325 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2973: finishing write tid 3 to smithi04268048-3 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2979: read oid 42 snap -1 2024-09-20T19:35:39.543 INFO:tasks.rados.rados.0.smithi042.stdout:2979: expect deleted 2024-09-20T19:35:39.544 INFO:tasks.rados.rados.0.smithi042.stdout:2980: snap_remove snap 328 2024-09-20T19:35:39.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:39.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:39.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:39 smithi042 bash[32075]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:39 smithi042 bash[32075]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:39 smithi042 bash[22706]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:39 smithi042 bash[22706]: cluster 2024-09-20T19:35:38.402923+0000 mgr.y (mgr.24491) 664 : cluster [DBG] pgmap v1243: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 341 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-20T19:35:39.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.755 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:39.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:39.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:39.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:39.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:39.993 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:35:39.993 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:35:39.993 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-20T19:35:39.993 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T19:35:39.993 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T19:35:39.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-20T19:35:40.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:40.026 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:35:40.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:40.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:40.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:40.405 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:40.506 INFO:tasks.rados.rados.0.smithi042.stdout:2980: done (7 left) 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2981: snap_create 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: finishing write tid 4 to smithi04268048-3 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: finishing write tid 5 to smithi04268048-3 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: oid 3 updating version 0 to 964 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: oid 3 updating version 964 to 966 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: oid 3 version 966 is already newer than 965 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: oid 3 updating version 966 to 967 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 967 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2973: left oid 3 (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2974: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2974: finishing copy_from to smithi04268048-50 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 587 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2976: finishing write tid 1 to smithi04268048-9 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:2975: expect (ObjNum 940 snap 314 seq_num 940) 2024-09-20T19:35:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:40.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.517 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[32075]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.518 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:40 smithi042 bash[22706]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:40.601 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:40.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:40.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:40.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:40.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.468246+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2024-09-20T19:35:40.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.653925+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:35:40.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.771 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:40 smithi162 bash[24557]: cluster 2024-09-20T19:35:39.656278+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: finishing write tid 2 to smithi04268048-9 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: finishing write tid 3 to smithi04268048-9 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: finishing write tid 4 to smithi04268048-9 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: finishing write tid 5 to smithi04268048-9 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: oid 9 updating version 0 to 1015 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: oid 9 version 1015 is already newer than 1014 2024-09-20T19:35:40.782 INFO:tasks.rados.rados.0.smithi042.stdout:2976: oid 9 updating version 1015 to 1016 2024-09-20T19:35:40.783 INFO:tasks.rados.rados.0.smithi042.stdout:2976: oid 9 updating version 1016 to 1017 2024-09-20T19:35:40.783 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1017 (ObjNum 1035 snap 340 seq_num 1035) dirty exists 2024-09-20T19:35:40.783 INFO:tasks.rados.rados.0.smithi042.stdout:2976: left oid 9 (ObjNum 1035 snap 340 seq_num 1035) 2024-09-20T19:35:40.783 INFO:tasks.rados.rados.0.smithi042.stdout:2977: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:40.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:40.829 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:35:40.829 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:35:40.847 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:40.881 INFO:tasks.rados.rados.0.smithi042.stdout:2978: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:35:40.881 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 864 (ObjNum 161 snap 0 seq_num 1415027344) dirty dne 2024-09-20T19:35:40.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:40.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:41.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:41.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:41.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:41.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:41.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:41.286 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:41.303 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:41.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:41.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:41.481 INFO:tasks.rados.rados.0.smithi042.stdout:2973: done (7 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2974: done (6 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2975: done (5 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2976: done (4 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2977: done (3 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2978: done (2 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2979: done (1 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2981: done (0 left) 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2982: copy_from oid 22 from oid 49 current snap is 341 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2983: write (excl) oid 25 current snap is 341 2024-09-20T19:35:41.482 INFO:tasks.rados.rados.0.smithi042.stdout:2983: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-20T19:35:41.496 INFO:tasks.rados.rados.0.smithi042.stdout:2983: writing smithi04268048-25 from 612052 to 1194024 tid 1 2024-09-20T19:35:41.508 INFO:tasks.rados.rados.0.smithi042.stdout:2983: writing smithi04268048-25 from 1657698 to 2155061 tid 2 2024-09-20T19:35:41.521 INFO:tasks.rados.rados.0.smithi042.stdout:2983: writing smithi04268048-25 from 2717044 to 3412052 tid 3 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:2984: snap_create 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:2982: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:2983: finishing write tid 1 to smithi04268048-25 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:2982: finishing copy_from to smithi04268048-22 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 882 (ObjNum 731 snap 240 seq_num 731) dirty exists 2024-09-20T19:35:41.522 INFO:tasks.rados.rados.0.smithi042.stdout:2983: finishing write tid 2 to smithi04268048-25 2024-09-20T19:35:41.526 INFO:tasks.rados.rados.0.smithi042.stdout:2983: finishing write tid 3 to smithi04268048-25 2024-09-20T19:35:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:2983: finishing write tid 4 to smithi04268048-25 2024-09-20T19:35:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:2983: finishing write tid 5 to smithi04268048-25 2024-09-20T19:35:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:2983: oid 25 updating version 0 to 983 2024-09-20T19:35:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:2983: oid 25 version 983 is already newer than 982 2024-09-20T19:35:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:2983: oid 25 updating version 983 to 984 2024-09-20T19:35:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:2983: oid 25 version 984 is already newer than 981 2024-09-20T19:35:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 984 (ObjNum 1036 snap 341 seq_num 1036) dirty exists 2024-09-20T19:35:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:2983: left oid 25 (ObjNum 1036 snap 341 seq_num 1036) 2024-09-20T19:35:41.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:41.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:41.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:41 smithi162 bash[24557]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:41.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:35:41.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[22706]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: audit 2024-09-20T19:35:39.769583+0000 mgr.y (mgr.24491) 665 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:41.761 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.403528+0000 mgr.y (mgr.24491) 666 : cluster [DBG] pgmap v1245: 178 pgs: 13 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 339 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.499565+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.666894+0000 osd.5 (osd.5) 257 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.762 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:41 smithi042 bash[32075]: cluster 2024-09-20T19:35:40.668431+0000 osd.5 (osd.5) 258 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:35:41.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:41.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:41.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:41.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:41.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:42.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:42.077 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:42.179 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:42.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:42.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:42.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:42.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:42.499 INFO:tasks.rados.rados.0.smithi042.stdout:2982: done (2 left) 2024-09-20T19:35:42.499 INFO:tasks.rados.rados.0.smithi042.stdout:2983: done (1 left) 2024-09-20T19:35:42.500 INFO:tasks.rados.rados.0.smithi042.stdout:2984: done (0 left) 2024-09-20T19:35:42.500 INFO:tasks.rados.rados.0.smithi042.stdout:2985: read oid 9 snap 341 2024-09-20T19:35:42.500 INFO:tasks.rados.rados.0.smithi042.stdout:2985: expect (ObjNum 1035 snap 340 seq_num 1035) 2024-09-20T19:35:42.500 INFO:tasks.rados.rados.0.smithi042.stdout:2986: delete oid 46 current snap is 342 2024-09-20T19:35:42.506 INFO:tasks.rados.rados.0.smithi042.stdout:2986: done (1 left) 2024-09-20T19:35:42.506 INFO:tasks.rados.rados.0.smithi042.stdout:2987: snap_create 2024-09-20T19:35:42.511 INFO:tasks.rados.rados.0.smithi042.stdout:2985: expect (ObjNum 1035 snap 340 seq_num 1035) 2024-09-20T19:35:42.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:42.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:42 smithi162 bash[24557]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:42.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.838 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[32075]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.479471+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.697110+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.698635+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.832445+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:42 smithi042 bash[22706]: cluster 2024-09-20T19:35:41.834024+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:35:42.856 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:42.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:42.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:42.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:43.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:43.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:43.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:43.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:43.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:43.390 INFO:tasks.rados.rados.0.smithi042.stdout:2985: done (1 left) 2024-09-20T19:35:43.390 INFO:tasks.rados.rados.0.smithi042.stdout:2987: done (0 left) 2024-09-20T19:35:43.390 INFO:tasks.rados.rados.0.smithi042.stdout:2988: copy_from oid 42 from oid 34 current snap is 343 2024-09-20T19:35:43.390 INFO:tasks.rados.rados.0.smithi042.stdout:2989: write oid 21 current snap is 343 2024-09-20T19:35:43.390 INFO:tasks.rados.rados.0.smithi042.stdout:2989: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-20T19:35:43.394 INFO:tasks.rados.rados.0.smithi042.stdout:2989: writing smithi04268048-21 from 645666 to 1378136 tid 1 2024-09-20T19:35:43.402 INFO:tasks.rados.rados.0.smithi042.stdout:2989: writing smithi04268048-21 from 2174662 to 2675694 tid 2 2024-09-20T19:35:43.407 INFO:tasks.rados.rados.0.smithi042.stdout:2989: writing smithi04268048-21 from 3288243 to 3445666 tid 3 2024-09-20T19:35:43.407 INFO:tasks.rados.rados.0.smithi042.stdout:2990: snap_create 2024-09-20T19:35:43.407 INFO:tasks.rados.rados.0.smithi042.stdout:2988: finishing copy_from to smithi04268048-42 2024-09-20T19:35:43.407 INFO:tasks.rados.rados.0.smithi042.stdout:2988: got expected ENOENT (src dne) 2024-09-20T19:35:43.408 INFO:tasks.rados.rados.0.smithi042.stdout:2988: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:35:43.408 INFO:tasks.rados.rados.0.smithi042.stdout:2989: finishing write tid 1 to smithi04268048-21 2024-09-20T19:35:43.409 INFO:tasks.rados.rados.0.smithi042.stdout:2989: finishing write tid 2 to smithi04268048-21 2024-09-20T19:35:43.410 INFO:tasks.rados.rados.0.smithi042.stdout:2989: finishing write tid 3 to smithi04268048-21 2024-09-20T19:35:43.410 INFO:tasks.rados.rados.0.smithi042.stdout:2989: finishing write tid 4 to smithi04268048-21 2024-09-20T19:35:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:2989: finishing write tid 5 to smithi04268048-21 2024-09-20T19:35:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:2989: oid 21 updating version 0 to 1232 2024-09-20T19:35:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:2989: oid 21 version 1232 is already newer than 1229 2024-09-20T19:35:43.412 INFO:tasks.rados.rados.0.smithi042.stdout:2989: oid 21 version 1232 is already newer than 1231 2024-09-20T19:35:43.412 INFO:tasks.rados.rados.0.smithi042.stdout:2989: oid 21 version 1232 is already newer than 1230 2024-09-20T19:35:43.412 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1232 (ObjNum 1038 snap 343 seq_num 1038) dirty exists 2024-09-20T19:35:43.412 INFO:tasks.rados.rados.0.smithi042.stdout:2989: left oid 21 (ObjNum 1038 snap 343 seq_num 1038) 2024-09-20T19:35:43.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:43.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:43.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:43 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:42.405108+0000 mgr.y (mgr.24491) 667 : cluster [DBG] pgmap v1248: 178 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 159 active+clean; 349 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:42.497862+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.210706+0000 osd.6 (osd.6) 178 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:35:43.750 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.214766+0000 osd.6 (osd.6) 179 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:35:43.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: audit 2024-09-20T19:35:43.298697+0000 mon.a (mon.0) 2386 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:43.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.751 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:43 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.385165+0000 mon.a (mon.0) 2387 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2024-09-20T19:35:43.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:43.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:43.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:44.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:44.069 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:35:44.070 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:35:44.086 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:35:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:35:44.102 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:35:44.102 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:35:44.118 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:35:44.119 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:35:44.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:44.139 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:35:44.139 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:35:44.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:44.157 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:35:44.157 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:35:44.175 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:35:44.175 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:35:44.192 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:35:44.193 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:35:44.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:44.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:44.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:44.384 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2988: done (2 left) 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2989: done (1 left) 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2990: done (0 left) 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2991: read oid 29 snap -1 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2991: expect deleted 2024-09-20T19:35:44.395 INFO:tasks.rados.rados.0.smithi042.stdout:2992: copy_from oid 38 from oid 9 current snap is 344 2024-09-20T19:35:44.396 INFO:tasks.rados.rados.0.smithi042.stdout:2993: snap_create 2024-09-20T19:35:44.402 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:44.416 INFO:tasks.rados.rados.0.smithi042.stdout:2992: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:35:44.422 INFO:tasks.rados.rados.0.smithi042.stdout:2992: finishing copy_from to smithi04268048-38 2024-09-20T19:35:44.422 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1017 (ObjNum 1035 snap 340 seq_num 1035) dirty exists 2024-09-20T19:35:44.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:44.523 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:44.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:44.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:44.755 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:44 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.858 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:43.669563+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.698576+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.699997+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.873771+0000 osd.4 (osd.4) 113 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:43.876593+0000 osd.4 (osd.4) 114 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:44.051344+0000 mon.a (mon.0) 2389 : audit [DBG] from='client.? v1:172.21.15.42:0/3056464022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:44.859 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.381412+0000 mon.a (mon.0) 2390 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.381448+0000 mon.a (mon.0) 2391 : cluster [INF] Cluster is now healthy 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: audit 2024-09-20T19:35:44.382933+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.42:0/103858095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.386980+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.421397+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:44 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.422847+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:35:44.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:44.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:45.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:45.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:45.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:45.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:35:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2991: done (2 left) 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2992: done (1 left) 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2993: done (0 left) 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2994: read oid 15 snap 339 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2994: expect (ObjNum 1027 snap 337 seq_num 1027) 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2995: read oid 28 snap -1 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2995: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:45.399 INFO:tasks.rados.rados.0.smithi042.stdout:2996: delete oid 47 current snap is 345 2024-09-20T19:35:45.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:45.432 INFO:tasks.rados.rados.0.smithi042.stdout:2996: done (2 left) 2024-09-20T19:35:45.432 INFO:tasks.rados.rados.0.smithi042.stdout:2997: snap_create 2024-09-20T19:35:45.432 INFO:tasks.rados.rados.0.smithi042.stdout:2995: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:45.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:45.535 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[22706]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:45 smithi042 bash[32075]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.560 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:45.585 INFO:tasks.rados.rados.0.smithi042.stdout:2994: expect (ObjNum 1027 snap 337 seq_num 1027) 2024-09-20T19:35:45.597 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1325599 2024-09-20T19:35:45.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:45.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:45.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.405601+0000 mgr.y (mgr.24491) 668 : cluster [DBG] pgmap v1252: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 20 op/s 2024-09-20T19:35:45.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.729697+0000 osd.5 (osd.5) 263 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:44.736987+0000 osd.5 (osd.5) 264 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.173548+0000 osd.6 (osd.6) 180 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.175806+0000 osd.6 (osd.6) 181 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.803 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:45 smithi162 bash[24557]: cluster 2024-09-20T19:35:45.395270+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2024-09-20T19:35:45.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:45.921 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:45.949 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:45.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:46.053 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:46.180 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:46.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:46.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2994: done (2 left) 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2995: done (1 left) 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2997: done (0 left) 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2998: read oid 5 snap -1 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2998: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:2999: rollback oid 34 current snap is 346 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 324 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:3000: read oid 48 snap -1 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:3000: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:35:46.405 INFO:tasks.rados.rados.0.smithi042.stdout:3001: delete oid 14 current snap is 346 2024-09-20T19:35:46.412 INFO:tasks.rados.rados.0.smithi042.stdout:2999: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:35:46.414 INFO:tasks.rados.rados.0.smithi042.stdout:3000: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:35:46.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:46.452 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:46.487 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1223297 2024-09-20T19:35:46.558 INFO:tasks.rados.rados.0.smithi042.stdout:2999: finishing rollback tid 2 to smithi04268048-34 2024-09-20T19:35:46.558 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1232 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:35:46.558 INFO:tasks.rados.rados.0.smithi042.stdout:2998: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:35:46.558 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:46.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:46.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:46.608 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 15542 2024-09-20T19:35:46.658 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:46.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:46.710 INFO:tasks.rados.rados.0.smithi042.stdout:2998: done (3 left) 2024-09-20T19:35:46.710 INFO:tasks.rados.rados.0.smithi042.stdout:2999: done (2 left) 2024-09-20T19:35:46.710 INFO:tasks.rados.rados.0.smithi042.stdout:3000: done (1 left) 2024-09-20T19:35:46.711 INFO:tasks.rados.rados.0.smithi042.stdout:3001: done (0 left) 2024-09-20T19:35:46.711 INFO:tasks.rados.rados.0.smithi042.stdout:3002: read oid 8 snap -1 2024-09-20T19:35:46.711 INFO:tasks.rados.rados.0.smithi042.stdout:3002: expect (ObjNum 1025 snap 336 seq_num 1025) 2024-09-20T19:35:46.711 INFO:tasks.rados.rados.0.smithi042.stdout:3003: copy_from oid 30 from oid 36 current snap is 346 2024-09-20T19:35:46.711 INFO:tasks.rados.rados.0.smithi042.stdout:3004: snap_create 2024-09-20T19:35:46.718 INFO:tasks.rados.rados.0.smithi042.stdout:3002: expect (ObjNum 1025 snap 336 seq_num 1025) 2024-09-20T19:35:46.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:46.756 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2872103 2024-09-20T19:35:46.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:46.837 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:46.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:46.902 INFO:tasks.rados.rados.0.smithi042.stdout:3003: finishing copy_from racing read to smithi04268048-30 2024-09-20T19:35:46.902 INFO:tasks.rados.rados.0.smithi042.stdout:3003: finishing copy_from to smithi04268048-30 2024-09-20T19:35:46.903 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 587 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:35:46.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.967 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:35:46.967 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T19:35:46.967 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:35:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:46.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:47.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:47.113 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:47.216 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:47.243 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:47.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:47.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:35:47.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:47.415 INFO:tasks.rados.rados.0.smithi042.stdout:3002: done (2 left) 2024-09-20T19:35:47.415 INFO:tasks.rados.rados.0.smithi042.stdout:3003: done (1 left) 2024-09-20T19:35:47.415 INFO:tasks.rados.rados.0.smithi042.stdout:3004: done (0 left) 2024-09-20T19:35:47.415 INFO:tasks.rados.rados.0.smithi042.stdout:3005: write oid 18 current snap is 347 2024-09-20T19:35:47.415 INFO:tasks.rados.rados.0.smithi042.stdout:3005: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-20T19:35:47.422 INFO:tasks.rados.rados.0.smithi042.stdout:3005: writing smithi04268048-18 from 696087 to 1254304 tid 1 2024-09-20T19:35:47.429 INFO:tasks.rados.rados.0.smithi042.stdout:3005: writing smithi04268048-18 from 1833755 to 2482114 tid 2 2024-09-20T19:35:47.435 INFO:tasks.rados.rados.0.smithi042.stdout:3005: writing smithi04268048-18 from 3137806 to 3496087 tid 3 2024-09-20T19:35:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:3006: write oid 41 current snap is 347 2024-09-20T19:35:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:3005: finishing write tid 1 to smithi04268048-18 2024-09-20T19:35:47.436 INFO:tasks.rados.rados.0.smithi042.stdout:3006: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-20T19:35:47.444 INFO:tasks.rados.rados.0.smithi042.stdout:3006: writing smithi04268048-41 from 712894 to 1462713 tid 1 2024-09-20T19:35:47.451 INFO:tasks.rados.rados.0.smithi042.stdout:3006: writing smithi04268048-41 from 2008590 to 2800607 tid 2 2024-09-20T19:35:47.456 INFO:tasks.rados.rados.0.smithi042.stdout:3006: writing smithi04268048-41 from 3281582 to 3512894 tid 3 2024-09-20T19:35:47.456 INFO:tasks.rados.rados.0.smithi042.stdout:3007: read oid 7 snap -1 2024-09-20T19:35:47.456 INFO:tasks.rados.rados.0.smithi042.stdout:3007: expect deleted 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3008: read oid 35 snap -1 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3008: expect deleted 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3009: snap_create 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: finishing write tid 2 to smithi04268048-18 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: finishing write tid 3 to smithi04268048-18 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: finishing write tid 4 to smithi04268048-18 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: finishing write tid 5 to smithi04268048-18 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: oid 18 updating version 0 to 1239 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: oid 18 version 1239 is already newer than 1238 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: oid 18 version 1239 is already newer than 1237 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: oid 18 version 1239 is already newer than 1236 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1239 (ObjNum 1041 snap 347 seq_num 1041) dirty exists 2024-09-20T19:35:47.457 INFO:tasks.rados.rados.0.smithi042.stdout:3005: left oid 18 (ObjNum 1041 snap 347 seq_num 1041) 2024-09-20T19:35:47.463 INFO:tasks.rados.rados.0.smithi042.stdout:3006: finishing write tid 1 to smithi04268048-41 2024-09-20T19:35:47.463 INFO:tasks.rados.rados.0.smithi042.stdout:3006: finishing write tid 2 to smithi04268048-41 2024-09-20T19:35:47.464 INFO:tasks.rados.rados.0.smithi042.stdout:3006: finishing write tid 3 to smithi04268048-41 2024-09-20T19:35:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:3006: finishing write tid 4 to smithi04268048-41 2024-09-20T19:35:47.466 INFO:tasks.rados.rados.0.smithi042.stdout:3006: finishing write tid 5 to smithi04268048-41 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:3006: oid 41 updating version 0 to 1222 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:3006: oid 41 version 1222 is already newer than 1221 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:3006: oid 41 version 1222 is already newer than 1220 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:3006: oid 41 updating version 1222 to 1223 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1223 (ObjNum 1042 snap 347 seq_num 1042) dirty exists 2024-09-20T19:35:47.467 INFO:tasks.rados.rados.0.smithi042.stdout:3006: left oid 41 (ObjNum 1042 snap 347 seq_num 1042) 2024-09-20T19:35:47.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.488 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[22706]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.489 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.490 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:47 smithi042 bash[32075]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.499 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.159361+0000 osd.1 (osd.1) 289 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:35:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.163283+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.190178+0000 osd.6 (osd.6) 182 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.197202+0000 osd.6 (osd.6) 183 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.403002+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.406172+0000 mgr.y (mgr.24491) 669 : cluster [DBG] pgmap v1255: 178 pgs: 4 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 164 active+clean; 348 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 1014 KiB/s wr, 4 op/s 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.708619+0000 osd.0 (osd.0) 71 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:47 smithi162 bash[24557]: cluster 2024-09-20T19:35:46.716994+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:35:47.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:47.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:47.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:47.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:47.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:47.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:47.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:47.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:35:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:48.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:48.154 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:48.256 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:48.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:48.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3005: done (4 left) 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3006: done (3 left) 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3007: done (2 left) 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3008: done (1 left) 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3009: done (0 left) 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3010: write (excl) oid 17 current snap is 348 2024-09-20T19:35:48.436 INFO:tasks.rados.rados.0.smithi042.stdout:3010: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-20T19:35:48.442 INFO:tasks.rados.rados.0.smithi042.stdout:3010: writing smithi04268048-17 from 729701 to 1154769 tid 1 2024-09-20T19:35:48.449 INFO:tasks.rados.rados.0.smithi042.stdout:3010: writing smithi04268048-17 from 1667072 to 2319100 tid 2 2024-09-20T19:35:48.455 INFO:tasks.rados.rados.0.smithi042.stdout:3010: writing smithi04268048-17 from 2909005 to 3394185 tid 3 2024-09-20T19:35:48.455 INFO:tasks.rados.rados.0.smithi042.stdout:3011: write oid 32 current snap is 348 2024-09-20T19:35:48.455 INFO:tasks.rados.rados.0.smithi042.stdout:3011: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-20T19:35:48.462 INFO:tasks.rados.rados.0.smithi042.stdout:3011: writing smithi04268048-32 from 746508 to 1246825 tid 1 2024-09-20T19:35:48.471 INFO:tasks.rados.rados.0.smithi042.stdout:3011: writing smithi04268048-32 from 2009201 to 2804887 tid 2 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3011: writing smithi04268048-32 from 3220075 to 3546508 tid 3 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3012: copy_from oid 26 from oid 31 current snap is 348 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3010: finishing write tid 1 to smithi04268048-17 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3010: finishing write tid 2 to smithi04268048-17 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3010: finishing write tid 3 to smithi04268048-17 2024-09-20T19:35:48.476 INFO:tasks.rados.rados.0.smithi042.stdout:3010: finishing write tid 4 to smithi04268048-17 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: finishing write tid 5 to smithi04268048-17 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: oid 17 updating version 0 to 1150 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: oid 17 updating version 1150 to 1151 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: oid 17 version 1151 is already newer than 1149 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: oid 17 version 1151 is already newer than 1148 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1151 (ObjNum 1043 snap 348 seq_num 1043) dirty exists 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: left oid 17 (ObjNum 1043 snap 348 seq_num 1043) 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3011: finishing write tid 1 to smithi04268048-32 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3010: done (2 left) 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3013: snap_create 2024-09-20T19:35:48.477 INFO:tasks.rados.rados.0.smithi042.stdout:3011: finishing write tid 2 to smithi04268048-32 2024-09-20T19:35:48.479 INFO:tasks.rados.rados.0.smithi042.stdout:3011: finishing write tid 3 to smithi04268048-32 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: finishing write tid 4 to smithi04268048-32 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: finishing write tid 5 to smithi04268048-32 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: oid 32 updating version 0 to 1409 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: oid 32 version 1409 is already newer than 1408 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: oid 32 version 1409 is already newer than 1407 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: oid 32 updating version 1409 to 1410 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1410 (ObjNum 1044 snap 348 seq_num 1044) dirty exists 2024-09-20T19:35:48.480 INFO:tasks.rados.rados.0.smithi042.stdout:3011: left oid 32 (ObjNum 1044 snap 348 seq_num 1044) 2024-09-20T19:35:48.493 INFO:tasks.rados.rados.0.smithi042.stdout:3012: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:35:48.498 INFO:tasks.rados.rados.0.smithi042.stdout:3012: finishing copy_from to smithi04268048-26 2024-09-20T19:35:48.498 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 904 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:35:48.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:48.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:48.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.629 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.630 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.631 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.632 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:48 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.664 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:48.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.084442+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.087845+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.134325+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.136545+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.231879+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.234076+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.408983+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.418512+0000 osd.2 (osd.2) 257 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.420678+0000 osd.2 (osd.2) 258 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.659675+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:35:48.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.661403+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.987965+0000 osd.4 (osd.4) 115 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:47.994930+0000 osd.4 (osd.4) 116 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.181982+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:48 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.183540+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:35:48.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:48.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:48.930 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:48.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:49.034 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:49.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:49.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:49.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:49.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3011: done (2 left) 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3012: done (1 left) 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3013: done (0 left) 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3014: read oid 28 snap -1 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3014: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:49.442 INFO:tasks.rados.rados.0.smithi042.stdout:3015: snap_create 2024-09-20T19:35:49.448 INFO:tasks.rados.rados.0.smithi042.stdout:3014: expect (ObjNum 1017 snap 334 seq_num 1017) 2024-09-20T19:35:49.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:49.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:49.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:49.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.701 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:49.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.407285+0000 mgr.y (mgr.24491) 670 : cluster [DBG] pgmap v1257: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 354 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.433796+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.742742+0000 osd.5 (osd.5) 265 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:48.747864+0000 osd.5 (osd.5) 266 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.173134+0000 osd.6 (osd.6) 188 : cluster [DBG] 7.d scrub starts 2024-09-20T19:35:49.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:49 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.178747+0000 osd.6 (osd.6) 189 : cluster [DBG] 7.d scrub ok 2024-09-20T19:35:49.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:49.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:49.978 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:49.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:50.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:35:50.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:50.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:50.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:50.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3014: done (1 left) 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3015: done (0 left) 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3016: read oid 35 snap -1 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3016: expect deleted 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3017: copy_from oid 49 from oid 50 current snap is 350 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3018: write (excl) oid 11 current snap is 350 2024-09-20T19:35:50.455 INFO:tasks.rados.rados.0.smithi042.stdout:3018: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-09-20T19:35:50.462 INFO:tasks.rados.rados.0.smithi042.stdout:3018: writing smithi04268048-11 from 763315 to 1338881 tid 1 2024-09-20T19:35:50.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.471 INFO:tasks.rados.rados.0.smithi042.stdout:3018: writing smithi04268048-11 from 2067683 to 2723380 tid 2 2024-09-20T19:35:50.476 INFO:tasks.rados.rados.0.smithi042.stdout:3018: writing smithi04268048-11 from 3247498 to 3563315 tid 3 2024-09-20T19:35:50.476 INFO:tasks.rados.rados.0.smithi042.stdout:3019: snap_create 2024-09-20T19:35:50.477 INFO:tasks.rados.rados.0.smithi042.stdout:3018: finishing write tid 1 to smithi04268048-11 2024-09-20T19:35:50.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:50 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.483 INFO:tasks.rados.rados.0.smithi042.stdout:3018: finishing write tid 2 to smithi04268048-11 2024-09-20T19:35:50.485 INFO:tasks.rados.rados.0.smithi042.stdout:3017: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:35:50.489 INFO:tasks.rados.rados.0.smithi042.stdout:3018: finishing write tid 3 to smithi04268048-11 2024-09-20T19:35:50.489 INFO:tasks.rados.rados.0.smithi042.stdout:3018: finishing write tid 4 to smithi04268048-11 2024-09-20T19:35:50.489 INFO:tasks.rados.rados.0.smithi042.stdout:3018: finishing write tid 5 to smithi04268048-11 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:3018: oid 11 updating version 0 to 1026 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:3018: oid 11 version 1026 is already newer than 1023 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:3018: oid 11 version 1026 is already newer than 1025 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:3018: oid 11 version 1026 is already newer than 1024 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1026 (ObjNum 1045 snap 350 seq_num 1045) dirty exists 2024-09-20T19:35:50.490 INFO:tasks.rados.rados.0.smithi042.stdout:3018: left oid 11 (ObjNum 1045 snap 350 seq_num 1045) 2024-09-20T19:35:50.491 INFO:tasks.rados.rados.0.smithi042.stdout:3017: finishing copy_from to smithi04268048-49 2024-09-20T19:35:50.491 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 587 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:35:50.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:50.541 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:35:50.541 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:35:50.542 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:35:50.542 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:35:50.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:50.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.123221+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.126027+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.435901+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.731024+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.735616+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.936435+0000 osd.4 (osd.4) 117 : cluster [DBG] 7.e scrub starts 2024-09-20T19:35:50.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:49.940565+0000 osd.4 (osd.4) 118 : cluster [DBG] 7.e scrub ok 2024-09-20T19:35:50.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.125241+0000 osd.6 (osd.6) 190 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:35:50.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:50 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.127679+0000 osd.6 (osd.6) 191 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:35:50.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:50.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:50.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:50.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:50.980 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:51.008 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:51.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:51.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:51.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:51.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:51.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:51.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:51.474 INFO:tasks.rados.rados.0.smithi042.stdout:3016: done (3 left) 2024-09-20T19:35:51.474 INFO:tasks.rados.rados.0.smithi042.stdout:3017: done (2 left) 2024-09-20T19:35:51.474 INFO:tasks.rados.rados.0.smithi042.stdout:3018: done (1 left) 2024-09-20T19:35:51.474 INFO:tasks.rados.rados.0.smithi042.stdout:3019: done (0 left) 2024-09-20T19:35:51.474 INFO:tasks.rados.rados.0.smithi042.stdout:3020: snap_remove snap 318 2024-09-20T19:35:51.507 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:51.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:51.589 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T19:35:51.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:51.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:51.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[22706]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.773 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.774 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:51 smithi042 bash[32075]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: audit 2024-09-20T19:35:49.774481+0000 mgr.y (mgr.24491) 671 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:35:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.121280+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.122662+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.407806+0000 mgr.y (mgr.24491) 672 : cluster [DBG] pgmap v1260: 178 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 173 active+clean; 357 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.453822+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.751076+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b scrub starts 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:51 smithi162 bash[24557]: cluster 2024-09-20T19:35:50.754267+0000 osd.0 (osd.0) 78 : cluster [DBG] 4.b scrub ok 2024-09-20T19:35:51.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:51.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:52.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:52.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:52.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:52.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:52.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:52.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:52.499 INFO:tasks.rados.rados.0.smithi042.stdout:3020: done (0 left) 2024-09-20T19:35:52.499 INFO:tasks.rados.rados.0.smithi042.stdout:3021: write (excl) oid 19 current snap is 351 2024-09-20T19:35:52.499 INFO:tasks.rados.rados.0.smithi042.stdout:3021: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-20T19:35:52.506 INFO:tasks.rados.rados.0.smithi042.stdout:3021: writing smithi04268048-19 from 780122 to 1430937 tid 1 2024-09-20T19:35:52.522 INFO:tasks.rados.rados.0.smithi042.stdout:3021: writing smithi04268048-19 from 2126165 to 2925520 tid 2 2024-09-20T19:35:52.522 INFO:tasks.rados.rados.0.smithi042.stdout:3022: write (excl) oid 43 current snap is 351 2024-09-20T19:35:52.522 INFO:tasks.rados.rados.0.smithi042.stdout:3021: finishing write tid 1 to smithi04268048-19 2024-09-20T19:35:52.522 INFO:tasks.rados.rados.0.smithi042.stdout:3022: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-20T19:35:52.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:52.536 INFO:tasks.rados.rados.0.smithi042.stdout:3022: writing smithi04268048-43 from 796929 to 1522993 tid 1 2024-09-20T19:35:52.550 INFO:tasks.rados.rados.0.smithi042.stdout:3022: writing smithi04268048-43 from 2184647 to 2844013 tid 2 2024-09-20T19:35:52.557 INFO:tasks.rados.rados.0.smithi042.stdout:3022: writing smithi04268048-43 from 3302344 to 3596929 tid 3 2024-09-20T19:35:52.557 INFO:tasks.rados.rados.0.smithi042.stdout:3023: snap_remove snap 330 2024-09-20T19:35:52.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:52.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:52.692 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:52.737 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:35:52.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:35:52.737 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-20T19:35:52.738 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:35:52.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.747 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:52 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.796 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.102294+0000 osd.1 (osd.1) 297 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.104314+0000 osd.1 (osd.1) 298 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.471730+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.724592+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:51.726400+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.129565+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:52 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.132276+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:35:52.821 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:52.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:52.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:52.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:53.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.080 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:53.184 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.212 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:53.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:53.392 INFO:tasks.rados.rados.0.smithi042.stdout:3023: done (2 left) 2024-09-20T19:35:53.392 INFO:tasks.rados.rados.0.smithi042.stdout:3024: write oid 7 current snap is 351 2024-09-20T19:35:53.393 INFO:tasks.rados.rados.0.smithi042.stdout:3024: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-20T19:35:53.402 INFO:tasks.rados.rados.0.smithi042.stdout:3024: writing smithi04268048-7 from 413736 to 815049 tid 1 2024-09-20T19:35:53.403 INFO:tasks.rados.rados.0.smithi042.stdout:3024: writing smithi04268048-7 from 1326776 to 1729800 tid 2 2024-09-20T19:35:53.411 INFO:tasks.rados.rados.0.smithi042.stdout:3024: writing smithi04268048-7 from 2413414 to 2871307 tid 3 2024-09-20T19:35:53.416 INFO:tasks.rados.rados.0.smithi042.stdout:3024: writing smithi04268048-7 from 3438716 to 3613736 tid 4 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: finishing write tid 2 to smithi04268048-19 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: finishing write tid 3 to smithi04268048-19 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: finishing write tid 4 to smithi04268048-19 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: oid 19 updating version 0 to 1163 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: oid 19 version 1163 is already newer than 1162 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: oid 19 updating version 1163 to 1164 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1164 (ObjNum 1046 snap 351 seq_num 1046) dirty exists 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3021: left oid 19 (ObjNum 1046 snap 351 seq_num 1046) 2024-09-20T19:35:53.418 INFO:tasks.rados.rados.0.smithi042.stdout:3022: finishing write tid 1 to smithi04268048-43 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: finishing write tid 2 to smithi04268048-43 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: finishing write tid 3 to smithi04268048-43 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: finishing write tid 4 to smithi04268048-43 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: finishing write tid 5 to smithi04268048-43 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: oid 43 updating version 0 to 743 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: oid 43 version 743 is already newer than 742 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: oid 43 updating version 743 to 744 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: oid 43 updating version 744 to 745 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 745 (ObjNum 1047 snap 351 seq_num 1047) dirty exists 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3022: left oid 43 (ObjNum 1047 snap 351 seq_num 1047) 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3025: read oid 36 snap -1 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3025: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:35:53.419 INFO:tasks.rados.rados.0.smithi042.stdout:3021: done (3 left) 2024-09-20T19:35:53.420 INFO:tasks.rados.rados.0.smithi042.stdout:3022: done (2 left) 2024-09-20T19:35:53.420 INFO:tasks.rados.rados.0.smithi042.stdout:3026: rollback oid 12 current snap is 351 2024-09-20T19:35:53.420 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 319 2024-09-20T19:35:53.420 INFO:tasks.rados.rados.0.smithi042.stdout:3027: snap_create 2024-09-20T19:35:53.423 INFO:tasks.rados.rados.0.smithi042.stdout:3026: finishing rollback tid 0 to smithi04268048-12 2024-09-20T19:35:53.425 INFO:tasks.rados.rados.0.smithi042.stdout:3026: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:35:53.425 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 744 (ObjNum 289 snap 0 seq_num 174359772) dirty dne 2024-09-20T19:35:53.428 INFO:tasks.rados.rados.0.smithi042.stdout:3025: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:35:53.443 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1347585 2024-09-20T19:35:53.444 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:53.466 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:53.503 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1825495 2024-09-20T19:35:53.519 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 1 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 2 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 3 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 4 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 5 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: finishing write tid 6 to smithi04268048-7 2024-09-20T19:35:53.568 INFO:tasks.rados.rados.0.smithi042.stdout:3024: oid 7 updating version 0 to 1230 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:3024: oid 7 version 1230 is already newer than 1229 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:3024: oid 7 version 1230 is already newer than 1228 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:3024: oid 7 updating version 1230 to 1232 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:3024: oid 7 version 1232 is already newer than 1231 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1232 (ObjNum 1048 snap 351 seq_num 1048) dirty exists 2024-09-20T19:35:53.569 INFO:tasks.rados.rados.0.smithi042.stdout:3024: left oid 7 (ObjNum 1048 snap 351 seq_num 1048) 2024-09-20T19:35:53.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.579 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:53 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.596 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:53.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.100366+0000 osd.1 (osd.1) 299 : cluster [DBG] 7.b scrub starts 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.105075+0000 osd.1 (osd.1) 300 : cluster [DBG] 7.b scrub ok 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.409403+0000 mgr.y (mgr.24491) 673 : cluster [DBG] pgmap v1263: 178 pgs: 178 active+clean; 369 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.487088+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2024-09-20T19:35:53.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.938887+0000 osd.4 (osd.4) 119 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:35:53.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:52.944555+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:35:53.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:53 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.390331+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2024-09-20T19:35:53.726 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:53.829 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.855 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:53.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:53.985 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:53.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:54.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:54.117 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:54.223 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:54.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:54.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:54.402 INFO:tasks.rados.rados.0.smithi042.stdout:3024: done (3 left) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3025: done (2 left) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3026: done (1 left) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3027: done (0 left) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3028: read oid 50 snap -1 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3028: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3029: rollback oid 19 current snap is 352 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 351 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3030: read oid 34 snap -1 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3030: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3031: write (excl) oid 49 current snap is 352 2024-09-20T19:35:54.403 INFO:tasks.rados.rados.0.smithi042.stdout:3031: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-20T19:35:54.409 INFO:tasks.rados.rados.0.smithi042.stdout:3031: writing smithi04268048-49 from 430543 to 907105 tid 1 2024-09-20T19:35:54.416 INFO:tasks.rados.rados.0.smithi042.stdout:3031: writing smithi04268048-49 from 1385258 to 2048293 tid 2 2024-09-20T19:35:54.427 INFO:tasks.rados.rados.0.smithi042.stdout:3031: writing smithi04268048-49 from 2840837 to 3510002 tid 3 2024-09-20T19:35:54.427 INFO:tasks.rados.rados.0.smithi042.stdout:3032: read oid 35 snap -1 2024-09-20T19:35:54.427 INFO:tasks.rados.rados.0.smithi042.stdout:3032: expect deleted 2024-09-20T19:35:54.427 INFO:tasks.rados.rados.0.smithi042.stdout:3033: delete oid 26 current snap is 352 2024-09-20T19:35:54.427 INFO:tasks.rados.rados.0.smithi042.stdout:3030: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:35:54.453 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3237922 2024-09-20T19:35:54.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:54.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:54.536 INFO:tasks.rados.rados.0.smithi042.stdout:3028: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:35:54.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:54.635 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 981646 2024-09-20T19:35:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: finishing write tid 1 to smithi04268048-49 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: finishing write tid 2 to smithi04268048-49 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: finishing write tid 3 to smithi04268048-49 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: finishing write tid 4 to smithi04268048-49 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: finishing write tid 5 to smithi04268048-49 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: oid 49 updating version 0 to 1424 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: oid 49 updating version 1424 to 1425 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: oid 49 version 1425 is already newer than 1422 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: oid 49 version 1425 is already newer than 1423 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1425 (ObjNum 1049 snap 352 seq_num 1049) dirty exists 2024-09-20T19:35:54.670 INFO:tasks.rados.rados.0.smithi042.stdout:3031: left oid 49 (ObjNum 1049 snap 352 seq_num 1049) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3029: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3029: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3029: finishing rollback tid 2 to smithi04268048-19 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1167 (ObjNum 1046 snap 351 seq_num 1046) dirty exists 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3028: done (5 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3029: done (4 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3030: done (3 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3031: done (2 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3032: done (1 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3033: done (0 left) 2024-09-20T19:35:54.671 INFO:tasks.rados.rados.0.smithi042.stdout:3034: snap_remove snap 334 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:54 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:54.761 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.761 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.910767+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.e scrub starts 2024-09-20T19:35:54.761 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.761 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:53.913559+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.e scrub ok 2024-09-20T19:35:54.761 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:54 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.400886+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2024-09-20T19:35:54.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:54.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:54.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:55.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:55.067 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:35:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:35:55.085 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:35:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:35:55.102 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:35:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:35:55.120 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:35:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:35:55.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:55.141 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:35:55.142 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:35:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:55.165 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:35:55.165 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:35:55.184 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:35:55.184 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:35:55.204 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:35:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:35:55.263 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:55.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:55.471 INFO:tasks.rados.rados.0.smithi042.stdout:3034: done (0 left) 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3035: read oid 17 snap 349 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3035: expect (ObjNum 1043 snap 348 seq_num 1043) 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3036: copy_from oid 6 from oid 35 current snap is 352 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3037: read oid 39 snap -1 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3037: expect deleted 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3038: copy_from oid 43 from oid 12 current snap is 352 2024-09-20T19:35:55.472 INFO:tasks.rados.rados.0.smithi042.stdout:3039: delete oid 9 current snap is 352 2024-09-20T19:35:55.474 INFO:tasks.rados.rados.0.smithi042.stdout:3036: finishing copy_from to smithi04268048-6 2024-09-20T19:35:55.474 INFO:tasks.rados.rados.0.smithi042.stdout:3036: got expected ENOENT (src dne) 2024-09-20T19:35:55.476 INFO:tasks.rados.rados.0.smithi042.stdout:3036: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:35:55.476 INFO:tasks.rados.rados.0.smithi042.stdout:3038: finishing copy_from to smithi04268048-43 2024-09-20T19:35:55.476 INFO:tasks.rados.rados.0.smithi042.stdout:3038: got expected ENOENT (src dne) 2024-09-20T19:35:55.476 INFO:tasks.rados.rados.0.smithi042.stdout:3038: finishing copy_from racing read to smithi04268048-43 2024-09-20T19:35:55.479 INFO:tasks.rados.rados.0.smithi042.stdout:3035: expect (ObjNum 1043 snap 348 seq_num 1043) 2024-09-20T19:35:55.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.523 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[22706]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.524 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:55 smithi042 bash[32075]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:55.575 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:55.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3035: done (4 left) 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3036: done (3 left) 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3038: done (2 left) 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3039: done (1 left) 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3040: write oid 2 current snap is 352 2024-09-20T19:35:55.745 INFO:tasks.rados.rados.0.smithi042.stdout:3040: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-20T19:35:55.750 INFO:tasks.rados.rados.0.smithi042.stdout:3040: writing smithi04268048-2 from 480964 to 899626 tid 1 2024-09-20T19:35:55.756 INFO:tasks.rados.rados.0.smithi042.stdout:3040: writing smithi04268048-2 from 1560704 to 1971066 tid 2 2024-09-20T19:35:55.762 INFO:tasks.rados.rados.0.smithi042.stdout:3040: writing smithi04268048-2 from 2523106 to 3142440 tid 3 2024-09-20T19:35:55.763 INFO:tasks.rados.rados.0.smithi042.stdout:3037: done (1 left) 2024-09-20T19:35:55.763 INFO:tasks.rados.rados.0.smithi042.stdout:3041: read oid 7 snap -1 2024-09-20T19:35:55.763 INFO:tasks.rados.rados.0.smithi042.stdout:3041: expect (ObjNum 1048 snap 351 seq_num 1048) 2024-09-20T19:35:55.763 INFO:tasks.rados.rados.0.smithi042.stdout:3042: write oid 50 current snap is 352 2024-09-20T19:35:55.763 INFO:tasks.rados.rados.0.smithi042.stdout:3042: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-20T19:35:55.763 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:55.768 INFO:tasks.rados.rados.0.smithi042.stdout:3042: writing smithi04268048-50 from 497771 to 991682 tid 1 2024-09-20T19:35:55.775 INFO:tasks.rados.rados.0.smithi042.stdout:3042: writing smithi04268048-50 from 1619186 to 2173206 tid 2 2024-09-20T19:35:55.782 INFO:tasks.rados.rados.0.smithi042.stdout:3042: writing smithi04268048-50 from 2950529 to 3381135 tid 3 2024-09-20T19:35:55.782 INFO:tasks.rados.rados.0.smithi042.stdout:3043: snap_remove snap 340 2024-09-20T19:35:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.410235+0000 mgr.y (mgr.24491) 674 : cluster [DBG] pgmap v1267: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1017 KiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.756261+0000 osd.5 (osd.5) 267 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.758551+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.871184+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.f scrub starts 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.874411+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.f scrub ok 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:55 smithi162 bash[24557]: audit 2024-09-20T19:35:55.047977+0000 mon.a (mon.0) 2404 : audit [DBG] from='client.? v1:172.21.15.42:0/2900265253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:35:55.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:55.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:56.040 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:56.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:56.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:56.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:56.326 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:56.429 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:56.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:35:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:35:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3043: done (3 left) 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3044: write (excl) oid 18 current snap is 352 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: finishing write tid 1 to smithi04268048-2 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: finishing write tid 2 to smithi04268048-2 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: finishing write tid 3 to smithi04268048-2 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: finishing write tid 4 to smithi04268048-2 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: finishing write tid 5 to smithi04268048-2 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: oid 2 updating version 0 to 1307 2024-09-20T19:35:56.477 INFO:tasks.rados.rados.0.smithi042.stdout:3040: oid 2 version 1307 is already newer than 1305 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3040: oid 2 updating version 1307 to 1308 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3040: oid 2 version 1308 is already newer than 1306 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1308 (ObjNum 1052 snap 352 seq_num 1052) dirty exists 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3040: left oid 2 (ObjNum 1052 snap 352 seq_num 1052) 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3042: finishing write tid 1 to smithi04268048-50 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3042: finishing write tid 2 to smithi04268048-50 2024-09-20T19:35:56.478 INFO:tasks.rados.rados.0.smithi042.stdout:3044: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-20T19:35:56.483 INFO:tasks.rados.rados.0.smithi042.stdout:3044: writing smithi04268048-18 from 514578 to 1083738 tid 1 2024-09-20T19:35:56.491 INFO:tasks.rados.rados.0.smithi042.stdout:3044: writing smithi04268048-18 from 1677668 to 2091699 tid 2 2024-09-20T19:35:56.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: writing smithi04268048-18 from 2577952 to 3219830 tid 3 2024-09-20T19:35:56.498 INFO:tasks.rados.rados.0.smithi042.stdout:3040: done (3 left) 2024-09-20T19:35:56.498 INFO:tasks.rados.rados.0.smithi042.stdout:3045: write (excl) oid 31 current snap is 352 2024-09-20T19:35:56.499 INFO:tasks.rados.rados.0.smithi042.stdout:3041: expect (ObjNum 1048 snap 351 seq_num 1048) 2024-09-20T19:35:56.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2585539 2024-09-20T19:35:56.528 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:56.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:56.570 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:56 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.577 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:35:56.584 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:56.617 INFO:tasks.rados.rados.0.smithi042.stdout:3042: finishing write tid 3 to smithi04268048-50 2024-09-20T19:35:56.617 INFO:tasks.rados.rados.0.smithi042.stdout:3042: finishing write tid 4 to smithi04268048-50 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: finishing write tid 5 to smithi04268048-50 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: oid 50 updating version 0 to 1033 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: oid 50 version 1033 is already newer than 1032 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: oid 50 updating version 1033 to 1035 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: oid 50 version 1035 is already newer than 1034 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1035 (ObjNum 1053 snap 352 seq_num 1053) dirty exists 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3042: left oid 50 (ObjNum 1053 snap 352 seq_num 1053) 2024-09-20T19:35:56.618 INFO:tasks.rados.rados.0.smithi042.stdout:3045: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-20T19:35:56.624 INFO:tasks.rados.rados.0.smithi042.stdout:3045: writing smithi04268048-31 from 531385 to 1175794 tid 1 2024-09-20T19:35:56.630 INFO:tasks.rados.rados.0.smithi042.stdout:3045: writing smithi04268048-31 from 1736150 to 2293839 tid 2 2024-09-20T19:35:56.640 INFO:tasks.rados.rados.0.smithi042.stdout:3045: writing smithi04268048-31 from 3005375 to 3574878 tid 3 2024-09-20T19:35:56.640 INFO:tasks.rados.rados.0.smithi042.stdout:3041: done (3 left) 2024-09-20T19:35:56.640 INFO:tasks.rados.rados.0.smithi042.stdout:3042: done (2 left) 2024-09-20T19:35:56.640 INFO:tasks.rados.rados.0.smithi042.stdout:3046: snap_remove snap 295 2024-09-20T19:35:56.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:56.704 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:56.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.529206+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:35:56.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:54.533538+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.468361+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.751380+0000 osd.0 (osd.0) 81 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:56 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.753885+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:35:56.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:56.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:56.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:57.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:57.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:35:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:57.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:57.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:57.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:57.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:57.490 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:35:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3046: done (2 left) 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3047: rollback oid 44 current snap is 352 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 325 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: finishing write tid 1 to smithi04268048-18 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: finishing write tid 2 to smithi04268048-18 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: finishing write tid 3 to smithi04268048-18 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: finishing write tid 4 to smithi04268048-18 2024-09-20T19:35:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3044: finishing write tid 5 to smithi04268048-18 2024-09-20T19:35:57.499 INFO:tasks.rados.rados.0.smithi042.stdout:3044: oid 18 updating version 0 to 1266 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:3044: oid 18 version 1266 is already newer than 1265 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:3044: oid 18 version 1266 is already newer than 1260 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:3044: oid 18 version 1266 is already newer than 1259 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1266 (ObjNum 1054 snap 352 seq_num 1054) dirty exists 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:3044: left oid 18 (ObjNum 1054 snap 352 seq_num 1054) 2024-09-20T19:35:57.500 INFO:tasks.rados.rados.0.smithi042.stdout:3045: finishing write tid 1 to smithi04268048-31 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: finishing write tid 2 to smithi04268048-31 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: finishing write tid 3 to smithi04268048-31 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: finishing write tid 4 to smithi04268048-31 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: finishing write tid 5 to smithi04268048-31 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: oid 31 updating version 0 to 927 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: oid 31 version 927 is already newer than 924 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: oid 31 version 927 is already newer than 926 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: oid 31 version 927 is already newer than 925 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 927 (ObjNum 1055 snap 352 seq_num 1055) dirty exists 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: left oid 31 (ObjNum 1055 snap 352 seq_num 1055) 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3044: done (2 left) 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3045: done (1 left) 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3048: copy_from oid 20 from oid 41 current snap is 352 2024-09-20T19:35:57.501 INFO:tasks.rados.rados.0.smithi042.stdout:3049: delete oid 8 current snap is 352 2024-09-20T19:35:57.502 INFO:tasks.rados.rados.0.smithi042.stdout:3049: done (2 left) 2024-09-20T19:35:57.502 INFO:tasks.rados.rados.0.smithi042.stdout:3050: rollback oid 48 current snap is 352 2024-09-20T19:35:57.502 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 335 2024-09-20T19:35:57.510 INFO:tasks.rados.rados.0.smithi042.stdout:3051: rollback oid 18 current snap is 352 2024-09-20T19:35:57.510 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 299 2024-09-20T19:35:57.510 INFO:tasks.rados.rados.0.smithi042.stdout:3052: write (excl) oid 33 current snap is 352 2024-09-20T19:35:57.510 INFO:tasks.rados.rados.0.smithi042.stdout:3052: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-20T19:35:57.510 INFO:tasks.rados.rados.0.smithi042.stdout:3052: writing smithi04268048-33 from 564999 to 1076259 tid 1 2024-09-20T19:35:57.518 INFO:tasks.rados.rados.0.smithi042.stdout:3052: writing smithi04268048-33 from 1853114 to 2414472 tid 2 2024-09-20T19:35:57.525 INFO:tasks.rados.rados.0.smithi042.stdout:3052: writing smithi04268048-33 from 3060221 to 3652268 tid 3 2024-09-20T19:35:57.525 INFO:tasks.rados.rados.0.smithi042.stdout:3053: snap_remove snap 343 2024-09-20T19:35:57.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.603 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.604 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.604 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.604 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.604 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:57 smithi162 bash[24557]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.616 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:35:57.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[22706]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.568833+0000 osd.2 (osd.2) 261 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:55.571603+0000 osd.2 (osd.2) 262 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.410884+0000 mgr.y (mgr.24491) 675 : cluster [DBG] pgmap v1269: 178 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 171 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1004 KiB/s rd, 705 KiB/s wr, 4 op/s 2024-09-20T19:35:57.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.475969+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.749893+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: cluster 2024-09-20T19:35:56.751362+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: audit 2024-09-20T19:35:56.983330+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.42:0/765998211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.686 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:57 smithi042 bash[32075]: audit 2024-09-20T19:35:56.983920+0000 mon.a (mon.0) 2407 : audit [INF] from='client.25883 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:35:57.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:57.748 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:57.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:57.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:57.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:57.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:58.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:58.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:58.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:58.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:58.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:35:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:35:58.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:58.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:35:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:35:58.485 INFO:tasks.rados.rados.0.smithi042.stdout:3053: done (5 left) 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3054: copy_from oid 50 from oid 3 current snap is 352 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3047: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3047: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1185 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3050: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3048: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3050: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3051: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3051: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3051: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1277 (ObjNum 846 snap 277 seq_num 846) dirty exists 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:3050: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:35:58.486 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 994 (ObjNum 962 snap 319 seq_num 962) dirty exists 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3048: finishing copy_from to smithi04268048-20 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1223 (ObjNum 1042 snap 347 seq_num 1042) dirty exists 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3052: finishing write tid 1 to smithi04268048-33 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3052: finishing write tid 2 to smithi04268048-33 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3047: done (5 left) 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3048: done (4 left) 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3050: done (3 left) 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3051: done (2 left) 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3055: read oid 46 snap -1 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3055: expect deleted 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3056: write oid 28 current snap is 352 2024-09-20T19:35:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3056: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-20T19:35:58.498 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3056: writing smithi04268048-28 from 581806 to 1168315 tid 1 2024-09-20T19:35:58.516 INFO:tasks.rados.rados.0.smithi042.stdout:3056: writing smithi04268048-28 from 1911596 to 2332965 tid 2 2024-09-20T19:35:58.528 INFO:tasks.rados.rados.0.smithi042.stdout:3056: writing smithi04268048-28 from 3087644 to 3490963 tid 3 2024-09-20T19:35:58.528 INFO:tasks.rados.rados.0.smithi042.stdout:3057: delete oid 11 current snap is 352 2024-09-20T19:35:58.528 INFO:tasks.rados.rados.0.smithi042.stdout:3052: finishing write tid 3 to smithi04268048-33 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: finishing write tid 4 to smithi04268048-33 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: finishing write tid 5 to smithi04268048-33 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: oid 33 updating version 0 to 1309 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: oid 33 updating version 1309 to 1312 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: oid 33 version 1312 is already newer than 1310 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:3052: oid 33 version 1312 is already newer than 1311 2024-09-20T19:35:58.529 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1312 (ObjNum 1057 snap 352 seq_num 1057) dirty exists 2024-09-20T19:35:58.530 INFO:tasks.rados.rados.0.smithi042.stdout:3052: left oid 33 (ObjNum 1057 snap 352 seq_num 1057) 2024-09-20T19:35:58.530 INFO:tasks.rados.rados.0.smithi042.stdout:3056: finishing write tid 1 to smithi04268048-28 2024-09-20T19:35:58.530 INFO:tasks.rados.rados.0.smithi042.stdout:3054: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:35:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:58.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:58.535 INFO:tasks.rados.rados.0.smithi042.stdout:3056: finishing write tid 2 to smithi04268048-28 2024-09-20T19:35:58.535 INFO:tasks.rados.rados.0.smithi042.stdout:3054: finishing copy_from to smithi04268048-50 2024-09-20T19:35:58.536 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 967 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-20T19:35:58.536 INFO:tasks.rados.rados.0.smithi042.stdout:3056: finishing write tid 3 to smithi04268048-28 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3052: done (4 left) 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3054: done (3 left) 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3055: done (2 left) 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3057: done (1 left) 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3058: write (excl) oid 37 current snap is 352 2024-09-20T19:35:58.537 INFO:tasks.rados.rados.0.smithi042.stdout:3058: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-20T19:35:58.545 INFO:tasks.rados.rados.0.smithi042.stdout:3058: writing smithi04268048-37 from 615420 to 1352427 tid 1 2024-09-20T19:35:58.550 INFO:tasks.rados.rados.0.smithi042.stdout:3058: writing smithi04268048-37 from 1912207 to 2337245 tid 2 2024-09-20T19:35:58.557 INFO:tasks.rados.rados.0.smithi042.stdout:3058: writing smithi04268048-37 from 3026137 to 3568353 tid 3 2024-09-20T19:35:58.557 INFO:tasks.rados.rados.0.smithi042.stdout:3059: read oid 1 snap -1 2024-09-20T19:35:58.558 INFO:tasks.rados.rados.0.smithi042.stdout:3059: expect (ObjNum 1015 snap 332 seq_num 1015) 2024-09-20T19:35:58.558 INFO:tasks.rados.rados.0.smithi042.stdout:3060: snap_remove snap 345 2024-09-20T19:35:58.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:58.657 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:58.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:35:58.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[32075]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:58 smithi042 bash[22706]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:58.784 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:35:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: cluster 2024-09-20T19:35:57.473997+0000 mon.a (mon.0) 2408 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: audit 2024-09-20T19:35:57.476658+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: cluster 2024-09-20T19:35:57.483067+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2024-09-20T19:35:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:58 smithi162 bash[24557]: audit 2024-09-20T19:35:58.316503+0000 mon.a (mon.0) 2411 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:35:58.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:58.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:59.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:59.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:59.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:35:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:35:59.265 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:59.293 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:59.396 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.422 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:59.488 INFO:tasks.rados.rados.0.smithi042.stdout:3060: done (3 left) 2024-09-20T19:35:59.488 INFO:tasks.rados.rados.0.smithi042.stdout:3061: snap_remove snap 309 2024-09-20T19:35:59.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[22706]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[22706]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[22706]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[22706]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[32075]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.537 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[32075]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[32075]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:35:59 smithi042 bash[32075]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:35:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:35:59.654 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:59.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:35:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:35:59.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:59.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:59 smithi162 bash[24557]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:59 smithi162 bash[24557]: cluster 2024-09-20T19:35:58.411876+0000 mgr.y (mgr.24491) 676 : cluster [DBG] pgmap v1272: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-20T19:35:59.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:59 smithi162 bash[24557]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:35:59 smithi162 bash[24557]: cluster 2024-09-20T19:35:58.483679+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2024-09-20T19:35:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:35:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:35:59.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:59.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:35:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:35:59.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:00.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:00.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:35:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:00.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:00.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:00.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:00.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:00.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:00.487 INFO:tasks.rados.rados.0.smithi042.stdout:3061: done (3 left) 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3062: delete oid 32 current snap is 352 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: finishing write tid 4 to smithi04268048-28 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: finishing write tid 5 to smithi04268048-28 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: oid 28 updating version 0 to 1168 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: oid 28 updating version 1168 to 1170 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: oid 28 version 1170 is already newer than 1169 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: oid 28 updating version 1170 to 1171 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1171 (ObjNum 1058 snap 352 seq_num 1058) dirty exists 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3056: left oid 28 (ObjNum 1058 snap 352 seq_num 1058) 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3058: finishing write tid 1 to smithi04268048-37 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3058: finishing write tid 2 to smithi04268048-37 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3058: finishing write tid 3 to smithi04268048-37 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3058: finishing write tid 4 to smithi04268048-37 2024-09-20T19:36:00.488 INFO:tasks.rados.rados.0.smithi042.stdout:3058: finishing write tid 5 to smithi04268048-37 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3058: oid 37 updating version 0 to 1007 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3058: oid 37 version 1007 is already newer than 1004 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3058: oid 37 version 1007 is already newer than 1005 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3058: oid 37 version 1007 is already newer than 1006 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1007 (ObjNum 1060 snap 352 seq_num 1060) dirty exists 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3058: left oid 37 (ObjNum 1060 snap 352 seq_num 1060) 2024-09-20T19:36:00.489 INFO:tasks.rados.rados.0.smithi042.stdout:3059: expect (ObjNum 1015 snap 332 seq_num 1015) 2024-09-20T19:36:00.554 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1763942 2024-09-20T19:36:00.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:00.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:00.601 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:36:00.601 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:36:00.602 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2922183 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3056: done (3 left) 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3058: done (2 left) 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3059: done (1 left) 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3062: done (0 left) 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3063: read oid 3 snap -1 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3063: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:00.623 INFO:tasks.rados.rados.0.smithi042.stdout:3064: rollback oid 18 current snap is 352 2024-09-20T19:36:00.624 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 333 2024-09-20T19:36:00.624 INFO:tasks.rados.rados.0.smithi042.stdout:3065: copy_from oid 12 from oid 1 current snap is 352 2024-09-20T19:36:00.624 INFO:tasks.rados.rados.0.smithi042.stdout:3066: rollback oid 50 current snap is 352 2024-09-20T19:36:00.624 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 346 2024-09-20T19:36:00.630 INFO:tasks.rados.rados.0.smithi042.stdout:3067: write oid 37 current snap is 352 2024-09-20T19:36:00.630 INFO:tasks.rados.rados.0.smithi042.stdout:3067: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-20T19:36:00.630 INFO:tasks.rados.rados.0.smithi042.stdout:3067: writing smithi04268048-37 from 649034 to 1136539 tid 1 2024-09-20T19:36:00.637 INFO:tasks.rados.rados.0.smithi042.stdout:3067: writing smithi04268048-37 from 1629171 to 2057878 tid 2 2024-09-20T19:36:00.644 INFO:tasks.rados.rados.0.smithi042.stdout:3067: writing smithi04268048-37 from 2680983 to 3245743 tid 3 2024-09-20T19:36:00.648 INFO:tasks.rados.rados.0.smithi042.stdout:3067: writing smithi04268048-37 from 3798768 to 3849034 tid 4 2024-09-20T19:36:00.648 INFO:tasks.rados.rados.0.smithi042.stdout:3068: rollback oid 20 current snap is 352 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 20 to 351 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3066: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3066: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3064: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3066: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1043 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3064: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3066: done (5 left) 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3069: read oid 8 snap -1 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3069: expect deleted 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3070: write oid 13 current snap is 352 2024-09-20T19:36:00.649 INFO:tasks.rados.rados.0.smithi042.stdout:3064: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:36:00.650 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1294 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:36:00.650 INFO:tasks.rados.rados.0.smithi042.stdout:3063: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:00.650 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 325665 2024-09-20T19:36:00.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[32075]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[22706]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:00 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 1 to smithi04268048-37 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3065: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3065: finishing copy_from to smithi04268048-12 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 1131 (ObjNum 1015 snap 332 seq_num 1015) dirty exists 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 2 to smithi04268048-37 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 3 to smithi04268048-37 2024-09-20T19:36:00.751 INFO:tasks.rados.rados.0.smithi042.stdout:3070: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2024-09-20T19:36:00.758 INFO:tasks.rados.rados.0.smithi042.stdout:3070: writing smithi04268048-13 from 665841 to 1228595 tid 1 2024-09-20T19:36:00.764 INFO:tasks.rados.rados.0.smithi042.stdout:3070: writing smithi04268048-13 from 1687653 to 2260018 tid 2 2024-09-20T19:36:00.771 INFO:tasks.rados.rados.0.smithi042.stdout:3070: writing smithi04268048-13 from 2708406 to 3484438 tid 3 2024-09-20T19:36:00.771 INFO:tasks.rados.rados.0.smithi042.stdout:3063: done (6 left) 2024-09-20T19:36:00.771 INFO:tasks.rados.rados.0.smithi042.stdout:3064: done (5 left) 2024-09-20T19:36:00.771 INFO:tasks.rados.rados.0.smithi042.stdout:3065: done (4 left) 2024-09-20T19:36:00.771 INFO:tasks.rados.rados.0.smithi042.stdout:3071: write (excl) oid 26 current snap is 352 2024-09-20T19:36:00.772 INFO:tasks.rados.rados.0.smithi042.stdout:3071: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-20T19:36:00.780 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:36:00.781 INFO:tasks.rados.rados.0.smithi042.stdout:3071: writing smithi04268048-26 from 682648 to 1320651 tid 1 2024-09-20T19:36:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:36:00.788 INFO:tasks.rados.rados.0.smithi042.stdout:3071: writing smithi04268048-26 from 2029782 to 2462158 tid 2 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3071: writing smithi04268048-26 from 3019476 to 3606780 tid 3 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3072: snap_create 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 4 to smithi04268048-37 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 5 to smithi04268048-37 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3068: finishing rollback tid 0 to smithi04268048-20 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3067: finishing write tid 6 to smithi04268048-37 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3067: oid 37 updating version 0 to 1011 2024-09-20T19:36:00.793 INFO:tasks.rados.rados.0.smithi042.stdout:3067: oid 37 updating version 1011 to 1012 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3067: oid 37 version 1012 is already newer than 1008 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3067: oid 37 version 1012 is already newer than 1010 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3067: oid 37 version 1012 is already newer than 1009 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1012 (ObjNum 1062 snap 352 seq_num 1062) dirty exists 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3067: left oid 37 (ObjNum 1062 snap 352 seq_num 1062) 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3068: finishing rollback tid 1 to smithi04268048-20 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3068: finishing rollback tid 2 to smithi04268048-20 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1226 (ObjNum 943 snap 314 seq_num 943) dirty exists 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3070: finishing write tid 1 to smithi04268048-13 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3070: finishing write tid 2 to smithi04268048-13 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3070: finishing write tid 3 to smithi04268048-13 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3070: finishing write tid 4 to smithi04268048-13 2024-09-20T19:36:00.794 INFO:tasks.rados.rados.0.smithi042.stdout:3070: finishing write tid 5 to smithi04268048-13 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3070: oid 13 updating version 0 to 596 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3070: oid 13 version 596 is already newer than 595 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3070: oid 13 updating version 596 to 598 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3070: oid 13 version 598 is already newer than 597 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 598 (ObjNum 1063 snap 352 seq_num 1063) dirty exists 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3070: left oid 13 (ObjNum 1063 snap 352 seq_num 1063) 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3071: finishing write tid 1 to smithi04268048-26 2024-09-20T19:36:00.795 INFO:tasks.rados.rados.0.smithi042.stdout:3071: finishing write tid 2 to smithi04268048-26 2024-09-20T19:36:00.797 INFO:tasks.rados.rados.0.smithi042.stdout:3071: finishing write tid 3 to smithi04268048-26 2024-09-20T19:36:00.798 INFO:tasks.rados.rados.0.smithi042.stdout:3071: finishing write tid 4 to smithi04268048-26 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: finishing write tid 5 to smithi04268048-26 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: oid 26 updating version 0 to 1235 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: oid 26 updating version 1235 to 1237 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: oid 26 version 1237 is already newer than 1234 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: oid 26 version 1237 is already newer than 1236 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1237 (ObjNum 1064 snap 352 seq_num 1064) dirty exists 2024-09-20T19:36:00.799 INFO:tasks.rados.rados.0.smithi042.stdout:3071: left oid 26 (ObjNum 1064 snap 352 seq_num 1064) 2024-09-20T19:36:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:00 smithi162 bash[24557]: cluster 2024-09-20T19:35:59.484272+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2024-09-20T19:36:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:00 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:00 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.486579+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2024-09-20T19:36:00.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:00.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:00.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:00.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:01.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:01.084 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:01.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:01.214 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:01.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:01.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:01.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:01.475 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:01.537 INFO:tasks.rados.rados.0.smithi042.stdout:3067: done (5 left) 2024-09-20T19:36:01.537 INFO:tasks.rados.rados.0.smithi042.stdout:3068: done (4 left) 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3069: done (3 left) 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3070: done (2 left) 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3071: done (1 left) 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3072: done (0 left) 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3073: rollback oid 31 current snap is 353 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 344 2024-09-20T19:36:01.538 INFO:tasks.rados.rados.0.smithi042.stdout:3074: snap_remove snap 341 2024-09-20T19:36:01.577 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:01.708 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[32075]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:01 smithi042 bash[22706]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.734 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: audit 2024-09-20T19:35:59.785859+0000 mgr.y (mgr.24491) 677 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.145914+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.147974+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:01 smithi162 bash[24557]: cluster 2024-09-20T19:36:00.412864+0000 mgr.y (mgr.24491) 678 : cluster [DBG] pgmap v1275: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-20T19:36:01.837 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:01.865 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:01.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:01.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:01.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:02.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:02.108 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:36:02.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:02.226 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:02.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:02.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:02.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:02.520 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1276,"stamp":"2024-09-20T19:36:00.482843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398200665,"num_objects":389,"num_object_clones":142,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":13,"num_read":3984,"num_read_kb":1331958,"num_write":2040,"num_write_kb":496007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25106,"ondisk_log_size":25106,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1178236,"kb_used_data":858576,"kb_used_omap":92,"kb_used_meta":319395,"kb_avail":655037828,"statfs":{"total":671965249536,"available":670758735872,"internally_reserved":0,"allocated":879181824,"data_stored":1241690210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":327061177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8258447,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":63,"num_read_kb":45164,"num_write":45,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009268"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":3738,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.511608+0000","last_change":"2024-09-20T19:35:56.511704+0000","last_active":"2024-09-20T19:35:56.511608+0000","last_peered":"2024-09-20T19:35:56.511608+0000","last_clean":"2024-09-20T19:35:56.511608+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:35:56.511608+0000","last_undegraded":"2024-09-20T19:35:56.511608+0000","last_fullsized":"2024-09-20T19:35:56.511608+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:35:50.127690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:50.127690+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:53.740879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013614246,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"7.22","version":"849'290","reported_seq":3461,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486157+0000","last_change":"2024-09-20T19:35:58.487677+0000","last_active":"2024-09-20T19:35:58.486157+0000","last_peered":"2024-09-20T19:35:58.486157+0000","last_clean":"2024-09-20T19:35:58.486157+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:35:58.486157+0000","last_undegraded":"2024-09-20T19:35:58.486157+0000","last_fullsized":"2024-09-20T19:35:58.486157+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"844'282","last_scrub_stamp":"2024-09-20T19:35:52.944565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:52.944565+0000","objects_scrubbed":5,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:53.057706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002478021,"stat_sum":{"num_bytes":8000726,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":9602,"num_write":15,"num_write_kb":6047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"129","length":"1"},{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"},{"start":"159","length":"1"}]},{"pgid":"7.21","version":"848'687","reported_seq":4312,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.522173+0000","last_change":"2024-09-20T19:35:56.522173+0000","last_active":"2024-09-20T19:35:56.522173+0000","last_peered":"2024-09-20T19:35:56.522173+0000","last_clean":"2024-09-20T19:35:56.522173+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:35:56.522173+0000","last_undegraded":"2024-09-20T19:35:56.522173+0000","last_fullsized":"2024-09-20T19:35:56.522173+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"832'681","last_scrub_stamp":"2024-09-20T19:35:48.747875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:48.747875+0000","objects_scrubbed":5,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:56.469273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.01546467,"stat_sum":{"num_bytes":7984794,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":48,"num_read_kb":56176,"num_write":10,"num_write_kb":3726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"7.20","version":"847'902","reported_seq":4449,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487111+0000","last_change":"2024-09-20T19:35:56.487370+0000","last_active":"2024-09-20T19:35:56.487111+0000","last_peered":"2024-09-20T19:35:56.487111+0000","last_clean":"2024-09-20T19:35:56.487111+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:35:56.487111+0000","last_undegraded":"2024-09-20T19:35:56.487111+0000","last_fullsized":"2024-09-20T19:35:56.487111+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"721'837","last_scrub_stamp":"2024-09-20T19:33:59.352105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:59.352105+0000","objects_scrubbed":2,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015958599999999999,"stat_sum":{"num_bytes":11782549,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":27241,"num_write":20,"num_write_kb":9249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"7.1a","version":"850'641","reported_seq":3964,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.540948+0000","last_change":"2024-09-20T19:35:58.489304+0000","last_active":"2024-09-20T19:35:58.540948+0000","last_peered":"2024-09-20T19:35:58.540948+0000","last_clean":"2024-09-20T19:35:58.540948+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:35:58.540948+0000","last_undegraded":"2024-09-20T19:35:58.540948+0000","last_fullsized":"2024-09-20T19:35:58.540948+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"832'621","last_scrub_stamp":"2024-09-20T19:35:47.994943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.994943+0000","objects_scrubbed":7,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:51.688440+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0036915339999999998,"stat_sum":{"num_bytes":17608793,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":79,"num_read_kb":61667,"num_write":47,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"129","length":"1"},{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"},{"start":"159","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":1091,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488942+0000","last_change":"2024-09-20T19:33:43.387357+0000","last_active":"2024-09-20T19:35:56.488942+0000","last_peered":"2024-09-20T19:35:56.488942+0000","last_clean":"2024-09-20T19:35:56.488942+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:35:56.488942+0000","last_undegraded":"2024-09-20T19:35:56.488942+0000","last_fullsized":"2024-09-20T19:35:56.488942+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:37.502365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:37.502365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1413,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497866+0000","last_change":"2024-09-20T19:34:58.530958+0000","last_active":"2024-09-20T19:35:57.497866+0000","last_peered":"2024-09-20T19:35:57.497866+0000","last_clean":"2024-09-20T19:35:57.497866+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:35:57.497866+0000","last_undegraded":"2024-09-20T19:35:57.497866+0000","last_fullsized":"2024-09-20T19:35:57.497866+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:13.393745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:13.393745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"523'27","reported_seq":1333,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485743+0000","last_change":"2024-09-20T19:35:55.571670+0000","last_active":"2024-09-20T19:35:56.485743+0000","last_peered":"2024-09-20T19:35:56.485743+0000","last_clean":"2024-09-20T19:35:56.485743+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:35:56.485743+0000","last_undegraded":"2024-09-20T19:35:56.485743+0000","last_fullsized":"2024-09-20T19:35:56.485743+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:35:55.571607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:55.571607+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:18.747573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1394,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.484557+0000","last_change":"2024-09-20T19:35:04.624469+0000","last_active":"2024-09-20T19:35:58.484557+0000","last_peered":"2024-09-20T19:35:58.484557+0000","last_clean":"2024-09-20T19:35:58.484557+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:35:58.484557+0000","last_undegraded":"2024-09-20T19:35:58.484557+0000","last_fullsized":"2024-09-20T19:35:58.484557+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.624374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.624374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.762788+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":4017,"reported_epoch":849,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:57.496987+0000","last_change":"2024-09-20T19:35:56.510943+0000","last_active":"2024-09-20T19:35:57.496987+0000","last_peered":"2024-09-20T19:35:57.496987+0000","last_clean":"2024-09-20T19:35:57.496987+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:35:57.496987+0000","last_undegraded":"2024-09-20T19:35:57.496987+0000","last_fullsized":"2024-09-20T19:35:57.496987+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:33:53.566287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:53.566287+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.019123048,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"2.1e","version":"0'0","reported_seq":1149,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.499627+0000","last_change":"2024-09-20T19:33:43.367928+0000","last_active":"2024-09-20T19:35:57.499627+0000","last_peered":"2024-09-20T19:35:57.499627+0000","last_clean":"2024-09-20T19:35:57.499627+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:35:57.499627+0000","last_undegraded":"2024-09-20T19:35:57.499627+0000","last_fullsized":"2024-09-20T19:35:57.499627+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:12.873972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:12.873972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"523'15","reported_seq":1498,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.484930+0000","last_change":"2024-09-20T19:34:47.537704+0000","last_active":"2024-09-20T19:35:58.484930+0000","last_peered":"2024-09-20T19:35:58.484930+0000","last_clean":"2024-09-20T19:35:58.484930+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:35:58.484930+0000","last_undegraded":"2024-09-20T19:35:58.484930+0000","last_fullsized":"2024-09-20T19:35:58.484930+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:47.537616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.537616+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.097439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1622,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.475713+0000","last_change":"2024-09-20T19:34:54.283813+0000","last_active":"2024-09-20T19:35:56.475713+0000","last_peered":"2024-09-20T19:35:56.475713+0000","last_clean":"2024-09-20T19:35:56.475713+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:35:56.475713+0000","last_undegraded":"2024-09-20T19:35:56.475713+0000","last_fullsized":"2024-09-20T19:35:56.475713+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.283695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.283695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.569450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1456,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485861+0000","last_change":"2024-09-20T19:34:58.623699+0000","last_active":"2024-09-20T19:35:58.485861+0000","last_peered":"2024-09-20T19:35:58.485861+0000","last_clean":"2024-09-20T19:35:58.485861+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:35:58.485861+0000","last_undegraded":"2024-09-20T19:35:58.485861+0000","last_fullsized":"2024-09-20T19:35:58.485861+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:58.623637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:58.623637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:23.832554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"849'1185","reported_seq":4681,"reported_epoch":849,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:57.506314+0000","last_change":"2024-09-20T19:35:56.511706+0000","last_active":"2024-09-20T19:35:57.506314+0000","last_peered":"2024-09-20T19:35:57.506314+0000","last_clean":"2024-09-20T19:35:57.506314+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:35:57.506314+0000","last_undegraded":"2024-09-20T19:35:57.506314+0000","last_fullsized":"2024-09-20T19:35:57.506314+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"632'1040","last_scrub_stamp":"2024-09-20T19:32:32.473748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:32.473748+0000","objects_scrubbed":2,"log_size":1185,"log_dups_size":0,"ondisk_log_size":1185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.020674207999999999,"stat_sum":{"num_bytes":15161715,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":73,"num_read_kb":53384,"num_write":70,"num_write_kb":28605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.1d","version":"0'0","reported_seq":1620,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.494500+0000","last_change":"2024-09-20T19:32:29.864003+0000","last_active":"2024-09-20T19:35:56.494500+0000","last_peered":"2024-09-20T19:35:56.494500+0000","last_clean":"2024-09-20T19:35:56.494500+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:35:56.494500+0000","last_undegraded":"2024-09-20T19:35:56.494500+0000","last_fullsized":"2024-09-20T19:35:56.494500+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:29.863944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.863944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"523'7","reported_seq":1493,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485813+0000","last_change":"2024-09-20T19:34:42.553244+0000","last_active":"2024-09-20T19:35:58.485813+0000","last_peered":"2024-09-20T19:35:58.485813+0000","last_clean":"2024-09-20T19:35:58.485813+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:35:58.485813+0000","last_undegraded":"2024-09-20T19:35:58.485813+0000","last_fullsized":"2024-09-20T19:35:58.485813+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:34:42.553118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:42.553118+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.964837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1612,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487708+0000","last_change":"2024-09-20T19:34:50.140557+0000","last_active":"2024-09-20T19:35:56.487708+0000","last_peered":"2024-09-20T19:35:56.487708+0000","last_clean":"2024-09-20T19:35:56.487708+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:35:56.487708+0000","last_undegraded":"2024-09-20T19:35:56.487708+0000","last_fullsized":"2024-09-20T19:35:56.487708+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:50.140498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:50.140498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1623,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.505979+0000","last_change":"2024-09-20T19:33:43.447076+0000","last_active":"2024-09-20T19:35:56.505979+0000","last_peered":"2024-09-20T19:35:56.505979+0000","last_clean":"2024-09-20T19:35:56.505979+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:35:56.505979+0000","last_undegraded":"2024-09-20T19:35:56.505979+0000","last_fullsized":"2024-09-20T19:35:56.505979+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:28.436481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:28.436481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.802489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"844'420","reported_seq":3971,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.514016+0000","last_change":"2024-09-20T19:35:56.514136+0000","last_active":"2024-09-20T19:35:56.514016+0000","last_peered":"2024-09-20T19:35:56.514016+0000","last_clean":"2024-09-20T19:35:56.514016+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:35:56.514016+0000","last_undegraded":"2024-09-20T19:35:56.514016+0000","last_fullsized":"2024-09-20T19:35:56.514016+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"829'418","last_scrub_stamp":"2024-09-20T19:35:43.214781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:43.214781+0000","objects_scrubbed":2,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:52.168276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018756220000000001,"stat_sum":{"num_bytes":3521610,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":46162,"num_write":32,"num_write_kb":8793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"2.1c","version":"0'0","reported_seq":1609,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.492676+0000","last_change":"2024-09-20T19:32:22.784663+0000","last_active":"2024-09-20T19:35:56.492676+0000","last_peered":"2024-09-20T19:35:56.492676+0000","last_clean":"2024-09-20T19:35:56.492676+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:35:56.492676+0000","last_undegraded":"2024-09-20T19:35:56.492676+0000","last_fullsized":"2024-09-20T19:35:56.492676+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:22.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:22.784543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"523'15","reported_seq":1534,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485194+0000","last_change":"2024-09-20T19:35:36.810092+0000","last_active":"2024-09-20T19:35:58.485194+0000","last_peered":"2024-09-20T19:35:58.485194+0000","last_clean":"2024-09-20T19:35:58.485194+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:35:58.485194+0000","last_undegraded":"2024-09-20T19:35:58.485194+0000","last_fullsized":"2024-09-20T19:35:58.485194+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:36.809949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:36.809949+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:51.268475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1604,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.509730+0000","last_change":"2024-09-20T19:32:20.560562+0000","last_active":"2024-09-20T19:35:56.509730+0000","last_peered":"2024-09-20T19:35:56.509730+0000","last_clean":"2024-09-20T19:35:56.509730+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:35:56.509730+0000","last_undegraded":"2024-09-20T19:35:56.509730+0000","last_fullsized":"2024-09-20T19:35:56.509730+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.560451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.560451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.786217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":582,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.495370+0000","last_change":"2024-09-20T19:32:23.370108+0000","last_active":"2024-09-20T19:35:57.495370+0000","last_peered":"2024-09-20T19:35:57.495370+0000","last_clean":"2024-09-20T19:35:57.495370+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:35:57.495370+0000","last_undegraded":"2024-09-20T19:35:57.495370+0000","last_fullsized":"2024-09-20T19:35:57.495370+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:23.370012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.370012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1239,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.504765+0000","last_change":"2024-09-20T19:34:07.057170+0000","last_active":"2024-09-20T19:35:56.504765+0000","last_peered":"2024-09-20T19:35:56.504765+0000","last_clean":"2024-09-20T19:35:56.504765+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:35:56.504765+0000","last_undegraded":"2024-09-20T19:35:56.504765+0000","last_fullsized":"2024-09-20T19:35:56.504765+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:07.057076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:07.057076+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"850'326","reported_seq":3900,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.503346+0000","last_change":"2024-09-20T19:35:58.505132+0000","last_active":"2024-09-20T19:35:58.503346+0000","last_peered":"2024-09-20T19:35:58.503346+0000","last_clean":"2024-09-20T19:35:58.503346+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:35:58.503346+0000","last_undegraded":"2024-09-20T19:35:58.503346+0000","last_fullsized":"2024-09-20T19:35:58.503346+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'320","last_scrub_stamp":"2024-09-20T19:35:49.940578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.940578+0000","objects_scrubbed":2,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:08.805628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018982780000000001,"stat_sum":{"num_bytes":8471125,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":5339,"num_write":6,"num_write_kb":5197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"129","length":"1"},{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"},{"start":"159","length":"1"}]},{"pgid":"4.d","version":"523'31","reported_seq":1476,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486301+0000","last_change":"2024-09-20T19:34:13.521735+0000","last_active":"2024-09-20T19:35:58.486301+0000","last_peered":"2024-09-20T19:35:58.486301+0000","last_clean":"2024-09-20T19:35:58.486301+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:35:58.486301+0000","last_undegraded":"2024-09-20T19:35:58.486301+0000","last_fullsized":"2024-09-20T19:35:58.486301+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:34:13.521620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.521620+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1612,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.490287+0000","last_change":"2024-09-20T19:34:49.185014+0000","last_active":"2024-09-20T19:35:56.490287+0000","last_peered":"2024-09-20T19:35:56.490287+0000","last_clean":"2024-09-20T19:35:56.490287+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:35:56.490287+0000","last_undegraded":"2024-09-20T19:35:56.490287+0000","last_fullsized":"2024-09-20T19:35:56.490287+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:49.184958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:49.184958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1624,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476221+0000","last_change":"2024-09-20T19:31:25.357653+0000","last_active":"2024-09-20T19:35:56.476221+0000","last_peered":"2024-09-20T19:35:56.476221+0000","last_clean":"2024-09-20T19:35:56.476221+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:35:56.476221+0000","last_undegraded":"2024-09-20T19:35:56.476221+0000","last_fullsized":"2024-09-20T19:35:56.476221+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:25.357600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:25.357600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.402096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":192,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486579+0000","last_change":"2024-09-20T19:34:26.402007+0000","last_active":"2024-09-20T19:35:56.486579+0000","last_peered":"2024-09-20T19:35:56.486579+0000","last_clean":"2024-09-20T19:35:56.486579+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:35:56.486579+0000","last_undegraded":"2024-09-20T19:35:56.486579+0000","last_fullsized":"2024-09-20T19:35:56.486579+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:06.544189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:06.544189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:14.318932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":1705,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488298+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:35:56.488298+0000","last_peered":"2024-09-20T19:35:56.488298+0000","last_clean":"2024-09-20T19:35:56.488298+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:35:56.488298+0000","last_undegraded":"2024-09-20T19:35:56.488298+0000","last_fullsized":"2024-09-20T19:35:56.488298+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"846'578","reported_seq":3908,"reported_epoch":849,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:57.497710+0000","last_change":"2024-09-20T19:35:56.509828+0000","last_active":"2024-09-20T19:35:57.497710+0000","last_peered":"2024-09-20T19:35:57.497710+0000","last_clean":"2024-09-20T19:35:57.497710+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:35:57.497710+0000","last_undegraded":"2024-09-20T19:35:57.497710+0000","last_fullsized":"2024-09-20T19:35:57.497710+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'544","last_scrub_stamp":"2024-09-20T19:33:54.617348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.617348+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.017522029000000001,"stat_sum":{"num_bytes":9042793,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":10989,"num_write":16,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"4.c","version":"523'20","reported_seq":1527,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485896+0000","last_change":"2024-09-20T19:35:32.719860+0000","last_active":"2024-09-20T19:35:58.485896+0000","last_peered":"2024-09-20T19:35:58.485896+0000","last_clean":"2024-09-20T19:35:58.485896+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:35:58.485896+0000","last_undegraded":"2024-09-20T19:35:58.485896+0000","last_fullsized":"2024-09-20T19:35:58.485896+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:35:32.719711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:32.719711+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:34.804796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1649,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478452+0000","last_change":"2024-09-20T19:34:57.369277+0000","last_active":"2024-09-20T19:35:56.478452+0000","last_peered":"2024-09-20T19:35:56.478452+0000","last_clean":"2024-09-20T19:35:56.478452+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:35:56.478452+0000","last_undegraded":"2024-09-20T19:35:56.478452+0000","last_fullsized":"2024-09-20T19:35:56.478452+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.369117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.369117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:07.185078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":1847,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486693+0000","last_change":"2024-09-20T19:34:17.102983+0000","last_active":"2024-09-20T19:35:56.486693+0000","last_peered":"2024-09-20T19:35:56.486693+0000","last_clean":"2024-09-20T19:35:56.486693+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:35:56.486693+0000","last_undegraded":"2024-09-20T19:35:56.486693+0000","last_fullsized":"2024-09-20T19:35:56.486693+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:34:17.102808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.102808+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1458,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486442+0000","last_change":"2024-09-20T19:34:45.548359+0000","last_active":"2024-09-20T19:35:58.486442+0000","last_peered":"2024-09-20T19:35:58.486442+0000","last_clean":"2024-09-20T19:35:58.486442+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:35:58.486442+0000","last_undegraded":"2024-09-20T19:35:58.486442+0000","last_fullsized":"2024-09-20T19:35:58.486442+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:45.548235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:45.548235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:59.055514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1210,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.506699+0000","last_change":"2024-09-20T19:34:10.011707+0000","last_active":"2024-09-20T19:35:56.506699+0000","last_peered":"2024-09-20T19:35:56.506699+0000","last_clean":"2024-09-20T19:35:56.506699+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:35:56.506699+0000","last_undegraded":"2024-09-20T19:35:56.506699+0000","last_fullsized":"2024-09-20T19:35:56.506699+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:10.011637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.011637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"844'894","reported_seq":4524,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.491086+0000","last_change":"2024-09-20T19:35:54.509910+0000","last_active":"2024-09-20T19:35:56.491086+0000","last_peered":"2024-09-20T19:35:56.491086+0000","last_clean":"2024-09-20T19:35:56.491086+0000","last_became_active":"2024-09-20T19:35:34.395504+0000","last_became_peered":"2024-09-20T19:35:34.395504+0000","last_unstale":"2024-09-20T19:35:56.491086+0000","last_undegraded":"2024-09-20T19:35:56.491086+0000","last_fullsized":"2024-09-20T19:35:56.491086+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":826,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'835","last_scrub_stamp":"2024-09-20T19:33:55.628320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.628320+0000","objects_scrubbed":2,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.094459285000000004,"stat_sum":{"num_bytes":12463144,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":8535,"num_write":18,"num_write_kb":6941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.f","version":"523'27","reported_seq":1715,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.477744+0000","last_change":"2024-09-20T19:34:56.365553+0000","last_active":"2024-09-20T19:35:56.477744+0000","last_peered":"2024-09-20T19:35:56.477744+0000","last_clean":"2024-09-20T19:35:56.477744+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:35:56.477744+0000","last_undegraded":"2024-09-20T19:35:56.477744+0000","last_fullsized":"2024-09-20T19:35:56.477744+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:56.365402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:56.365402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.638116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1649,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476031+0000","last_change":"2024-09-20T19:34:46.261824+0000","last_active":"2024-09-20T19:35:56.476031+0000","last_peered":"2024-09-20T19:35:56.476031+0000","last_clean":"2024-09-20T19:35:56.476031+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:35:56.476031+0000","last_undegraded":"2024-09-20T19:35:56.476031+0000","last_fullsized":"2024-09-20T19:35:56.476031+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:46.261770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.261770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:04.225781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":1646,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486630+0000","last_change":"2024-09-20T19:35:53.913698+0000","last_active":"2024-09-20T19:35:58.486630+0000","last_peered":"2024-09-20T19:35:58.486630+0000","last_clean":"2024-09-20T19:35:58.486630+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:35:58.486630+0000","last_undegraded":"2024-09-20T19:35:58.486630+0000","last_fullsized":"2024-09-20T19:35:58.486630+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:53.913572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:53.913572+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:56.477761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1616,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.510596+0000","last_change":"2024-09-20T19:34:17.113285+0000","last_active":"2024-09-20T19:35:56.510596+0000","last_peered":"2024-09-20T19:35:56.510596+0000","last_clean":"2024-09-20T19:35:56.510596+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:35:56.510596+0000","last_undegraded":"2024-09-20T19:35:56.510596+0000","last_fullsized":"2024-09-20T19:35:56.510596+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:17.113170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.113170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"848'594","reported_seq":4185,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.514326+0000","last_change":"2024-09-20T19:35:56.514326+0000","last_active":"2024-09-20T19:35:56.514326+0000","last_peered":"2024-09-20T19:35:56.514326+0000","last_clean":"2024-09-20T19:35:56.514326+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:35:56.514326+0000","last_undegraded":"2024-09-20T19:35:56.514326+0000","last_fullsized":"2024-09-20T19:35:56.514326+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'590","last_scrub_stamp":"2024-09-20T19:35:49.178758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.178758+0000","objects_scrubbed":4,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:12.148539+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.073997172999999999,"stat_sum":{"num_bytes":10031323,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":24,"num_write_kb":6943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.e","version":"523'19","reported_seq":1500,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486821+0000","last_change":"2024-09-20T19:31:35.807162+0000","last_active":"2024-09-20T19:35:58.486821+0000","last_peered":"2024-09-20T19:35:58.486821+0000","last_clean":"2024-09-20T19:35:58.486821+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:35:58.486821+0000","last_undegraded":"2024-09-20T19:35:58.486821+0000","last_fullsized":"2024-09-20T19:35:58.486821+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:31:35.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.807096+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:27.666205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":278,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486981+0000","last_change":"2024-09-20T19:33:43.387910+0000","last_active":"2024-09-20T19:35:56.486981+0000","last_peered":"2024-09-20T19:35:56.486981+0000","last_clean":"2024-09-20T19:35:56.486981+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:35:56.486981+0000","last_undegraded":"2024-09-20T19:35:56.486981+0000","last_fullsized":"2024-09-20T19:35:56.486981+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:26.549387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:26.549387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1620,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.512447+0000","last_change":"2024-09-20T19:34:48.070464+0000","last_active":"2024-09-20T19:35:56.512447+0000","last_peered":"2024-09-20T19:35:56.512447+0000","last_clean":"2024-09-20T19:35:56.512447+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:35:56.512447+0000","last_undegraded":"2024-09-20T19:35:56.512447+0000","last_fullsized":"2024-09-20T19:35:56.512447+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.070340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.070340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:14.947230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1087,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.497941+0000","last_change":"2024-09-20T19:33:43.352629+0000","last_active":"2024-09-20T19:35:56.497941+0000","last_peered":"2024-09-20T19:35:56.497941+0000","last_clean":"2024-09-20T19:35:56.497941+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:35:56.497941+0000","last_undegraded":"2024-09-20T19:35:56.497941+0000","last_fullsized":"2024-09-20T19:35:56.497941+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:27:56.229505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:27:56.229505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.525029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":448,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.523777+0000","last_change":"2024-09-20T19:35:03.930343+0000","last_active":"2024-09-20T19:35:56.523777+0000","last_peered":"2024-09-20T19:35:56.523777+0000","last_clean":"2024-09-20T19:35:56.523777+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:35:56.523777+0000","last_undegraded":"2024-09-20T19:35:56.523777+0000","last_fullsized":"2024-09-20T19:35:56.523777+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:03.930289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.930289+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.856845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":3305,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.512413+0000","last_change":"2024-09-20T19:35:56.512777+0000","last_active":"2024-09-20T19:35:56.512413+0000","last_peered":"2024-09-20T19:35:56.512413+0000","last_clean":"2024-09-20T19:35:56.512413+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:35:56.512413+0000","last_undegraded":"2024-09-20T19:35:56.512413+0000","last_fullsized":"2024-09-20T19:35:56.512413+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:35:33.305049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:33.305049+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:37.098577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.034931402,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"4.1","version":"523'28","reported_seq":1469,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485061+0000","last_change":"2024-09-20T19:32:20.372534+0000","last_active":"2024-09-20T19:35:58.485061+0000","last_peered":"2024-09-20T19:35:58.485061+0000","last_clean":"2024-09-20T19:35:58.485061+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:35:58.485061+0000","last_undegraded":"2024-09-20T19:35:58.485061+0000","last_fullsized":"2024-09-20T19:35:58.485061+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'28","last_scrub_stamp":"2024-09-20T19:32:20.372307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.372307+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:29.697913+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1611,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.501431+0000","last_change":"2024-09-20T19:28:02.698201+0000","last_active":"2024-09-20T19:35:56.501431+0000","last_peered":"2024-09-20T19:35:56.501431+0000","last_clean":"2024-09-20T19:35:56.501431+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:35:56.501431+0000","last_undegraded":"2024-09-20T19:35:56.501431+0000","last_fullsized":"2024-09-20T19:35:56.501431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:02.698051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:02.698051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:13.574522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":526,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.475318+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:35:56.475318+0000","last_peered":"2024-09-20T19:35:56.475318+0000","last_clean":"2024-09-20T19:35:56.475318+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:35:56.475318+0000","last_undegraded":"2024-09-20T19:35:56.475318+0000","last_fullsized":"2024-09-20T19:35:56.475318+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1598,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.489523+0000","last_change":"2024-09-20T19:34:16.144529+0000","last_active":"2024-09-20T19:35:56.489523+0000","last_peered":"2024-09-20T19:35:56.489523+0000","last_clean":"2024-09-20T19:35:56.489523+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:35:56.489523+0000","last_undegraded":"2024-09-20T19:35:56.489523+0000","last_fullsized":"2024-09-20T19:35:56.489523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:16.144473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:16.144473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1031,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486543+0000","last_change":"2024-09-20T19:35:07.564250+0000","last_active":"2024-09-20T19:35:58.486543+0000","last_peered":"2024-09-20T19:35:58.486543+0000","last_clean":"2024-09-20T19:35:58.486543+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:35:58.486543+0000","last_undegraded":"2024-09-20T19:35:58.486543+0000","last_fullsized":"2024-09-20T19:35:58.486543+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:07.564190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.564190+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:29.024831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"848'437","reported_seq":3838,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.494704+0000","last_change":"2024-09-20T19:35:56.495058+0000","last_active":"2024-09-20T19:35:56.494704+0000","last_peered":"2024-09-20T19:35:56.494704+0000","last_clean":"2024-09-20T19:35:56.494704+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:35:56.494704+0000","last_undegraded":"2024-09-20T19:35:56.494704+0000","last_fullsized":"2024-09-20T19:35:56.494704+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'421","last_scrub_stamp":"2024-09-20T19:34:59.428803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.428803+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:04.110307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090266400000000007,"stat_sum":{"num_bytes":4982359,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":7624,"num_write":11,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"4.0","version":"523'30","reported_seq":1337,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487524+0000","last_change":"2024-09-20T19:33:43.476313+0000","last_active":"2024-09-20T19:35:56.487524+0000","last_peered":"2024-09-20T19:35:56.487524+0000","last_clean":"2024-09-20T19:35:56.487524+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:35:56.487524+0000","last_undegraded":"2024-09-20T19:35:56.487524+0000","last_fullsized":"2024-09-20T19:35:56.487524+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:32:20.900888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.900888+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":1627,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.477365+0000","last_change":"2024-09-20T19:35:47.136622+0000","last_active":"2024-09-20T19:35:56.477365+0000","last_peered":"2024-09-20T19:35:56.477365+0000","last_clean":"2024-09-20T19:35:56.477365+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:35:56.477365+0000","last_undegraded":"2024-09-20T19:35:56.477365+0000","last_fullsized":"2024-09-20T19:35:56.477365+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:35:47.136550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.136550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:52.713879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1436,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.484525+0000","last_change":"2024-09-20T19:35:41.834087+0000","last_active":"2024-09-20T19:35:58.484525+0000","last_peered":"2024-09-20T19:35:58.484525+0000","last_clean":"2024-09-20T19:35:58.484525+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:35:58.484525+0000","last_undegraded":"2024-09-20T19:35:58.484525+0000","last_fullsized":"2024-09-20T19:35:58.484525+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:41.834028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:41.834028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:57.042766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1390,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485283+0000","last_change":"2024-09-20T19:34:59.643830+0000","last_active":"2024-09-20T19:35:58.485283+0000","last_peered":"2024-09-20T19:35:58.485283+0000","last_clean":"2024-09-20T19:35:58.485283+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:35:58.485283+0000","last_undegraded":"2024-09-20T19:35:58.485283+0000","last_fullsized":"2024-09-20T19:35:58.485283+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.643704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.643704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1331,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.499264+0000","last_change":"2024-09-20T19:35:00.419430+0000","last_active":"2024-09-20T19:35:56.499264+0000","last_peered":"2024-09-20T19:35:56.499264+0000","last_clean":"2024-09-20T19:35:56.499264+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:35:56.499264+0000","last_undegraded":"2024-09-20T19:35:56.499264+0000","last_fullsized":"2024-09-20T19:35:56.499264+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:00.419230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:00.419230+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:26.608264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"844'896","reported_seq":4659,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.522002+0000","last_change":"2024-09-20T19:35:56.522145+0000","last_active":"2024-09-20T19:35:56.522002+0000","last_peered":"2024-09-20T19:35:56.522002+0000","last_clean":"2024-09-20T19:35:56.522002+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:35:56.522002+0000","last_undegraded":"2024-09-20T19:35:56.522002+0000","last_fullsized":"2024-09-20T19:35:56.522002+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'829","last_scrub_stamp":"2024-09-20T19:34:01.119894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:01.119894+0000","objects_scrubbed":1,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017704561000000001,"stat_sum":{"num_bytes":13877560,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":50,"num_read_kb":56717,"num_write":22,"num_write_kb":4742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.3","version":"523'29","reported_seq":1431,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.496681+0000","last_change":"2024-09-20T19:32:30.464174+0000","last_active":"2024-09-20T19:35:57.496681+0000","last_peered":"2024-09-20T19:35:57.496681+0000","last_clean":"2024-09-20T19:35:57.496681+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:35:57.496681+0000","last_undegraded":"2024-09-20T19:35:57.496681+0000","last_fullsized":"2024-09-20T19:35:57.496681+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:32:30.464013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.464013+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1576,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486574+0000","last_change":"2024-09-20T19:34:05.253077+0000","last_active":"2024-09-20T19:35:56.486574+0000","last_peered":"2024-09-20T19:35:56.486574+0000","last_clean":"2024-09-20T19:35:56.486574+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:35:56.486574+0000","last_undegraded":"2024-09-20T19:35:56.486574+0000","last_fullsized":"2024-09-20T19:35:56.486574+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:05.253020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:05.253020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1622,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.495550+0000","last_change":"2024-09-20T19:35:48.183610+0000","last_active":"2024-09-20T19:35:56.495550+0000","last_peered":"2024-09-20T19:35:56.495550+0000","last_clean":"2024-09-20T19:35:56.495550+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:35:56.495550+0000","last_undegraded":"2024-09-20T19:35:56.495550+0000","last_fullsized":"2024-09-20T19:35:56.495550+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:48.183545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:48.183545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:15.933730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1617,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.475701+0000","last_change":"2024-09-20T19:34:55.325695+0000","last_active":"2024-09-20T19:35:56.475701+0000","last_peered":"2024-09-20T19:35:56.475701+0000","last_clean":"2024-09-20T19:35:56.475701+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:35:56.475701+0000","last_undegraded":"2024-09-20T19:35:56.475701+0000","last_fullsized":"2024-09-20T19:35:56.475701+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.325573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.325573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:35:58.162804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1324,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.500112+0000","last_change":"2024-09-20T19:33:43.353150+0000","last_active":"2024-09-20T19:35:56.500112+0000","last_peered":"2024-09-20T19:35:56.500112+0000","last_clean":"2024-09-20T19:35:56.500112+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:35:56.500112+0000","last_undegraded":"2024-09-20T19:35:56.500112+0000","last_fullsized":"2024-09-20T19:35:56.500112+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:27:54.666885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:27:54.666885+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:15.514065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":4102,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485993+0000","last_change":"2024-09-20T19:35:58.488073+0000","last_active":"2024-09-20T19:35:58.485993+0000","last_peered":"2024-09-20T19:35:58.485993+0000","last_clean":"2024-09-20T19:35:58.485993+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:35:58.485993+0000","last_undegraded":"2024-09-20T19:35:58.485993+0000","last_fullsized":"2024-09-20T19:35:58.485993+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:35:43.876617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:43.876617+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:03.977617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002866352,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"129","length":"1"},{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"},{"start":"159","length":"1"}]},{"pgid":"4.2","version":"523'20","reported_seq":1671,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478294+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:35:56.478294+0000","last_peered":"2024-09-20T19:35:56.478294+0000","last_clean":"2024-09-20T19:35:56.478294+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:35:56.478294+0000","last_undegraded":"2024-09-20T19:35:56.478294+0000","last_fullsized":"2024-09-20T19:35:56.478294+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1626,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476684+0000","last_change":"2024-09-20T19:34:48.237100+0000","last_active":"2024-09-20T19:35:56.476684+0000","last_peered":"2024-09-20T19:35:56.476684+0000","last_clean":"2024-09-20T19:35:56.476684+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:35:56.476684+0000","last_undegraded":"2024-09-20T19:35:56.476684+0000","last_fullsized":"2024-09-20T19:35:56.476684+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.236983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.236983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.560188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":1562,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497287+0000","last_change":"2024-09-20T19:32:29.434161+0000","last_active":"2024-09-20T19:35:57.497287+0000","last_peered":"2024-09-20T19:35:57.497287+0000","last_clean":"2024-09-20T19:35:57.497287+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:35:57.497287+0000","last_undegraded":"2024-09-20T19:35:57.497287+0000","last_fullsized":"2024-09-20T19:35:57.497287+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:32:29.434004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:29.434004+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1424,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.495812+0000","last_change":"2024-09-20T19:35:47.661521+0000","last_active":"2024-09-20T19:35:57.495812+0000","last_peered":"2024-09-20T19:35:57.495812+0000","last_clean":"2024-09-20T19:35:57.495812+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:35:57.495812+0000","last_undegraded":"2024-09-20T19:35:57.495812+0000","last_fullsized":"2024-09-20T19:35:57.495812+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.661410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.661410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:13.023728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1213,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.505470+0000","last_change":"2024-09-20T19:32:21.523083+0000","last_active":"2024-09-20T19:35:56.505470+0000","last_peered":"2024-09-20T19:35:56.505470+0000","last_clean":"2024-09-20T19:35:56.505470+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:35:56.505470+0000","last_undegraded":"2024-09-20T19:35:56.505470+0000","last_fullsized":"2024-09-20T19:35:56.505470+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:21.522955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.522955+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:26.787771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"848'557","reported_seq":4223,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.514506+0000","last_change":"2024-09-20T19:35:56.514506+0000","last_active":"2024-09-20T19:35:56.514506+0000","last_peered":"2024-09-20T19:35:56.514506+0000","last_clean":"2024-09-20T19:35:56.514506+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:35:56.514506+0000","last_undegraded":"2024-09-20T19:35:56.514506+0000","last_fullsized":"2024-09-20T19:35:56.514506+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"834'548","last_scrub_stamp":"2024-09-20T19:35:46.197213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:46.197213+0000","objects_scrubbed":8,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:01.156473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.010303022,"stat_sum":{"num_bytes":15908696,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":99,"num_read_kb":68567,"num_write":68,"num_write_kb":22680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.4","version":"715'45","reported_seq":1961,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476573+0000","last_change":"2024-09-20T19:34:17.143462+0000","last_active":"2024-09-20T19:35:56.476573+0000","last_peered":"2024-09-20T19:35:56.476573+0000","last_clean":"2024-09-20T19:35:56.476573+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:35:56.476573+0000","last_undegraded":"2024-09-20T19:35:56.476573+0000","last_fullsized":"2024-09-20T19:35:56.476573+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'45","last_scrub_stamp":"2024-09-20T19:34:17.143380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:17.143380+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1626,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.508452+0000","last_change":"2024-09-20T19:34:14.039743+0000","last_active":"2024-09-20T19:35:56.508452+0000","last_peered":"2024-09-20T19:35:56.508452+0000","last_clean":"2024-09-20T19:35:56.508452+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:35:56.508452+0000","last_undegraded":"2024-09-20T19:35:56.508452+0000","last_fullsized":"2024-09-20T19:35:56.508452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:14.039620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:14.039620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1414,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497610+0000","last_change":"2024-09-20T19:35:49.735749+0000","last_active":"2024-09-20T19:35:57.497610+0000","last_peered":"2024-09-20T19:35:57.497610+0000","last_clean":"2024-09-20T19:35:57.497610+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:35:57.497610+0000","last_undegraded":"2024-09-20T19:35:57.497610+0000","last_fullsized":"2024-09-20T19:35:57.497610+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:49.735628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.735628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:07.556679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1072,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486495+0000","last_change":"2024-09-20T19:33:43.347560+0000","last_active":"2024-09-20T19:35:58.486495+0000","last_peered":"2024-09-20T19:35:58.486495+0000","last_clean":"2024-09-20T19:35:58.486495+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:35:58.486495+0000","last_undegraded":"2024-09-20T19:35:58.486495+0000","last_fullsized":"2024-09-20T19:35:58.486495+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:14.826722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:14.826722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.580288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":1683,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488870+0000","last_change":"2024-09-20T19:32:31.871758+0000","last_active":"2024-09-20T19:35:56.488870+0000","last_peered":"2024-09-20T19:35:56.488870+0000","last_clean":"2024-09-20T19:35:56.488870+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:35:56.488870+0000","last_undegraded":"2024-09-20T19:35:56.488870+0000","last_fullsized":"2024-09-20T19:35:56.488870+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:32:31.871626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:32:31.871626+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"849'994","reported_seq":4237,"reported_epoch":849,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:57.511983+0000","last_change":"2024-09-20T19:35:56.512750+0000","last_active":"2024-09-20T19:35:57.511983+0000","last_peered":"2024-09-20T19:35:57.511983+0000","last_clean":"2024-09-20T19:35:57.511983+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:35:57.511983+0000","last_undegraded":"2024-09-20T19:35:57.511983+0000","last_fullsized":"2024-09-20T19:35:57.511983+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'984","last_scrub_stamp":"2024-09-20T19:35:46.717005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:46.717005+0000","objects_scrubbed":10,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:10.537034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.016982369000000001,"stat_sum":{"num_bytes":18067270,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":119,"num_read_kb":111744,"num_write":73,"num_write_kb":26224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.7","version":"523'17","reported_seq":1587,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487294+0000","last_change":"2024-09-20T19:33:04.118188+0000","last_active":"2024-09-20T19:35:56.487294+0000","last_peered":"2024-09-20T19:35:56.487294+0000","last_clean":"2024-09-20T19:35:56.487294+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:35:56.487294+0000","last_undegraded":"2024-09-20T19:35:56.487294+0000","last_fullsized":"2024-09-20T19:35:56.487294+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:33:04.117998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:33:04.117998+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.306669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":2682,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.475486+0000","last_change":"2024-09-20T19:35:02.415570+0000","last_active":"2024-09-20T19:35:56.475486+0000","last_peered":"2024-09-20T19:35:56.475486+0000","last_clean":"2024-09-20T19:35:56.475486+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:35:56.475486+0000","last_undegraded":"2024-09-20T19:35:56.475486+0000","last_fullsized":"2024-09-20T19:35:56.475486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:35:02.415432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:35:02.415432+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:22.696507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1051,"num_read_kb":1046,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1646,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485791+0000","last_change":"2024-09-20T19:35:47.420774+0000","last_active":"2024-09-20T19:35:56.485791+0000","last_peered":"2024-09-20T19:35:56.485791+0000","last_clean":"2024-09-20T19:35:56.485791+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:35:56.485791+0000","last_undegraded":"2024-09-20T19:35:56.485791+0000","last_fullsized":"2024-09-20T19:35:56.485791+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.420685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.420685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:06.821962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":200,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486579+0000","last_change":"2024-09-20T19:35:36.267944+0000","last_active":"2024-09-20T19:35:56.486579+0000","last_peered":"2024-09-20T19:35:56.486579+0000","last_clean":"2024-09-20T19:35:56.486579+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:35:56.486579+0000","last_undegraded":"2024-09-20T19:35:56.486579+0000","last_fullsized":"2024-09-20T19:35:56.486579+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:36.267819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:36.267819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:48.661733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1616,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.493369+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:35:56.493369+0000","last_peered":"2024-09-20T19:35:56.493369+0000","last_clean":"2024-09-20T19:35:56.493369+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:35:56.493369+0000","last_undegraded":"2024-09-20T19:35:56.493369+0000","last_fullsized":"2024-09-20T19:35:56.493369+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1216,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478394+0000","last_change":"2024-09-20T19:31:28.342297+0000","last_active":"2024-09-20T19:35:56.478394+0000","last_peered":"2024-09-20T19:35:56.478394+0000","last_clean":"2024-09-20T19:35:56.478394+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:35:56.478394+0000","last_undegraded":"2024-09-20T19:35:56.478394+0000","last_fullsized":"2024-09-20T19:35:56.478394+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:31:28.342212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:31:28.342212+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:27.380299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"847'1035","reported_seq":4633,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.507717+0000","last_change":"2024-09-20T19:35:56.507864+0000","last_active":"2024-09-20T19:35:56.507717+0000","last_peered":"2024-09-20T19:35:56.507717+0000","last_clean":"2024-09-20T19:35:56.507717+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:35:56.507717+0000","last_undegraded":"2024-09-20T19:35:56.507717+0000","last_fullsized":"2024-09-20T19:35:56.507717+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"589'825","last_scrub_stamp":"2024-09-20T19:31:39.884442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:31:39.884442+0000","objects_scrubbed":4,"log_size":1035,"log_dups_size":0,"ondisk_log_size":1035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014341371,"stat_sum":{"num_bytes":18496739,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":88,"num_read_kb":51543,"num_write":72,"num_write_kb":23671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.6","version":"523'24","reported_seq":1446,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.498696+0000","last_change":"2024-09-20T19:34:26.408655+0000","last_active":"2024-09-20T19:35:57.498696+0000","last_peered":"2024-09-20T19:35:57.498696+0000","last_clean":"2024-09-20T19:35:57.498696+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:35:57.498696+0000","last_undegraded":"2024-09-20T19:35:57.498696+0000","last_fullsized":"2024-09-20T19:35:57.498696+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:32:57.496961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:57.496961+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1360,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.499273+0000","last_change":"2024-09-20T19:34:43.943794+0000","last_active":"2024-09-20T19:35:57.499273+0000","last_peered":"2024-09-20T19:35:57.499273+0000","last_clean":"2024-09-20T19:35:57.499273+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:35:57.499273+0000","last_undegraded":"2024-09-20T19:35:57.499273+0000","last_fullsized":"2024-09-20T19:35:57.499273+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:34:43.943661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.943661+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1621,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487955+0000","last_change":"2024-09-20T19:32:16.795244+0000","last_active":"2024-09-20T19:35:56.487955+0000","last_peered":"2024-09-20T19:35:56.487955+0000","last_clean":"2024-09-20T19:35:56.487955+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:35:56.487955+0000","last_undegraded":"2024-09-20T19:35:56.487955+0000","last_fullsized":"2024-09-20T19:35:56.487955+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:16.795124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.795124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1621,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.512691+0000","last_change":"2024-09-20T19:35:40.668511+0000","last_active":"2024-09-20T19:35:56.512691+0000","last_peered":"2024-09-20T19:35:56.512691+0000","last_clean":"2024-09-20T19:35:56.512691+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:35:56.512691+0000","last_undegraded":"2024-09-20T19:35:56.512691+0000","last_fullsized":"2024-09-20T19:35:56.512691+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:40.668435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:40.668435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:02.957592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1645,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476072+0000","last_change":"2024-09-20T19:35:04.500375+0000","last_active":"2024-09-20T19:35:56.476072+0000","last_peered":"2024-09-20T19:35:56.476072+0000","last_clean":"2024-09-20T19:35:56.476072+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:35:56.476072+0000","last_undegraded":"2024-09-20T19:35:56.476072+0000","last_fullsized":"2024-09-20T19:35:56.476072+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:04.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:04.500313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.786196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":3509,"reported_epoch":849,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:57.500056+0000","last_change":"2024-09-20T19:35:56.512740+0000","last_active":"2024-09-20T19:35:57.500056+0000","last_peered":"2024-09-20T19:35:57.500056+0000","last_clean":"2024-09-20T19:35:57.500056+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:35:57.500056+0000","last_undegraded":"2024-09-20T19:35:57.500056+0000","last_fullsized":"2024-09-20T19:35:57.500056+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:35:06.711015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:06.711015+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:24.417785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.019439878000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.5","version":"523'30","reported_seq":1873,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.498851+0000","last_change":"2024-09-20T19:34:55.460785+0000","last_active":"2024-09-20T19:35:56.498851+0000","last_peered":"2024-09-20T19:35:56.498851+0000","last_clean":"2024-09-20T19:35:56.498851+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:35:56.498851+0000","last_undegraded":"2024-09-20T19:35:56.498851+0000","last_fullsized":"2024-09-20T19:35:56.498851+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:34:55.460588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.460588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.240558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":473,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.507830+0000","last_change":"2024-09-20T19:35:41.698701+0000","last_active":"2024-09-20T19:35:56.507830+0000","last_peered":"2024-09-20T19:35:56.507830+0000","last_clean":"2024-09-20T19:35:56.507830+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:35:56.507830+0000","last_undegraded":"2024-09-20T19:35:56.507830+0000","last_fullsized":"2024-09-20T19:35:56.507830+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:35:41.698640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:35:41.698640+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:03.031172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1610,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.510272+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:35:56.510272+0000","last_peered":"2024-09-20T19:35:56.510272+0000","last_clean":"2024-09-20T19:35:56.510272+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:35:56.510272+0000","last_undegraded":"2024-09-20T19:35:56.510272+0000","last_fullsized":"2024-09-20T19:35:56.510272+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":1671,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488200+0000","last_change":"2024-09-20T19:34:48.152641+0000","last_active":"2024-09-20T19:35:56.488200+0000","last_peered":"2024-09-20T19:35:56.488200+0000","last_clean":"2024-09-20T19:35:56.488200+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:35:56.488200+0000","last_undegraded":"2024-09-20T19:35:56.488200+0000","last_fullsized":"2024-09-20T19:35:56.488200+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:48.152576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.152576+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.506442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1594,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.492163+0000","last_change":"2024-09-20T19:32:20.802111+0000","last_active":"2024-09-20T19:35:56.492163+0000","last_peered":"2024-09-20T19:35:56.492163+0000","last_clean":"2024-09-20T19:35:56.492163+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:35:56.492163+0000","last_undegraded":"2024-09-20T19:35:56.492163+0000","last_fullsized":"2024-09-20T19:35:56.492163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:20.802051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:20.802051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1623,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.521097+0000","last_change":"2024-09-20T19:34:15.087033+0000","last_active":"2024-09-20T19:35:56.521097+0000","last_peered":"2024-09-20T19:35:56.521097+0000","last_clean":"2024-09-20T19:35:56.521097+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:35:56.521097+0000","last_undegraded":"2024-09-20T19:35:56.521097+0000","last_fullsized":"2024-09-20T19:35:56.521097+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:15.086902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:15.086902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"848'389","reported_seq":3659,"reported_epoch":849,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:57.499175+0000","last_change":"2024-09-20T19:35:56.500572+0000","last_active":"2024-09-20T19:35:57.499175+0000","last_peered":"2024-09-20T19:35:57.499175+0000","last_clean":"2024-09-20T19:35:57.499175+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:35:57.499175+0000","last_undegraded":"2024-09-20T19:35:57.499175+0000","last_fullsized":"2024-09-20T19:35:57.499175+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'375","last_scrub_stamp":"2024-09-20T19:34:53.903004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.903004+0000","objects_scrubbed":2,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:17.869274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.069034204000000002,"stat_sum":{"num_bytes":2454020,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":6,"num_write":32,"num_write_kb":7400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.a","version":"523'35","reported_seq":1654,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.496075+0000","last_change":"2024-09-20T19:28:03.711457+0000","last_active":"2024-09-20T19:35:56.496075+0000","last_peered":"2024-09-20T19:35:56.496075+0000","last_clean":"2024-09-20T19:35:56.496075+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:35:56.496075+0000","last_undegraded":"2024-09-20T19:35:56.496075+0000","last_fullsized":"2024-09-20T19:35:56.496075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-20T19:28:03.711366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:28:03.711366+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:11.545395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1639,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488731+0000","last_change":"2024-09-20T19:34:47.295550+0000","last_active":"2024-09-20T19:35:56.488731+0000","last_peered":"2024-09-20T19:35:56.488731+0000","last_clean":"2024-09-20T19:35:56.488731+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:35:56.488731+0000","last_undegraded":"2024-09-20T19:35:56.488731+0000","last_fullsized":"2024-09-20T19:35:56.488731+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:47.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.295412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485960+0000","last_change":"2024-09-20T19:34:26.390149+0000","last_active":"2024-09-20T19:35:56.485960+0000","last_peered":"2024-09-20T19:35:56.485960+0000","last_clean":"2024-09-20T19:35:56.485960+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:35:56.485960+0000","last_undegraded":"2024-09-20T19:35:56.485960+0000","last_fullsized":"2024-09-20T19:35:56.485960+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.553131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.553131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:32.225480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1600,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488064+0000","last_change":"2024-09-20T19:33:43.369554+0000","last_active":"2024-09-20T19:35:56.488064+0000","last_peered":"2024-09-20T19:35:56.488064+0000","last_clean":"2024-09-20T19:35:56.488064+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:35:56.488064+0000","last_undegraded":"2024-09-20T19:35:56.488064+0000","last_fullsized":"2024-09-20T19:35:56.488064+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:15.777743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:15.777743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"848'1433","reported_seq":4965,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.502469+0000","last_change":"2024-09-20T19:35:56.502754+0000","last_active":"2024-09-20T19:35:56.502469+0000","last_peered":"2024-09-20T19:35:56.502469+0000","last_clean":"2024-09-20T19:35:56.502469+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:35:56.502469+0000","last_undegraded":"2024-09-20T19:35:56.502469+0000","last_fullsized":"2024-09-20T19:35:56.502469+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'1322","last_scrub_stamp":"2024-09-20T19:34:52.301450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:52.301450+0000","objects_scrubbed":6,"log_size":1433,"log_dups_size":0,"ondisk_log_size":1433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.017376926000000001,"stat_sum":{"num_bytes":26639517,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":159,"num_read_kb":117584,"num_write":120,"num_write_kb":53066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"4.b","version":"523'15","reported_seq":1429,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497161+0000","last_change":"2024-09-20T19:35:50.754361+0000","last_active":"2024-09-20T19:35:57.497161+0000","last_peered":"2024-09-20T19:35:57.497161+0000","last_clean":"2024-09-20T19:35:57.497161+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:35:57.497161+0000","last_undegraded":"2024-09-20T19:35:57.497161+0000","last_fullsized":"2024-09-20T19:35:57.497161+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:50.754272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:50.754272+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:03.897038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1646,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.479027+0000","last_change":"2024-09-20T19:34:26.406213+0000","last_active":"2024-09-20T19:35:56.479027+0000","last_peered":"2024-09-20T19:35:56.479027+0000","last_clean":"2024-09-20T19:35:56.479027+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:35:56.479027+0000","last_undegraded":"2024-09-20T19:35:56.479027+0000","last_fullsized":"2024-09-20T19:35:56.479027+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:31:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:32.324134+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1646,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486172+0000","last_change":"2024-09-20T19:34:44.295768+0000","last_active":"2024-09-20T19:35:56.486172+0000","last_peered":"2024-09-20T19:35:56.486172+0000","last_clean":"2024-09-20T19:35:56.486172+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:35:56.486172+0000","last_undegraded":"2024-09-20T19:35:56.486172+0000","last_fullsized":"2024-09-20T19:35:56.486172+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.295697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.295697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1430,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.499371+0000","last_change":"2024-09-20T19:35:51.726463+0000","last_active":"2024-09-20T19:35:57.499371+0000","last_peered":"2024-09-20T19:35:57.499371+0000","last_clean":"2024-09-20T19:35:57.499371+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:35:57.499371+0000","last_undegraded":"2024-09-20T19:35:57.499371+0000","last_fullsized":"2024-09-20T19:35:57.499371+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:35:51.726405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:51.726405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:02.238429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"848'588","reported_seq":4136,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.512243+0000","last_change":"2024-09-20T19:35:56.512243+0000","last_active":"2024-09-20T19:35:56.512243+0000","last_peered":"2024-09-20T19:35:56.512243+0000","last_clean":"2024-09-20T19:35:56.512243+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:35:56.512243+0000","last_undegraded":"2024-09-20T19:35:56.512243+0000","last_fullsized":"2024-09-20T19:35:56.512243+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'582","last_scrub_stamp":"2024-09-20T19:35:52.105089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:52.105089+0000","objects_scrubbed":7,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:02.598597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0333221,"stat_sum":{"num_bytes":12286709,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":52,"num_read_kb":46958,"num_write":24,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"4.8","version":"523'27","reported_seq":1665,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.505247+0000","last_change":"2024-09-20T19:34:47.083599+0000","last_active":"2024-09-20T19:35:56.505247+0000","last_peered":"2024-09-20T19:35:56.505247+0000","last_clean":"2024-09-20T19:35:56.505247+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:35:56.505247+0000","last_undegraded":"2024-09-20T19:35:56.505247+0000","last_fullsized":"2024-09-20T19:35:56.505247+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:34:47.083533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:47.083533+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.559644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":204,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.486593+0000","last_change":"2024-09-20T19:35:33.276159+0000","last_active":"2024-09-20T19:35:56.486593+0000","last_peered":"2024-09-20T19:35:56.486593+0000","last_clean":"2024-09-20T19:35:56.486593+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:35:56.486593+0000","last_undegraded":"2024-09-20T19:35:56.486593+0000","last_fullsized":"2024-09-20T19:35:56.486593+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:35:33.276021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:33.276021+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:47.350803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":1840,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487972+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:35:56.487972+0000","last_peered":"2024-09-20T19:35:56.487972+0000","last_clean":"2024-09-20T19:35:56.487972+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:35:56.487972+0000","last_undegraded":"2024-09-20T19:35:56.487972+0000","last_fullsized":"2024-09-20T19:35:56.487972+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1617,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.507123+0000","last_change":"2024-09-20T19:34:53.009990+0000","last_active":"2024-09-20T19:35:56.507123+0000","last_peered":"2024-09-20T19:35:56.507123+0000","last_clean":"2024-09-20T19:35:56.507123+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:35:56.507123+0000","last_undegraded":"2024-09-20T19:35:56.507123+0000","last_fullsized":"2024-09-20T19:35:56.507123+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:53.009934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:53.009934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:06.466332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":3653,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487046+0000","last_change":"2024-09-20T19:35:56.487046+0000","last_active":"2024-09-20T19:35:56.487046+0000","last_peered":"2024-09-20T19:35:56.487046+0000","last_clean":"2024-09-20T19:35:56.487046+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:35:56.487046+0000","last_undegraded":"2024-09-20T19:35:56.487046+0000","last_fullsized":"2024-09-20T19:35:56.487046+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:35:07.477259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:07.477259+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.926817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014831530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"4.9","version":"523'18","reported_seq":1496,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.484937+0000","last_change":"2024-09-20T19:33:43.369566+0000","last_active":"2024-09-20T19:35:58.484937+0000","last_peered":"2024-09-20T19:35:58.484937+0000","last_clean":"2024-09-20T19:35:58.484937+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:35:58.484937+0000","last_undegraded":"2024-09-20T19:35:58.484937+0000","last_fullsized":"2024-09-20T19:35:58.484937+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:32:26.295666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.295666+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:30.693243+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1492,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.487990+0000","last_change":"2024-09-20T19:35:54.874567+0000","last_active":"2024-09-20T19:35:58.487990+0000","last_peered":"2024-09-20T19:35:58.487990+0000","last_clean":"2024-09-20T19:35:58.487990+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:35:58.487990+0000","last_undegraded":"2024-09-20T19:35:58.487990+0000","last_fullsized":"2024-09-20T19:35:58.487990+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:35:54.874421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:54.874421+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:56.432880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1624,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485758+0000","last_change":"2024-09-20T19:33:06.111877+0000","last_active":"2024-09-20T19:35:56.485758+0000","last_peered":"2024-09-20T19:35:56.485758+0000","last_clean":"2024-09-20T19:35:56.485758+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:35:56.485758+0000","last_undegraded":"2024-09-20T19:35:56.485758+0000","last_fullsized":"2024-09-20T19:35:56.485758+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:06.111753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:33:06.111753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:19.369782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1104,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.489209+0000","last_change":"2024-09-20T19:33:43.447359+0000","last_active":"2024-09-20T19:35:56.489209+0000","last_peered":"2024-09-20T19:35:56.489209+0000","last_clean":"2024-09-20T19:35:56.489209+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:35:56.489209+0000","last_undegraded":"2024-09-20T19:35:56.489209+0000","last_fullsized":"2024-09-20T19:35:56.489209+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:11.827920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:11.827920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"846'1210","reported_seq":5129,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.516628+0000","last_change":"2024-09-20T19:35:56.516748+0000","last_active":"2024-09-20T19:35:56.516628+0000","last_peered":"2024-09-20T19:35:56.516628+0000","last_clean":"2024-09-20T19:35:56.516628+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:35:56.516628+0000","last_undegraded":"2024-09-20T19:35:56.516628+0000","last_fullsized":"2024-09-20T19:35:56.516628+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'1072","last_scrub_stamp":"2024-09-20T19:34:09.034773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:09.034773+0000","objects_scrubbed":12,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.022529304999999999,"stat_sum":{"num_bytes":18576860,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":170,"num_read_kb":118271,"num_write":150,"num_write_kb":54700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.16","version":"523'15","reported_seq":1423,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497938+0000","last_change":"2024-09-20T19:33:43.390311+0000","last_active":"2024-09-20T19:35:57.497938+0000","last_peered":"2024-09-20T19:35:57.497938+0000","last_clean":"2024-09-20T19:35:57.497938+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:35:57.497938+0000","last_undegraded":"2024-09-20T19:35:57.497938+0000","last_fullsized":"2024-09-20T19:35:57.497938+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:32:16.493879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.493879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1655,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485960+0000","last_change":"2024-09-20T19:34:54.365631+0000","last_active":"2024-09-20T19:35:56.485960+0000","last_peered":"2024-09-20T19:35:56.485960+0000","last_clean":"2024-09-20T19:35:56.485960+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:35:56.485960+0000","last_undegraded":"2024-09-20T19:35:56.485960+0000","last_fullsized":"2024-09-20T19:35:56.485960+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.365568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.365568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1099,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.477647+0000","last_change":"2024-09-20T19:34:10.220726+0000","last_active":"2024-09-20T19:35:56.477647+0000","last_peered":"2024-09-20T19:35:56.477647+0000","last_clean":"2024-09-20T19:35:56.477647+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:35:56.477647+0000","last_undegraded":"2024-09-20T19:35:56.477647+0000","last_fullsized":"2024-09-20T19:35:56.477647+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:10.220603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:10.220603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1619,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488296+0000","last_change":"2024-09-20T19:35:44.422895+0000","last_active":"2024-09-20T19:35:56.488296+0000","last_peered":"2024-09-20T19:35:56.488296+0000","last_clean":"2024-09-20T19:35:56.488296+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:35:56.488296+0000","last_undegraded":"2024-09-20T19:35:56.488296+0000","last_fullsized":"2024-09-20T19:35:56.488296+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:44.422851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:44.422851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:53.014994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"848'1268","reported_seq":4879,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.523749+0000","last_change":"2024-09-20T19:35:56.524095+0000","last_active":"2024-09-20T19:35:56.523749+0000","last_peered":"2024-09-20T19:35:56.523749+0000","last_clean":"2024-09-20T19:35:56.523749+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:35:56.523749+0000","last_undegraded":"2024-09-20T19:35:56.523749+0000","last_fullsized":"2024-09-20T19:35:56.523749+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1069","last_scrub_stamp":"2024-09-20T19:34:11.169527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.169527+0000","objects_scrubbed":7,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035993241000000002,"stat_sum":{"num_bytes":34789855,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":207,"num_read_kb":134078,"num_write":139,"num_write_kb":58795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"2.11","version":"0'0","reported_seq":1621,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.496396+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:35:56.496396+0000","last_peered":"2024-09-20T19:35:56.496396+0000","last_clean":"2024-09-20T19:35:56.496396+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:35:56.496396+0000","last_undegraded":"2024-09-20T19:35:56.496396+0000","last_fullsized":"2024-09-20T19:35:56.496396+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.149622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"523'12","reported_seq":1272,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.502174+0000","last_change":"2024-09-20T19:33:43.464473+0000","last_active":"2024-09-20T19:35:56.502174+0000","last_peered":"2024-09-20T19:35:56.502174+0000","last_clean":"2024-09-20T19:35:56.502174+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:35:56.502174+0000","last_undegraded":"2024-09-20T19:35:56.502174+0000","last_fullsized":"2024-09-20T19:35:56.502174+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:31:41.454765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:41.454765+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1635,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.511749+0000","last_change":"2024-09-20T19:35:54.758684+0000","last_active":"2024-09-20T19:35:56.511749+0000","last_peered":"2024-09-20T19:35:56.511749+0000","last_clean":"2024-09-20T19:35:56.511749+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:35:56.511749+0000","last_undegraded":"2024-09-20T19:35:56.511749+0000","last_fullsized":"2024-09-20T19:35:56.511749+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:54.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:54.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:20.244421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1600,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.493122+0000","last_change":"2024-09-20T19:34:44.066655+0000","last_active":"2024-09-20T19:35:56.493122+0000","last_peered":"2024-09-20T19:35:56.493122+0000","last_clean":"2024-09-20T19:35:56.493122+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:35:56.493122+0000","last_undegraded":"2024-09-20T19:35:56.493122+0000","last_fullsized":"2024-09-20T19:35:56.493122+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:00.702253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"848'703","reported_seq":4163,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.522735+0000","last_change":"2024-09-20T19:35:56.508283+0000","last_active":"2024-09-20T19:35:56.522735+0000","last_peered":"2024-09-20T19:35:56.522735+0000","last_clean":"2024-09-20T19:35:56.522735+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:35:56.522735+0000","last_undegraded":"2024-09-20T19:35:56.522735+0000","last_fullsized":"2024-09-20T19:35:56.522735+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"616'500","last_scrub_stamp":"2024-09-20T19:32:10.788448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:32:10.788448+0000","objects_scrubbed":6,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.1403049839999999,"stat_sum":{"num_bytes":24965612,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":31635,"num_write":57,"num_write_kb":26303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"523'20","reported_seq":1289,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.476416+0000","last_change":"2024-09-20T19:33:43.463851+0000","last_active":"2024-09-20T19:35:56.476416+0000","last_peered":"2024-09-20T19:35:56.476416+0000","last_clean":"2024-09-20T19:35:56.476416+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:35:56.476416+0000","last_undegraded":"2024-09-20T19:35:56.476416+0000","last_fullsized":"2024-09-20T19:35:56.476416+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:31:40.419578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:31:40.419578+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:28.333955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1635,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.508163+0000","last_change":"2024-09-20T19:34:11.058115+0000","last_active":"2024-09-20T19:35:56.508163+0000","last_peered":"2024-09-20T19:35:56.508163+0000","last_clean":"2024-09-20T19:35:56.508163+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:35:56.508163+0000","last_undegraded":"2024-09-20T19:35:56.508163+0000","last_fullsized":"2024-09-20T19:35:56.508163+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:11.058052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.058052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":1847,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.510824+0000","last_change":"2024-09-20T19:35:39.656351+0000","last_active":"2024-09-20T19:35:56.510824+0000","last_peered":"2024-09-20T19:35:56.510824+0000","last_clean":"2024-09-20T19:35:56.510824+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:35:56.510824+0000","last_undegraded":"2024-09-20T19:35:56.510824+0000","last_fullsized":"2024-09-20T19:35:56.510824+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:35:39.656283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:39.656283+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:58.730668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1403,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.500695+0000","last_change":"2024-09-20T19:35:56.751414+0000","last_active":"2024-09-20T19:35:57.500695+0000","last_peered":"2024-09-20T19:35:57.500695+0000","last_clean":"2024-09-20T19:35:57.500695+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:35:57.500695+0000","last_undegraded":"2024-09-20T19:35:57.500695+0000","last_fullsized":"2024-09-20T19:35:57.500695+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:56.751367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:56.751367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:04.985651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"850'577","reported_seq":3884,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.510983+0000","last_change":"2024-09-20T19:35:58.512161+0000","last_active":"2024-09-20T19:35:58.510983+0000","last_peered":"2024-09-20T19:35:58.510983+0000","last_clean":"2024-09-20T19:35:58.510983+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:35:58.510983+0000","last_undegraded":"2024-09-20T19:35:58.510983+0000","last_fullsized":"2024-09-20T19:35:58.510983+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'490","last_scrub_stamp":"2024-09-20T19:34:46.528129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:46.528129+0000","objects_scrubbed":6,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:15.501726+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.024265304000000001,"stat_sum":{"num_bytes":18601364,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":132,"num_read_kb":94556,"num_write":82,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"129","length":"1"},{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"},{"start":"159","length":"1"}]},{"pgid":"4.15","version":"523'15","reported_seq":1673,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.510919+0000","last_change":"2024-09-20T19:34:08.010996+0000","last_active":"2024-09-20T19:35:56.510919+0000","last_peered":"2024-09-20T19:35:56.510919+0000","last_clean":"2024-09-20T19:35:56.510919+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:35:56.510919+0000","last_undegraded":"2024-09-20T19:35:56.510919+0000","last_fullsized":"2024-09-20T19:35:56.510919+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:34:08.010906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.010906+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.788559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1368,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.497406+0000","last_change":"2024-09-20T19:33:43.369959+0000","last_active":"2024-09-20T19:35:57.497406+0000","last_peered":"2024-09-20T19:35:57.497406+0000","last_clean":"2024-09-20T19:35:57.497406+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:35:57.497406+0000","last_undegraded":"2024-09-20T19:35:57.497406+0000","last_fullsized":"2024-09-20T19:35:57.497406+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:26.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:32:26.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":297,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.497115+0000","last_change":"2024-09-20T19:35:52.132424+0000","last_active":"2024-09-20T19:35:56.497115+0000","last_peered":"2024-09-20T19:35:56.497115+0000","last_clean":"2024-09-20T19:35:56.497115+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:35:56.497115+0000","last_undegraded":"2024-09-20T19:35:56.497115+0000","last_fullsized":"2024-09-20T19:35:56.497115+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'17","last_scrub_stamp":"2024-09-20T19:35:52.132287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:52.132287+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:09.618654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1454,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.487873+0000","last_change":"2024-09-20T19:34:55.537273+0000","last_active":"2024-09-20T19:35:58.487873+0000","last_peered":"2024-09-20T19:35:58.487873+0000","last_clean":"2024-09-20T19:35:58.487873+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:35:58.487873+0000","last_undegraded":"2024-09-20T19:35:58.487873+0000","last_fullsized":"2024-09-20T19:35:58.487873+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:55.537217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:55.537217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:13.844779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"848'551","reported_seq":4122,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.516744+0000","last_change":"2024-09-20T19:35:56.516744+0000","last_active":"2024-09-20T19:35:56.516744+0000","last_peered":"2024-09-20T19:35:56.516744+0000","last_clean":"2024-09-20T19:35:56.516744+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:35:56.516744+0000","last_undegraded":"2024-09-20T19:35:56.516744+0000","last_fullsized":"2024-09-20T19:35:56.516744+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"517'448","last_scrub_stamp":"2024-09-20T19:30:30.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:30:30.252579+0000","objects_scrubbed":2,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.090366399,"stat_sum":{"num_bytes":18152613,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":48701,"num_write":33,"num_write_kb":15091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"4.12","version":"523'15","reported_seq":1651,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478194+0000","last_change":"2024-09-20T19:35:49.126133+0000","last_active":"2024-09-20T19:35:56.478194+0000","last_peered":"2024-09-20T19:35:56.478194+0000","last_clean":"2024-09-20T19:35:56.478194+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:35:56.478194+0000","last_undegraded":"2024-09-20T19:35:56.478194+0000","last_fullsized":"2024-09-20T19:35:56.478194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:49.126032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.126032+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:00.260372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1620,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.498252+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:35:56.498252+0000","last_peered":"2024-09-20T19:35:56.498252+0000","last_clean":"2024-09-20T19:35:56.498252+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:35:56.498252+0000","last_undegraded":"2024-09-20T19:35:56.498252+0000","last_fullsized":"2024-09-20T19:35:56.498252+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:21.355670+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1123,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.496151+0000","last_change":"2024-09-20T19:33:43.447329+0000","last_active":"2024-09-20T19:35:57.496151+0000","last_peered":"2024-09-20T19:35:57.496151+0000","last_clean":"2024-09-20T19:35:57.496151+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:35:57.496151+0000","last_undegraded":"2024-09-20T19:35:57.496151+0000","last_fullsized":"2024-09-20T19:35:57.496151+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:35.565324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:31:35.565324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:15.558468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1378,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.495917+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:35:57.495917+0000","last_peered":"2024-09-20T19:35:57.495917+0000","last_clean":"2024-09-20T19:35:57.495917+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:35:57.495917+0000","last_undegraded":"2024-09-20T19:35:57.495917+0000","last_fullsized":"2024-09-20T19:35:57.495917+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"844'726","reported_seq":4247,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.485626+0000","last_change":"2024-09-20T19:35:56.486391+0000","last_active":"2024-09-20T19:35:56.485626+0000","last_peered":"2024-09-20T19:35:56.485626+0000","last_clean":"2024-09-20T19:35:56.485626+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:35:56.485626+0000","last_undegraded":"2024-09-20T19:35:56.485626+0000","last_fullsized":"2024-09-20T19:35:56.485626+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"744'658","last_scrub_stamp":"2024-09-20T19:34:13.122172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:13.122172+0000","objects_scrubbed":3,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010712930000000001,"stat_sum":{"num_bytes":9527172,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":44348,"num_write":48,"num_write_kb":13747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"},{"start":"156","length":"1"}]},{"pgid":"4.13","version":"523'19","reported_seq":1461,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485605+0000","last_change":"2024-09-20T19:34:41.550328+0000","last_active":"2024-09-20T19:35:58.485605+0000","last_peered":"2024-09-20T19:35:58.485605+0000","last_clean":"2024-09-20T19:35:58.485605+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:35:58.485605+0000","last_undegraded":"2024-09-20T19:35:58.485605+0000","last_fullsized":"2024-09-20T19:35:58.485605+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:34:41.550266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.550266+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":1641,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.479019+0000","last_change":"2024-09-20T19:35:37.372791+0000","last_active":"2024-09-20T19:35:56.479019+0000","last_peered":"2024-09-20T19:35:56.479019+0000","last_clean":"2024-09-20T19:35:56.479019+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:35:56.479019+0000","last_undegraded":"2024-09-20T19:35:56.479019+0000","last_fullsized":"2024-09-20T19:35:56.479019+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:35:37.372647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:37.372647+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:00.338644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1635,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478018+0000","last_change":"2024-09-20T19:34:59.381856+0000","last_active":"2024-09-20T19:35:56.478018+0000","last_peered":"2024-09-20T19:35:56.478018+0000","last_clean":"2024-09-20T19:35:56.478018+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:35:56.478018+0000","last_undegraded":"2024-09-20T19:35:56.478018+0000","last_fullsized":"2024-09-20T19:35:56.478018+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:59.381728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:59.381728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:24.871997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1127,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.499096+0000","last_change":"2024-09-20T19:34:54.942515+0000","last_active":"2024-09-20T19:35:57.499096+0000","last_peered":"2024-09-20T19:35:57.499096+0000","last_clean":"2024-09-20T19:35:57.499096+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:35:57.499096+0000","last_undegraded":"2024-09-20T19:35:57.499096+0000","last_fullsized":"2024-09-20T19:35:57.499096+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:54.942431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:54.942431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:12.629890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"845'492","reported_seq":4045,"reported_epoch":849,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:57.496454+0000","last_change":"2024-09-20T19:35:56.510441+0000","last_active":"2024-09-20T19:35:57.496454+0000","last_peered":"2024-09-20T19:35:57.496454+0000","last_clean":"2024-09-20T19:35:57.496454+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:35:57.496454+0000","last_undegraded":"2024-09-20T19:35:57.496454+0000","last_fullsized":"2024-09-20T19:35:57.496454+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"777'478","last_scrub_stamp":"2024-09-20T19:34:51.901991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.901991+0000","objects_scrubbed":2,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.119858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.01910017,"stat_sum":{"num_bytes":6980097,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":7,"num_write":42,"num_write_kb":11327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"4.10","version":"523'8","reported_seq":1281,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.475699+0000","last_change":"2024-09-20T19:34:43.272725+0000","last_active":"2024-09-20T19:35:56.475699+0000","last_peered":"2024-09-20T19:35:56.475699+0000","last_clean":"2024-09-20T19:35:56.475699+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:35:56.475699+0000","last_undegraded":"2024-09-20T19:35:56.475699+0000","last_fullsized":"2024-09-20T19:35:56.475699+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'8","last_scrub_stamp":"2024-09-20T19:34:43.272551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.272551+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1623,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.511960+0000","last_change":"2024-09-20T19:35:43.700067+0000","last_active":"2024-09-20T19:35:56.511960+0000","last_peered":"2024-09-20T19:35:56.511960+0000","last_clean":"2024-09-20T19:35:56.511960+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:35:56.511960+0000","last_undegraded":"2024-09-20T19:35:56.511960+0000","last_fullsized":"2024-09-20T19:35:56.511960+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:43.700002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:43.700002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:09.757070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1613,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.495956+0000","last_change":"2024-09-20T19:35:03.419683+0000","last_active":"2024-09-20T19:35:56.495956+0000","last_peered":"2024-09-20T19:35:56.495956+0000","last_clean":"2024-09-20T19:35:56.495956+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:35:56.495956+0000","last_undegraded":"2024-09-20T19:35:56.495956+0000","last_fullsized":"2024-09-20T19:35:56.495956+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:03.419628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.419628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:25.847292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1592,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488873+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:35:56.488873+0000","last_peered":"2024-09-20T19:35:56.488873+0000","last_clean":"2024-09-20T19:35:56.488873+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:35:56.488873+0000","last_undegraded":"2024-09-20T19:35:56.488873+0000","last_fullsized":"2024-09-20T19:35:56.488873+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":3602,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.512717+0000","last_change":"2024-09-20T19:35:56.512853+0000","last_active":"2024-09-20T19:35:56.512717+0000","last_peered":"2024-09-20T19:35:56.512717+0000","last_clean":"2024-09-20T19:35:56.512717+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:35:56.512717+0000","last_undegraded":"2024-09-20T19:35:56.512717+0000","last_fullsized":"2024-09-20T19:35:56.512717+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:35:45.175816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:45.175816+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:06.900784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017057273000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"4.11","version":"523'19","reported_seq":488,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.488415+0000","last_change":"2024-09-20T19:33:43.373453+0000","last_active":"2024-09-20T19:35:56.488415+0000","last_peered":"2024-09-20T19:35:56.488415+0000","last_clean":"2024-09-20T19:35:56.488415+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:35:56.488415+0000","last_undegraded":"2024-09-20T19:35:56.488415+0000","last_fullsized":"2024-09-20T19:35:56.488415+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:32:16.785134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:32:16.785134+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1610,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.501162+0000","last_change":"2024-09-20T19:28:06.699682+0000","last_active":"2024-09-20T19:35:56.501162+0000","last_peered":"2024-09-20T19:35:56.501162+0000","last_clean":"2024-09-20T19:35:56.501162+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:35:56.501162+0000","last_undegraded":"2024-09-20T19:35:56.501162+0000","last_fullsized":"2024-09-20T19:35:56.501162+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:28:06.699550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:28:06.699550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.449775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1612,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487188+0000","last_change":"2024-09-20T19:32:27.828625+0000","last_active":"2024-09-20T19:35:56.487188+0000","last_peered":"2024-09-20T19:35:56.487188+0000","last_clean":"2024-09-20T19:35:56.487188+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:35:56.487188+0000","last_undegraded":"2024-09-20T19:35:56.487188+0000","last_fullsized":"2024-09-20T19:35:56.487188+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:27.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:27.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1135,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.496793+0000","last_change":"2024-09-20T19:33:43.367812+0000","last_active":"2024-09-20T19:35:57.496793+0000","last_peered":"2024-09-20T19:35:57.496793+0000","last_clean":"2024-09-20T19:35:57.496793+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:35:57.496793+0000","last_undegraded":"2024-09-20T19:35:57.496793+0000","last_fullsized":"2024-09-20T19:35:57.496793+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.767876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.767876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"848'666","reported_seq":4221,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.513007+0000","last_change":"2024-09-20T19:35:56.513007+0000","last_active":"2024-09-20T19:35:56.513007+0000","last_peered":"2024-09-20T19:35:56.513007+0000","last_clean":"2024-09-20T19:35:56.513007+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:35:56.513007+0000","last_undegraded":"2024-09-20T19:35:56.513007+0000","last_fullsized":"2024-09-20T19:35:56.513007+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"721'544","last_scrub_stamp":"2024-09-20T19:34:00.387013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:00.387013+0000","objects_scrubbed":3,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.033448023,"stat_sum":{"num_bytes":17549184,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":45775,"num_write":39,"num_write_kb":20040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"}]},{"pgid":"6.1c","version":"77'1","reported_seq":1599,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.491572+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:35:56.491572+0000","last_peered":"2024-09-20T19:35:56.491572+0000","last_clean":"2024-09-20T19:35:56.491572+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:35:56.491572+0000","last_undegraded":"2024-09-20T19:35:56.491572+0000","last_fullsized":"2024-09-20T19:35:56.491572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1636,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.505526+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:35:56.505526+0000","last_peered":"2024-09-20T19:35:56.505526+0000","last_clean":"2024-09-20T19:35:56.505526+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:35:56.505526+0000","last_undegraded":"2024-09-20T19:35:56.505526+0000","last_fullsized":"2024-09-20T19:35:56.505526+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:03.363084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"523'20","reported_seq":1422,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.498448+0000","last_change":"2024-09-20T19:35:05.740151+0000","last_active":"2024-09-20T19:35:57.498448+0000","last_peered":"2024-09-20T19:35:57.498448+0000","last_clean":"2024-09-20T19:35:57.498448+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:35:57.498448+0000","last_undegraded":"2024-09-20T19:35:57.498448+0000","last_fullsized":"2024-09-20T19:35:57.498448+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:35:05.739968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:05.739968+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:18.343465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":1844,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.500715+0000","last_change":"2024-09-20T19:30:31.369879+0000","last_active":"2024-09-20T19:35:56.500715+0000","last_peered":"2024-09-20T19:35:56.500715+0000","last_clean":"2024-09-20T19:35:56.500715+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:35:56.500715+0000","last_undegraded":"2024-09-20T19:35:56.500715+0000","last_fullsized":"2024-09-20T19:35:56.500715+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:13.657532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:26:13.657532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.435375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"848'927","reported_seq":4677,"reported_epoch":848,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:35:56.648331+0000","last_change":"2024-09-20T19:35:56.520210+0000","last_active":"2024-09-20T19:35:56.648331+0000","last_peered":"2024-09-20T19:35:56.648331+0000","last_clean":"2024-09-20T19:35:56.648331+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:35:56.648331+0000","last_undegraded":"2024-09-20T19:35:56.648331+0000","last_fullsized":"2024-09-20T19:35:56.648331+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"832'916","last_scrub_stamp":"2024-09-20T19:35:44.736999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:44.736999+0000","objects_scrubbed":7,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:50.375385+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":1.034948344,"stat_sum":{"num_bytes":15556840,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":66,"num_read_kb":48104,"num_write":34,"num_write_kb":8975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"6.1d","version":"0'0","reported_seq":1613,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478564+0000","last_change":"2024-09-20T19:31:22.298803+0000","last_active":"2024-09-20T19:35:56.478564+0000","last_peered":"2024-09-20T19:35:56.478564+0000","last_clean":"2024-09-20T19:35:56.478564+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:35:56.478564+0000","last_undegraded":"2024-09-20T19:35:56.478564+0000","last_fullsized":"2024-09-20T19:35:56.478564+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:22.298730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:22.298730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:20.466563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":450,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.496290+0000","last_change":"2024-09-20T19:34:43.489833+0000","last_active":"2024-09-20T19:35:56.496290+0000","last_peered":"2024-09-20T19:35:56.496290+0000","last_clean":"2024-09-20T19:35:56.496290+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:35:56.496290+0000","last_undegraded":"2024-09-20T19:35:56.496290+0000","last_fullsized":"2024-09-20T19:35:56.496290+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:34:43.489670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:43.489670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:08.767313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"523'19","reported_seq":1660,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.500893+0000","last_change":"2024-09-20T19:28:23.498718+0000","last_active":"2024-09-20T19:35:56.500893+0000","last_peered":"2024-09-20T19:35:56.500893+0000","last_clean":"2024-09-20T19:35:56.500893+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:35:56.500893+0000","last_undegraded":"2024-09-20T19:35:56.500893+0000","last_fullsized":"2024-09-20T19:35:56.500893+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:26:23.786947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:26:23.786947+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:09.474818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1415,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.500482+0000","last_change":"2024-09-20T19:35:55.754020+0000","last_active":"2024-09-20T19:35:57.500482+0000","last_peered":"2024-09-20T19:35:57.500482+0000","last_clean":"2024-09-20T19:35:57.500482+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:35:57.500482+0000","last_undegraded":"2024-09-20T19:35:57.500482+0000","last_fullsized":"2024-09-20T19:35:57.500482+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:55.753897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:55.753897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:57.648058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"846'651","reported_seq":4000,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.514172+0000","last_change":"2024-09-20T19:35:56.514290+0000","last_active":"2024-09-20T19:35:56.514172+0000","last_peered":"2024-09-20T19:35:56.514172+0000","last_clean":"2024-09-20T19:35:56.514172+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:35:56.514172+0000","last_undegraded":"2024-09-20T19:35:56.514172+0000","last_fullsized":"2024-09-20T19:35:56.514172+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'580","last_scrub_stamp":"2024-09-20T19:34:41.516352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:41.516352+0000","objects_scrubbed":4,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:05.121106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014513391,"stat_sum":{"num_bytes":16123119,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":23,"num_read_kb":24721,"num_write":14,"num_write_kb":7581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"140","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":1435,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.486596+0000","last_change":"2024-09-20T19:31:33.781008+0000","last_active":"2024-09-20T19:35:58.486596+0000","last_peered":"2024-09-20T19:35:58.486596+0000","last_clean":"2024-09-20T19:35:58.486596+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:35:58.486596+0000","last_undegraded":"2024-09-20T19:35:58.486596+0000","last_fullsized":"2024-09-20T19:35:58.486596+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:31:33.780948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:31:33.780948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:26.698833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1629,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.496377+0000","last_change":"2024-09-20T19:35:47.234215+0000","last_active":"2024-09-20T19:35:56.496377+0000","last_peered":"2024-09-20T19:35:56.496377+0000","last_clean":"2024-09-20T19:35:56.496377+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:35:56.496377+0000","last_undegraded":"2024-09-20T19:35:56.496377+0000","last_fullsized":"2024-09-20T19:35:56.496377+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.234087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.234087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:49.482843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"523'27","reported_seq":216,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.487718+0000","last_change":"2024-09-20T19:35:54.533638+0000","last_active":"2024-09-20T19:35:56.487718+0000","last_peered":"2024-09-20T19:35:56.487718+0000","last_clean":"2024-09-20T19:35:56.487718+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:35:56.487718+0000","last_undegraded":"2024-09-20T19:35:56.487718+0000","last_fullsized":"2024-09-20T19:35:56.487718+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:35:54.533544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:54.533544+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:24.042351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1632,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478835+0000","last_change":"2024-09-20T19:35:50.122718+0000","last_active":"2024-09-20T19:35:56.478835+0000","last_peered":"2024-09-20T19:35:56.478835+0000","last_clean":"2024-09-20T19:35:56.478835+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:35:56.478835+0000","last_undegraded":"2024-09-20T19:35:56.478835+0000","last_fullsized":"2024-09-20T19:35:56.478835+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:50.122667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:50.122667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:37:19.928797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":3805,"reported_epoch":848,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:35:56.512342+0000","last_change":"2024-09-20T19:35:56.512387+0000","last_active":"2024-09-20T19:35:56.512342+0000","last_peered":"2024-09-20T19:35:56.512342+0000","last_clean":"2024-09-20T19:35:56.512342+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:35:56.512342+0000","last_undegraded":"2024-09-20T19:35:56.512342+0000","last_fullsized":"2024-09-20T19:35:56.512342+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:35:51.104320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:51.104320+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:57.583114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035710069999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"140","length":"1"},{"start":"14c","length":"1"},{"start":"150","length":"1"}]},{"pgid":"6.1f","version":"0'0","reported_seq":260,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-20T19:35:57.495309+0000","last_change":"2024-09-20T19:33:58.560877+0000","last_active":"2024-09-20T19:35:57.495309+0000","last_peered":"2024-09-20T19:35:57.495309+0000","last_clean":"2024-09-20T19:35:57.495309+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:35:57.495309+0000","last_undegraded":"2024-09-20T19:35:57.495309+0000","last_fullsized":"2024-09-20T19:35:57.495309+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:58.560821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:58.560821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1105,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.490149+0000","last_change":"2024-09-20T19:34:51.148618+0000","last_active":"2024-09-20T19:35:56.490149+0000","last_peered":"2024-09-20T19:35:56.490149+0000","last_clean":"2024-09-20T19:35:56.490149+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:35:56.490149+0000","last_undegraded":"2024-09-20T19:35:56.490149+0000","last_fullsized":"2024-09-20T19:35:56.490149+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.148494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.148494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:16.914652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"523'18","reported_seq":501,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-09-20T19:35:56.478923+0000","last_change":"2024-09-20T19:35:46.163489+0000","last_active":"2024-09-20T19:35:56.478923+0000","last_peered":"2024-09-20T19:35:56.478923+0000","last_clean":"2024-09-20T19:35:56.478923+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:35:56.478923+0000","last_undegraded":"2024-09-20T19:35:56.478923+0000","last_fullsized":"2024-09-20T19:35:56.478923+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:35:46.163297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:46.163297+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:36:56.125733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1449,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-20T19:35:58.485710+0000","last_change":"2024-09-20T19:34:44.512596+0000","last_active":"2024-09-20T19:35:58.485710+0000","last_peered":"2024-09-20T19:35:58.485710+0000","last_clean":"2024-09-20T19:35:58.485710+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:35:58.485710+0000","last_undegraded":"2024-09-20T19:35:58.485710+0000","last_fullsized":"2024-09-20T19:35:58.485710+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.512457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.512457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":397604158,"num_objects":190,"num_object_clones":142,"num_object_copies":570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":13,"num_read":1817,"num_read_kb":1330039,"num_write":1318,"num_write_kb":495962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":929677312,"data_stored":1324275533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":23424,"ondisk_log_size":23424,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1103,"num_read_kb":860,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1051,"num_read_kb":1046,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907167,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213492,"kb_used_data":160096,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93531660,"statfs":{"total":95995035648,"available":95776419840,"internally_reserved":0,"allocated":163938304,"data_stored":218141312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234218,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":151252,"kb_used_data":106168,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93593900,"statfs":{"total":95995035648,"available":95840153600,"internally_reserved":0,"allocated":108716032,"data_stored":158732152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561268,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165144,"kb_used_data":107588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93580008,"statfs":{"total":95995035648,"available":95825928192,"internally_reserved":0,"allocated":110170112,"data_stored":160810814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928579,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120268,"kb_used_data":92404,"kb_used_omap":35,"kb_used_meta":27804,"kb_avail":93624884,"statfs":{"total":95995035648,"available":95871881216,"internally_reserved":0,"allocated":94621696,"data_stored":127999052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":28472241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378904,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189128,"kb_used_data":144048,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93556024,"statfs":{"total":95995035648,"available":95801368576,"internally_reserved":0,"allocated":147505152,"data_stored":193243179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575140,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169124,"kb_used_data":115720,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93576028,"statfs":{"total":95995035648,"available":95821852672,"internally_reserved":0,"allocated":118497280,"data_stored":185700509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648071,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169828,"kb_used_data":132552,"kb_used_omap":49,"kb_used_meta":37198,"kb_avail":93575324,"statfs":{"total":95995035648,"available":95821131776,"internally_reserved":0,"allocated":135733248,"data_stored":197063192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":38091305},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134774784,"data_stored":196519875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115675136,"data_stored":183450428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146288640,"data_stored":192444612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":93659136,"data_stored":127449163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":108638208,"data_stored":159684467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":105730048,"data_stored":155000357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":162672640,"data_stored":217278300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:36:02.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pgp_num 11 2024-09-20T19:36:02.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T19:36:02.546 INFO:tasks.rados.rados.0.smithi042.stdout:3074: done (1 left) 2024-09-20T19:36:02.546 INFO:tasks.rados.rados.0.smithi042.stdout:3075: snap_remove snap 333 2024-09-20T19:36:02.618 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:02.639 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:02.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:02.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:02 smithi162 bash[24557]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:02 smithi162 bash[24557]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:02 smithi162 bash[24557]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:02 smithi162 bash[24557]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:02.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[32075]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[32075]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[32075]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[32075]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[22706]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[22706]: audit 2024-09-20T19:36:00.768342+0000 mgr.y (mgr.24491) 679 : audit [DBG] from='client.25889 v1:172.21.15.42:0/3583424726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:36:02.883 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[22706]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.884 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:02 smithi042 bash[22706]: cluster 2024-09-20T19:36:01.534939+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2024-09-20T19:36:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:02.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:03.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:03.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:03.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:03.266 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:03.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:03.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:03.405 INFO:tasks.rados.rados.0.smithi042.stdout:3075: done (1 left) 2024-09-20T19:36:03.405 INFO:tasks.rados.rados.0.smithi042.stdout:3076: read oid 44 snap -1 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:3076: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:3077: delete oid 19 current snap is 353 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:3073: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:3073: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:3073: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:36:03.406 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 937 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:36:03.426 INFO:tasks.rados.rados.0.smithi042.stdout:3076: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:03.426 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:03.459 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3073: done (2 left) 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3076: done (1 left) 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3077: done (0 left) 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3078: read oid 18 snap -1 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3078: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3079: read oid 22 snap -1 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3079: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:36:03.515 INFO:tasks.rados.rados.0.smithi042.stdout:3080: copy_from oid 37 from oid 42 current snap is 353 2024-09-20T19:36:03.516 INFO:tasks.rados.rados.0.smithi042.stdout:3081: delete oid 43 current snap is 353 2024-09-20T19:36:03.516 INFO:tasks.rados.rados.0.smithi042.stdout:3080: finishing copy_from to smithi04268048-37 2024-09-20T19:36:03.516 INFO:tasks.rados.rados.0.smithi042.stdout:3080: got expected ENOENT (src dne) 2024-09-20T19:36:03.517 INFO:tasks.rados.rados.0.smithi042.stdout:3080: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:36:03.522 INFO:tasks.rados.rados.0.smithi042.stdout:3078: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:36:03.522 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.557 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:03.588 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.620 INFO:tasks.rados.rados.0.smithi042.stdout:3079: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:36:03.662 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.671 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3078: done (3 left) 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3079: done (2 left) 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3080: done (1 left) 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3081: done (0 left) 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3082: copy_from oid 37 from oid 35 current snap is 353 2024-09-20T19:36:03.776 INFO:tasks.rados.rados.0.smithi042.stdout:3083: delete oid 28 current snap is 353 2024-09-20T19:36:03.777 INFO:tasks.rados.rados.0.smithi042.stdout:3082: finishing copy_from to smithi04268048-37 2024-09-20T19:36:03.777 INFO:tasks.rados.rados.0.smithi042.stdout:3082: got expected ENOENT (src dne) 2024-09-20T19:36:03.777 INFO:tasks.rados.rados.0.smithi042.stdout:3082: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3082: done (1 left) 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3083: done (0 left) 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3084: rollback oid 21 current snap is 353 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 352 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3085: read oid 13 snap 299 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3085: expect deleted 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3086: read oid 31 snap -1 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3086: expect (ObjNum 1023 snap 336 seq_num 1023) 2024-09-20T19:36:03.781 INFO:tasks.rados.rados.0.smithi042.stdout:3087: snap_create 2024-09-20T19:36:03.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:03.791 INFO:tasks.rados.rados.0.smithi042.stdout:3084: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:36:03.791 INFO:tasks.rados.rados.0.smithi042.stdout:3084: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:36:03.791 INFO:tasks.rados.rados.0.smithi042.stdout:3084: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:36:03.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1310 (ObjNum 1038 snap 343 seq_num 1038) dirty exists 2024-09-20T19:36:03.796 INFO:tasks.rados.rados.0.smithi042.stdout:3086: expect (ObjNum 1023 snap 336 seq_num 1023) 2024-09-20T19:36:03.796 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:03 smithi162 bash[24557]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:03.919 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[22706]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:02.414503+0000 mgr.y (mgr.24491) 680 : cluster [DBG] pgmap v1278: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 366 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:02.541803+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2024-09-20T19:36:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:03 smithi042 bash[32075]: cluster 2024-09-20T19:36:03.398811+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2024-09-20T19:36:03.940 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:03.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:03.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:04.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:04.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:04.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:04.221 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:04.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:04.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:04.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:04.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:04.554 INFO:tasks.rados.rados.0.smithi042.stdout:3084: done (3 left) 2024-09-20T19:36:04.554 INFO:tasks.rados.rados.0.smithi042.stdout:3085: done (2 left) 2024-09-20T19:36:04.554 INFO:tasks.rados.rados.0.smithi042.stdout:3086: done (1 left) 2024-09-20T19:36:04.554 INFO:tasks.rados.rados.0.smithi042.stdout:3087: done (0 left) 2024-09-20T19:36:04.555 INFO:tasks.rados.rados.0.smithi042.stdout:3088: write (excl) oid 35 current snap is 354 2024-09-20T19:36:04.555 INFO:tasks.rados.rados.0.smithi042.stdout:3088: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-20T19:36:04.560 INFO:tasks.rados.rados.0.smithi042.stdout:3088: writing smithi04268048-35 from 749876 to 1405228 tid 1 2024-09-20T19:36:04.567 INFO:tasks.rados.rados.0.smithi042.stdout:3088: writing smithi04268048-35 from 1863710 to 2587071 tid 2 2024-09-20T19:36:04.575 INFO:tasks.rados.rados.0.smithi042.stdout:3088: writing smithi04268048-35 from 3012815 to 3761560 tid 3 2024-09-20T19:36:04.575 INFO:tasks.rados.rados.0.smithi042.stdout:3089: snap_create 2024-09-20T19:36:04.575 INFO:tasks.rados.rados.0.smithi042.stdout:3088: finishing write tid 1 to smithi04268048-35 2024-09-20T19:36:04.575 INFO:tasks.rados.rados.0.smithi042.stdout:3088: finishing write tid 2 to smithi04268048-35 2024-09-20T19:36:04.581 INFO:tasks.rados.rados.0.smithi042.stdout:3088: finishing write tid 3 to smithi04268048-35 2024-09-20T19:36:04.581 INFO:tasks.rados.rados.0.smithi042.stdout:3088: finishing write tid 4 to smithi04268048-35 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: finishing write tid 5 to smithi04268048-35 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: oid 35 updating version 0 to 1235 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: oid 35 updating version 1235 to 1236 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: oid 35 version 1236 is already newer than 1233 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: oid 35 version 1236 is already newer than 1234 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1236 (ObjNum 1068 snap 354 seq_num 1068) dirty exists 2024-09-20T19:36:04.582 INFO:tasks.rados.rados.0.smithi042.stdout:3088: left oid 35 (ObjNum 1068 snap 354 seq_num 1068) 2024-09-20T19:36:04.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:04.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:04.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:04.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:04.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:04.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:04.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:04.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:05.060 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:05.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:05.121 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:05.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:05.352 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:05.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:05.565 INFO:tasks.rados.rados.0.smithi042.stdout:3088: done (1 left) 2024-09-20T19:36:05.565 INFO:tasks.rados.rados.0.smithi042.stdout:3089: done (0 left) 2024-09-20T19:36:05.565 INFO:tasks.rados.rados.0.smithi042.stdout:3090: snap_remove snap 325 2024-09-20T19:36:05.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.627 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:05 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.642 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:05.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:05.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:04.415482+0000 mgr.y (mgr.24491) 681 : cluster [DBG] pgmap v1281: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:04.551023+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.205228+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:05 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.206658+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:05.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:05.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:05.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:06.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.036 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:06.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:06.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:06.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:06.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:06.411 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:06.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:06.561 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:06.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:06 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.577 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:06 smithi042 bash[22706]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.585 INFO:tasks.rados.rados.0.smithi042.stdout:3090: done (0 left) 2024-09-20T19:36:06.586 INFO:tasks.rados.rados.0.smithi042.stdout:3091: read oid 1 snap -1 2024-09-20T19:36:06.586 INFO:tasks.rados.rados.0.smithi042.stdout:3091: expect (ObjNum 1015 snap 332 seq_num 1015) 2024-09-20T19:36:06.586 INFO:tasks.rados.rados.0.smithi042.stdout:3092: write (excl) oid 7 current snap is 355 2024-09-20T19:36:06.586 INFO:tasks.rados.rados.0.smithi042.stdout:3092: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2024-09-20T19:36:06.597 INFO:tasks.rados.rados.0.smithi042.stdout:3092: writing smithi04268048-7 from 766683 to 1497284 tid 1 2024-09-20T19:36:06.606 INFO:tasks.rados.rados.0.smithi042.stdout:3092: writing smithi04268048-7 from 1922192 to 2505564 tid 2 2024-09-20T19:36:06.616 INFO:tasks.rados.rados.0.smithi042.stdout:3092: writing smithi04268048-7 from 3156591 to 3832961 tid 3 2024-09-20T19:36:06.617 INFO:tasks.rados.rados.0.smithi042.stdout:3093: read oid 39 snap -1 2024-09-20T19:36:06.617 INFO:tasks.rados.rados.0.smithi042.stdout:3093: expect deleted 2024-09-20T19:36:06.617 INFO:tasks.rados.rados.0.smithi042.stdout:3094: rollback oid 50 current snap is 355 2024-09-20T19:36:06.617 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 353 2024-09-20T19:36:06.617 INFO:tasks.rados.rados.0.smithi042.stdout:3091: expect (ObjNum 1015 snap 332 seq_num 1015) 2024-09-20T19:36:06.631 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1910121 2024-09-20T19:36:06.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:06 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:06 smithi042 bash[32075]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:06.706 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: finishing write tid 1 to smithi04268048-7 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: finishing write tid 2 to smithi04268048-7 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3094: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: finishing write tid 3 to smithi04268048-7 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3094: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3094: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1049 (ObjNum 995 snap 326 seq_num 995) dirty exists 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: finishing write tid 4 to smithi04268048-7 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: finishing write tid 5 to smithi04268048-7 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: oid 7 updating version 0 to 1238 2024-09-20T19:36:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:3092: oid 7 updating version 1238 to 1239 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3092: oid 7 updating version 1239 to 1241 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3092: oid 7 version 1241 is already newer than 1240 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1241 (ObjNum 1069 snap 355 seq_num 1069) dirty exists 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3092: left oid 7 (ObjNum 1069 snap 355 seq_num 1069) 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3091: done (3 left) 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3092: done (2 left) 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3093: done (1 left) 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3094: done (0 left) 2024-09-20T19:36:06.757 INFO:tasks.rados.rados.0.smithi042.stdout:3095: snap_create 2024-09-20T19:36:06.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:06 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:06 smithi162 bash[24557]: cluster 2024-09-20T19:36:05.562817+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2024-09-20T19:36:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:06.919 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.944 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:06.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:07.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:07.075 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:07.105 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:36:07.105 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:36:07.107 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:07.107 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:07.108 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:07.131 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:36:07.131 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:36:07.148 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:36:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:36:07.165 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:36:07.165 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:36:07.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:07.186 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:36:07.186 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:36:07.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:07.205 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:36:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:36:07.224 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:36:07.224 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:36:07.242 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:36:07.242 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:36:07.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:07.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:07.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:07.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.585 INFO:tasks.rados.rados.0.smithi042.stdout:3095: done (0 left) 2024-09-20T19:36:07.585 INFO:tasks.rados.rados.0.smithi042.stdout:3096: snap_create 2024-09-20T19:36:07.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:07.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:07.706 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.706 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.706 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[22706]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:07 smithi042 bash[32075]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:07.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: cluster 2024-09-20T19:36:06.416254+0000 mgr.y (mgr.24491) 682 : cluster [DBG] pgmap v1284: 178 pgs: 10 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 153 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: cluster 2024-09-20T19:36:06.577277+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.838 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:07 smithi162 bash[24557]: audit 2024-09-20T19:36:07.084969+0000 mon.a (mon.0) 2421 : audit [DBG] from='client.? v1:172.21.15.42:0/3829290952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:07.854 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:07.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.981 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:07.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:08.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:08.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:08.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:08.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:08.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:08.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:08.474 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:08.503 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:08.584 INFO:tasks.rados.rados.0.smithi042.stdout:3096: done (0 left) 2024-09-20T19:36:08.584 INFO:tasks.rados.rados.0.smithi042.stdout:3097: snap_remove snap 337 2024-09-20T19:36:08.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:08.620 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:08 smithi162 bash[24557]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.620 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:08 smithi162 bash[24557]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.620 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:08 smithi162 bash[24557]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.620 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:08 smithi162 bash[24557]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:08.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:08.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:08.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:08.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[22706]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[22706]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[22706]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[22706]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[32075]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[32075]: cluster 2024-09-20T19:36:07.577694+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2024-09-20T19:36:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[32075]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:08 smithi042 bash[32075]: cluster 2024-09-20T19:36:08.580279+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2024-09-20T19:36:08.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:09.021 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:09.124 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:09.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:09.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:09.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:09.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:09.595 INFO:tasks.rados.rados.0.smithi042.stdout:3097: done (0 left) 2024-09-20T19:36:09.595 INFO:tasks.rados.rados.0.smithi042.stdout:3098: write oid 43 current snap is 357 2024-09-20T19:36:09.596 INFO:tasks.rados.rados.0.smithi042.stdout:3098: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-09-20T19:36:09.596 INFO:tasks.rados.rados.0.smithi042.stdout:3098: writing smithi04268048-43 from 783490 to 1189340 tid 1 2024-09-20T19:36:09.606 INFO:tasks.rados.rados.0.smithi042.stdout:3098: writing smithi04268048-43 from 1980674 to 2707704 tid 2 2024-09-20T19:36:09.613 INFO:tasks.rados.rados.0.smithi042.stdout:3098: writing smithi04268048-43 from 3184014 to 3671656 tid 3 2024-09-20T19:36:09.614 INFO:tasks.rados.rados.0.smithi042.stdout:3099: snap_remove snap 338 2024-09-20T19:36:09.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[32075]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[32075]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[32075]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[32075]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[22706]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[22706]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[22706]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:09 smithi042 bash[22706]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:09.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:09.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:09 smithi162 bash[24557]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:09 smithi162 bash[24557]: cluster 2024-09-20T19:36:08.417288+0000 mgr.y (mgr.24491) 683 : cluster [DBG] pgmap v1287: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 520 KiB/s wr, 14 op/s 2024-09-20T19:36:09.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:09 smithi162 bash[24557]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:09 smithi162 bash[24557]: cluster 2024-09-20T19:36:09.583298+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2024-09-20T19:36:09.813 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:09.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:09.927 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:09.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:09.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:10.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:10.075 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:10.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:10.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:10.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:10.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:10.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:10.610 INFO:tasks.rados.rados.0.smithi042.stdout:3099: done (1 left) 2024-09-20T19:36:10.610 INFO:tasks.rados.rados.0.smithi042.stdout:3100: snap_remove snap 348 2024-09-20T19:36:10.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:10.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:10.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:10.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:10.889 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:36:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:36:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:36:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:36:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2304 4.b 4:d201668e:::obj_delete_at_hint.0000000023:head [call lock.lock in=8b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14291257500003, 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.05466185, 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2304 2024-09-20T19:36:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743425+0000", 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743426+0000", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0299999999999997e-07 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743430+0000", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8550000000000001e-06 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743439+0000", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3930000000000008e-06 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743454+0000", 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4882000000000001e-05 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743510+0000", 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6486000000000001e-05 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743518+0000", 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4100000000000002e-06 2024-09-20T19:36:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.763011+0000", 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019493656000000002 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.763015+0000", 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6650000000000001e-06 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793210+0000", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0301954090000001 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793332+0000", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012176999999999999 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793851+0000", 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00051850500000000005 2024-09-20T19:36:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795617+0000", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00176593 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797863+0000", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002246255 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797953+0000", 2024-09-20T19:36:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.9913999999999993e-05 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798087+0000", 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000133817 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2381 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.142869754, 2024-09-20T19:36:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0519627060000001, 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2381 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743468+0000", 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0100000000000004e-07 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743479+0000", 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0648999999999999e-05 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743488+0000", 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6479999999999992e-06 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743494+0000", 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.798e-06 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743541+0000", 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6360000000000003e-05 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743548+0000", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.7710000000000001e-06 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762973+0000", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019425462000000001 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762984+0000", 2024-09-20T19:36:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1403000000000001e-05 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795280+0000", 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0322955819999999 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795430+0000", 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015023200000000001 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14478.0:2544 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94817798635392 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14283311200001, 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.050798683, 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14478", 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:36:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 2544 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743504+0000", 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743505+0000", 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1200000000000003e-07 2024-09-20T19:36:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743508+0000", 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2389999999999998e-06 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743515+0000", 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.5860000000000001e-06 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743521+0000", 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.827e-06 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743576+0000", 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4317000000000002e-05 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.743583+0000", 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4059999999999999e-06 2024-09-20T19:36:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762986+0000", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019403238 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.762995+0000", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8000000000000004e-06 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.789018+0000", 2024-09-20T19:36:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.026022295 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.789172+0000", 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000154189 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.794303+0000", 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0051311739999999996 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3928 7.4 7:24603624:::smithi04268048-48:head [write 591745~402925,stat] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14170497200001, 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0551826959999999, 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3928 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744632+0000", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744634+0000", 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9250000000000002e-06 2024-09-20T19:36:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744981+0000", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000346468 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.744993+0000", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1878e-05 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745001+0000", 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8340000000000005e-06 2024-09-20T19:36:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746485+0000", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001483617 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746497+0000", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2424e-05 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769303+0000", 2024-09-20T19:36:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0228058929999999 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769761+0000", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00045766200000000001 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791851+0000", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.022090143 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791978+0000", 2024-09-20T19:36:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012683 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792888+0000", 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00091 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792995+0000", 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107441 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.793430+0000", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00043459800000000002 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798331+0000", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0049014260000000004 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799616+0000", 2024-09-20T19:36:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0012846260000000001 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799686+0000", 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9567000000000007e-05 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799815+0000", 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000129364 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3929 7.4 7:24603624:::smithi04268048-48:head [write 1790337~563515] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14132279400002, 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0564495810000001, 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3929 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745015+0000", 2024-09-20T19:36:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1500000000000003e-07 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745320+0000", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00030476499999999997 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745330+0000", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5389999999999996e-06 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745338+0000", 2024-09-20T19:36:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3319999999999992e-06 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746527+0000", 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011885789999999999 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746535+0000", 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2160000000000006e-06 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.769872+0000", 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0233368110000001 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770035+0000", 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016329300000000001 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792008+0000", 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021972551999999999 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792074+0000", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.6872000000000006e-05 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795026+0000", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0029514490000000001 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.795812+0000", 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000786212 2024-09-20T19:36:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796112+0000", 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00030015199999999998 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799835+0000", 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0037226220000000001 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801207+0000", 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013722389999999999 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801282+0000", 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4697999999999995e-05 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801464+0000", 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018233499999999999 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3930 7.4 7:24603624:::smithi04268048-48:head [setxattr _header (58) in=7b,truncate 2991745] snapc ae=[ae,ad,ac,ab,aa,a9,a7,a6,a5,a3,a1,9f,9d,9c,98,88,72] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14098944800003, 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.05644351, 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3930 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745348+0000", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745349+0000", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0999999999999997e-07 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745354+0000", 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8039999999999998e-06 2024-09-20T19:36:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745360+0000", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8170000000000004e-06 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745366+0000", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.3299999999999998e-06 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746559+0000", 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001192993 2024-09-20T19:36:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746566+0000", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.1459999999999999e-06 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770610+0000", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0240446439999999 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770771+0000", 2024-09-20T19:36:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016099199999999999 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792101+0000", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021329839999999999 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792166+0000", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4751000000000004e-05 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797311+0000", 2024-09-20T19:36:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0051449699999999996 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797491+0000", 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018041800000000001 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1", 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797775+0000", 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00028395999999999998 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.799852+0000", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002076788 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801527+0000", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016751229999999999 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801613+0000", 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.6012000000000003e-05 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801791+0000", 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017811199999999999 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:3931 7.4 7:24603624:::smithi04268048-48:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14096554899999, 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0582288870000001, 2024-09-20T19:36:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 3931 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:36:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745372+0000", 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6400000000000003e-07 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745374+0000", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.232e-06 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745382+0000", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.807e-06 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745396+0000", 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3362e-05 2024-09-20T19:36:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746595+0000", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011997240000000001 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.746603+0000", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0139999999999998e-06 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.770850+0000", 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0242465510000001 2024-09-20T19:36:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.771097+0000", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00024681600000000002 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792188+0000", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.021091675000000001 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792323+0000", 2024-09-20T19:36:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00013485099999999999 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798407+0000", 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060842170000000003 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.798643+0000", 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00023551600000000001 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801873+0000", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0032300290000000001 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.802054+0000", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000180927 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.803601+0000", 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0015467020000000001 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4004 7.9 7:94ee8d8f:::smithi04268048-13:head [stat,delete] snapc b2=[b2,b1,b0,af,ae,ad,ac,ab,aa,a9,a7,a6,a3,a1,9f,9d,9c,98,88] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 362.14093451700001, 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.056561592, 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4004 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745403+0000", 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5700000000000002e-07 2024-09-20T19:36:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745406+0000", 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5830000000000001e-06 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745412+0000", 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7049999999999996e-06 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.745416+0000", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.7190000000000001e-06 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.747326+0000", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001909656 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:08.747332+0000", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1090000000000003e-06 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.791891+0000", 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0445585660000001 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.792090+0000", 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019927399999999999 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796389+0000", 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0042991349999999999 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.796569+0000", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000179848 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797353+0000", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00078369699999999997 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.797493+0000", 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000140536 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:09.801964+0000", 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0044712069999999996 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14652.0:21 3.1 3:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e518)", 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 340.58400808599998, 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0601109449999999, 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "reached pg", 2024-09-20T19:36:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14652", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/577469704", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 21 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302329+0000", 2024-09-20T19:36:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302332+0000", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6019999999999998e-06 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302342+0000", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7159999999999998e-06 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302360+0000", 2024-09-20T19:36:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8530999999999999e-05 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.302375+0000", 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4506e-05 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.305035+0000", 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.002660875 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:30.305076+0000", 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0312999999999998e-05 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.362285+0000", 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.057209316 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.362440+0000", 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000155086 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4234 7.13 7:cf582c87:::smithi04268048-12:head [copy-from ver 626] snapc ba=[ba,b8,b7,b5,b3,b2,b0,ad,ab,aa,a9,9f,9c] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 339.52671598799998, 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.519326363, 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4234 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359621+0000", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359624+0000", 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7130000000000001e-06 2024-09-20T19:36:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359632+0000", 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9400000000000002e-06 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359648+0000", 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5883e-05 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359656+0000", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5210000000000002e-06 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359747+0000", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0464999999999995e-05 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359899+0000", 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015204800000000001 2024-09-20T19:36:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,7", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.868922+0000", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.509023121 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876934+0000", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0080117520000000005 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876976+0000", 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2599999999999999e-05 2024-09-20T19:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878852+0000", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0018758570000000001 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878890+0000", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7568999999999999e-05 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.878948+0000", 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7893999999999998e-05 2024-09-20T19:36:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4235 7.13 7:cf582c87:::smithi04268048-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 339.52666423800002, 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.517188041, 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4235 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359673+0000", 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359674+0000", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4099999999999998e-07 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359677+0000", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0400000000000001e-06 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359683+0000", 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8030000000000002e-06 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.359687+0000", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6829999999999999e-06 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360140+0000", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000452373 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360199+0000", 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9703000000000001e-05 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876675+0000", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.516475835 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876753+0000", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8146999999999996e-05 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.876861+0000", 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000107816 2024-09-20T19:36:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(osd.0.500:1 7.16 7:6c48c6fe:::smithi04268048-7:head [assert-version v626,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 339.52589105800001, 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5039987930000001, 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360446+0000", 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360448+0000", 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4020000000000001e-06 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360453+0000", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1699999999999996e-06 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360463+0000", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0509000000000001e-05 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360470+0000", 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.109e-06 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360530+0000", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9468000000000001e-05 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:31.360539+0000", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5869999999999999e-06 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.861867+0000", 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5013275930000001 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.862028+0000", 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016082499999999999 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:32.864445+0000", 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00241713 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5407 7.4 7:23ed6a02:::smithi04268048-25:head [copy-from ver 767] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:36:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 217.598846138, 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030478080000000001, 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5407 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:36:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287491+0000", 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287494+0000", 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5909999999999998e-06 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287502+0000", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9279999999999993e-06 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287512+0000", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0562e-05 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287518+0000", 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.0800000000000002e-06 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288171+0000", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00065281200000000001 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.288179+0000", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2949999999999994e-06 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295330+0000", 2024-09-20T19:36:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0071509679999999997 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295627+0000", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029681800000000002 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,4", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.304712+0000", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0090846569999999995 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.314385+0000", 2024-09-20T19:36:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.009672702 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.315214+0000", 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00082897899999999998 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317811+0000", 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0025975809999999998 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317893+0000", 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1588999999999994e-05 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317969+0000", 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6518000000000002e-05 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5422 7.13 7:cf582c87:::smithi04268048-12:head [copy-from ver 721] snapc f8=[f8,f7,f6,f5,f4,f3,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 216.50283582, 2024-09-20T19:36:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037795502000000002, 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5422 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383501+0000", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383506+0000", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0629999999999999e-06 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383520+0000", 2024-09-20T19:36:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.4568e-05 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383551+0000", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1041999999999999e-05 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.383567+0000", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5488000000000001e-05 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384047+0000", 2024-09-20T19:36:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000480113 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384060+0000", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3049e-05 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.388415+0000", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.004354803 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389079+0000", 2024-09-20T19:36:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00066449499999999997 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,7", 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.411175+0000", 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.022096352999999999 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.420015+0000", 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0088399080000000005 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.420138+0000", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000122879 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421052+0000", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00091352000000000002 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421197+0000", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014492500000000001 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.421297+0000", 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100296 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5423 7.13 7:cf582c87:::smithi04268048-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 216.50221875299999, 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.035737366, 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5423 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384119+0000", 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384120+0000", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8220000000000001e-06 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384129+0000", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8380000000000008e-06 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384155+0000", 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5514999999999999e-05 2024-09-20T19:36:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384472+0000", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031773400000000002 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384555+0000", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3004999999999996e-05 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.384563+0000", 2024-09-20T19:36:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.328e-06 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389341+0000", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0047780000000000001 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.389385+0000", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4147000000000003e-05 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419500+0000", 2024-09-20T19:36:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030114867 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419666+0000", 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016647499999999999 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:34.419856+0000", 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018963500000000001 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.6.0:1304 7.7 e698/550 7:e18d1735:::smithi04268048-41:f8 v 698'447, mlcod=694'444)", 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:26.373915+0000", 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 164.51242183599999, 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.044470757999999999, 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373915+0000", 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373915+0000", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373917+0000", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.717e-06 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373933+0000", 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.562e-05 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373952+0000", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.9076000000000002e-05 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.373975+0000", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3119999999999999e-05 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.374180+0000", 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000205305 2024-09-20T19:36:10.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.374254+0000", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3886000000000004e-05 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.418312+0000", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.044057790999999999 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.418386+0000", 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.4243000000000001e-05 2024-09-20T19:36:10.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.6.0:1305 7.7 e698/550 7:e75b8502:::smithi04268048-39:10d v 698'449, mlcod=694'444)", 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:26.375361+0000", 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 164.510976403, 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.043064814, 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375361+0000", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375361+0000", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375363+0000", 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5229999999999998e-06 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375394+0000", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1013000000000003e-05 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375416+0000", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1149999999999999e-05 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375439+0000", 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3689999999999998e-05 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375584+0000", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000145072 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.375648+0000", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3150000000000004e-05 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.418392+0000", 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.042744503000000003 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.418426+0000", 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.3713000000000001e-05 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:6435 7.18 7:1f0aed4a:::smithi04268048-44:head [copy-from ver 887] snapc 125=[125,124,123,122,120,11e,11d,118] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:34:13.341965+0000", 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 117.544372329, 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.034882338999999998, 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 6435 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.341965+0000", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.341965+0000", 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.341968+0000", 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7980000000000001e-06 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.341977+0000", 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3370000000000006e-06 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.341993+0000", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6345999999999999e-05 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.342005+0000", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1266e-05 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.342086+0000", 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1606000000000004e-05 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.342094+0000", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.1089999999999998e-06 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.346912+0000", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0048174029999999996 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.347063+0000", 2024-09-20T19:36:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015124999999999999 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 6,7", 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.360626+0000", 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01356252 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.367706+0000", 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070801240000000001 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.371805+0000", 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0040996210000000003 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.376747+0000", 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0049415930000000002 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.376788+0000", 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0810999999999999e-05 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:13.376847+0000", 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9555000000000002e-05 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:6470 7.13 7:cf582c87:::smithi04268048-12:head [rollback 124] snapc 127=[127,126,124,123,120,11d,118] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:34:18.378236+0000", 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 112.50810175700001, 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.027913074999999999, 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 6470 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378236+0000", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378236+0000", 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378237+0000", 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0079999999999999e-06 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378241+0000", 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2749999999999997e-06 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378249+0000", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9140000000000008e-06 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378256+0000", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.3069999999999997e-06 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378360+0000", 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010344399999999999 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.378366+0000", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.3640000000000004e-06 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.383936+0000", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.005569989 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.383991+0000", 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4648000000000003e-05 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 5,7", 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.384114+0000", 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012390099999999999 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.384959+0000", 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00084457900000000003 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.405027+0000", 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020068044 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.406039+0000", 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010115790000000001 2024-09-20T19:36:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.406114+0000", 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5185000000000006e-05 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:18.406149+0000", 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4838000000000001e-05 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:6653 7.18 7:1f0aed4a:::smithi04268048-44:head [copy-from ver 918] snapc 130=[130,12f,12e,12d,12a,129,124,120,118] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:34:34.348719+0000", 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 96.537618227999999, 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030574655999999999, 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 6653 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:36:11.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348719+0000", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348719+0000", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348723+0000", 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9349999999999996e-06 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348735+0000", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1871999999999999e-05 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348756+0000", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0934e-05 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348766+0000", 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.7450000000000007e-06 2024-09-20T19:36:11.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348835+0000", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9610000000000006e-05 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.348841+0000", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5300000000000004e-06 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.351148+0000", 2024-09-20T19:36:11.016 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0023076049999999999 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.351475+0000", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00032676500000000002 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 6,7", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.358106+0000", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.006630894 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.365505+0000", 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.007398529 2024-09-20T19:36:11.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.370974+0000", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0054694289999999996 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.379198+0000", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0082239280000000001 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.379237+0000", 2024-09-20T19:36:11.018 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8865000000000001e-05 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:34:34.379294+0000", 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7015000000000002e-05 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:36:11.019 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:36:11.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:11.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:11.123 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:36:11.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:11.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:11.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:11.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:11.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:11.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:set pool 1 pgp_num to 11 2024-09-20T19:36:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3100: done (1 left) 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3101: rollback oid 38 current snap is 357 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 299 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3102: read oid 50 snap 346 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3102: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3103: copy_from oid 47 from oid 26 current snap is 357 2024-09-20T19:36:11.616 INFO:tasks.rados.rados.0.smithi042.stdout:3104: snap_create 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: finishing write tid 1 to smithi04268048-43 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: finishing write tid 2 to smithi04268048-43 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: finishing write tid 3 to smithi04268048-43 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: finishing write tid 4 to smithi04268048-43 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: finishing write tid 5 to smithi04268048-43 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: oid 43 updating version 0 to 749 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: oid 43 version 749 is already newer than 748 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: oid 43 version 749 is already newer than 747 2024-09-20T19:36:11.617 INFO:tasks.rados.rados.0.smithi042.stdout:3098: oid 43 updating version 749 to 750 2024-09-20T19:36:11.618 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 750 (ObjNum 1070 snap 357 seq_num 1070) dirty exists 2024-09-20T19:36:11.619 INFO:tasks.rados.rados.0.smithi042.stdout:3098: left oid 43 (ObjNum 1070 snap 357 seq_num 1070) 2024-09-20T19:36:11.647 INFO:tasks.rados.rados.0.smithi042.stdout:3101: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:36:11.647 INFO:tasks.rados.rados.0.smithi042.stdout:3101: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:36:11.647 INFO:tasks.rados.rados.0.smithi042.stdout:3101: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:36:11.647 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1152 (ObjNum 904 snap 298 seq_num 904) dirty exists 2024-09-20T19:36:11.653 INFO:tasks.rados.rados.0.smithi042.stdout:3102: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:11.653 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:11.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:11.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.673 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:11 smithi162 bash[24557]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.683 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:11.742 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2652057 2024-09-20T19:36:11.742 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:11.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:11.791 INFO:tasks.rados.rados.0.smithi042.stdout:3103: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:36:11.791 INFO:tasks.rados.rados.0.smithi042.stdout:3103: finishing copy_from to smithi04268048-47 2024-09-20T19:36:11.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1237 (ObjNum 1064 snap 352 seq_num 1064) dirty exists 2024-09-20T19:36:11.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:11.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:11.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[22706]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:09.794530+0000 mgr.y (mgr.24491) 684 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: cluster 2024-09-20T19:36:10.418155+0000 mgr.y (mgr.24491) 685 : cluster [DBG] pgmap v1290: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 350 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1006 KiB/s wr, 16 op/s 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: cluster 2024-09-20T19:36:10.609607+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:10.910598+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:11 smithi042 bash[32075]: audit 2024-09-20T19:36:11.109907+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-20T19:36:11.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:11.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:12.041 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:12.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:36:12.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:12.197 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:12.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:12.319 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:12.422 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:12.554 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:12.616 INFO:tasks.rados.rados.0.smithi042.stdout:3098: done (4 left) 2024-09-20T19:36:12.616 INFO:tasks.rados.rados.0.smithi042.stdout:3101: done (3 left) 2024-09-20T19:36:12.617 INFO:tasks.rados.rados.0.smithi042.stdout:3102: done (2 left) 2024-09-20T19:36:12.617 INFO:tasks.rados.rados.0.smithi042.stdout:3103: done (1 left) 2024-09-20T19:36:12.617 INFO:tasks.rados.rados.0.smithi042.stdout:3104: done (0 left) 2024-09-20T19:36:12.617 INFO:tasks.rados.rados.0.smithi042.stdout:3105: delete oid 16 current snap is 358 2024-09-20T19:36:12.625 INFO:tasks.rados.rados.0.smithi042.stdout:3105: done (0 left) 2024-09-20T19:36:12.625 INFO:tasks.rados.rados.0.smithi042.stdout:3106: write oid 9 current snap is 358 2024-09-20T19:36:12.625 INFO:tasks.rados.rados.0.smithi042.stdout:3106: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-09-20T19:36:12.631 INFO:tasks.rados.rados.0.smithi042.stdout:3106: writing smithi04268048-9 from 417104 to 1089805 tid 1 2024-09-20T19:36:12.637 INFO:tasks.rados.rados.0.smithi042.stdout:3106: writing smithi04268048-9 from 1813991 to 2017104 tid 2 2024-09-20T19:36:12.637 INFO:tasks.rados.rados.0.smithi042.stdout:3107: snap_create 2024-09-20T19:36:12.643 INFO:tasks.rados.rados.0.smithi042.stdout:3106: finishing write tid 1 to smithi04268048-9 2024-09-20T19:36:12.650 INFO:tasks.rados.rados.0.smithi042.stdout:3106: finishing write tid 2 to smithi04268048-9 2024-09-20T19:36:12.650 INFO:tasks.rados.rados.0.smithi042.stdout:3106: finishing write tid 3 to smithi04268048-9 2024-09-20T19:36:12.651 INFO:tasks.rados.rados.0.smithi042.stdout:3106: finishing write tid 4 to smithi04268048-9 2024-09-20T19:36:12.651 INFO:tasks.rados.rados.0.smithi042.stdout:3106: oid 9 updating version 0 to 1056 2024-09-20T19:36:12.652 INFO:tasks.rados.rados.0.smithi042.stdout:3106: oid 9 version 1056 is already newer than 1055 2024-09-20T19:36:12.652 INFO:tasks.rados.rados.0.smithi042.stdout:3106: oid 9 version 1056 is already newer than 1054 2024-09-20T19:36:12.652 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1056 (ObjNum 1072 snap 358 seq_num 1072) dirty exists 2024-09-20T19:36:12.652 INFO:tasks.rados.rados.0.smithi042.stdout:3106: left oid 9 (ObjNum 1072 snap 358 seq_num 1072) 2024-09-20T19:36:12.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:12.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:12.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:12.829 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.829 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[32075]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.830 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:12 smithi042 bash[22706]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:12.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:12.958 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.958 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:11.606580+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:12.958 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.958 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: audit 2024-09-20T19:36:11.608493+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? v1:172.21.15.42:0/1501203467' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:12.958 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: audit 2024-09-20T19:36:11.608596+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.42:0/4107546445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T19:36:12.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:11.613432+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2024-09-20T19:36:12.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.959 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:12 smithi162 bash[24557]: cluster 2024-09-20T19:36:12.614234+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2024-09-20T19:36:12.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:12.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:13.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:13.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:13.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:13.229 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:13.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:13.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:13.402 INFO:tasks.rados.rados.0.smithi042.stdout:3106: done (1 left) 2024-09-20T19:36:13.402 INFO:tasks.rados.rados.0.smithi042.stdout:3107: done (0 left) 2024-09-20T19:36:13.402 INFO:tasks.rados.rados.0.smithi042.stdout:3108: snap_remove snap 332 2024-09-20T19:36:13.465 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:13.486 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:13.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:13.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:13.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[22706]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:13.846 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:13.870 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:13.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:13 smithi042 bash[32075]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: cluster 2024-09-20T19:36:12.419811+0000 mgr.y (mgr.24491) 686 : cluster [DBG] pgmap v1293: 178 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 160 active+clean; 359 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:12.770240+0000 mon.a (mon.0) 2433 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.346148+0000 mon.a (mon.0) 2434 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:13.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: cluster 2024-09-20T19:36:13.397308+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.522986+0000 mon.a (mon.0) 2436 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.523715+0000 mon.a (mon.0) 2437 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:13 smithi162 bash[24557]: audit 2024-09-20T19:36:13.528982+0000 mon.a (mon.0) 2438 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:13.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:14.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:14.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:14.260 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:14.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:14.391 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:14.405 INFO:tasks.rados.rados.0.smithi042.stdout:3108: done (0 left) 2024-09-20T19:36:14.406 INFO:tasks.rados.rados.0.smithi042.stdout:3109: read oid 8 snap -1 2024-09-20T19:36:14.406 INFO:tasks.rados.rados.0.smithi042.stdout:3109: expect deleted 2024-09-20T19:36:14.410 INFO:tasks.rados.rados.0.smithi042.stdout:3110: write (excl) oid 7 current snap is 359 2024-09-20T19:36:14.410 INFO:tasks.rados.rados.0.smithi042.stdout:3110: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-20T19:36:14.429 INFO:tasks.rados.rados.0.smithi042.stdout:3110: writing smithi04268048-7 from 433911 to 1181861 tid 1 2024-09-20T19:36:14.442 INFO:tasks.rados.rados.0.smithi042.stdout:3110: writing smithi04268048-7 from 1756120 to 2033911 tid 2 2024-09-20T19:36:14.443 INFO:tasks.rados.rados.0.smithi042.stdout:3111: snap_create 2024-09-20T19:36:14.443 INFO:tasks.rados.rados.0.smithi042.stdout:3110: finishing write tid 1 to smithi04268048-7 2024-09-20T19:36:14.446 INFO:tasks.rados.rados.0.smithi042.stdout:3110: finishing write tid 2 to smithi04268048-7 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: finishing write tid 3 to smithi04268048-7 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: finishing write tid 4 to smithi04268048-7 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: oid 7 updating version 0 to 1244 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: oid 7 version 1244 is already newer than 1242 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: oid 7 version 1244 is already newer than 1243 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1244 (ObjNum 1073 snap 359 seq_num 1073) dirty exists 2024-09-20T19:36:14.447 INFO:tasks.rados.rados.0.smithi042.stdout:3110: left oid 7 (ObjNum 1073 snap 359 seq_num 1073) 2024-09-20T19:36:14.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:14.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:14.649 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:14.753 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:14.782 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:14.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:14.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:14.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:15.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:15.085 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:36:15.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:15.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:15.294 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:15.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:15.404 INFO:tasks.rados.rados.0.smithi042.stdout:3109: done (2 left) 2024-09-20T19:36:15.405 INFO:tasks.rados.rados.0.smithi042.stdout:3110: done (1 left) 2024-09-20T19:36:15.405 INFO:tasks.rados.rados.0.smithi042.stdout:3111: done (0 left) 2024-09-20T19:36:15.405 INFO:tasks.rados.rados.0.smithi042.stdout:3112: write (excl) oid 9 current snap is 360 2024-09-20T19:36:15.405 INFO:tasks.rados.rados.0.smithi042.stdout:3112: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-20T19:36:15.420 INFO:tasks.rados.rados.0.smithi042.stdout:3112: writing smithi04268048-9 from 450718 to 873917 tid 1 2024-09-20T19:36:15.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:15.443 INFO:tasks.rados.rados.0.smithi042.stdout:3112: writing smithi04268048-9 from 1414602 to 2050718 tid 2 2024-09-20T19:36:15.444 INFO:tasks.rados.rados.0.smithi042.stdout:3113: write (excl) oid 30 current snap is 360 2024-09-20T19:36:15.444 INFO:tasks.rados.rados.0.smithi042.stdout:3112: finishing write tid 1 to smithi04268048-9 2024-09-20T19:36:15.444 INFO:tasks.rados.rados.0.smithi042.stdout:3113: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-20T19:36:15.459 INFO:tasks.rados.rados.0.smithi042.stdout:3113: writing smithi04268048-30 from 467525 to 965973 tid 1 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3113: writing smithi04268048-30 from 1473084 to 2067463 tid 2 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3114: rollback oid 21 current snap is 360 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 21 to 346 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3112: finishing write tid 2 to smithi04268048-9 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3115: copy_from oid 1 from oid 11 current snap is 360 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3112: finishing write tid 3 to smithi04268048-9 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3112: finishing write tid 4 to smithi04268048-9 2024-09-20T19:36:15.477 INFO:tasks.rados.rados.0.smithi042.stdout:3112: oid 9 updating version 0 to 1062 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3112: oid 9 version 1062 is already newer than 1061 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3112: oid 9 version 1062 is already newer than 1060 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1062 (ObjNum 1074 snap 360 seq_num 1074) dirty exists 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3112: left oid 9 (ObjNum 1074 snap 360 seq_num 1074) 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3113: finishing write tid 1 to smithi04268048-30 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3112: done (3 left) 2024-09-20T19:36:15.478 INFO:tasks.rados.rados.0.smithi042.stdout:3116: snap_remove snap 347 2024-09-20T19:36:15.529 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:15.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:15.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:15 smithi162 bash[24557]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:15 smithi162 bash[24557]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:15 smithi162 bash[24557]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.672 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:15 smithi162 bash[24557]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[32075]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[32075]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[32075]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[32075]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[22706]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[22706]: cluster 2024-09-20T19:36:14.403205+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2024-09-20T19:36:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[22706]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:15 smithi042 bash[22706]: cluster 2024-09-20T19:36:14.421837+0000 mgr.y (mgr.24491) 687 : cluster [DBG] pgmap v1297: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2024-09-20T19:36:15.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:15.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:15.821 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:15.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.952 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:15.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:16.054 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:16.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:16.215 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:16.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:16.412 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3116: done (3 left) 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3117: delete oid 31 current snap is 360 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3115: finishing copy_from to smithi04268048-1 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3115: got expected ENOENT (src dne) 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3115: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:36:16.431 INFO:tasks.rados.rados.0.smithi042.stdout:3113: finishing write tid 2 to smithi04268048-30 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: finishing write tid 3 to smithi04268048-30 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: finishing write tid 4 to smithi04268048-30 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: oid 30 updating version 0 to 1342 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: oid 30 version 1342 is already newer than 1340 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: oid 30 version 1342 is already newer than 1341 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1342 (ObjNum 1075 snap 360 seq_num 1075) dirty exists 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3113: left oid 30 (ObjNum 1075 snap 360 seq_num 1075) 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3114: finishing rollback tid 0 to smithi04268048-21 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3114: finishing rollback tid 1 to smithi04268048-21 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:3114: finishing rollback tid 2 to smithi04268048-21 2024-09-20T19:36:16.432 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1346 (ObjNum 1038 snap 343 seq_num 1038) dirty exists 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3113: done (3 left) 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3114: done (2 left) 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3115: done (1 left) 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3117: done (0 left) 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3118: read oid 37 snap -1 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3118: expect (ObjNum 1062 snap 352 seq_num 1062) 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3119: rollback oid 26 current snap is 360 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 26 to 353 2024-09-20T19:36:16.442 INFO:tasks.rados.rados.0.smithi042.stdout:3120: read oid 45 snap -1 2024-09-20T19:36:16.443 INFO:tasks.rados.rados.0.smithi042.stdout:3120: expect deleted 2024-09-20T19:36:16.443 INFO:tasks.rados.rados.0.smithi042.stdout:3121: rollback oid 44 current snap is 360 2024-09-20T19:36:16.443 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 299 2024-09-20T19:36:16.443 INFO:tasks.rados.rados.0.smithi042.stdout:3122: write (excl) oid 9 current snap is 360 2024-09-20T19:36:16.443 INFO:tasks.rados.rados.0.smithi042.stdout:3122: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-09-20T19:36:16.446 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:16.460 INFO:tasks.rados.rados.0.smithi042.stdout:3122: writing smithi04268048-9 from 501139 to 1150085 tid 1 2024-09-20T19:36:16.474 INFO:tasks.rados.rados.0.smithi042.stdout:3122: writing smithi04268048-9 from 1873695 to 2101139 tid 2 2024-09-20T19:36:16.474 INFO:tasks.rados.rados.0.smithi042.stdout:3123: read oid 5 snap 359 2024-09-20T19:36:16.474 INFO:tasks.rados.rados.0.smithi042.stdout:3123: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:36:16.474 INFO:tasks.rados.rados.0.smithi042.stdout:3124: snap_create 2024-09-20T19:36:16.474 INFO:tasks.rados.rados.0.smithi042.stdout:3118: expect (ObjNum 1062 snap 352 seq_num 1062) 2024-09-20T19:36:16.480 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:16.515 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2804192 2024-09-20T19:36:16.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:16 smithi162 bash[24557]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:16 smithi162 bash[24557]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:16.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:16.622 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:16.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:16 smithi042 bash[32075]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:16 smithi042 bash[32075]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:16 smithi042 bash[22706]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:16 smithi042 bash[22706]: cluster 2024-09-20T19:36:15.402258+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3121: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3119: finishing rollback tid 0 to smithi04268048-26 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3119: finishing rollback tid 1 to smithi04268048-26 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3121: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3119: finishing rollback tid 2 to smithi04268048-26 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1247 (ObjNum 1064 snap 352 seq_num 1064) dirty exists 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3121: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1199 (ObjNum 868 snap 288 seq_num 868) dirty exists 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: finishing write tid 1 to smithi04268048-9 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: finishing write tid 2 to smithi04268048-9 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: finishing write tid 3 to smithi04268048-9 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: finishing write tid 4 to smithi04268048-9 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: oid 9 updating version 0 to 1065 2024-09-20T19:36:16.705 INFO:tasks.rados.rados.0.smithi042.stdout:3122: oid 9 version 1065 is already newer than 1063 2024-09-20T19:36:16.706 INFO:tasks.rados.rados.0.smithi042.stdout:3122: oid 9 version 1065 is already newer than 1064 2024-09-20T19:36:16.706 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1065 (ObjNum 1077 snap 360 seq_num 1077) dirty exists 2024-09-20T19:36:16.706 INFO:tasks.rados.rados.0.smithi042.stdout:3122: left oid 9 (ObjNum 1077 snap 360 seq_num 1077) 2024-09-20T19:36:16.706 INFO:tasks.rados.rados.0.smithi042.stdout:3123: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:36:16.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:16.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:16.766 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1627754 2024-09-20T19:36:16.794 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:16.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:16.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:16.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:16.881 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:36:16.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:17.000 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:17.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:17.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:17.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:17.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:17.433 INFO:tasks.rados.rados.0.smithi042.stdout:3118: done (6 left) 2024-09-20T19:36:17.433 INFO:tasks.rados.rados.0.smithi042.stdout:3119: done (5 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3120: done (4 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3121: done (3 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3122: done (2 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3123: done (1 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3124: done (0 left) 2024-09-20T19:36:17.434 INFO:tasks.rados.rados.0.smithi042.stdout:3125: snap_remove snap 336 2024-09-20T19:36:17.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:17.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:17.652 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:17.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:17.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:17.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:17.897 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[32075]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:17.897 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[32075]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:17.897 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[32075]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:17.898 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[32075]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:17.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[22706]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:17.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[22706]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:17.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[22706]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:17.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:17 smithi042 bash[22706]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:17.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:18.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:18.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:17 smithi162 bash[24557]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:18.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:17 smithi162 bash[24557]: cluster 2024-09-20T19:36:16.422850+0000 mgr.y (mgr.24491) 688 : cluster [DBG] pgmap v1300: 178 pgs: 10 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 158 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail 2024-09-20T19:36:18.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:17 smithi162 bash[24557]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:18.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:17 smithi162 bash[24557]: cluster 2024-09-20T19:36:16.424660+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2024-09-20T19:36:18.046 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:18.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:36:18.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:36:18.076 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-20T19:36:18.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-20T19:36:18.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:18.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:18.297 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:18.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.425 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:18.448 INFO:tasks.rados.rados.0.smithi042.stdout:3125: done (0 left) 2024-09-20T19:36:18.448 INFO:tasks.rados.rados.0.smithi042.stdout:3126: write (excl) oid 30 current snap is 361 2024-09-20T19:36:18.448 INFO:tasks.rados.rados.0.smithi042.stdout:3126: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-09-20T19:36:18.455 INFO:tasks.rados.rados.0.smithi042.stdout:3126: writing smithi04268048-30 from 517946 to 1242141 tid 1 2024-09-20T19:36:18.461 INFO:tasks.rados.rados.0.smithi042.stdout:3126: writing smithi04268048-30 from 1932177 to 2117946 tid 2 2024-09-20T19:36:18.461 INFO:tasks.rados.rados.0.smithi042.stdout:3127: snap_remove snap 351 2024-09-20T19:36:18.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:18.530 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:36:18.530 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:36:18.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:18.548 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:36:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:36:18.565 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:36:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:36:18.582 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:36:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:36:18.601 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:36:18.601 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:36:18.620 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:36:18.620 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:36:18.637 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:36:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:36:18.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.655 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:36:18.655 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:36:18.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:18 smithi042 bash[32075]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.658 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:18 smithi042 bash[32075]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:18 smithi042 bash[22706]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:18 smithi042 bash[22706]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.672 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:18.730 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:36:18.730 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:36:18.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:18 smithi162 bash[24557]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:18 smithi162 bash[24557]: cluster 2024-09-20T19:36:17.429166+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2024-09-20T19:36:18.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:18.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:18.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:18.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:18.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:18.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:19.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:19.053 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:19.156 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:19.182 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:19.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:19.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:19.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3127: done (1 left) 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3128: read oid 5 snap 350 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3128: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3129: delete oid 44 current snap is 361 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3126: finishing write tid 1 to smithi04268048-30 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3126: finishing write tid 2 to smithi04268048-30 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3126: finishing write tid 3 to smithi04268048-30 2024-09-20T19:36:19.466 INFO:tasks.rados.rados.0.smithi042.stdout:3126: finishing write tid 4 to smithi04268048-30 2024-09-20T19:36:19.467 INFO:tasks.rados.rados.0.smithi042.stdout:3126: oid 30 updating version 0 to 1356 2024-09-20T19:36:19.467 INFO:tasks.rados.rados.0.smithi042.stdout:3126: oid 30 version 1356 is already newer than 1355 2024-09-20T19:36:19.467 INFO:tasks.rados.rados.0.smithi042.stdout:3126: oid 30 version 1356 is already newer than 1354 2024-09-20T19:36:19.467 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1356 (ObjNum 1078 snap 361 seq_num 1078) dirty exists 2024-09-20T19:36:19.467 INFO:tasks.rados.rados.0.smithi042.stdout:3126: left oid 30 (ObjNum 1078 snap 361 seq_num 1078) 2024-09-20T19:36:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:3126: done (2 left) 2024-09-20T19:36:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:3129: done (1 left) 2024-09-20T19:36:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:3130: copy_from oid 32 from oid 15 current snap is 361 2024-09-20T19:36:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:3131: read oid 34 snap 358 2024-09-20T19:36:19.470 INFO:tasks.rados.rados.0.smithi042.stdout:3131: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3132: read oid 38 snap 319 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3132: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3133: read oid 33 snap -1 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3133: expect (ObjNum 1057 snap 352 seq_num 1057) 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3134: rollback oid 9 current snap is 361 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 354 2024-09-20T19:36:19.471 INFO:tasks.rados.rados.0.smithi042.stdout:3135: snap_remove snap 346 2024-09-20T19:36:19.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[24557]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T19:36:19.569 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:19.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:19.697 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[32075]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: cluster 2024-09-20T19:36:18.423476+0000 mgr.y (mgr.24491) 689 : cluster [DBG] pgmap v1302: 178 pgs: 9 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 165 active+clean; 364 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 948 KiB/s wr, 13 op/s 2024-09-20T19:36:19.699 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.700 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: cluster 2024-09-20T19:36:18.447113+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2024-09-20T19:36:19.700 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.700 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:19 smithi042 bash[22706]: audit 2024-09-20T19:36:18.510590+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.42:0/3372689413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:19.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:19.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:19.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:19.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:20.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:20.087 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:20.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:20.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:20.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:20.478 INFO:tasks.rados.rados.0.smithi042.stdout:3135: done (6 left) 2024-09-20T19:36:20.478 INFO:tasks.rados.rados.0.smithi042.stdout:3136: snap_remove snap 342 2024-09-20T19:36:20.482 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:20.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:20.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:20 smithi042 bash[32075]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:20 smithi042 bash[32075]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:20 smithi042 bash[22706]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:20 smithi042 bash[22706]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.615 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:20.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:20.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:20 smithi162 bash[24557]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:20 smithi162 bash[24557]: cluster 2024-09-20T19:36:19.462117+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2024-09-20T19:36:20.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:20.877 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:20.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:20.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:21.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:21.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:21.140 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:21.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:21.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:21.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:21.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:21.482 INFO:tasks.rados.rados.0.smithi042.stdout:3136: done (6 left) 2024-09-20T19:36:21.482 INFO:tasks.rados.rados.0.smithi042.stdout:3137: read oid 13 snap -1 2024-09-20T19:36:21.482 INFO:tasks.rados.rados.0.smithi042.stdout:3137: expect (ObjNum 1063 snap 352 seq_num 1063) 2024-09-20T19:36:21.482 INFO:tasks.rados.rados.0.smithi042.stdout:3138: write (excl) oid 20 current snap is 361 2024-09-20T19:36:21.482 INFO:tasks.rados.rados.0.smithi042.stdout:3138: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-20T19:36:21.489 INFO:tasks.rados.rados.0.smithi042.stdout:3138: writing smithi04268048-20 from 551560 to 1142606 tid 1 2024-09-20T19:36:21.495 INFO:tasks.rados.rados.0.smithi042.stdout:3138: writing smithi04268048-20 from 1765494 to 2151560 tid 2 2024-09-20T19:36:21.495 INFO:tasks.rados.rados.0.smithi042.stdout:3139: write (excl) oid 47 current snap is 361 2024-09-20T19:36:21.496 INFO:tasks.rados.rados.0.smithi042.stdout:3139: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-09-20T19:36:21.503 INFO:tasks.rados.rados.0.smithi042.stdout:3139: writing smithi04268048-47 from 568367 to 1234662 tid 1 2024-09-20T19:36:21.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:21.509 INFO:tasks.rados.rados.0.smithi042.stdout:3139: writing smithi04268048-47 from 1707623 to 2168367 tid 2 2024-09-20T19:36:21.509 INFO:tasks.rados.rados.0.smithi042.stdout:3140: copy_from oid 4 from oid 48 current snap is 361 2024-09-20T19:36:21.509 INFO:tasks.rados.rados.0.smithi042.stdout:3128: expect (ObjNum 977 snap 324 seq_num 977) 2024-09-20T19:36:21.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:21 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.581 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:21.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:21.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:36:21.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:21.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: audit 2024-09-20T19:36:19.797302+0000 mgr.y (mgr.24491) 690 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.023352+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.025167+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.424233+0000 mgr.y (mgr.24491) 691 : cluster [DBG] pgmap v1305: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 365 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:20.474819+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.095493+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:21.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:21 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.096912+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:21.680 INFO:tasks.rados.rados.0.smithi042.stdout:3131: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:21.680 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:21.721 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 903763 2024-09-20T19:36:21.746 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:21.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:21.785 INFO:tasks.rados.rados.0.smithi042.stdout:3133: expect (ObjNum 1057 snap 352 seq_num 1057) 2024-09-20T19:36:21.785 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:21.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:21.856 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:21.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:21.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:21.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:21.939 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 887763 2024-09-20T19:36:21.998 INFO:tasks.rados.rados.0.smithi042.stdout:3134: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:36:21.999 INFO:tasks.rados.rados.0.smithi042.stdout:3134: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:36:21.999 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1068 (ObjNum 1201 snap 0 seq_num 1415775984) dirty dne 2024-09-20T19:36:21.999 INFO:tasks.rados.rados.0.smithi042.stdout:3132: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:36:22.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:22.046 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:22.072 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:22.147 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:22.149 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:22.222 INFO:tasks.rados.rados.0.smithi042.stdout:3130: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:36:22.222 INFO:tasks.rados.rados.0.smithi042.stdout:3130: finishing copy_from to smithi04268048-32 2024-09-20T19:36:22.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1144 (ObjNum 1027 snap 337 seq_num 1027) dirty exists 2024-09-20T19:36:22.222 INFO:tasks.rados.rados.0.smithi042.stdout:3137: expect (ObjNum 1063 snap 352 seq_num 1063) 2024-09-20T19:36:22.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:22.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:22.310 INFO:tasks.rados.rados.0.smithi042.stdout:3138: finishing write tid 1 to smithi04268048-20 2024-09-20T19:36:22.310 INFO:tasks.rados.rados.0.smithi042.stdout:3138: finishing write tid 2 to smithi04268048-20 2024-09-20T19:36:22.310 INFO:tasks.rados.rados.0.smithi042.stdout:3138: finishing write tid 3 to smithi04268048-20 2024-09-20T19:36:22.310 INFO:tasks.rados.rados.0.smithi042.stdout:3138: finishing write tid 4 to smithi04268048-20 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3138: oid 20 updating version 0 to 1228 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3138: oid 20 updating version 1228 to 1229 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3138: oid 20 version 1229 is already newer than 1227 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1229 (ObjNum 1080 snap 361 seq_num 1080) dirty exists 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3138: left oid 20 (ObjNum 1080 snap 361 seq_num 1080) 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: finishing write tid 1 to smithi04268048-47 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: finishing write tid 2 to smithi04268048-47 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: finishing write tid 3 to smithi04268048-47 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: finishing write tid 4 to smithi04268048-47 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: oid 47 updating version 0 to 1239 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: oid 47 version 1239 is already newer than 1238 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:3139: oid 47 updating version 1239 to 1240 2024-09-20T19:36:22.311 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1240 (ObjNum 1081 snap 361 seq_num 1081) dirty exists 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3139: left oid 47 (ObjNum 1081 snap 361 seq_num 1081) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3140: finishing copy_from racing read to smithi04268048-4 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3140: finishing copy_from to smithi04268048-4 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 994 (ObjNum 962 snap 319 seq_num 962) dirty exists 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3128: done (9 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3130: done (8 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3131: done (7 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3132: done (6 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3133: done (5 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3134: done (4 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3137: done (3 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3138: done (2 left) 2024-09-20T19:36:22.312 INFO:tasks.rados.rados.0.smithi042.stdout:3139: done (1 left) 2024-09-20T19:36:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:3140: done (0 left) 2024-09-20T19:36:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:3141: read oid 6 snap -1 2024-09-20T19:36:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:3141: expect deleted 2024-09-20T19:36:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:3142: write oid 18 current snap is 361 2024-09-20T19:36:22.313 INFO:tasks.rados.rados.0.smithi042.stdout:3142: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-09-20T19:36:22.331 INFO:tasks.rados.rados.0.smithi042.stdout:3142: writing smithi04268048-18 from 585174 to 1326718 tid 1 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:3142: writing smithi04268048-18 from 1766105 to 2185174 tid 2 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:3143: rollback oid 44 current snap is 361 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 359 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:3142: finishing write tid 1 to smithi04268048-18 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:3141: done (2 left) 2024-09-20T19:36:22.346 INFO:tasks.rados.rados.0.smithi042.stdout:3144: read oid 48 snap -1 2024-09-20T19:36:22.347 INFO:tasks.rados.rados.0.smithi042.stdout:3144: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:36:22.347 INFO:tasks.rados.rados.0.smithi042.stdout:3145: copy_from oid 37 from oid 39 current snap is 361 2024-09-20T19:36:22.347 INFO:tasks.rados.rados.0.smithi042.stdout:3146: copy_from oid 32 from oid 15 current snap is 361 2024-09-20T19:36:22.347 INFO:tasks.rados.rados.0.smithi042.stdout:3147: write oid 24 current snap is 361 2024-09-20T19:36:22.347 INFO:tasks.rados.rados.0.smithi042.stdout:3147: seq_num 1083 ranges {601981=416793,1424587=492702} 2024-09-20T19:36:22.363 INFO:tasks.rados.rados.0.smithi042.stdout:3147: writing smithi04268048-24 from 601981 to 1018774 tid 1 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3147: writing smithi04268048-24 from 1424587 to 1917289 tid 2 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3148: copy_from oid 47 from oid 35 current snap is 361 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3145: finishing copy_from to smithi04268048-37 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3145: got expected ENOENT (src dne) 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3145: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3145: done (6 left) 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3149: rollback oid 7 current snap is 361 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 331 2024-09-20T19:36:22.375 INFO:tasks.rados.rados.0.smithi042.stdout:3144: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:36:22.384 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 723186 2024-09-20T19:36:22.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:22.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:22.477 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3143: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3143: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1203 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3142: finishing write tid 2 to smithi04268048-18 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3142: finishing write tid 3 to smithi04268048-18 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3142: finishing write tid 4 to smithi04268048-18 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3142: oid 18 updating version 0 to 1383 2024-09-20T19:36:22.519 INFO:tasks.rados.rados.0.smithi042.stdout:3142: oid 18 updating version 1383 to 1384 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3142: oid 18 version 1384 is already newer than 1382 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1384 (ObjNum 1082 snap 361 seq_num 1082) dirty exists 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3142: left oid 18 (ObjNum 1082 snap 361 seq_num 1082) 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3146: finishing copy_from racing read to smithi04268048-32 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3142: done (6 left) 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3143: done (5 left) 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3144: done (4 left) 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3150: delete oid 44 current snap is 361 2024-09-20T19:36:22.520 INFO:tasks.rados.rados.0.smithi042.stdout:3146: finishing copy_from to smithi04268048-32 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1144 (ObjNum 1027 snap 337 seq_num 1027) dirty exists 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: finishing write tid 1 to smithi04268048-24 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3149: finishing rollback tid 0 to smithi04268048-7 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: finishing write tid 2 to smithi04268048-24 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: finishing write tid 3 to smithi04268048-24 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: finishing write tid 4 to smithi04268048-24 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: oid 24 updating version 0 to 1153 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: oid 24 updating version 1153 to 1154 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: oid 24 updating version 1154 to 1155 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1155 (ObjNum 1083 snap 361 seq_num 1083) dirty exists 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3147: left oid 24 (ObjNum 1083 snap 361 seq_num 1083) 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:3149: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:36:22.521 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1249 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3148: finishing copy_from racing read to smithi04268048-47 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3148: finishing copy_from to smithi04268048-47 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1236 (ObjNum 1068 snap 354 seq_num 1068) dirty exists 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3146: done (4 left) 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3147: done (3 left) 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3148: done (2 left) 2024-09-20T19:36:22.522 INFO:tasks.rados.rados.0.smithi042.stdout:3149: done (1 left) 2024-09-20T19:36:22.523 INFO:tasks.rados.rados.0.smithi042.stdout:3150: done (0 left) 2024-09-20T19:36:22.523 INFO:tasks.rados.rados.0.smithi042.stdout:3151: delete oid 2 current snap is 361 2024-09-20T19:36:22.526 INFO:tasks.rados.rados.0.smithi042.stdout:3151: done (0 left) 2024-09-20T19:36:22.526 INFO:tasks.rados.rados.0.smithi042.stdout:3152: snap_create 2024-09-20T19:36:22.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:22.550 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:22.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:22 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:22 smithi162 bash[24557]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:22 smithi162 bash[24557]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:22 smithi162 bash[24557]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:22.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:22.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:22.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[32075]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[32075]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.796 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[32075]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[22706]: cluster 2024-09-20T19:36:21.479575+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2024-09-20T19:36:22.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[22706]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:22 smithi042 bash[22706]: audit 2024-09-20T19:36:22.098655+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:36:22.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:22.911 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:22.938 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:22.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:23.041 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:23.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:23.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:23.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:23.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:23.414 INFO:tasks.rados.rados.0.smithi042.stdout:3152: done (0 left) 2024-09-20T19:36:23.415 INFO:tasks.rados.rados.0.smithi042.stdout:3153: write oid 10 current snap is 362 2024-09-20T19:36:23.415 INFO:tasks.rados.rados.0.smithi042.stdout:3153: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-20T19:36:23.431 INFO:tasks.rados.rados.0.smithi042.stdout:3153: writing smithi04268048-10 from 652402 to 1294942 tid 1 2024-09-20T19:36:23.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3153: writing smithi04268048-10 from 1883680 to 2252402 tid 2 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3154: read oid 27 snap -1 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3154: expect (ObjNum 1021 snap 335 seq_num 1021) 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3155: read oid 41 snap -1 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3155: expect (ObjNum 1042 snap 347 seq_num 1042) 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3156: delete oid 9 current snap is 362 2024-09-20T19:36:23.449 INFO:tasks.rados.rados.0.smithi042.stdout:3153: finishing write tid 1 to smithi04268048-10 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: finishing write tid 2 to smithi04268048-10 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: finishing write tid 3 to smithi04268048-10 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: finishing write tid 4 to smithi04268048-10 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: oid 10 updating version 0 to 912 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: oid 10 version 912 is already newer than 910 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: oid 10 version 912 is already newer than 911 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 912 (ObjNum 1086 snap 362 seq_num 1086) dirty exists 2024-09-20T19:36:23.456 INFO:tasks.rados.rados.0.smithi042.stdout:3153: left oid 10 (ObjNum 1086 snap 362 seq_num 1086) 2024-09-20T19:36:23.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3153: done (3 left) 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3156: done (2 left) 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3157: read oid 34 snap -1 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3157: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3158: read oid 3 snap -1 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3158: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:23.472 INFO:tasks.rados.rados.0.smithi042.stdout:3159: delete oid 21 current snap is 362 2024-09-20T19:36:23.477 INFO:tasks.rados.rados.0.smithi042.stdout:3157: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:23.494 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2194286 2024-09-20T19:36:23.550 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 596462 2024-09-20T19:36:23.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:23.578 INFO:tasks.rados.rados.0.smithi042.stdout:3154: expect (ObjNum 1021 snap 335 seq_num 1021) 2024-09-20T19:36:23.581 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 492913 2024-09-20T19:36:23.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:23.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:23 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:23.764 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:23.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:23.835 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.836 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.425901+0000 mgr.y (mgr.24491) 692 : cluster [DBG] pgmap v1308: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 372 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T19:36:23.837 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.474573+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: audit 2024-09-20T19:36:22.476273+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? v1:172.21.15.42:0/3002558601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.480665+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.544076+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:22.545226+0000 osd.5 (osd.5) 270 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.034365+0000 osd.1 (osd.1) 303 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.838 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.036970+0000 osd.1 (osd.1) 304 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.038118+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.040957+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.839 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:23 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.405507+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2024-09-20T19:36:23.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:23.859 INFO:tasks.rados.rados.0.smithi042.stdout:3155: expect (ObjNum 1042 snap 347 seq_num 1042) 2024-09-20T19:36:23.934 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:23.949 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:23.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:23.974 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:23.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:24.037 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:24.077 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:24.103 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:24.125 INFO:tasks.rados.rados.0.smithi042.stdout:3158: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:24.134 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1660748 2024-09-20T19:36:24.187 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:24.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:24.223 INFO:tasks.rados.rados.0.smithi042.stdout:3154: done (4 left) 2024-09-20T19:36:24.223 INFO:tasks.rados.rados.0.smithi042.stdout:3155: done (3 left) 2024-09-20T19:36:24.223 INFO:tasks.rados.rados.0.smithi042.stdout:3157: done (2 left) 2024-09-20T19:36:24.223 INFO:tasks.rados.rados.0.smithi042.stdout:3158: done (1 left) 2024-09-20T19:36:24.223 INFO:tasks.rados.rados.0.smithi042.stdout:3159: done (0 left) 2024-09-20T19:36:24.224 INFO:tasks.rados.rados.0.smithi042.stdout:3160: rollback oid 24 current snap is 362 2024-09-20T19:36:24.224 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 360 2024-09-20T19:36:24.224 INFO:tasks.rados.rados.0.smithi042.stdout:3161: write oid 25 current snap is 362 2024-09-20T19:36:24.224 INFO:tasks.rados.rados.0.smithi042.stdout:3161: seq_num 1089 ranges {702823=584640,2059126=243697} 2024-09-20T19:36:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:24.233 INFO:tasks.rados.rados.0.smithi042.stdout:3161: writing smithi04268048-25 from 702823 to 1287463 tid 1 2024-09-20T19:36:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3161: writing smithi04268048-25 from 2059126 to 2302823 tid 2 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3162: read oid 6 snap -1 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3162: expect deleted 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3163: delete oid 10 current snap is 362 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3160: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:36:24.242 INFO:tasks.rados.rados.0.smithi042.stdout:3160: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:36:24.243 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1157 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:36:24.243 INFO:tasks.rados.rados.0.smithi042.stdout:3161: finishing write tid 1 to smithi04268048-25 2024-09-20T19:36:24.245 INFO:tasks.rados.rados.0.smithi042.stdout:3160: done (3 left) 2024-09-20T19:36:24.245 INFO:tasks.rados.rados.0.smithi042.stdout:3162: done (2 left) 2024-09-20T19:36:24.245 INFO:tasks.rados.rados.0.smithi042.stdout:3163: done (1 left) 2024-09-20T19:36:24.245 INFO:tasks.rados.rados.0.smithi042.stdout:3164: delete oid 36 current snap is 362 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: finishing write tid 2 to smithi04268048-25 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: finishing write tid 3 to smithi04268048-25 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: finishing write tid 4 to smithi04268048-25 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: oid 25 updating version 0 to 1026 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: oid 25 version 1026 is already newer than 1024 2024-09-20T19:36:24.246 INFO:tasks.rados.rados.0.smithi042.stdout:3161: oid 25 version 1026 is already newer than 1025 2024-09-20T19:36:24.247 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 1026 (ObjNum 1089 snap 362 seq_num 1089) dirty exists 2024-09-20T19:36:24.247 INFO:tasks.rados.rados.0.smithi042.stdout:3161: left oid 25 (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:36:24.249 INFO:tasks.rados.rados.0.smithi042.stdout:3161: done (1 left) 2024-09-20T19:36:24.249 INFO:tasks.rados.rados.0.smithi042.stdout:3164: done (0 left) 2024-09-20T19:36:24.249 INFO:tasks.rados.rados.0.smithi042.stdout:3165: snap_create 2024-09-20T19:36:24.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:24.350 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:36:24.363 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:24.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:24.499 INFO:tasks.rados.rados.0.smithi042.stdout:3165: done (0 left) 2024-09-20T19:36:24.499 INFO:tasks.rados.rados.0.smithi042.stdout:3166: read oid 16 snap -1 2024-09-20T19:36:24.499 INFO:tasks.rados.rados.0.smithi042.stdout:3166: expect deleted 2024-09-20T19:36:24.499 INFO:tasks.rados.rados.0.smithi042.stdout:3167: write (excl) oid 2 current snap is 363 2024-09-20T19:36:24.499 INFO:tasks.rados.rados.0.smithi042.stdout:3167: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-20T19:36:24.505 INFO:tasks.rados.rados.0.smithi042.stdout:3167: writing smithi04268048-2 from 753244 to 1163631 tid 1 2024-09-20T19:36:24.512 INFO:tasks.rados.rados.0.smithi042.stdout:3167: writing smithi04268048-2 from 1834572 to 2353244 tid 2 2024-09-20T19:36:24.512 INFO:tasks.rados.rados.0.smithi042.stdout:3168: snap_remove snap 356 2024-09-20T19:36:24.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:24.624 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:24.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.739 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:24 smithi162 bash[24557]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:24.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:24.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:24.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[32075]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.580811+0000 osd.5 (osd.5) 271 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.582262+0000 osd.5 (osd.5) 272 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:36:24.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.688340+0000 osd.2 (osd.2) 265 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:36:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:24 smithi042 bash[22706]: cluster 2024-09-20T19:36:23.692247+0000 osd.2 (osd.2) 266 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:36:24.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:25.012 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:25.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:25.144 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:25.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:25.272 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:25.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:25.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:25.461 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:36:25.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:3168: done (2 left) 2024-09-20T19:36:25.507 INFO:tasks.rados.rados.0.smithi042.stdout:3169: snap_remove snap 358 2024-09-20T19:36:25.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:25.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:25.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:36:25.752 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:36:25.753 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.754 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.755 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.756 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:25 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.018798+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.022519+0000 osd.0 (osd.0) 88 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.426822+0000 mgr.y (mgr.24491) 693 : cluster [DBG] pgmap v1311: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 21 op/s 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.498035+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.655655+0000 osd.2 (osd.2) 267 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:24.659021+0000 osd.2 (osd.2) 268 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.009881+0000 osd.1 (osd.1) 305 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:25 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.013645+0000 osd.1 (osd.1) 306 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:36:25.787 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:25.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:25.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:25.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:26.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:26.041 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:26.130 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:36:26.130 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:36:26.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:26.173 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:26.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:26.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:26.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:26.419 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:26.512 INFO:tasks.rados.rados.0.smithi042.stdout:3169: done (2 left) 2024-09-20T19:36:26.512 INFO:tasks.rados.rados.0.smithi042.stdout:3170: delete oid 17 current snap is 363 2024-09-20T19:36:26.512 INFO:tasks.rados.rados.0.smithi042.stdout:3167: finishing write tid 1 to smithi04268048-2 2024-09-20T19:36:26.512 INFO:tasks.rados.rados.0.smithi042.stdout:3167: finishing write tid 2 to smithi04268048-2 2024-09-20T19:36:26.512 INFO:tasks.rados.rados.0.smithi042.stdout:3167: finishing write tid 3 to smithi04268048-2 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:3167: finishing write tid 4 to smithi04268048-2 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:3167: oid 2 updating version 0 to 1315 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:3167: oid 2 version 1315 is already newer than 1314 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:3167: oid 2 updating version 1315 to 1316 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1316 (ObjNum 1092 snap 363 seq_num 1092) dirty exists 2024-09-20T19:36:26.513 INFO:tasks.rados.rados.0.smithi042.stdout:3167: left oid 2 (ObjNum 1092 snap 363 seq_num 1092) 2024-09-20T19:36:26.523 INFO:tasks.rados.rados.0.smithi042.stdout:3166: done (2 left) 2024-09-20T19:36:26.523 INFO:tasks.rados.rados.0.smithi042.stdout:3167: done (1 left) 2024-09-20T19:36:26.523 INFO:tasks.rados.rados.0.smithi042.stdout:3170: done (0 left) 2024-09-20T19:36:26.523 INFO:tasks.rados.rados.0.smithi042.stdout:3171: delete oid 7 current snap is 363 2024-09-20T19:36:26.533 INFO:tasks.rados.rados.0.smithi042.stdout:3171: done (0 left) 2024-09-20T19:36:26.533 INFO:tasks.rados.rados.0.smithi042.stdout:3172: write (excl) oid 47 current snap is 363 2024-09-20T19:36:26.533 INFO:tasks.rados.rados.0.smithi042.stdout:3172: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-20T19:36:26.539 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:26.543 INFO:tasks.rados.rados.0.smithi042.stdout:3172: writing smithi04268048-47 from 403665 to 1156152 tid 1 2024-09-20T19:36:26.552 INFO:tasks.rados.rados.0.smithi042.stdout:3172: writing smithi04268048-47 from 1610018 to 2124734 tid 2 2024-09-20T19:36:26.552 INFO:tasks.rados.rados.0.smithi042.stdout:3173: delete oid 7 current snap is 363 2024-09-20T19:36:26.552 INFO:tasks.rados.rados.0.smithi042.stdout:3172: finishing write tid 1 to smithi04268048-47 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3173: done (1 left) 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3174: read oid 21 snap 354 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3174: expect (ObjNum 1038 snap 343 seq_num 1038) 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3175: read oid 7 snap -1 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3175: expect deleted 2024-09-20T19:36:26.557 INFO:tasks.rados.rados.0.smithi042.stdout:3176: snap_create 2024-09-20T19:36:26.558 INFO:tasks.rados.rados.0.smithi042.stdout:3172: finishing write tid 2 to smithi04268048-47 2024-09-20T19:36:26.558 INFO:tasks.rados.rados.0.smithi042.stdout:3172: finishing write tid 3 to smithi04268048-47 2024-09-20T19:36:26.559 INFO:tasks.rados.rados.0.smithi042.stdout:3172: finishing write tid 4 to smithi04268048-47 2024-09-20T19:36:26.559 INFO:tasks.rados.rados.0.smithi042.stdout:3172: oid 47 updating version 0 to 1241 2024-09-20T19:36:26.559 INFO:tasks.rados.rados.0.smithi042.stdout:3172: oid 47 updating version 1241 to 1243 2024-09-20T19:36:26.559 INFO:tasks.rados.rados.0.smithi042.stdout:3172: oid 47 version 1243 is already newer than 1242 2024-09-20T19:36:26.560 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1243 (ObjNum 1095 snap 363 seq_num 1095) dirty exists 2024-09-20T19:36:26.560 INFO:tasks.rados.rados.0.smithi042.stdout:3172: left oid 47 (ObjNum 1095 snap 363 seq_num 1095) 2024-09-20T19:36:26.565 INFO:tasks.rados.rados.0.smithi042.stdout:3174: expect (ObjNum 1038 snap 343 seq_num 1038) 2024-09-20T19:36:26.565 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:26.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:36:26.646 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:36:26.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-20T19:36:26.667 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2350484 2024-09-20T19:36:26.667 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.677 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:26 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:26.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:26.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.504877+0000 mon.a (mon.0) 2453 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.577462+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.580695+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.629367+0000 osd.2 (osd.2) 269 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.632147+0000 osd.2 (osd.2) 270 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.995108+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:36:26.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:25.997668+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:36:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:26 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.510884+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2024-09-20T19:36:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:26.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:26.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T19:36:26.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:26.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:27.045 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:27.075 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:27.123 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:27.178 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:27.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:27.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:27.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:27.463 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:27.540 INFO:tasks.rados.rados.0.smithi042.stdout:3172: done (3 left) 2024-09-20T19:36:27.540 INFO:tasks.rados.rados.0.smithi042.stdout:3174: done (2 left) 2024-09-20T19:36:27.540 INFO:tasks.rados.rados.0.smithi042.stdout:3175: done (1 left) 2024-09-20T19:36:27.540 INFO:tasks.rados.rados.0.smithi042.stdout:3176: done (0 left) 2024-09-20T19:36:27.540 INFO:tasks.rados.rados.0.smithi042.stdout:3177: write oid 5 current snap is 364 2024-09-20T19:36:27.541 INFO:tasks.rados.rados.0.smithi042.stdout:3177: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-20T19:36:27.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.545 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.546 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.557 INFO:tasks.rados.rados.0.smithi042.stdout:3177: writing smithi04268048-5 from 437279 to 940264 tid 1 2024-09-20T19:36:27.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:27.575 INFO:tasks.rados.rados.0.smithi042.stdout:3177: writing smithi04268048-5 from 1610629 to 2129014 tid 2 2024-09-20T19:36:27.576 INFO:tasks.rados.rados.0.smithi042.stdout:3178: read oid 42 snap -1 2024-09-20T19:36:27.576 INFO:tasks.rados.rados.0.smithi042.stdout:3178: expect deleted 2024-09-20T19:36:27.576 INFO:tasks.rados.rados.0.smithi042.stdout:3179: delete oid 48 current snap is 364 2024-09-20T19:36:27.576 INFO:tasks.rados.rados.0.smithi042.stdout:3177: finishing write tid 1 to smithi04268048-5 2024-09-20T19:36:27.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.427331+0000 mgr.y (mgr.24491) 694 : cluster [DBG] pgmap v1314: 178 pgs: 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 165 active+clean; 373 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 239 KiB/s wr, 6 op/s 2024-09-20T19:36:27.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.976824+0000 osd.1 (osd.1) 307 : cluster [DBG] 5.c scrub starts 2024-09-20T19:36:27.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:26.979088+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.c scrub ok 2024-09-20T19:36:27.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:27 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.581 INFO:tasks.rados.rados.0.smithi042.stdout:3177: finishing write tid 2 to smithi04268048-5 2024-09-20T19:36:27.582 INFO:tasks.rados.rados.0.smithi042.stdout:3177: finishing write tid 3 to smithi04268048-5 2024-09-20T19:36:27.582 INFO:tasks.rados.rados.0.smithi042.stdout:3178: done (2 left) 2024-09-20T19:36:27.582 INFO:tasks.rados.rados.0.smithi042.stdout:3179: done (1 left) 2024-09-20T19:36:27.582 INFO:tasks.rados.rados.0.smithi042.stdout:3180: snap_remove snap 350 2024-09-20T19:36:27.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:27.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:27.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:27.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.044271+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:36:27.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:27 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.045949+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:36:27.850 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:27.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:27.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:27.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:28.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:28.111 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:28.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:28.240 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:28.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:28.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:28.414 INFO:tasks.rados.rados.0.smithi042.stdout:3180: done (1 left) 2024-09-20T19:36:28.414 INFO:tasks.rados.rados.0.smithi042.stdout:3181: read oid 41 snap -1 2024-09-20T19:36:28.415 INFO:tasks.rados.rados.0.smithi042.stdout:3181: expect (ObjNum 1042 snap 347 seq_num 1042) 2024-09-20T19:36:28.415 INFO:tasks.rados.rados.0.smithi042.stdout:3182: write (excl) oid 39 current snap is 364 2024-09-20T19:36:28.415 INFO:tasks.rados.rados.0.smithi042.stdout:3182: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-20T19:36:28.437 INFO:tasks.rados.rados.0.smithi042.stdout:3182: writing smithi04268048-39 from 470893 to 1124376 tid 1 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:3182: writing smithi04268048-39 from 1727593 to 2249647 tid 2 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:3183: rollback oid 31 current snap is 364 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 339 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:3184: read oid 24 snap -1 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:3184: expect deleted 2024-09-20T19:36:28.451 INFO:tasks.rados.rados.0.smithi042.stdout:3185: snap_remove snap 354 2024-09-20T19:36:28.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:28.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:28.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:28.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:28.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:28 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:28.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:28.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.869 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: cluster 2024-09-20T19:36:27.537431+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: audit 2024-09-20T19:36:28.367657+0000 mon.a (mon.0) 2456 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:28.870 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.871 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:28 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.407622+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2024-09-20T19:36:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:28.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:28.896 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:36:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:36:28.914 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:36:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:36:28.931 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:36:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:36:28.949 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:36:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:36:28.970 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:36:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:36:28.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:28.991 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:36:28.991 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:36:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:29.013 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:36:29.014 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:36:29.035 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:36:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:36:29.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:29.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:29.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:29.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:29.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:29.432 INFO:tasks.rados.rados.0.smithi042.stdout:3185: done (5 left) 2024-09-20T19:36:29.432 INFO:tasks.rados.rados.0.smithi042.stdout:3186: snap_remove snap 361 2024-09-20T19:36:29.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:29.527 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:29.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[22706]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:29 smithi042 bash[32075]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:29.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:29.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.428287+0000 mgr.y (mgr.24491) 695 : cluster [DBG] pgmap v1318: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: audit 2024-09-20T19:36:28.874350+0000 mon.a (mon.0) 2458 : audit [DBG] from='client.? v1:172.21.15.42:0/87889857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.901499+0000 osd.7 (osd.7) 201 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:28.907359+0000 osd.7 (osd.7) 202 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.045746+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.e scrub starts 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.049765+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.e scrub ok 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[24557]: cluster 2024-09-20T19:36:29.429962+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2024-09-20T19:36:29.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:29.915 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:29.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:30.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:30.052 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:30.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:30.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:30.288 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:30.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3186: done (5 left) 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3187: rollback oid 13 current snap is 364 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 352 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3177: finishing write tid 4 to smithi04268048-5 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3177: oid 5 updating version 0 to 1128 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3177: oid 5 version 1128 is already newer than 1127 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3177: oid 5 version 1128 is already newer than 1126 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1128 (ObjNum 1097 snap 364 seq_num 1097) dirty exists 2024-09-20T19:36:30.445 INFO:tasks.rados.rados.0.smithi042.stdout:3177: left oid 5 (ObjNum 1097 snap 364 seq_num 1097) 2024-09-20T19:36:30.446 INFO:tasks.rados.rados.0.smithi042.stdout:3181: expect (ObjNum 1042 snap 347 seq_num 1042) 2024-09-20T19:36:30.446 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:30.454 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:30.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.571 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.572 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:30 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:30.677 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2999892 2024-09-20T19:36:30.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.695 INFO:tasks.rados.rados.0.smithi042.stdout:3182: finishing write tid 1 to smithi04268048-39 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3183: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: finishing write tid 2 to smithi04268048-39 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3183: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 981 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: finishing write tid 3 to smithi04268048-39 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: finishing write tid 4 to smithi04268048-39 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: oid 39 updating version 0 to 1226 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: oid 39 version 1226 is already newer than 1224 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: oid 39 version 1226 is already newer than 1225 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1226 (ObjNum 1099 snap 364 seq_num 1099) dirty exists 2024-09-20T19:36:30.696 INFO:tasks.rados.rados.0.smithi042.stdout:3182: left oid 39 (ObjNum 1099 snap 364 seq_num 1099) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3187: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3177: done (5 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3181: done (4 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3182: done (3 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3183: done (2 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3184: done (1 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3188: copy_from oid 18 from oid 25 current snap is 364 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3187: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3187: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 601 (ObjNum 1063 snap 352 seq_num 1063) dirty exists 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3187: done (1 left) 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3189: rollback oid 37 current snap is 364 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 37 to 335 2024-09-20T19:36:30.697 INFO:tasks.rados.rados.0.smithi042.stdout:3190: snap_remove snap 352 2024-09-20T19:36:30.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.014251+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:36:30.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.018541+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:36:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:30 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.433312+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2024-09-20T19:36:30.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:30.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:30.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:30.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:31.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:31.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:31.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:31.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:31.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.370 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:31.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:31.486 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:31.500 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:31.567 INFO:tasks.rados.rados.0.smithi042.stdout:3190: done (2 left) 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3191: rollback oid 38 current snap is 364 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 357 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3189: finishing rollback tid 0 to smithi04268048-37 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3189: finishing rollback tid 1 to smithi04268048-37 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3189: finishing rollback tid 2 to smithi04268048-37 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1015 (ObjNum 999 snap 327 seq_num 999) dirty exists 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3188: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3188: finishing copy_from to smithi04268048-18 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1026 (ObjNum 1089 snap 362 seq_num 1089) dirty exists 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3188: done (2 left) 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3189: done (1 left) 2024-09-20T19:36:31.568 INFO:tasks.rados.rados.0.smithi042.stdout:3192: snap_remove snap 359 2024-09-20T19:36:31.604 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[22706]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.614 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.615 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:31 smithi042 bash[32075]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:31.725 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:36:31.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: audit 2024-09-20T19:36:29.806635+0000 mgr.y (mgr.24491) 696 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.429034+0000 mgr.y (mgr.24491) 697 : cluster [DBG] pgmap v1320: 178 pgs: 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 167 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-09-20T19:36:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.551424+0000 osd.2 (osd.2) 271 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:36:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:31 smithi162 bash[24557]: cluster 2024-09-20T19:36:30.561187+0000 osd.2 (osd.2) 272 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:36:31.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.883 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:31.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:31.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:36:32.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:32.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:32.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:32.247 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:32.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:32.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3192: done (1 left) 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3193: rollback oid 2 current snap is 364 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 344 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3194: read oid 36 snap -1 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3194: expect deleted 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3195: copy_from oid 41 from oid 43 current snap is 364 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3196: copy_from oid 49 from oid 7 current snap is 364 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3197: read oid 46 snap -1 2024-09-20T19:36:32.580 INFO:tasks.rados.rados.0.smithi042.stdout:3197: expect deleted 2024-09-20T19:36:32.581 INFO:tasks.rados.rados.0.smithi042.stdout:3198: delete oid 15 current snap is 364 2024-09-20T19:36:32.581 INFO:tasks.rados.rados.0.smithi042.stdout:3191: finishing rollback tid 0 to smithi04268048-38 2024-09-20T19:36:32.581 INFO:tasks.rados.rados.0.smithi042.stdout:3191: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:36:32.581 INFO:tasks.rados.rados.0.smithi042.stdout:3191: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:36:32.581 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1155 (ObjNum 904 snap 298 seq_num 904) dirty exists 2024-09-20T19:36:32.589 INFO:tasks.rados.rados.0.smithi042.stdout:3193: finishing rollback tid 0 to smithi04268048-2 2024-09-20T19:36:32.589 INFO:tasks.rados.rados.0.smithi042.stdout:3193: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:36:32.589 INFO:tasks.rados.rados.0.smithi042.stdout:3193: finishing rollback tid 2 to smithi04268048-2 2024-09-20T19:36:32.589 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 1319 (ObjNum 1025 snap 336 seq_num 1025) dirty exists 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3191: done (6 left) 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3193: done (5 left) 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3194: done (4 left) 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3197: done (3 left) 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3198: done (2 left) 2024-09-20T19:36:32.591 INFO:tasks.rados.rados.0.smithi042.stdout:3199: read oid 25 snap -1 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3199: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3200: read oid 20 snap -1 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3200: expect (ObjNum 1080 snap 361 seq_num 1080) 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3201: rollback oid 23 current snap is 364 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 362 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3202: delete oid 27 current snap is 364 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3196: finishing copy_from to smithi04268048-49 2024-09-20T19:36:32.592 INFO:tasks.rados.rados.0.smithi042.stdout:3196: got expected ENOENT (src dne) 2024-09-20T19:36:32.595 INFO:tasks.rados.rados.0.smithi042.stdout:3199: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:36:32.596 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:32.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:32.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:32.670 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1271991 2024-09-20T19:36:32.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3196: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3201: finishing rollback tid 0 to smithi04268048-23 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3201: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3201: finishing rollback tid 2 to smithi04268048-23 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1155 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3195: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:36:32.752 INFO:tasks.rados.rados.0.smithi042.stdout:3200: expect (ObjNum 1080 snap 361 seq_num 1080) 2024-09-20T19:36:32.775 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:32.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:32.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:32 smithi162 bash[24557]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:32.828 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 707255 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3195: finishing copy_from to smithi04268048-41 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 750 (ObjNum 1070 snap 357 seq_num 1070) dirty exists 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3195: done (5 left) 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3196: done (4 left) 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3199: done (3 left) 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3200: done (2 left) 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3201: done (1 left) 2024-09-20T19:36:32.859 INFO:tasks.rados.rados.0.smithi042.stdout:3202: done (0 left) 2024-09-20T19:36:32.860 INFO:tasks.rados.rados.0.smithi042.stdout:3203: read oid 28 snap -1 2024-09-20T19:36:32.860 INFO:tasks.rados.rados.0.smithi042.stdout:3203: expect deleted 2024-09-20T19:36:32.860 INFO:tasks.rados.rados.0.smithi042.stdout:3204: snap_create 2024-09-20T19:36:32.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:32.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:32.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[32075]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.556772+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.558915+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:36:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.564005+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2024-09-20T19:36:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.858674+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:36:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:32 smithi042 bash[22706]: cluster 2024-09-20T19:36:31.860524+0000 osd.7 (osd.7) 204 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:36:33.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:33.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.167 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:33.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:33.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:33.410 INFO:tasks.rados.rados.0.smithi042.stdout:3203: done (1 left) 2024-09-20T19:36:33.410 INFO:tasks.rados.rados.0.smithi042.stdout:3204: done (0 left) 2024-09-20T19:36:33.410 INFO:tasks.rados.rados.0.smithi042.stdout:3205: write oid 12 current snap is 365 2024-09-20T19:36:33.410 INFO:tasks.rados.rados.0.smithi042.stdout:3205: seq_num 1102 ranges {521314=479230,1786686=669381} 2024-09-20T19:36:33.415 INFO:tasks.rados.rados.0.smithi042.stdout:3205: writing smithi04268048-12 from 521314 to 1000544 tid 1 2024-09-20T19:36:33.425 INFO:tasks.rados.rados.0.smithi042.stdout:3205: writing smithi04268048-12 from 1786686 to 2456067 tid 2 2024-09-20T19:36:33.425 INFO:tasks.rados.rados.0.smithi042.stdout:3206: write oid 44 current snap is 365 2024-09-20T19:36:33.425 INFO:tasks.rados.rados.0.smithi042.stdout:3205: finishing write tid 1 to smithi04268048-12 2024-09-20T19:36:33.425 INFO:tasks.rados.rados.0.smithi042.stdout:3206: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-20T19:36:33.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:33.433 INFO:tasks.rados.rados.0.smithi042.stdout:3206: writing smithi04268048-44 from 538121 to 1208953 tid 1 2024-09-20T19:36:33.441 INFO:tasks.rados.rados.0.smithi042.stdout:3206: writing smithi04268048-44 from 1961521 to 2374560 tid 2 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3207: delete oid 22 current snap is 365 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: finishing write tid 2 to smithi04268048-12 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: finishing write tid 3 to smithi04268048-12 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: finishing write tid 4 to smithi04268048-12 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: oid 12 updating version 0 to 1132 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: oid 12 updating version 1132 to 1133 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: oid 12 updating version 1133 to 1134 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 1134 (ObjNum 1102 snap 365 seq_num 1102) dirty exists 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3205: left oid 12 (ObjNum 1102 snap 365 seq_num 1102) 2024-09-20T19:36:33.442 INFO:tasks.rados.rados.0.smithi042.stdout:3206: finishing write tid 1 to smithi04268048-44 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: finishing write tid 2 to smithi04268048-44 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: finishing write tid 3 to smithi04268048-44 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: finishing write tid 4 to smithi04268048-44 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: oid 44 updating version 0 to 1213 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: oid 44 version 1213 is already newer than 1212 2024-09-20T19:36:33.445 INFO:tasks.rados.rados.0.smithi042.stdout:3206: oid 44 version 1213 is already newer than 1211 2024-09-20T19:36:33.446 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1213 (ObjNum 1103 snap 365 seq_num 1103) dirty exists 2024-09-20T19:36:33.446 INFO:tasks.rados.rados.0.smithi042.stdout:3206: left oid 44 (ObjNum 1103 snap 365 seq_num 1103) 2024-09-20T19:36:33.447 INFO:tasks.rados.rados.0.smithi042.stdout:3205: done (2 left) 2024-09-20T19:36:33.447 INFO:tasks.rados.rados.0.smithi042.stdout:3206: done (1 left) 2024-09-20T19:36:33.447 INFO:tasks.rados.rados.0.smithi042.stdout:3207: done (0 left) 2024-09-20T19:36:33.448 INFO:tasks.rados.rados.0.smithi042.stdout:3208: write oid 13 current snap is 365 2024-09-20T19:36:33.448 INFO:tasks.rados.rados.0.smithi042.stdout:3208: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-20T19:36:33.453 INFO:tasks.rados.rados.0.smithi042.stdout:3208: writing smithi04268048-13 from 571735 to 993065 tid 1 2024-09-20T19:36:33.459 INFO:tasks.rados.rados.0.smithi042.stdout:3208: writing smithi04268048-13 from 1562132 to 1978840 tid 2 2024-09-20T19:36:33.459 INFO:tasks.rados.rados.0.smithi042.stdout:3209: write (excl) oid 11 current snap is 365 2024-09-20T19:36:33.459 INFO:tasks.rados.rados.0.smithi042.stdout:3209: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-20T19:36:33.465 INFO:tasks.rados.rados.0.smithi042.stdout:3209: writing smithi04268048-11 from 588542 to 1085121 tid 1 2024-09-20T19:36:33.472 INFO:tasks.rados.rados.0.smithi042.stdout:3209: writing smithi04268048-11 from 1620614 to 2297333 tid 2 2024-09-20T19:36:33.472 INFO:tasks.rados.rados.0.smithi042.stdout:3210: snap_remove snap 364 2024-09-20T19:36:33.531 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:33.554 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:33.596 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:36:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:36:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2234 4.1a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.lock in=8b] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.24276730299999, 2024-09-20T19:36:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0273693370000001, 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2234 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352497+0000", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352499+0000", 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4540000000000001e-06 2024-09-20T19:36:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352507+0000", 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0539999999999995e-06 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352518+0000", 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0154e-05 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352527+0000", 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0250000000000008e-06 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352886+0000", 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035973599999999999 2024-09-20T19:36:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352896+0000", 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4369999999999996e-06 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.371744+0000", 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0188486000000001 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.371982+0000", 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023746300000000001 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.372358+0000", 2024-09-20T19:36:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037604499999999998 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.378302+0000", 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059445499999999998 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379623+0000", 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001320821 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379724+0000", 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010050700000000001 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379866+0000", 2024-09-20T19:36:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014249099999999999 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2592 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94817798633088 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.242719239, 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0268885000000001, 2024-09-20T19:36:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2592 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352545+0000", 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352546+0000", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8299999999999996e-07 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352549+0000", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4929999999999998e-06 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352555+0000", 2024-09-20T19:36:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2230000000000004e-06 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.352562+0000", 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1920000000000002e-06 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.354341+0000", 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017789629999999999 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:30.354350+0000", 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2469999999999997e-06 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.351955+0000", 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99760481000000001 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.352331+0000", 2024-09-20T19:36:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000375922 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:31.379433+0000", 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027102866999999999 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14904.0:4471 7.8 e547/522 7:17ea136f:::smithi04268048-32:head v 547'1046, mlcod=546'1042)", 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 338.245878006, 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018403108000000001, 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349386+0000", 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.349387+0000", 2024-09-20T19:36:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4799999999999998e-07 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360853+0000", 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011466551 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360875+0000", 2024-09-20T19:36:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1763000000000002e-05 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360886+0000", 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360959+0000", 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2787000000000006e-05 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.360986+0000", 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7191000000000001e-05 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.367755+0000", 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067691649999999997 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:55.367789+0000", 2024-09-20T19:36:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3827e-05 2024-09-20T19:36:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6108 7.16 7:6dcf8a02:::smithi04268048-26:head [copy-from ver 1194] snapc 115=[115,112,110,10e] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:33:45.365608+0000", 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.229655686, 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024198581, 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6108 2024-09-20T19:36:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365608+0000", 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365608+0000", 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365610+0000", 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8509999999999999e-06 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365620+0000", 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.431e-06 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365635+0000", 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5262000000000001e-05 2024-09-20T19:36:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365649+0000", 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365769+0000", 2024-09-20T19:36:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120019 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365888+0000", 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119103 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.378953+0000", 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013065257 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.387013+0000", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080599520000000004 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.388587+0000", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015741450000000001 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.389714+0000", 2024-09-20T19:36:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011268109999999999 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.389765+0000", 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0637000000000003e-05 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.389807+0000", 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2187000000000001e-05 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6109 7.16 7:6dcf8a02:::smithi04268048-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e717)", 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:33:45.365686+0000", 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.229578452, 2024-09-20T19:36:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019126311, 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6109 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365686+0000", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365686+0000", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365687+0000", 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3769999999999999e-06 2024-09-20T19:36:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365693+0000", 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0739999999999998e-06 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365707+0000", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3951000000000001e-05 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.365720+0000", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2591e-05 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.366298+0000", 2024-09-20T19:36:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057877099999999995 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.366428+0000", 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129281 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.384600+0000", 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018172421000000001 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.384698+0000", 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7860000000000002e-05 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:45.384812+0000", 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113985 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6182 7.1a 7:5fdc189b:::smithi04268048-19:head [stat,delete] snapc 117=[117,116,10e] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:33:50.419814+0000", 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.17544974800001, 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022056574999999998, 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6182 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419814+0000", 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419814+0000", 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419820+0000", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7130000000000002e-06 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419837+0000", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6753e-05 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419878+0000", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0629e-05 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.419902+0000", 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4609000000000001e-05 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.420082+0000", 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018001700000000001 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.420101+0000", 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.872e-05 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.438550+0000", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01844925 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.438677+0000", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126718 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.438945+0000", 2024-09-20T19:36:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026838699999999998 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.440898+0000", 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019529409999999999 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.440915+0000", 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6711e-05 2024-09-20T19:36:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.441611+0000", 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069590799999999996 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.441787+0000", 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000175728 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:33:50.441871+0000", 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4491000000000001e-05 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6520 7.1a 7:5fdc189b:::smithi04268048-19:head [copy-from ver 816] snapc 128=[128,127,126,124,123,120,11d,118] ondisk+write+known_if_redirected+supports_pool_eio e751)", 2024-09-20T19:36:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:34:19.853382+0000", 2024-09-20T19:36:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 133.74188213100001, 2024-09-20T19:36:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026473921000000001, 2024-09-20T19:36:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6520 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853382+0000", 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853382+0000", 2024-09-20T19:36:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853386+0000", 2024-09-20T19:36:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8530000000000002e-06 2024-09-20T19:36:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853400+0000", 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3633e-05 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853431+0000", 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1612000000000002e-05 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853448+0000", 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6707000000000001e-05 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853629+0000", 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018125699999999999 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853798+0000", 2024-09-20T19:36:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016865200000000001 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.861590+0000", 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077926540000000004 2024-09-20T19:36:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.879320+0000", 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017729465999999999 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.879381+0000", 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0708000000000002e-05 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.879535+0000", 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154406 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.879746+0000", 2024-09-20T19:36:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211094 2024-09-20T19:36:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.879856+0000", 2024-09-20T19:36:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109879 2024-09-20T19:36:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6521 7.1a 7:5fdc189b:::smithi04268048-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e751)", 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:34:19.853488+0000", 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 133.74177575499999, 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021476861, 2024-09-20T19:36:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:36:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6521 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853488+0000", 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853488+0000", 2024-09-20T19:36:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853490+0000", 2024-09-20T19:36:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2139999999999999e-06 2024-09-20T19:36:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853495+0000", 2024-09-20T19:36:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2460000000000003e-06 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853506+0000", 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1483e-05 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.853515+0000", 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5299999999999996e-06 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.854064+0000", 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054898700000000002 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.854146+0000", 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2378000000000003e-05 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.874632+0000", 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020485546 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.874788+0000", 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015590499999999999 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:19.874965+0000", 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000177572 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:6697 7.16 7:6dcf8a02:::smithi04268048-26:head [write 712861~733540,stat] snapc 134=[134,133,132,131,130,12f,12e,12d,12a,129,124,120,118] ondisk+write+known_if_redirected+supports_pool_eio e770)", 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:34:38.399801+0000", 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.19546293400001, 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020162274000000001, 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6697 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.399801+0000", 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.399801+0000", 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.399808+0000", 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9959999999999996e-06 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.401312+0000", 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001503573 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.401352+0000", 2024-09-20T19:36:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0159e-05 2024-09-20T19:36:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.401379+0000", 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6582999999999999e-05 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.401540+0000", 2024-09-20T19:36:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016131199999999999 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.401556+0000", 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6195999999999999e-05 2024-09-20T19:36:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.408369+0000", 2024-09-20T19:36:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068132770000000004 2024-09-20T19:36:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.408461+0000", 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2118000000000001e-05 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.408639+0000", 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000177644 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.412319+0000", 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003680362 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.415874+0000", 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035548519999999998 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.419847+0000", 2024-09-20T19:36:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039722450000000001 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.419902+0000", 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5593000000000001e-05 2024-09-20T19:36:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:34:38.419964+0000", 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1364000000000004e-05 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.682 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7007 7.1a 7:5a4763d9:::smithi04268048-28:12f [read 0~2823774,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792)", 2024-09-20T19:36:33.683 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:00.556318+0000", 2024-09-20T19:36:33.683 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.038945884, 2024-09-20T19:36:33.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019217430000000001, 2024-09-20T19:36:33.683 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.683 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7007 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.684 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556318+0000", 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556318+0000", 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556320+0000", 2024-09-20T19:36:33.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-09-20T19:36:33.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556330+0000", 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0159e-05 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556347+0000", 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.696e-05 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.556362+0000", 2024-09-20T19:36:33.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4696e-05 2024-09-20T19:36:33.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.574042+0000", 2024-09-20T19:36:33.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017680626000000001 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.574330+0000", 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000287235 2024-09-20T19:36:33.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:00.575536+0000", 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001206164 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.692 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7401 7.22 7:47f606bc:::smithi04268048-22:head [copy-from ver 875] snapc 14f=[14f,14e,14d,14c,14a,148,147,146,141,140,137,131,12d,129,124] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:30.430671+0000", 2024-09-20T19:36:33.693 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.164592726000002, 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021881879999999999, 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.694 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7401 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430671+0000", 2024-09-20T19:36:33.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430671+0000", 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430672+0000", 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7899999999999999e-07 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430674+0000", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.419e-06 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430678+0000", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.693e-06 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.430681+0000", 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-20T19:36:33.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.433583+0000", 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029019670000000001 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.433709+0000", 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125838 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-09-20T19:36:33.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.441792+0000", 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080834189999999997 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.451019+0000", 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092266559999999997 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.451118+0000", 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9161999999999997e-05 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.452443+0000", 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001325286 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.452503+0000", 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9206999999999997e-05 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:30.452553+0000", 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0661000000000002e-05 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7558 7.1a 7:5a4763d9:::smithi04268048-28:head [sparse-read 0~1387872,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e837)", 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:45.402000+0000", 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.193264450000001, 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021118446999999999, 2024-09-20T19:36:33.702 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7558 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.402000+0000", 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.402000+0000", 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.402004+0000", 2024-09-20T19:36:33.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6890000000000003e-06 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.402020+0000", 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5906999999999999e-05 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.422481+0000", 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020461044000000001 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.422503+0000", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1641000000000002e-05 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.422701+0000", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000197746 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.422840+0000", 2024-09-20T19:36:33.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139395 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:45.423118+0000", 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000278025 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7605 7.16 7:6dcf8a02:::smithi04268048-26:head [copy-from ver 904] snapc 15d=[15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,14c,147,146,141,140,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e840)", 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:48.478844+0000", 2024-09-20T19:36:33.707 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.116420345000002, 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022031044999999999, 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7605 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478844+0000", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478844+0000", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478846+0000", 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0719999999999998e-06 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478853+0000", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0110000000000003e-06 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478867+0000", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4103e-05 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478873+0000", 2024-09-20T19:36:33.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4389999999999997e-06 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.478941+0000", 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7507000000000003e-05 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.479025+0000", 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4056999999999998e-05 2024-09-20T19:36:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.490302+0000", 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011277002 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.497148+0000", 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068457240000000001 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.500300+0000", 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031527019999999998 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.500743+0000", 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000442423 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.500816+0000", 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3162000000000002e-05 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:48.500875+0000", 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8842999999999999e-05 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.714 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.2.0:1375 7.8 e845/522 7:17ea136f:::smithi04268048-32:15d v 845'1420, mlcod=844'1414)", 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:53.399080+0000", 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.196184312, 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027890439999999999, 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399080+0000", 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399080+0000", 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399082+0000", 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7400000000000001e-06 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399095+0000", 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3563e-05 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399119+0000", 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3750000000000001e-05 2024-09-20T19:36:33.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.399137+0000", 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8096000000000001e-05 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.403151+0000", 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040139800000000003 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.405601+0000", 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024504850000000001 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.426904+0000", 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021302979999999999 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.426970+0000", 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5846000000000006e-05 2024-09-20T19:36:33.721 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7634 7.16 7:6c48c6fe:::smithi04268048-7:head [write 413736~401313] snapc 160=[160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:53.405526+0000", 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.189738609000003, 2024-09-20T19:36:33.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028613184, 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7634 2024-09-20T19:36:33.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.405526+0000", 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.405526+0000", 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.405530+0000", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.583e-06 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.406782+0000", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001252003 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.407206+0000", 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042381199999999997 2024-09-20T19:36:33.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.407230+0000", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4102999999999999e-05 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.411370+0000", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041404170000000004 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.411622+0000", 2024-09-20T19:36:33.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000252017 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.412045+0000", 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042278299999999998 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.427829+0000", 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015783701000000001 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.427888+0000", 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9466999999999998e-05 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.433919+0000", 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060310299999999997 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.434046+0000", 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012621299999999999 2024-09-20T19:36:33.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.434139+0000", 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3054999999999996e-05 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7635 7.16 7:6c48c6fe:::smithi04268048-7:head [write 1326776~403024] snapc 160=[160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:53.408718+0000", 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.186546038000003, 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025624747999999999, 2024-09-20T19:36:33.729 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7635 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.408718+0000", 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.408718+0000", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.408722+0000", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8700000000000002e-06 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.409553+0000", 2024-09-20T19:36:33.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00083130200000000002 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.409588+0000", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4335999999999998e-05 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.409610+0000", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1923e-05 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.414728+0000", 2024-09-20T19:36:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051183319999999997 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.415002+0000", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027456699999999998 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.415342+0000", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033918499999999999 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.427929+0000", 2024-09-20T19:36:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012587666000000001 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.431723+0000", 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003793224 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.434161+0000", 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024387459999999999 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.434262+0000", 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100503 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.434343+0000", 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1094000000000007e-05 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7636 7.16 7:6c48c6fe:::smithi04268048-7:head [write 2413414~457893] snapc 160=[160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-20T19:36:33.735 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:53.414918+0000", 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.180346045999997, 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021032845000000001, 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7636 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.414918+0000", 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.414918+0000", 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.414923+0000", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6630000000000001e-06 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.415750+0000", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082757000000000002 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.415785+0000", 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4391999999999999e-05 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.415808+0000", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2906000000000001e-05 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.427966+0000", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012158331 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.428107+0000", 2024-09-20T19:36:33.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014059399999999999 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.428283+0000", 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017646599999999999 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.431798+0000", 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003515369 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.435868+0000", 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004069189 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.435878+0000", 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0346000000000001e-05 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.435913+0000", 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5277000000000001e-05 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:53.435951+0000", 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7741999999999999e-05 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7649 7.1a 7:5fdc189b:::smithi04268048-19:head [append 0~0] snapc 161=[161,160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:54.404986+0000", 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.190278419999999, 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033902863999999998, 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.742 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7649 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.404986+0000", 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.404986+0000", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.404991+0000", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5130000000000004e-06 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405007+0000", 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5803000000000001e-05 2024-09-20T19:36:33.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405056+0000", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9165000000000001e-05 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405080+0000", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3617000000000001e-05 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405372+0000", 2024-09-20T19:36:33.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029200599999999998 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405390+0000", 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7889000000000001e-05 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.421583+0000", 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016193203999999999 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.421766+0000", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018294599999999999 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.422180+0000", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041382600000000001 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.429135+0000", 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069548429999999996 2024-09-20T19:36:33.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.433661+0000", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045260450000000002 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.438709+0000", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050484709999999997 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.438803+0000", 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3694000000000001e-05 2024-09-20T19:36:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.438889+0000", 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5841999999999996e-05 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7650 7.1a 7:5fdc189b:::smithi04268048-19:head [rollback 161] snapc 161=[161,160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:54.405136+0000", 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.190127943999997, 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034073622999999997, 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.749 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7650 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405136+0000", 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405136+0000", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405139+0000", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3329999999999999e-06 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405147+0000", 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9400000000000008e-06 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405167+0000", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9573e-05 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405183+0000", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5650000000000001e-05 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405516+0000", 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033362499999999999 2024-09-20T19:36:33.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405532+0000", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5693e-05 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.424822+0000", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019290275999999999 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.425093+0000", 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027018700000000002 2024-09-20T19:36:33.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.425631+0000", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053812599999999999 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.429513+0000", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038828439999999999 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.436690+0000", 2024-09-20T19:36:33.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071767109999999997 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439033+0000", 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023425820000000002 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439136+0000", 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010367200000000001 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439210+0000", 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3411000000000006e-05 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14904.0:7651 7.1a 7:5fdc189b:::smithi04268048-19:head [append 0~0] snapc 161=[161,160,15f,15e,15d,15c,15b,15a,159,158,157,156,155,154,153,152,151,150,14f,14e,14d,147,146,141,137,12d,129] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:35:54.405204+0000", 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.190059882, 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034259398000000003, 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:36:33.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14904", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7651 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405204+0000", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405204+0000", 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:36:33.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405206+0000", 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.513e-06 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405212+0000", 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3230000000000003e-06 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:36:33.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405228+0000", 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6010000000000001e-05 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405241+0000", 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2957e-05 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405599+0000", 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035743100000000001 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.405614+0000", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4989e-05 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.426900+0000", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021286431000000001 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.427119+0000", 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021855999999999999 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.427550+0000", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043151700000000002 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.429529+0000", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019792 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.436873+0000", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073440140000000003 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:36:33.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439311+0000", 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024374700000000002 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439397+0000", 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5914000000000004e-05 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:35:54.439464+0000", 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7069000000000004e-05 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:36:33.763 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:36:33.763 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:36:33.763 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:33.791 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:33.792 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.794 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.795 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:33 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.430556+0000 mgr.y (mgr.24491) 698 : cluster [DBG] pgmap v1323: 178 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 159 active+clean; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.576441+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.591581+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.b scrub starts 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:32.597287+0000 osd.2 (osd.2) 276 : cluster [DBG] 5.b scrub ok 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:33 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.406290+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2024-09-20T19:36:33.897 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:33.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:34.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:34.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.173 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:34.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.300 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:36:34.322 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:36:34.323 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:36:34.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:36:34.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.405 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:36:34.412 INFO:tasks.rados.rados.0.smithi042.stdout:3210: done (2 left) 2024-09-20T19:36:34.412 INFO:tasks.rados.rados.0.smithi042.stdout:3211: copy_from oid 16 from oid 49 current snap is 365 2024-09-20T19:36:34.412 INFO:tasks.rados.rados.0.smithi042.stdout:3208: finishing write tid 1 to smithi04268048-13 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: finishing write tid 2 to smithi04268048-13 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: finishing write tid 3 to smithi04268048-13 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: finishing write tid 4 to smithi04268048-13 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: oid 13 updating version 0 to 604 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: oid 13 version 604 is already newer than 602 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: oid 13 version 604 is already newer than 603 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 604 (ObjNum 1105 snap 365 seq_num 1105) dirty exists 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3208: left oid 13 (ObjNum 1105 snap 365 seq_num 1105) 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3209: finishing write tid 1 to smithi04268048-11 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3209: finishing write tid 2 to smithi04268048-11 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3209: finishing write tid 3 to smithi04268048-11 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3209: finishing write tid 4 to smithi04268048-11 2024-09-20T19:36:34.413 INFO:tasks.rados.rados.0.smithi042.stdout:3209: oid 11 updating version 0 to 1090 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3209: oid 11 version 1090 is already newer than 1088 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3209: oid 11 version 1090 is already newer than 1089 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1090 (ObjNum 1106 snap 365 seq_num 1106) dirty exists 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3209: left oid 11 (ObjNum 1106 snap 365 seq_num 1106) 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3208: done (2 left) 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3209: done (1 left) 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3212: write oid 42 current snap is 365 2024-09-20T19:36:34.414 INFO:tasks.rados.rados.0.smithi042.stdout:3212: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-09-20T19:36:34.420 INFO:tasks.rados.rados.0.smithi042.stdout:3212: writing smithi04268048-42 from 605349 to 1177177 tid 1 2024-09-20T19:36:34.426 INFO:tasks.rados.rados.0.smithi042.stdout:3212: writing smithi04268048-42 from 1679096 to 2099473 tid 2 2024-09-20T19:36:34.427 INFO:tasks.rados.rados.0.smithi042.stdout:3213: rollback oid 8 current snap is 365 2024-09-20T19:36:34.427 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 357 2024-09-20T19:36:34.427 INFO:tasks.rados.rados.0.smithi042.stdout:3214: copy_from oid 26 from oid 13 current snap is 365 2024-09-20T19:36:34.427 INFO:tasks.rados.rados.0.smithi042.stdout:3215: write oid 1 current snap is 365 2024-09-20T19:36:34.427 INFO:tasks.rados.rados.0.smithi042.stdout:3215: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-20T19:36:34.431 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:34.435 INFO:tasks.rados.rados.0.smithi042.stdout:3215: writing smithi04268048-1 from 622156 to 1269233 tid 1 2024-09-20T19:36:34.442 INFO:tasks.rados.rados.0.smithi042.stdout:3215: writing smithi04268048-1 from 1737578 to 2417966 tid 2 2024-09-20T19:36:34.442 INFO:tasks.rados.rados.0.smithi042.stdout:3216: snap_create 2024-09-20T19:36:34.443 INFO:tasks.rados.rados.0.smithi042.stdout:3213: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:36:34.443 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 929 (ObjNum 225 snap 0 seq_num 1415671008) dirty dne 2024-09-20T19:36:34.444 INFO:tasks.rados.rados.0.smithi042.stdout:3212: finishing write tid 1 to smithi04268048-42 2024-09-20T19:36:34.445 INFO:tasks.rados.rados.0.smithi042.stdout:3211: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:36:34.448 INFO:tasks.rados.rados.0.smithi042.stdout:3214: finishing copy_from racing read to smithi04268048-26 2024-09-20T19:36:34.451 INFO:tasks.rados.rados.0.smithi042.stdout:3212: finishing write tid 2 to smithi04268048-42 2024-09-20T19:36:34.451 INFO:tasks.rados.rados.0.smithi042.stdout:3212: finishing write tid 3 to smithi04268048-42 2024-09-20T19:36:34.451 INFO:tasks.rados.rados.0.smithi042.stdout:3212: finishing write tid 4 to smithi04268048-42 2024-09-20T19:36:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:3212: oid 42 updating version 0 to 1246 2024-09-20T19:36:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:3212: oid 42 version 1246 is already newer than 1245 2024-09-20T19:36:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:3212: oid 42 updating version 1246 to 1247 2024-09-20T19:36:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1247 (ObjNum 1107 snap 365 seq_num 1107) dirty exists 2024-09-20T19:36:34.452 INFO:tasks.rados.rados.0.smithi042.stdout:3212: left oid 42 (ObjNum 1107 snap 365 seq_num 1107) 2024-09-20T19:36:34.458 INFO:tasks.rados.rados.0.smithi042.stdout:3215: finishing write tid 1 to smithi04268048-1 2024-09-20T19:36:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:3211: finishing copy_from to smithi04268048-16 2024-09-20T19:36:34.460 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1425 (ObjNum 1049 snap 352 seq_num 1049) dirty exists 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: finishing write tid 2 to smithi04268048-1 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: finishing write tid 3 to smithi04268048-1 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: finishing write tid 4 to smithi04268048-1 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: oid 1 updating version 0 to 1134 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: oid 1 version 1134 is already newer than 1132 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: oid 1 version 1134 is already newer than 1133 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1134 (ObjNum 1108 snap 365 seq_num 1108) dirty exists 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3215: left oid 1 (ObjNum 1108 snap 365 seq_num 1108) 2024-09-20T19:36:34.461 INFO:tasks.rados.rados.0.smithi042.stdout:3214: finishing copy_from to smithi04268048-26 2024-09-20T19:36:34.462 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 604 (ObjNum 1105 snap 365 seq_num 1105) dirty exists 2024-09-20T19:36:34.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:36:34.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:34.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:34 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:34.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:34.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.025907+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:36:34.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:33.028051+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:36:34.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: audit 2024-09-20T19:36:33.616360+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.025327+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub starts 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.027251+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.d scrub ok 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.402797+0000 mon.a (mon.0) 2465 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.402824+0000 mon.a (mon.0) 2466 : cluster [INF] Cluster is now healthy 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: audit 2024-09-20T19:36:34.404190+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.42:0/3646328516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.800 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:34 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.408101+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2024-09-20T19:36:34.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:34.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:34.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:34.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:35.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:35.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:35.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:35.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:35.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:35.422 INFO:tasks.rados.rados.0.smithi042.stdout:3211: done (5 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3212: done (4 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3213: done (3 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3214: done (2 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3215: done (1 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3216: done (0 left) 2024-09-20T19:36:35.423 INFO:tasks.rados.rados.0.smithi042.stdout:3217: delete oid 49 current snap is 366 2024-09-20T19:36:35.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.442 INFO:tasks.rados.rados.0.smithi042.stdout:3217: done (0 left) 2024-09-20T19:36:35.442 INFO:tasks.rados.rados.0.smithi042.stdout:3218: read oid 50 snap -1 2024-09-20T19:36:35.442 INFO:tasks.rados.rados.0.smithi042.stdout:3218: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:35.443 INFO:tasks.rados.rados.0.smithi042.stdout:3219: copy_from oid 37 from oid 23 current snap is 366 2024-09-20T19:36:35.443 INFO:tasks.rados.rados.0.smithi042.stdout:3220: delete oid 1 current snap is 366 2024-09-20T19:36:35.448 INFO:tasks.rados.rados.0.smithi042.stdout:3220: done (2 left) 2024-09-20T19:36:35.448 INFO:tasks.rados.rados.0.smithi042.stdout:3221: copy_from oid 3 from oid 36 current snap is 366 2024-09-20T19:36:35.448 INFO:tasks.rados.rados.0.smithi042.stdout:3222: snap_remove snap 365 2024-09-20T19:36:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:35.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:35.573 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[22706]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:35.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:35.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:35.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:35.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:35.839 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:35 smithi042 bash[32075]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:36:35.868 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:36:35.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-20T19:36:35.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.986 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:35.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:36.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:36.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.431131+0000 mgr.y (mgr.24491) 699 : cluster [DBG] pgmap v1327: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-09-20T19:36:36.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:36.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.636476+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.d scrub starts 2024-09-20T19:36:36.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:36.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:34.638352+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.d scrub ok 2024-09-20T19:36:36.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:36.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:35 smithi162 bash[24557]: cluster 2024-09-20T19:36:35.420438+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2024-09-20T19:36:36.088 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:36.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:36.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:36.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:36.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:36.359 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:36.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:36.424 INFO:tasks.rados.rados.0.smithi042.stdout:3222: done (3 left) 2024-09-20T19:36:36.424 INFO:tasks.rados.rados.0.smithi042.stdout:3223: read oid 10 snap -1 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:3223: expect deleted 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:3224: rollback oid 35 current snap is 366 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 353 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:3225: read oid 25 snap -1 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:3225: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:36:36.425 INFO:tasks.rados.rados.0.smithi042.stdout:3226: snap_create 2024-09-20T19:36:36.427 INFO:tasks.rados.rados.0.smithi042.stdout:3221: finishing copy_from to smithi04268048-3 2024-09-20T19:36:36.427 INFO:tasks.rados.rados.0.smithi042.stdout:3221: got expected ENOENT (src dne) 2024-09-20T19:36:36.427 INFO:tasks.rados.rados.0.smithi042.stdout:3221: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:36:36.427 INFO:tasks.rados.rados.0.smithi042.stdout:3218: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:36.464 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1216195 2024-09-20T19:36:36.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:36.506 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:36.529 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 573502 2024-09-20T19:36:36.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:36.618 INFO:tasks.rados.rados.0.smithi042.stdout:3219: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:36:36.618 INFO:tasks.rados.rados.0.smithi042.stdout:3219: finishing copy_from to smithi04268048-37 2024-09-20T19:36:36.618 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1155 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:36:36.618 INFO:tasks.rados.rados.0.smithi042.stdout:3225: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:36:36.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:36.724 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:36.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:36.778 INFO:tasks.rados.rados.0.smithi042.stdout:3224: finishing rollback tid 0 to smithi04268048-35 2024-09-20T19:36:36.778 INFO:tasks.rados.rados.0.smithi042.stdout:3224: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:36:36.779 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1238 (ObjNum 129 snap 0 seq_num 174542016) dirty dne 2024-09-20T19:36:36.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:36.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:36.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:36.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:37.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:37.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:37.157 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:37.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:37.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:37.413 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3218: done (6 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3219: done (5 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3221: done (4 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3223: done (3 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3224: done (2 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3225: done (1 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3226: done (0 left) 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3227: read oid 49 snap -1 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3227: expect deleted 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:3228: rollback oid 19 current snap is 367 2024-09-20T19:36:37.433 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 355 2024-09-20T19:36:37.434 INFO:tasks.rados.rados.0.smithi042.stdout:3229: delete oid 39 current snap is 367 2024-09-20T19:36:37.436 INFO:tasks.rados.rados.0.smithi042.stdout:3228: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:36:37.436 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1172 (ObjNum 289 snap 0 seq_num 173687120) dirty dne 2024-09-20T19:36:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:3227: done (2 left) 2024-09-20T19:36:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:3228: done (1 left) 2024-09-20T19:36:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:3229: done (0 left) 2024-09-20T19:36:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:3230: copy_from oid 31 from oid 22 current snap is 367 2024-09-20T19:36:37.450 INFO:tasks.rados.rados.0.smithi042.stdout:3231: rollback oid 13 current snap is 367 2024-09-20T19:36:37.451 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 299 2024-09-20T19:36:37.451 INFO:tasks.rados.rados.0.smithi042.stdout:3232: delete oid 12 current snap is 367 2024-09-20T19:36:37.452 INFO:tasks.rados.rados.0.smithi042.stdout:3230: finishing copy_from to smithi04268048-31 2024-09-20T19:36:37.453 INFO:tasks.rados.rados.0.smithi042.stdout:3230: got expected ENOENT (src dne) 2024-09-20T19:36:37.453 INFO:tasks.rados.rados.0.smithi042.stdout:3230: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:36:37.454 INFO:tasks.rados.rados.0.smithi042.stdout:3231: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:36:37.454 INFO:tasks.rados.rados.0.smithi042.stdout:3230: done (2 left) 2024-09-20T19:36:37.454 INFO:tasks.rados.rados.0.smithi042.stdout:3232: done (1 left) 2024-09-20T19:36:37.455 INFO:tasks.rados.rados.0.smithi042.stdout:3233: delete oid 46 current snap is 367 2024-09-20T19:36:37.455 INFO:tasks.rados.rados.0.smithi042.stdout:3231: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:36:37.455 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 606 (ObjNum 129 snap 0 seq_num 1415457976) dirty dne 2024-09-20T19:36:37.458 INFO:tasks.rados.rados.0.smithi042.stdout:3231: done (1 left) 2024-09-20T19:36:37.458 INFO:tasks.rados.rados.0.smithi042.stdout:3233: done (0 left) 2024-09-20T19:36:37.458 INFO:tasks.rados.rados.0.smithi042.stdout:3234: write (excl) oid 24 current snap is 367 2024-09-20T19:36:37.458 INFO:tasks.rados.rados.0.smithi042.stdout:3234: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-20T19:36:37.465 INFO:tasks.rados.rados.0.smithi042.stdout:3234: writing smithi04268048-24 from 722998 to 1137922 tid 1 2024-09-20T19:36:37.470 INFO:tasks.rados.rados.0.smithi042.stdout:3234: writing smithi04268048-24 from 1572117 to 2147159 tid 2 2024-09-20T19:36:37.470 INFO:tasks.rados.rados.0.smithi042.stdout:3235: snap_create 2024-09-20T19:36:37.471 INFO:tasks.rados.rados.0.smithi042.stdout:3234: finishing write tid 1 to smithi04268048-24 2024-09-20T19:36:37.477 INFO:tasks.rados.rados.0.smithi042.stdout:3234: finishing write tid 2 to smithi04268048-24 2024-09-20T19:36:37.477 INFO:tasks.rados.rados.0.smithi042.stdout:3234: finishing write tid 3 to smithi04268048-24 2024-09-20T19:36:37.477 INFO:tasks.rados.rados.0.smithi042.stdout:3234: finishing write tid 4 to smithi04268048-24 2024-09-20T19:36:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3234: oid 24 updating version 0 to 1428 2024-09-20T19:36:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3234: oid 24 version 1428 is already newer than 1426 2024-09-20T19:36:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3234: oid 24 version 1428 is already newer than 1427 2024-09-20T19:36:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1428 (ObjNum 1114 snap 367 seq_num 1114) dirty exists 2024-09-20T19:36:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3234: left oid 24 (ObjNum 1114 snap 367 seq_num 1114) 2024-09-20T19:36:37.516 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:37.543 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:37.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:37.673 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:37 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:37.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.020263+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:36:37.787 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.022987+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.419882+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:36.432815+0000 mgr.y (mgr.24491) 700 : cluster [DBG] pgmap v1330: 178 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 154 active+clean; 384 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.014409+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.788 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:37 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.017047+0000 osd.7 (osd.7) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:37.880 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:36:37.908 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:37.953 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:37.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:38.056 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:38.082 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:38.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:38.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:38.315 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:38.397 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:38.412 INFO:tasks.rados.rados.0.smithi042.stdout:3234: done (1 left) 2024-09-20T19:36:38.412 INFO:tasks.rados.rados.0.smithi042.stdout:3235: done (0 left) 2024-09-20T19:36:38.412 INFO:tasks.rados.rados.0.smithi042.stdout:3236: copy_from oid 2 from oid 46 current snap is 368 2024-09-20T19:36:38.413 INFO:tasks.rados.rados.0.smithi042.stdout:3237: delete oid 24 current snap is 368 2024-09-20T19:36:38.434 INFO:tasks.rados.rados.0.smithi042.stdout:3237: done (1 left) 2024-09-20T19:36:38.434 INFO:tasks.rados.rados.0.smithi042.stdout:3238: write oid 25 current snap is 368 2024-09-20T19:36:38.434 INFO:tasks.rados.rados.0.smithi042.stdout:3238: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-20T19:36:38.442 INFO:tasks.rados.rados.0.smithi042.stdout:3238: writing smithi04268048-25 from 756612 to 1322034 tid 1 2024-09-20T19:36:38.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.450 INFO:tasks.rados.rados.0.smithi042.stdout:3238: writing smithi04268048-25 from 2089081 to 2667792 tid 2 2024-09-20T19:36:38.450 INFO:tasks.rados.rados.0.smithi042.stdout:3239: write (excl) oid 26 current snap is 368 2024-09-20T19:36:38.450 INFO:tasks.rados.rados.0.smithi042.stdout:3239: seq_num 1117 ranges {773419=640671,2147563=438722} 2024-09-20T19:36:38.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:38 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.458 INFO:tasks.rados.rados.0.smithi042.stdout:3239: writing smithi04268048-26 from 773419 to 1414090 tid 1 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3239: writing smithi04268048-26 from 2147563 to 2586285 tid 2 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3240: read oid 34 snap -1 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3240: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3241: write oid 9 current snap is 368 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3236: finishing copy_from to smithi04268048-2 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3236: got expected ENOENT (src dne) 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3236: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3238: finishing write tid 1 to smithi04268048-25 2024-09-20T19:36:38.465 INFO:tasks.rados.rados.0.smithi042.stdout:3241: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-20T19:36:38.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.472 INFO:tasks.rados.rados.0.smithi042.stdout:3241: writing smithi04268048-9 from 790226 to 1222499 tid 1 2024-09-20T19:36:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3241: writing smithi04268048-9 from 1806045 to 2388425 tid 2 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3236: done (4 left) 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3242: rollback oid 15 current snap is 368 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 331 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: finishing write tid 2 to smithi04268048-25 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: finishing write tid 3 to smithi04268048-25 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: finishing write tid 4 to smithi04268048-25 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: oid 25 updating version 0 to 1044 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: oid 25 version 1044 is already newer than 1043 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:3238: oid 25 version 1044 is already newer than 1042 2024-09-20T19:36:38.479 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 1044 (ObjNum 1116 snap 368 seq_num 1116) dirty exists 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3238: left oid 25 (ObjNum 1116 snap 368 seq_num 1116) 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3239: finishing write tid 1 to smithi04268048-26 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3238: done (4 left) 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3243: write oid 41 current snap is 368 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3239: finishing write tid 2 to smithi04268048-26 2024-09-20T19:36:38.480 INFO:tasks.rados.rados.0.smithi042.stdout:3243: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-09-20T19:36:38.484 INFO:tasks.rados.rados.0.smithi042.stdout:3243: writing smithi04268048-41 from 407033 to 914555 tid 1 2024-09-20T19:36:38.489 INFO:tasks.rados.rados.0.smithi042.stdout:3243: writing smithi04268048-41 from 1464527 to 1906918 tid 2 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3243: writing smithi04268048-41 from 2345976 to 2807033 tid 3 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3244: copy_from oid 36 from oid 23 current snap is 368 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: finishing write tid 3 to smithi04268048-26 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: finishing write tid 4 to smithi04268048-26 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: oid 26 updating version 0 to 1251 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: oid 26 updating version 1251 to 1252 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: oid 26 version 1252 is already newer than 1250 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1252 (ObjNum 1117 snap 368 seq_num 1117) dirty exists 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: left oid 26 (ObjNum 1117 snap 368 seq_num 1117) 2024-09-20T19:36:38.495 INFO:tasks.rados.rados.0.smithi042.stdout:3239: done (5 left) 2024-09-20T19:36:38.496 INFO:tasks.rados.rados.0.smithi042.stdout:3245: copy_from oid 1 from oid 18 current snap is 368 2024-09-20T19:36:38.496 INFO:tasks.rados.rados.0.smithi042.stdout:3240: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:38.496 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 471898 2024-09-20T19:36:38.529 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:38.571 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1124729 2024-09-20T19:36:38.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.594 INFO:tasks.rados.rados.0.smithi042.stdout:3241: finishing write tid 1 to smithi04268048-9 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3242: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3242: finishing rollback tid 2 to smithi04268048-15 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1147 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: finishing write tid 2 to smithi04268048-9 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: finishing write tid 3 to smithi04268048-9 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: finishing write tid 4 to smithi04268048-9 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: oid 9 updating version 0 to 1095 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: oid 9 version 1095 is already newer than 1093 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: oid 9 version 1095 is already newer than 1094 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1095 (ObjNum 1118 snap 368 seq_num 1118) dirty exists 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3241: left oid 9 (ObjNum 1118 snap 368 seq_num 1118) 2024-09-20T19:36:38.595 INFO:tasks.rados.rados.0.smithi042.stdout:3243: finishing write tid 1 to smithi04268048-41 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: finishing write tid 2 to smithi04268048-41 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: finishing write tid 3 to smithi04268048-41 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: finishing write tid 4 to smithi04268048-41 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: finishing write tid 5 to smithi04268048-41 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: oid 41 updating version 0 to 1231 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: oid 41 version 1231 is already newer than 1228 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: oid 41 version 1231 is already newer than 1229 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: oid 41 version 1231 is already newer than 1230 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1231 (ObjNum 1119 snap 368 seq_num 1119) dirty exists 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3243: left oid 41 (ObjNum 1119 snap 368 seq_num 1119) 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3245: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3244: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3245: finishing copy_from to smithi04268048-1 2024-09-20T19:36:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1026 (ObjNum 1089 snap 362 seq_num 1089) dirty exists 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3244: finishing copy_from to smithi04268048-36 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 1155 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3240: done (5 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3241: done (4 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3242: done (3 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3243: done (2 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3244: done (1 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3245: done (0 left) 2024-09-20T19:36:38.597 INFO:tasks.rados.rados.0.smithi042.stdout:3246: delete oid 47 current snap is 368 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3246: done (0 left) 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3247: rollback oid 23 current snap is 368 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 23 to 335 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3248: read oid 16 snap -1 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3248: expect (ObjNum 1049 snap 352 seq_num 1049) 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3249: read oid 2 snap -1 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3249: expect (ObjNum 1025 snap 336 seq_num 1025) 2024-09-20T19:36:38.599 INFO:tasks.rados.rados.0.smithi042.stdout:3250: read oid 36 snap -1 2024-09-20T19:36:38.600 INFO:tasks.rados.rados.0.smithi042.stdout:3250: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:38.600 INFO:tasks.rados.rados.0.smithi042.stdout:3251: snap_remove snap 366 2024-09-20T19:36:38.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:38.695 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.696 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.431814+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.826064+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.827676+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:38 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.408663+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2024-09-20T19:36:38.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.731 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:38.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:38.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:38.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:38.991 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:38.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:39.093 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:39.119 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:39.222 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:39.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:39.380 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:39.418 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:39.440 INFO:tasks.rados.rados.0.smithi042.stdout:3251: done (4 left) 2024-09-20T19:36:39.440 INFO:tasks.rados.rados.0.smithi042.stdout:3252: read oid 10 snap -1 2024-09-20T19:36:39.440 INFO:tasks.rados.rados.0.smithi042.stdout:3252: expect deleted 2024-09-20T19:36:39.440 INFO:tasks.rados.rados.0.smithi042.stdout:3253: write oid 7 current snap is 368 2024-09-20T19:36:39.440 INFO:tasks.rados.rados.0.smithi042.stdout:3253: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-20T19:36:39.451 INFO:tasks.rados.rados.0.smithi042.stdout:3253: writing smithi04268048-7 from 440647 to 1098667 tid 1 2024-09-20T19:36:39.457 INFO:tasks.rados.rados.0.smithi042.stdout:3253: writing smithi04268048-7 from 1581491 to 2027551 tid 2 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:3253: writing smithi04268048-7 from 2800822 to 2840647 tid 3 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:3254: rollback oid 18 current snap is 368 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 339 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:3247: finishing rollback tid 0 to smithi04268048-23 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:3247: finishing rollback tid 1 to smithi04268048-23 2024-09-20T19:36:39.462 INFO:tasks.rados.rados.0.smithi042.stdout:3247: finishing rollback tid 2 to smithi04268048-23 2024-09-20T19:36:39.463 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1158 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:36:39.463 INFO:tasks.rados.rados.0.smithi042.stdout:3250: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:39.465 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1753444 2024-09-20T19:36:39.478 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 4530 2024-09-20T19:36:39.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.491 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[32075]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.492 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.493 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.494 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:39 smithi042 bash[22706]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.502 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:39.504 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:39.530 INFO:tasks.rados.rados.0.smithi042.stdout:3248: expect (ObjNum 1049 snap 352 seq_num 1049) 2024-09-20T19:36:39.546 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1924401 2024-09-20T19:36:39.589 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:39.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.666591+0000 osd.2 (osd.2) 279 : cluster [DBG] 5.a scrub starts 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:37.668692+0000 osd.2 (osd.2) 280 : cluster [DBG] 5.a scrub ok 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.433401+0000 mgr.y (mgr.24491) 701 : cluster [DBG] pgmap v1333: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.474740+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.476152+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.599314+0000 osd.5 (osd.5) 275 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:36:39.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[24557]: cluster 2024-09-20T19:36:38.603422+0000 osd.5 (osd.5) 276 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:36:39.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:39.720 INFO:tasks.rados.rados.0.smithi042.stdout:3249: expect (ObjNum 1025 snap 336 seq_num 1025) 2024-09-20T19:36:39.720 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:39.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:39.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:39.850 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:39.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:39.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3253: finishing write tid 1 to smithi04268048-7 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3253: finishing write tid 2 to smithi04268048-7 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3247: done (6 left) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3248: done (5 left) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3249: done (4 left) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3250: done (3 left) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3252: done (2 left) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3255: read oid 2 snap 355 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3255: expect (ObjNum 1052 snap 352 seq_num 1052) 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3256: read oid 48 snap -1 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3256: expect deleted 2024-09-20T19:36:39.917 INFO:tasks.rados.rados.0.smithi042.stdout:3257: read oid 50 snap -1 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3257: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3258: delete oid 43 current snap is 368 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: finishing write tid 3 to smithi04268048-7 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: finishing write tid 4 to smithi04268048-7 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: finishing write tid 5 to smithi04268048-7 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: oid 7 updating version 0 to 1253 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: oid 7 updating version 1253 to 1255 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: oid 7 updating version 1255 to 1256 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: oid 7 version 1256 is already newer than 1254 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1256 (ObjNum 1121 snap 368 seq_num 1121) dirty exists 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3253: left oid 7 (ObjNum 1121 snap 368 seq_num 1121) 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3254: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3254: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:36:39.918 INFO:tasks.rados.rados.0.smithi042.stdout:3254: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:36:39.919 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1430 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:36:39.920 INFO:tasks.rados.rados.0.smithi042.stdout:3253: done (5 left) 2024-09-20T19:36:39.920 INFO:tasks.rados.rados.0.smithi042.stdout:3254: done (4 left) 2024-09-20T19:36:39.920 INFO:tasks.rados.rados.0.smithi042.stdout:3256: done (3 left) 2024-09-20T19:36:39.920 INFO:tasks.rados.rados.0.smithi042.stdout:3258: done (2 left) 2024-09-20T19:36:39.920 INFO:tasks.rados.rados.0.smithi042.stdout:3259: copy_from oid 49 from oid 29 current snap is 368 2024-09-20T19:36:39.921 INFO:tasks.rados.rados.0.smithi042.stdout:3260: write oid 27 current snap is 368 2024-09-20T19:36:39.921 INFO:tasks.rados.rados.0.smithi042.stdout:3260: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-09-20T19:36:39.925 INFO:tasks.rados.rados.0.smithi042.stdout:3260: writing smithi04268048-27 from 474261 to 882779 tid 1 2024-09-20T19:36:39.932 INFO:tasks.rados.rados.0.smithi042.stdout:3260: writing smithi04268048-27 from 1582102 to 2031831 tid 2 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3260: writing smithi04268048-27 from 2739315 to 2874261 tid 3 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3261: delete oid 41 current snap is 368 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3259: finishing copy_from to smithi04268048-49 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3259: got expected ENOENT (src dne) 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3259: finishing copy_from racing read to smithi04268048-49 2024-09-20T19:36:39.938 INFO:tasks.rados.rados.0.smithi042.stdout:3257: expect (ObjNum 995 snap 326 seq_num 995) 2024-09-20T19:36:39.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:39.996 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:40.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:40.075 INFO:tasks.rados.rados.0.smithi042.stdout:3255: expect (ObjNum 1052 snap 352 seq_num 1052) 2024-09-20T19:36:40.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:40.154 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:40.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.266 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2706182 2024-09-20T19:36:40.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:40.301 INFO:tasks.rados.rados.0.smithi042.stdout:3260: finishing write tid 1 to smithi04268048-27 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: finishing write tid 2 to smithi04268048-27 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: finishing write tid 3 to smithi04268048-27 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: finishing write tid 4 to smithi04268048-27 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: finishing write tid 5 to smithi04268048-27 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: oid 27 updating version 0 to 1252 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: oid 27 updating version 1252 to 1253 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: oid 27 version 1253 is already newer than 1251 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: oid 27 updating version 1253 to 1254 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1254 (ObjNum 1123 snap 368 seq_num 1123) dirty exists 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3260: left oid 27 (ObjNum 1123 snap 368 seq_num 1123) 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3255: done (4 left) 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3257: done (3 left) 2024-09-20T19:36:40.302 INFO:tasks.rados.rados.0.smithi042.stdout:3259: done (2 left) 2024-09-20T19:36:40.303 INFO:tasks.rados.rados.0.smithi042.stdout:3260: done (1 left) 2024-09-20T19:36:40.303 INFO:tasks.rados.rados.0.smithi042.stdout:3261: done (0 left) 2024-09-20T19:36:40.303 INFO:tasks.rados.rados.0.smithi042.stdout:3262: snap_create 2024-09-20T19:36:40.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:40.431 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3262: done (0 left) 2024-09-20T19:36:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3263: read oid 28 snap -1 2024-09-20T19:36:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3263: expect deleted 2024-09-20T19:36:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3264: write (excl) oid 50 current snap is 369 2024-09-20T19:36:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3264: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3264: writing smithi04268048-50 from 507875 to 1066891 tid 1 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3264: writing smithi04268048-50 from 1699066 to 2436111 tid 2 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3265: rollback oid 10 current snap is 369 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 10 to 368 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3263: done (2 left) 2024-09-20T19:36:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3266: read oid 29 snap 355 2024-09-20T19:36:40.491 INFO:tasks.rados.rados.0.smithi042.stdout:3266: expect deleted 2024-09-20T19:36:40.491 INFO:tasks.rados.rados.0.smithi042.stdout:3267: snap_remove snap 299 2024-09-20T19:36:40.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:40 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:40.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.641 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.438416+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.737065+0000 osd.7 (osd.7) 209 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:39.739606+0000 osd.7 (osd.7) 210 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:36:40.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.112224+0000 osd.6 (osd.6) 196 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:36:40.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:40 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.118756+0000 osd.6 (osd.6) 197 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:36:40.657 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:40.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:40.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:40.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:40.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:41.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:41.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:41.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:41.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:41.446 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3267: done (3 left) 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3268: snap_create 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3265: finishing rollback tid 1 to smithi04268048-10 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 913 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3264: finishing write tid 1 to smithi04268048-50 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3264: finishing write tid 2 to smithi04268048-50 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3264: finishing write tid 3 to smithi04268048-50 2024-09-20T19:36:41.474 INFO:tasks.rados.rados.0.smithi042.stdout:3264: finishing write tid 4 to smithi04268048-50 2024-09-20T19:36:41.475 INFO:tasks.rados.rados.0.smithi042.stdout:3264: oid 50 updating version 0 to 1098 2024-09-20T19:36:41.475 INFO:tasks.rados.rados.0.smithi042.stdout:3264: oid 50 updating version 1098 to 1099 2024-09-20T19:36:41.475 INFO:tasks.rados.rados.0.smithi042.stdout:3264: oid 50 version 1099 is already newer than 1097 2024-09-20T19:36:41.475 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1099 (ObjNum 1125 snap 369 seq_num 1125) dirty exists 2024-09-20T19:36:41.475 INFO:tasks.rados.rados.0.smithi042.stdout:3264: left oid 50 (ObjNum 1125 snap 369 seq_num 1125) 2024-09-20T19:36:41.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:41.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:41 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.575 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:41.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:41.706 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.707 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.708 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: audit 2024-09-20T19:36:39.814843+0000 mgr.y (mgr.24491) 702 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.434008+0000 mgr.y (mgr.24491) 703 : cluster [DBG] pgmap v1335: 178 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 167 active+clean; 404 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.3 MiB/s rd, 4.8 MiB/s wr, 21 op/s 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.443912+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.611884+0000 osd.5 (osd.5) 277 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.613730+0000 osd.5 (osd.5) 278 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.783660+0000 osd.7 (osd.7) 211 : cluster [DBG] 7.5 scrub starts 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.791816+0000 osd.7 (osd.7) 212 : cluster [DBG] 7.5 scrub ok 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.970290+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:40.973892+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.710 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.071819+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:36:41.711 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.711 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:41 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.073489+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:36:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:41.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:41.840 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:41.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:41.971 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:41.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:42.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:42.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:42.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:42.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:42.376 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:36:42.376 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:36:42.395 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:36:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:36:42.413 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:36:42.414 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:36:42.431 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:36:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:36:42.452 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:36:42.452 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:36:42.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3264: done (3 left) 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3265: done (2 left) 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3266: done (1 left) 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3268: done (0 left) 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3269: write oid 16 current snap is 370 2024-09-20T19:36:42.472 INFO:tasks.rados.rados.0.smithi042.stdout:3269: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-20T19:36:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:42.479 INFO:tasks.rados.rados.0.smithi042.stdout:3269: writing smithi04268048-16 from 524682 to 1275300 tid 1 2024-09-20T19:36:42.482 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:36:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:36:42.485 INFO:tasks.rados.rados.0.smithi042.stdout:3269: writing smithi04268048-16 from 1757548 to 2354604 tid 2 2024-09-20T19:36:42.485 INFO:tasks.rados.rados.0.smithi042.stdout:3270: write (excl) oid 39 current snap is 370 2024-09-20T19:36:42.485 INFO:tasks.rados.rados.0.smithi042.stdout:3270: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-20T19:36:42.492 INFO:tasks.rados.rados.0.smithi042.stdout:3270: writing smithi04268048-39 from 541489 to 967356 tid 1 2024-09-20T19:36:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:42.501 INFO:tasks.rados.rados.0.smithi042.stdout:3270: writing smithi04268048-39 from 1416030 to 2156744 tid 2 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3270: writing smithi04268048-39 from 2732654 to 2941489 tid 3 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3271: read oid 12 snap -1 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3271: expect deleted 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3272: read oid 6 snap -1 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3272: expect deleted 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3273: read oid 22 snap 357 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3273: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3274: write (excl) oid 32 current snap is 370 2024-09-20T19:36:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3274: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-09-20T19:36:42.513 INFO:tasks.rados.rados.0.smithi042.stdout:3274: writing smithi04268048-32 from 558296 to 1059412 tid 1 2024-09-20T19:36:42.519 INFO:tasks.rados.rados.0.smithi042.stdout:3274: writing smithi04268048-32 from 1474512 to 2075237 tid 2 2024-09-20T19:36:42.523 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:36:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:36:42.526 INFO:tasks.rados.rados.0.smithi042.stdout:3274: writing smithi04268048-32 from 2760077 to 2958296 tid 3 2024-09-20T19:36:42.526 INFO:tasks.rados.rados.0.smithi042.stdout:3275: read oid 49 snap -1 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3275: expect deleted 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3276: snap_create 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: finishing write tid 1 to smithi04268048-16 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: finishing write tid 2 to smithi04268048-16 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: finishing write tid 3 to smithi04268048-16 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3270: finishing write tid 1 to smithi04268048-39 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: finishing write tid 4 to smithi04268048-16 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: oid 16 updating version 0 to 1430 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: oid 16 updating version 1430 to 1431 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: oid 16 updating version 1431 to 1432 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1432 (ObjNum 1126 snap 370 seq_num 1126) dirty exists 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3269: left oid 16 (ObjNum 1126 snap 370 seq_num 1126) 2024-09-20T19:36:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3270: finishing write tid 2 to smithi04268048-39 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: finishing write tid 3 to smithi04268048-39 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: finishing write tid 4 to smithi04268048-39 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3274: finishing write tid 1 to smithi04268048-32 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: finishing write tid 5 to smithi04268048-39 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: oid 39 updating version 0 to 1233 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: oid 39 updating version 1233 to 1234 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: oid 39 updating version 1234 to 1236 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: oid 39 version 1236 is already newer than 1235 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1236 (ObjNum 1127 snap 370 seq_num 1127) dirty exists 2024-09-20T19:36:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3270: left oid 39 (ObjNum 1127 snap 370 seq_num 1127) 2024-09-20T19:36:42.529 INFO:tasks.rados.rados.0.smithi042.stdout:3273: expect (ObjNum 731 snap 240 seq_num 731) 2024-09-20T19:36:42.546 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:36:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:36:42.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:42 smithi162 bash[24557]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.581 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 439899 2024-09-20T19:36:42.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:42.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:42.675 INFO:tasks.rados.rados.0.smithi042.stdout:3274: finishing write tid 2 to smithi04268048-32 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: finishing write tid 3 to smithi04268048-32 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: finishing write tid 4 to smithi04268048-32 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: finishing write tid 5 to smithi04268048-32 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: oid 32 updating version 0 to 1511 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: oid 32 updating version 1511 to 1512 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: oid 32 updating version 1512 to 1514 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: oid 32 version 1514 is already newer than 1513 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1514 (ObjNum 1128 snap 370 seq_num 1128) dirty exists 2024-09-20T19:36:42.676 INFO:tasks.rados.rados.0.smithi042.stdout:3274: left oid 32 (ObjNum 1128 snap 370 seq_num 1128) 2024-09-20T19:36:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[32075]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.464476+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.582651+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.584109+0000 osd.5 (osd.5) 280 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.983854+0000 osd.4 (osd.4) 139 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:41.997943+0000 osd.4 (osd.4) 140 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.049106+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:36:42.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.051898+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:36:42.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:42 smithi042 bash[22706]: audit 2024-09-20T19:36:42.356257+0000 mon.a (mon.0) 2476 : audit [DBG] from='client.? v1:172.21.15.42:0/3091280940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:36:42.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:42.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:42.887 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:42.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:42.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:43.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:43.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:43.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:43.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:43.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:43.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:43.405 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3269: done (7 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3270: done (6 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3271: done (5 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3272: done (4 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3273: done (3 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3274: done (2 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3275: done (1 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3276: done (0 left) 2024-09-20T19:36:43.411 INFO:tasks.rados.rados.0.smithi042.stdout:3277: snap_create 2024-09-20T19:36:43.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:43.533 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:43.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:43.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:43.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:43 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:43.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.434855+0000 mgr.y (mgr.24491) 704 : cluster [DBG] pgmap v1338: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 401 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:42.470418+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2024-09-20T19:36:43.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: audit 2024-09-20T19:36:43.403043+0000 mon.a (mon.0) 2478 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:43.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: audit 2024-09-20T19:36:43.405661+0000 mon.a (mon.0) 2479 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:43.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:43 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.410186+0000 mon.a (mon.0) 2480 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2024-09-20T19:36:43.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:43.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:43.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:44.025 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:44.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:44.089 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:36:44.089 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:36:44.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:44.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:44.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:44.307 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:44.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:44.417 INFO:tasks.rados.rados.0.smithi042.stdout:3277: done (0 left) 2024-09-20T19:36:44.417 INFO:tasks.rados.rados.0.smithi042.stdout:3278: copy_from oid 27 from oid 46 current snap is 372 2024-09-20T19:36:44.417 INFO:tasks.rados.rados.0.smithi042.stdout:3279: snap_create 2024-09-20T19:36:44.428 INFO:tasks.rados.rados.0.smithi042.stdout:3278: finishing copy_from to smithi04268048-27 2024-09-20T19:36:44.429 INFO:tasks.rados.rados.0.smithi042.stdout:3278: got expected ENOENT (src dne) 2024-09-20T19:36:44.429 INFO:tasks.rados.rados.0.smithi042.stdout:3278: finishing copy_from racing read to smithi04268048-27 2024-09-20T19:36:44.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:44.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:44.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:44.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.616 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.617 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:44 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:44.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:44.802 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.029911+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.033555+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:44 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.415989+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2024-09-20T19:36:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:44.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:44.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:44.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:45.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:45.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:45.325 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:45.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:45.422 INFO:tasks.rados.rados.0.smithi042.stdout:3278: done (1 left) 2024-09-20T19:36:45.422 INFO:tasks.rados.rados.0.smithi042.stdout:3279: done (0 left) 2024-09-20T19:36:45.422 INFO:tasks.rados.rados.0.smithi042.stdout:3280: snap_remove snap 362 2024-09-20T19:36:45.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:45.582 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:45.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.653 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[32075]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.654 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:45 smithi042 bash[22706]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:45.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.910016+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:43.911417+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.435797+0000 mgr.y (mgr.24491) 705 : cluster [DBG] pgmap v1342: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.557899+0000 osd.5 (osd.5) 281 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.560057+0000 osd.5 (osd.5) 282 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:45 smithi162 bash[24557]: cluster 2024-09-20T19:36:45.420970+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2024-09-20T19:36:45.840 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:45.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:45.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:45.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:45.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:46.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:46.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:46.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:46.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:46.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:46.375 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:46.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:46.426 INFO:tasks.rados.rados.0.smithi042.stdout:3280: done (0 left) 2024-09-20T19:36:46.426 INFO:tasks.rados.rados.0.smithi042.stdout:3281: snap_remove snap 360 2024-09-20T19:36:46.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:46 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:46.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:46.633 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.905198+0000 osd.1 (osd.1) 315 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:44.910625+0000 osd.1 (osd.1) 316 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:36:46.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:46 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.424153+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2024-09-20T19:36:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:46.738 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:36:46.738 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:36:46.738 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:36:46.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:36:46.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:46.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:46.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:46.905 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:46.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:47.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:47.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:47.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:47.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:47.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:47.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:47.423 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:3281: done (0 left) 2024-09-20T19:36:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:3282: read oid 40 snap 331 2024-09-20T19:36:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:3282: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:36:47.439 INFO:tasks.rados.rados.0.smithi042.stdout:3283: snap_remove snap 353 2024-09-20T19:36:47.446 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:36:47.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:47.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:47 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:47.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:47.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:47.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.057057+0000 osd.0 (osd.0) 91 : cluster [DBG] 7.18 scrub starts 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.063452+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.18 scrub ok 2024-09-20T19:36:47.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.436288+0000 mgr.y (mgr.24491) 706 : cluster [DBG] pgmap v1345: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 563 KiB/s wr, 4 op/s 2024-09-20T19:36:47.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:47 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.437267+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2024-09-20T19:36:47.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:47.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:48.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:48.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:48.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:48.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:48.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:48.419 INFO:tasks.rados.rados.0.smithi042.stdout:3283: done (1 left) 2024-09-20T19:36:48.419 INFO:tasks.rados.rados.0.smithi042.stdout:3284: snap_remove snap 339 2024-09-20T19:36:48.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:48.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:48.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.582 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.583 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:48 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:36:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:36:48.619 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:36:48.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-20T19:36:48.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:48.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.697626+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:46.701265+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.073660+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.075949+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.025844+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.027585+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:36:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:48 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.417889+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2024-09-20T19:36:48.833 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:48.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.980 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:48.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:49.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:49.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:49.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:49.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:49.346 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:49.422 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3284: done (1 left) 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3285: copy_from oid 6 from oid 34 current snap is 373 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3286: read oid 18 snap 371 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3286: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3287: read oid 44 snap -1 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3287: expect (ObjNum 1103 snap 365 seq_num 1103) 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3288: read oid 50 snap -1 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3288: expect (ObjNum 1125 snap 369 seq_num 1125) 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3289: read oid 16 snap 331 2024-09-20T19:36:49.423 INFO:tasks.rados.rados.0.smithi042.stdout:3289: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:49.424 INFO:tasks.rados.rados.0.smithi042.stdout:3290: delete oid 2 current snap is 373 2024-09-20T19:36:49.424 INFO:tasks.rados.rados.0.smithi042.stdout:3282: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:36:49.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:49.481 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.499 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:49.516 INFO:tasks.rados.rados.0.smithi042.stdout:3286: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:36:49.527 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1313736 2024-09-20T19:36:49.541 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.572 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:49.605 INFO:tasks.rados.rados.0.smithi042.stdout:3287: expect (ObjNum 1103 snap 365 seq_num 1103) 2024-09-20T19:36:49.605 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:49.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.670 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[32075]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.672 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.673 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:49 smithi042 bash[22706]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.681 INFO:tasks.rados.rados.0.smithi042.stdout:3288: expect (ObjNum 1125 snap 369 seq_num 1125) 2024-09-20T19:36:49.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:49.742 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.727958+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:47.730610+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.437272+0000 mgr.y (mgr.24491) 707 : cluster [DBG] pgmap v1348: 178 pgs: 8 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 161 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 6 op/s 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.502131+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.d scrub starts 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.503643+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.d scrub ok 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:48.912584+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.42:0/3724540527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:48.912934+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25961 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:36:49.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.944991+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:48.949451+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.009328+0000 osd.4 (osd.4) 145 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.011615+0000 osd.4 (osd.4) 146 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.409646+0000 mon.a (mon.0) 2487 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: audit 2024-09-20T19:36:49.411287+0000 mon.a (mon.0) 2488 : audit [INF] from='client.25961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[24557]: cluster 2024-09-20T19:36:49.415976+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2024-09-20T19:36:49.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:49.815 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 849217 2024-09-20T19:36:49.863 INFO:tasks.rados.rados.0.smithi042.stdout:3289: expect (ObjNum 967 snap 320 seq_num 967) 2024-09-20T19:36:49.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:49.870 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1347898 2024-09-20T19:36:49.880 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.890 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:49.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:49.904 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:49.929 INFO:tasks.rados.rados.0.smithi042.stdout:3285: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:36:49.929 INFO:tasks.rados.rados.0.smithi042.stdout:3285: finishing copy_from to smithi04268048-6 2024-09-20T19:36:49.929 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1232 (ObjNum 911 snap 299 seq_num 911) dirty exists 2024-09-20T19:36:49.929 INFO:tasks.rados.rados.0.smithi042.stdout:3282: done (6 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3285: done (5 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3286: done (4 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3287: done (3 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3288: done (2 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3289: done (1 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3290: done (0 left) 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3291: rollback oid 13 current snap is 373 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 357 2024-09-20T19:36:49.930 INFO:tasks.rados.rados.0.smithi042.stdout:3292: write (excl) oid 19 current snap is 373 2024-09-20T19:36:49.931 INFO:tasks.rados.rados.0.smithi042.stdout:3292: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-20T19:36:49.936 INFO:tasks.rados.rados.0.smithi042.stdout:3292: writing smithi04268048-19 from 591910 to 1243524 tid 1 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3292: writing smithi04268048-19 from 1875123 to 2479517 tid 2 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3293: read oid 39 snap -1 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3293: expect (ObjNum 1127 snap 370 seq_num 1127) 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3294: delete oid 43 current snap is 373 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3291: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3291: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 608 (ObjNum 1063 snap 352 seq_num 1063) dirty exists 2024-09-20T19:36:49.943 INFO:tasks.rados.rados.0.smithi042.stdout:3292: finishing write tid 1 to smithi04268048-19 2024-09-20T19:36:49.946 INFO:tasks.rados.rados.0.smithi042.stdout:3291: done (3 left) 2024-09-20T19:36:49.946 INFO:tasks.rados.rados.0.smithi042.stdout:3294: done (2 left) 2024-09-20T19:36:49.946 INFO:tasks.rados.rados.0.smithi042.stdout:3295: write (excl) oid 49 current snap is 373 2024-09-20T19:36:49.946 INFO:tasks.rados.rados.0.smithi042.stdout:3295: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-09-20T19:36:49.952 INFO:tasks.rados.rados.0.smithi042.stdout:3295: writing smithi04268048-49 from 625524 to 1027636 tid 1 2024-09-20T19:36:49.959 INFO:tasks.rados.rados.0.smithi042.stdout:3295: writing smithi04268048-49 from 1592087 to 2200150 tid 2 2024-09-20T19:36:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:3295: writing smithi04268048-49 from 2869769 to 3025524 tid 3 2024-09-20T19:36:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:3296: snap_create 2024-09-20T19:36:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:3292: finishing write tid 2 to smithi04268048-19 2024-09-20T19:36:49.964 INFO:tasks.rados.rados.0.smithi042.stdout:3293: expect (ObjNum 1127 snap 370 seq_num 1127) 2024-09-20T19:36:49.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:36:50.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:3292: finishing write tid 3 to smithi04268048-19 2024-09-20T19:36:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:3292: finishing write tid 4 to smithi04268048-19 2024-09-20T19:36:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:3292: oid 19 updating version 0 to 1176 2024-09-20T19:36:50.037 INFO:tasks.rados.rados.0.smithi042.stdout:3292: oid 19 version 1176 is already newer than 1174 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3292: oid 19 version 1176 is already newer than 1175 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1176 (ObjNum 1130 snap 373 seq_num 1130) dirty exists 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3292: left oid 19 (ObjNum 1130 snap 373 seq_num 1130) 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: finishing write tid 1 to smithi04268048-49 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: finishing write tid 2 to smithi04268048-49 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: finishing write tid 3 to smithi04268048-49 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: finishing write tid 4 to smithi04268048-49 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: finishing write tid 5 to smithi04268048-49 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: oid 49 updating version 0 to 1528 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: oid 49 updating version 1528 to 1530 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: oid 49 version 1530 is already newer than 1527 2024-09-20T19:36:50.038 INFO:tasks.rados.rados.0.smithi042.stdout:3295: oid 49 version 1530 is already newer than 1529 2024-09-20T19:36:50.039 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1530 (ObjNum 1132 snap 373 seq_num 1132) dirty exists 2024-09-20T19:36:50.039 INFO:tasks.rados.rados.0.smithi042.stdout:3295: left oid 49 (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:36:50.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:50.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:50.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:50.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:50.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3292: done (3 left) 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3293: done (2 left) 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3295: done (1 left) 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3296: done (0 left) 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3297: read oid 9 snap -1 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3297: expect (ObjNum 1118 snap 368 seq_num 1118) 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3298: write (excl) oid 16 current snap is 374 2024-09-20T19:36:50.493 INFO:tasks.rados.rados.0.smithi042.stdout:3298: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-09-20T19:36:50.504 INFO:tasks.rados.rados.0.smithi042.stdout:3298: writing smithi04268048-16 from 642331 to 1236045 tid 1 2024-09-20T19:36:50.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.516 INFO:tasks.rados.rados.0.smithi042.stdout:3298: writing smithi04268048-16 from 1766922 to 2518643 tid 2 2024-09-20T19:36:50.516 INFO:tasks.rados.rados.0.smithi042.stdout:3299: write (excl) oid 45 current snap is 374 2024-09-20T19:36:50.516 INFO:tasks.rados.rados.0.smithi042.stdout:3299: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-20T19:36:50.527 INFO:tasks.rados.rados.0.smithi042.stdout:3299: writing smithi04268048-45 from 659138 to 1328101 tid 1 2024-09-20T19:36:50.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:50.540 INFO:tasks.rados.rados.0.smithi042.stdout:3299: writing smithi04268048-45 from 2109051 to 2720783 tid 2 2024-09-20T19:36:50.540 INFO:tasks.rados.rados.0.smithi042.stdout:3300: snap_create 2024-09-20T19:36:50.540 INFO:tasks.rados.rados.0.smithi042.stdout:3297: expect (ObjNum 1118 snap 368 seq_num 1118) 2024-09-20T19:36:50.575 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1652283 2024-09-20T19:36:50.601 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1786241 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: finishing write tid 1 to smithi04268048-16 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: finishing write tid 2 to smithi04268048-16 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: finishing write tid 3 to smithi04268048-16 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: finishing write tid 4 to smithi04268048-16 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: oid 16 updating version 0 to 1435 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: oid 16 version 1435 is already newer than 1434 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: oid 16 version 1435 is already newer than 1433 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1435 (ObjNum 1133 snap 374 seq_num 1133) dirty exists 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3298: left oid 16 (ObjNum 1133 snap 374 seq_num 1133) 2024-09-20T19:36:50.618 INFO:tasks.rados.rados.0.smithi042.stdout:3299: finishing write tid 1 to smithi04268048-45 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: finishing write tid 2 to smithi04268048-45 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: finishing write tid 3 to smithi04268048-45 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: finishing write tid 4 to smithi04268048-45 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: oid 45 updating version 0 to 1532 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: oid 45 updating version 1532 to 1533 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: oid 45 version 1533 is already newer than 1531 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1533 (ObjNum 1134 snap 374 seq_num 1134) dirty exists 2024-09-20T19:36:50.619 INFO:tasks.rados.rados.0.smithi042.stdout:3299: left oid 45 (ObjNum 1134 snap 374 seq_num 1134) 2024-09-20T19:36:50.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:50.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:50.919 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:50.947 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:50.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:51.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:51.079 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:51.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:51.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:51.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:51.497 INFO:tasks.rados.rados.0.smithi042.stdout:3297: done (3 left) 2024-09-20T19:36:51.497 INFO:tasks.rados.rados.0.smithi042.stdout:3298: done (2 left) 2024-09-20T19:36:51.497 INFO:tasks.rados.rados.0.smithi042.stdout:3299: done (1 left) 2024-09-20T19:36:51.497 INFO:tasks.rados.rados.0.smithi042.stdout:3300: done (0 left) 2024-09-20T19:36:51.497 INFO:tasks.rados.rados.0.smithi042.stdout:3301: read oid 31 snap -1 2024-09-20T19:36:51.498 INFO:tasks.rados.rados.0.smithi042.stdout:3301: expect (ObjNum 1023 snap 336 seq_num 1023) 2024-09-20T19:36:51.498 INFO:tasks.rados.rados.0.smithi042.stdout:3302: copy_from oid 9 from oid 41 current snap is 375 2024-09-20T19:36:51.498 INFO:tasks.rados.rados.0.smithi042.stdout:3303: snap_remove snap 344 2024-09-20T19:36:51.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:51.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.714 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:51 smithi162 bash[24557]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:51.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.859 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:51.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[32075]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: audit 2024-09-20T19:36:49.822732+0000 mgr.y (mgr.24491) 708 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: cluster 2024-09-20T19:36:50.438349+0000 mgr.y (mgr.24491) 709 : cluster [DBG] pgmap v1350: 178 pgs: 10 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 157 active+clean; 380 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 979 KiB/s wr, 6 op/s 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:51 smithi042 bash[22706]: cluster 2024-09-20T19:36:50.491361+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2024-09-20T19:36:51.964 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:51.999 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:51.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:52.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.124 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:52.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:52.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:52.490 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.502 INFO:tasks.rados.rados.0.smithi042.stdout:3303: done (2 left) 2024-09-20T19:36:52.503 INFO:tasks.rados.rados.0.smithi042.stdout:3304: snap_remove snap 335 2024-09-20T19:36:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:52.618 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:36:52.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:52.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:52.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[22706]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[22706]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[22706]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[22706]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[32075]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[32075]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[32075]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:52 smithi042 bash[32075]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:52 smithi162 bash[24557]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:52 smithi162 bash[24557]: cluster 2024-09-20T19:36:51.495033+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2024-09-20T19:36:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:52 smithi162 bash[24557]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:52 smithi162 bash[24557]: cluster 2024-09-20T19:36:52.498484+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2024-09-20T19:36:52.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:52.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:52.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:53.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:53.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:53.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:53.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:53.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:53.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:53.425 INFO:tasks.rados.rados.0.smithi042.stdout:3304: done (2 left) 2024-09-20T19:36:53.425 INFO:tasks.rados.rados.0.smithi042.stdout:3305: rollback oid 28 current snap is 375 2024-09-20T19:36:53.425 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 324 2024-09-20T19:36:53.426 INFO:tasks.rados.rados.0.smithi042.stdout:3302: finishing copy_from to smithi04268048-9 2024-09-20T19:36:53.426 INFO:tasks.rados.rados.0.smithi042.stdout:3302: got expected ENOENT (src dne) 2024-09-20T19:36:53.426 INFO:tasks.rados.rados.0.smithi042.stdout:3302: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:36:53.426 INFO:tasks.rados.rados.0.smithi042.stdout:3301: expect (ObjNum 1023 snap 336 seq_num 1023) 2024-09-20T19:36:53.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:53.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:53.643 INFO:tasks.rados.rados.0.smithi042.stdout:3305: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:36:53.643 INFO:tasks.rados.rados.0.smithi042.stdout:3305: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1178 (ObjNum 956 snap 318 seq_num 956) dirty exists 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:3301: done (2 left) 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:3302: done (1 left) 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:3305: done (0 left) 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:3306: copy_from oid 14 from oid 15 current snap is 375 2024-09-20T19:36:53.644 INFO:tasks.rados.rados.0.smithi042.stdout:3307: snap_remove snap 374 2024-09-20T19:36:53.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:53.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:53 smithi162 bash[24557]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:53 smithi162 bash[24557]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:53 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:53 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[22706]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[22706]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[32075]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[32075]: cluster 2024-09-20T19:36:52.440007+0000 mgr.y (mgr.24491) 710 : cluster [DBG] pgmap v1353: 178 pgs: 12 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 150 active+clean; 385 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-09-20T19:36:53.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.810 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:53 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.419404+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2024-09-20T19:36:53.824 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:53.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:53.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:54.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:54.085 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:54.188 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:54.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:54.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:54.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:54.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:54.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:54.533 INFO:tasks.rados.rados.0.smithi042.stdout:3307: done (1 left) 2024-09-20T19:36:54.533 INFO:tasks.rados.rados.0.smithi042.stdout:3308: read oid 33 snap -1 2024-09-20T19:36:54.533 INFO:tasks.rados.rados.0.smithi042.stdout:3308: expect (ObjNum 1057 snap 352 seq_num 1057) 2024-09-20T19:36:54.542 INFO:tasks.rados.rados.0.smithi042.stdout:3309: rollback oid 35 current snap is 375 2024-09-20T19:36:54.542 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 35 to 369 2024-09-20T19:36:54.542 INFO:tasks.rados.rados.0.smithi042.stdout:3310: write (excl) oid 8 current snap is 375 2024-09-20T19:36:54.543 INFO:tasks.rados.rados.0.smithi042.stdout:3310: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-09-20T19:36:54.552 INFO:tasks.rados.rados.0.smithi042.stdout:3310: writing smithi04268048-8 from 675945 to 1420157 tid 1 2024-09-20T19:36:54.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:54.575 INFO:tasks.rados.rados.0.smithi042.stdout:3310: writing smithi04268048-8 from 2167533 to 2922923 tid 2 2024-09-20T19:36:54.576 INFO:tasks.rados.rados.0.smithi042.stdout:3311: write (excl) oid 30 current snap is 375 2024-09-20T19:36:54.576 INFO:tasks.rados.rados.0.smithi042.stdout:3311: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-20T19:36:54.590 INFO:tasks.rados.rados.0.smithi042.stdout:3311: writing smithi04268048-30 from 692752 to 1112213 tid 1 2024-09-20T19:36:54.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:54.605 INFO:tasks.rados.rados.0.smithi042.stdout:3311: writing smithi04268048-30 from 1826015 to 2441416 tid 2 2024-09-20T19:36:54.611 INFO:tasks.rados.rados.0.smithi042.stdout:3311: writing smithi04268048-30 from 2979461 to 3092752 tid 3 2024-09-20T19:36:54.611 INFO:tasks.rados.rados.0.smithi042.stdout:3312: read oid 42 snap -1 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3312: expect (ObjNum 1107 snap 365 seq_num 1107) 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3313: read oid 34 snap -1 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3313: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3314: write oid 43 current snap is 375 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3306: finishing copy_from racing read to smithi04268048-14 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3306: finishing copy_from to smithi04268048-14 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1147 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3309: finishing rollback tid 1 to smithi04268048-35 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1238 (ObjNum 129 snap 0 seq_num 1415628120) dirty dne 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3310: finishing write tid 1 to smithi04268048-8 2024-09-20T19:36:54.612 INFO:tasks.rados.rados.0.smithi042.stdout:3308: expect (ObjNum 1057 snap 352 seq_num 1057) 2024-09-20T19:36:54.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:54.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:54.843 INFO:tasks.rados.rados.0.smithi042.stdout:3310: finishing write tid 2 to smithi04268048-8 2024-09-20T19:36:54.843 INFO:tasks.rados.rados.0.smithi042.stdout:3310: finishing write tid 3 to smithi04268048-8 2024-09-20T19:36:54.843 INFO:tasks.rados.rados.0.smithi042.stdout:3310: finishing write tid 4 to smithi04268048-8 2024-09-20T19:36:54.843 INFO:tasks.rados.rados.0.smithi042.stdout:3310: oid 8 updating version 0 to 1235 2024-09-20T19:36:54.843 INFO:tasks.rados.rados.0.smithi042.stdout:3310: oid 8 version 1235 is already newer than 1234 2024-09-20T19:36:54.844 INFO:tasks.rados.rados.0.smithi042.stdout:3310: oid 8 version 1235 is already newer than 1233 2024-09-20T19:36:54.844 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1235 (ObjNum 1135 snap 375 seq_num 1135) dirty exists 2024-09-20T19:36:54.844 INFO:tasks.rados.rados.0.smithi042.stdout:3310: left oid 8 (ObjNum 1135 snap 375 seq_num 1135) 2024-09-20T19:36:54.844 INFO:tasks.rados.rados.0.smithi042.stdout:3311: finishing write tid 1 to smithi04268048-30 2024-09-20T19:36:54.844 INFO:tasks.rados.rados.0.smithi042.stdout:3314: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-09-20T19:36:54.850 INFO:tasks.rados.rados.0.smithi042.stdout:3314: writing smithi04268048-43 from 709559 to 1204269 tid 1 2024-09-20T19:36:54.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:54.866 INFO:tasks.rados.rados.0.smithi042.stdout:3314: writing smithi04268048-43 from 1884497 to 2643556 tid 2 2024-09-20T19:36:54.866 INFO:tasks.rados.rados.0.smithi042.stdout:3306: done (7 left) 2024-09-20T19:36:54.866 INFO:tasks.rados.rados.0.smithi042.stdout:3308: done (6 left) 2024-09-20T19:36:54.867 INFO:tasks.rados.rados.0.smithi042.stdout:3309: done (5 left) 2024-09-20T19:36:54.867 INFO:tasks.rados.rados.0.smithi042.stdout:3310: done (4 left) 2024-09-20T19:36:54.867 INFO:tasks.rados.rados.0.smithi042.stdout:3315: copy_from oid 50 from oid 3 current snap is 375 2024-09-20T19:36:54.867 INFO:tasks.rados.rados.0.smithi042.stdout:3311: finishing write tid 2 to smithi04268048-30 2024-09-20T19:36:54.867 INFO:tasks.rados.rados.0.smithi042.stdout:3312: expect (ObjNum 1107 snap 365 seq_num 1107) 2024-09-20T19:36:54.892 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:54.949 INFO:tasks.rados.rados.0.smithi042.stdout:3311: finishing write tid 3 to smithi04268048-30 2024-09-20T19:36:54.949 INFO:tasks.rados.rados.0.smithi042.stdout:3311: finishing write tid 4 to smithi04268048-30 2024-09-20T19:36:54.949 INFO:tasks.rados.rados.0.smithi042.stdout:3313: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:36:54.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:54.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:54.983 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 290363 2024-09-20T19:36:55.028 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2146943 2024-09-20T19:36:55.041 INFO:tasks.rados.rados.0.smithi042.stdout:3311: finishing write tid 5 to smithi04268048-30 2024-09-20T19:36:55.041 INFO:tasks.rados.rados.0.smithi042.stdout:3311: oid 30 updating version 0 to 1472 2024-09-20T19:36:55.041 INFO:tasks.rados.rados.0.smithi042.stdout:3311: oid 30 version 1472 is already newer than 1471 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3311: oid 30 updating version 1472 to 1473 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3311: oid 30 version 1473 is already newer than 1470 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1473 (ObjNum 1136 snap 375 seq_num 1136) dirty exists 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3311: left oid 30 (ObjNum 1136 snap 375 seq_num 1136) 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: finishing write tid 1 to smithi04268048-43 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3311: done (4 left) 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3312: done (3 left) 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3313: done (2 left) 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3316: copy_from oid 36 from oid 22 current snap is 375 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: finishing write tid 2 to smithi04268048-43 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: finishing write tid 3 to smithi04268048-43 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: finishing write tid 4 to smithi04268048-43 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: oid 43 updating version 0 to 754 2024-09-20T19:36:55.042 INFO:tasks.rados.rados.0.smithi042.stdout:3314: oid 43 version 754 is already newer than 752 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3314: oid 43 version 754 is already newer than 753 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 754 (ObjNum 1137 snap 375 seq_num 1137) dirty exists 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3314: left oid 43 (ObjNum 1137 snap 375 seq_num 1137) 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3315: finishing copy_from racing read to smithi04268048-50 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3315: finishing copy_from to smithi04268048-50 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 967 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3314: done (2 left) 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3315: done (1 left) 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3317: write oid 10 current snap is 375 2024-09-20T19:36:55.043 INFO:tasks.rados.rados.0.smithi042.stdout:3317: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-09-20T19:36:55.048 INFO:tasks.rados.rados.0.smithi042.stdout:3317: writing smithi04268048-10 from 726366 to 1296325 tid 1 2024-09-20T19:36:55.054 INFO:tasks.rados.rados.0.smithi042.stdout:3317: writing smithi04268048-10 from 1826626 to 2329343 tid 2 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3317: writing smithi04268048-10 from 2801601 to 3126366 tid 3 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3318: rollback oid 39 current snap is 375 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 331 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3316: finishing copy_from to smithi04268048-36 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3316: got expected ENOENT (src dne) 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3316: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3317: finishing write tid 1 to smithi04268048-10 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3316: done (2 left) 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3319: write (excl) oid 37 current snap is 375 2024-09-20T19:36:55.059 INFO:tasks.rados.rados.0.smithi042.stdout:3319: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-20T19:36:55.067 INFO:tasks.rados.rados.0.smithi042.stdout:3319: writing smithi04268048-37 from 743173 to 1388381 tid 1 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3319: writing smithi04268048-37 from 1885108 to 2647836 tid 2 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3320: delete oid 42 current snap is 375 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: finishing write tid 2 to smithi04268048-10 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: finishing write tid 3 to smithi04268048-10 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: finishing write tid 4 to smithi04268048-10 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: finishing write tid 5 to smithi04268048-10 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: oid 10 updating version 0 to 936 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: oid 10 version 936 is already newer than 934 2024-09-20T19:36:55.075 INFO:tasks.rados.rados.0.smithi042.stdout:3317: oid 10 version 936 is already newer than 935 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3317: oid 10 version 936 is already newer than 933 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 936 (ObjNum 1138 snap 375 seq_num 1138) dirty exists 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3317: left oid 10 (ObjNum 1138 snap 375 seq_num 1138) 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3318: finishing rollback tid 0 to smithi04268048-39 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3318: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3318: finishing rollback tid 2 to smithi04268048-39 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1239 (ObjNum 900 snap 296 seq_num 900) dirty exists 2024-09-20T19:36:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:3319: finishing write tid 1 to smithi04268048-37 2024-09-20T19:36:55.079 INFO:tasks.rados.rados.0.smithi042.stdout:3317: done (3 left) 2024-09-20T19:36:55.079 INFO:tasks.rados.rados.0.smithi042.stdout:3318: done (2 left) 2024-09-20T19:36:55.079 INFO:tasks.rados.rados.0.smithi042.stdout:3320: done (1 left) 2024-09-20T19:36:55.079 INFO:tasks.rados.rados.0.smithi042.stdout:3321: write oid 14 current snap is 375 2024-09-20T19:36:55.079 INFO:tasks.rados.rados.0.smithi042.stdout:3321: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-20T19:36:55.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:55.085 INFO:tasks.rados.rados.0.smithi042.stdout:3321: writing smithi04268048-14 from 776787 to 1288846 tid 1 2024-09-20T19:36:55.092 INFO:tasks.rados.rados.0.smithi042.stdout:3321: writing smithi04268048-14 from 1718425 to 2484822 tid 2 2024-09-20T19:36:55.096 INFO:tasks.rados.rados.0.smithi042.stdout:3321: writing smithi04268048-14 from 3000223 to 3176787 tid 3 2024-09-20T19:36:55.096 INFO:tasks.rados.rados.0.smithi042.stdout:3322: read oid 40 snap -1 2024-09-20T19:36:55.096 INFO:tasks.rados.rados.0.smithi042.stdout:3322: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:36:55.096 INFO:tasks.rados.rados.0.smithi042.stdout:3323: write (excl) oid 46 current snap is 375 2024-09-20T19:36:55.096 INFO:tasks.rados.rados.0.smithi042.stdout:3319: finishing write tid 2 to smithi04268048-37 2024-09-20T19:36:55.097 INFO:tasks.rados.rados.0.smithi042.stdout:3323: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-20T19:36:55.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:55.105 INFO:tasks.rados.rados.0.smithi042.stdout:3323: writing smithi04268048-46 from 793594 to 1380902 tid 1 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3323: writing smithi04268048-46 from 2060554 to 2570609 tid 2 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3324: rollback oid 28 current snap is 375 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 324 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: finishing write tid 3 to smithi04268048-37 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: finishing write tid 4 to smithi04268048-37 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: oid 37 updating version 0 to 1157 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: oid 37 version 1157 is already newer than 1156 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: oid 37 updating version 1157 to 1158 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1158 (ObjNum 1139 snap 375 seq_num 1139) dirty exists 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: left oid 37 (ObjNum 1139 snap 375 seq_num 1139) 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3321: finishing write tid 1 to smithi04268048-14 2024-09-20T19:36:55.114 INFO:tasks.rados.rados.0.smithi042.stdout:3319: done (4 left) 2024-09-20T19:36:55.115 INFO:tasks.rados.rados.0.smithi042.stdout:3325: snap_remove snap 372 2024-09-20T19:36:55.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:55.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:55.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:55.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:55.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:3325: done (4 left) 2024-09-20T19:36:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:3326: snap_remove snap 370 2024-09-20T19:36:55.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:55.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:55.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:55.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:55 smithi162 bash[24557]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[32075]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.867865+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:53.869920+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:54.441045+0000 mgr.y (mgr.24491) 711 : cluster [DBG] pgmap v1356: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 26 op/s 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:55 smithi042 bash[22706]: cluster 2024-09-20T19:36:54.530901+0000 mon.a (mon.0) 2494 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2024-09-20T19:36:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:55.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:55.991 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:56.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:56.122 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:56.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:56.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:56.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:56.385 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:36:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:36:56.404 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:56.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:56.532 INFO:tasks.rados.rados.0.smithi042.stdout:3326: done (4 left) 2024-09-20T19:36:56.532 INFO:tasks.rados.rados.0.smithi042.stdout:3327: write oid 33 current snap is 375 2024-09-20T19:36:56.532 INFO:tasks.rados.rados.0.smithi042.stdout:3322: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:36:56.537 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:56.539 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:36:56.539 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:36:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:56.573 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:56.623 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2186357 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: finishing write tid 2 to smithi04268048-14 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: finishing write tid 3 to smithi04268048-14 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: finishing write tid 4 to smithi04268048-14 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: finishing write tid 5 to smithi04268048-14 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: oid 14 updating version 0 to 1150 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: oid 14 updating version 1150 to 1151 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: oid 14 version 1151 is already newer than 1149 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: oid 14 version 1151 is already newer than 1148 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1151 (ObjNum 1141 snap 375 seq_num 1141) dirty exists 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3321: left oid 14 (ObjNum 1141 snap 375 seq_num 1141) 2024-09-20T19:36:56.632 INFO:tasks.rados.rados.0.smithi042.stdout:3327: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-09-20T19:36:56.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:56.643 INFO:tasks.rados.rados.0.smithi042.stdout:3327: writing smithi04268048-33 from 410401 to 1072958 tid 1 2024-09-20T19:36:56.660 INFO:tasks.rados.rados.0.smithi042.stdout:3327: writing smithi04268048-33 from 1719036 to 2489102 tid 2 2024-09-20T19:36:56.669 INFO:tasks.rados.rados.0.smithi042.stdout:3327: writing smithi04268048-33 from 2938716 to 3210401 tid 3 2024-09-20T19:36:56.669 INFO:tasks.rados.rados.0.smithi042.stdout:3321: done (4 left) 2024-09-20T19:36:56.669 INFO:tasks.rados.rados.0.smithi042.stdout:3322: done (3 left) 2024-09-20T19:36:56.669 INFO:tasks.rados.rados.0.smithi042.stdout:3328: snap_remove snap 371 2024-09-20T19:36:56.669 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[22706]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[22706]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[22706]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[22706]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[32075]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[32075]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[32075]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.671 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:56 smithi042 bash[32075]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:56.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:56.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:56 smithi162 bash[24557]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:56 smithi162 bash[24557]: cluster 2024-09-20T19:36:55.532686+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2024-09-20T19:36:56.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:56 smithi162 bash[24557]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:56 smithi162 bash[24557]: cluster 2024-09-20T19:36:56.530557+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2024-09-20T19:36:56.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:56.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:56.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:56.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:57.034 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:57.062 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:57.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:57.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:36:57.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:57.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:57.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:57.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:57.560 INFO:tasks.rados.rados.0.smithi042.stdout:3328: done (3 left) 2024-09-20T19:36:57.560 INFO:tasks.rados.rados.0.smithi042.stdout:3329: write (excl) oid 30 current snap is 375 2024-09-20T19:36:57.560 INFO:tasks.rados.rados.0.smithi042.stdout:3323: finishing write tid 1 to smithi04268048-46 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: finishing write tid 2 to smithi04268048-46 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: finishing write tid 3 to smithi04268048-46 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: finishing write tid 4 to smithi04268048-46 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: oid 46 updating version 0 to 1223 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: oid 46 version 1223 is already newer than 1221 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: oid 46 version 1223 is already newer than 1222 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1223 (ObjNum 1142 snap 375 seq_num 1142) dirty exists 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3323: left oid 46 (ObjNum 1142 snap 375 seq_num 1142) 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3324: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3324: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3324: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1181 (ObjNum 956 snap 318 seq_num 956) dirty exists 2024-09-20T19:36:57.561 INFO:tasks.rados.rados.0.smithi042.stdout:3327: finishing write tid 1 to smithi04268048-33 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: finishing write tid 2 to smithi04268048-33 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: finishing write tid 3 to smithi04268048-33 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: finishing write tid 4 to smithi04268048-33 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: finishing write tid 5 to smithi04268048-33 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: oid 33 updating version 0 to 1322 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: oid 33 version 1322 is already newer than 1321 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: oid 33 updating version 1322 to 1323 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: oid 33 updating version 1323 to 1324 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1324 (ObjNum 1143 snap 375 seq_num 1143) dirty exists 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3327: left oid 33 (ObjNum 1143 snap 375 seq_num 1143) 2024-09-20T19:36:57.562 INFO:tasks.rados.rados.0.smithi042.stdout:3329: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-09-20T19:36:57.567 INFO:tasks.rados.rados.0.smithi042.stdout:3329: writing smithi04268048-30 from 427208 to 1165014 tid 1 2024-09-20T19:36:57.574 INFO:tasks.rados.rados.0.smithi042.stdout:3329: writing smithi04268048-30 from 1777518 to 2291242 tid 2 2024-09-20T19:36:57.580 INFO:tasks.rados.rados.0.smithi042.stdout:3329: writing smithi04268048-30 from 2966139 to 3227208 tid 3 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3323: done (3 left) 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3324: done (2 left) 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3327: done (1 left) 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3330: delete oid 44 current snap is 375 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3329: finishing write tid 1 to smithi04268048-30 2024-09-20T19:36:57.581 INFO:tasks.rados.rados.0.smithi042.stdout:3329: finishing write tid 2 to smithi04268048-30 2024-09-20T19:36:57.589 INFO:tasks.rados.rados.0.smithi042.stdout:3330: done (1 left) 2024-09-20T19:36:57.589 INFO:tasks.rados.rados.0.smithi042.stdout:3331: snap_remove snap 373 2024-09-20T19:36:57.590 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:57.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.723 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:36:57.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:36:57.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:57 smithi162 bash[24557]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:57 smithi162 bash[24557]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:57.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:57 smithi042 bash[32075]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:57 smithi042 bash[32075]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:57 smithi042 bash[22706]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:57 smithi042 bash[22706]: cluster 2024-09-20T19:36:56.441599+0000 mgr.y (mgr.24491) 712 : cluster [DBG] pgmap v1359: 178 pgs: 10 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 154 active+clean; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 307 KiB/s wr, 3 op/s 2024-09-20T19:36:57.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:57.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:58.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:36:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:36:58.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:58.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:58.339 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:58.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:58.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:58.491 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:58.565 INFO:tasks.rados.rados.0.smithi042.stdout:3331: done (1 left) 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3332: snap_create 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: finishing write tid 3 to smithi04268048-30 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: finishing write tid 4 to smithi04268048-30 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: finishing write tid 5 to smithi04268048-30 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: oid 30 updating version 0 to 1483 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: oid 30 version 1483 is already newer than 1482 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: oid 30 version 1483 is already newer than 1480 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: oid 30 version 1483 is already newer than 1481 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1483 (ObjNum 1144 snap 375 seq_num 1144) dirty exists 2024-09-20T19:36:58.566 INFO:tasks.rados.rados.0.smithi042.stdout:3329: left oid 30 (ObjNum 1144 snap 375 seq_num 1144) 2024-09-20T19:36:58.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:58.617 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:36:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:36:58.641 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:58.641 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:58.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:58.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:36:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:36:58.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:58.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:58.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:58.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:58.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[22706]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:58.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:58 smithi042 bash[32075]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:58.876 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:58.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:58.979 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:59.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:59.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: cluster 2024-09-20T19:36:57.558668+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2024-09-20T19:36:59.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:59.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: audit 2024-09-20T19:36:58.430792+0000 mon.a (mon.0) 2498 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:36:59.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:59.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:58 smithi162 bash[24557]: audit 2024-09-20T19:36:58.432075+0000 mon.a (mon.0) 2499 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:36:59.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:59.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:59.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:59.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:59.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:36:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:36:59.501 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.529 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3329: done (1 left) 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3332: done (0 left) 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3333: read oid 50 snap -1 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3333: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3334: read oid 26 snap -1 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3334: expect (ObjNum 1117 snap 368 seq_num 1117) 2024-09-20T19:36:59.561 INFO:tasks.rados.rados.0.smithi042.stdout:3335: write oid 20 current snap is 376 2024-09-20T19:36:59.562 INFO:tasks.rados.rados.0.smithi042.stdout:3335: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2024-09-20T19:36:59.566 INFO:tasks.rados.rados.0.smithi042.stdout:3335: writing smithi04268048-20 from 460822 to 949126 tid 1 2024-09-20T19:36:59.572 INFO:tasks.rados.rados.0.smithi042.stdout:3335: writing smithi04268048-20 from 1378129 to 1895522 tid 2 2024-09-20T19:36:59.579 INFO:tasks.rados.rados.0.smithi042.stdout:3335: writing smithi04268048-20 from 2504632 to 2910594 tid 3 2024-09-20T19:36:59.579 INFO:tasks.rados.rados.0.smithi042.stdout:3336: write oid 11 current snap is 376 2024-09-20T19:36:59.579 INFO:tasks.rados.rados.0.smithi042.stdout:3336: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-20T19:36:59.585 INFO:tasks.rados.rados.0.smithi042.stdout:3336: writing smithi04268048-11 from 477629 to 1041182 tid 1 2024-09-20T19:36:59.595 INFO:tasks.rados.rados.0.smithi042.stdout:3336: writing smithi04268048-11 from 1836611 to 2614015 tid 2 2024-09-20T19:36:59.600 INFO:tasks.rados.rados.0.smithi042.stdout:3336: writing smithi04268048-11 from 3048408 to 3277629 tid 3 2024-09-20T19:36:59.600 INFO:tasks.rados.rados.0.smithi042.stdout:3337: write (excl) oid 45 current snap is 376 2024-09-20T19:36:59.600 INFO:tasks.rados.rados.0.smithi042.stdout:3337: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-20T19:36:59.608 INFO:tasks.rados.rados.0.smithi042.stdout:3337: writing smithi04268048-45 from 494436 to 1249591 tid 1 2024-09-20T19:36:59.616 INFO:tasks.rados.rados.0.smithi042.stdout:3337: writing smithi04268048-45 from 2011446 to 2532508 tid 2 2024-09-20T19:36:59.621 INFO:tasks.rados.rados.0.smithi042.stdout:3337: writing smithi04268048-45 from 3075831 to 3294436 tid 3 2024-09-20T19:36:59.621 INFO:tasks.rados.rados.0.smithi042.stdout:3338: copy_from oid 19 from oid 31 current snap is 376 2024-09-20T19:36:59.621 INFO:tasks.rados.rados.0.smithi042.stdout:3334: expect (ObjNum 1117 snap 368 seq_num 1117) 2024-09-20T19:36:59.621 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 143532 2024-09-20T19:36:59.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:59.654 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:36:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:36:59.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.768 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[24557]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: finishing write tid 1 to smithi04268048-20 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: finishing write tid 2 to smithi04268048-20 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: finishing write tid 3 to smithi04268048-20 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: finishing write tid 4 to smithi04268048-20 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: finishing write tid 5 to smithi04268048-20 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: oid 20 updating version 0 to 1232 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: oid 20 version 1232 is already newer than 1230 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: oid 20 updating version 1232 to 1233 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: oid 20 version 1233 is already newer than 1231 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1233 (ObjNum 1146 snap 376 seq_num 1146) dirty exists 2024-09-20T19:36:59.772 INFO:tasks.rados.rados.0.smithi042.stdout:3335: left oid 20 (ObjNum 1146 snap 376 seq_num 1146) 2024-09-20T19:36:59.773 INFO:tasks.rados.rados.0.smithi042.stdout:3334: done (5 left) 2024-09-20T19:36:59.773 INFO:tasks.rados.rados.0.smithi042.stdout:3335: done (4 left) 2024-09-20T19:36:59.773 INFO:tasks.rados.rados.0.smithi042.stdout:3339: snap_create 2024-09-20T19:36:59.773 INFO:tasks.rados.rados.0.smithi042.stdout:3333: expect (ObjNum 1034 snap 340 seq_num 1034) 2024-09-20T19:36:59.773 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:59.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:36:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:36:59.829 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: finishing write tid 1 to smithi04268048-11 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: finishing write tid 2 to smithi04268048-11 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: finishing write tid 3 to smithi04268048-11 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: finishing write tid 4 to smithi04268048-11 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: finishing write tid 5 to smithi04268048-11 2024-09-20T19:36:59.864 INFO:tasks.rados.rados.0.smithi042.stdout:3336: oid 11 updating version 0 to 1117 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3336: oid 11 updating version 1117 to 1118 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3336: oid 11 version 1118 is already newer than 1116 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3336: oid 11 version 1118 is already newer than 1115 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1118 (ObjNum 1147 snap 376 seq_num 1147) dirty exists 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3336: left oid 11 (ObjNum 1147 snap 376 seq_num 1147) 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: finishing write tid 1 to smithi04268048-45 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: finishing write tid 2 to smithi04268048-45 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: finishing write tid 3 to smithi04268048-45 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: finishing write tid 4 to smithi04268048-45 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: finishing write tid 5 to smithi04268048-45 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: oid 45 updating version 0 to 1545 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: oid 45 updating version 1545 to 1546 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: oid 45 updating version 1546 to 1547 2024-09-20T19:36:59.865 INFO:tasks.rados.rados.0.smithi042.stdout:3337: oid 45 updating version 1547 to 1548 2024-09-20T19:36:59.866 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1548 (ObjNum 1148 snap 376 seq_num 1148) dirty exists 2024-09-20T19:36:59.866 INFO:tasks.rados.rados.0.smithi042.stdout:3337: left oid 45 (ObjNum 1148 snap 376 seq_num 1148) 2024-09-20T19:36:59.866 INFO:tasks.rados.rados.0.smithi042.stdout:3338: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:36:59.866 INFO:tasks.rados.rados.0.smithi042.stdout:3338: finishing copy_from to smithi04268048-19 2024-09-20T19:36:59.866 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 981 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:36:59.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.899 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[32075]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:58.442198+0000 mgr.y (mgr.24491) 713 : cluster [DBG] pgmap v1362: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2024-09-20T19:36:59.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:58.557312+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2024-09-20T19:36:59.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.901 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:36:59 smithi042 bash[22706]: cluster 2024-09-20T19:36:59.559590+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2024-09-20T19:36:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:36:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:36:59.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:00.015 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:00.042 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:00.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:36:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:00.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:00.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:00.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:37:00.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:00.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.468 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:00.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:00.586 INFO:tasks.rados.rados.0.smithi042.stdout:3333: done (4 left) 2024-09-20T19:37:00.586 INFO:tasks.rados.rados.0.smithi042.stdout:3336: done (3 left) 2024-09-20T19:37:00.586 INFO:tasks.rados.rados.0.smithi042.stdout:3337: done (2 left) 2024-09-20T19:37:00.586 INFO:tasks.rados.rados.0.smithi042.stdout:3338: done (1 left) 2024-09-20T19:37:00.587 INFO:tasks.rados.rados.0.smithi042.stdout:3339: done (0 left) 2024-09-20T19:37:00.587 INFO:tasks.rados.rados.0.smithi042.stdout:3340: write oid 44 current snap is 377 2024-09-20T19:37:00.587 INFO:tasks.rados.rados.0.smithi042.stdout:3340: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-20T19:37:00.587 INFO:tasks.rados.rados.0.smithi042.stdout:3340: writing smithi04268048-44 from 511243 to 941647 tid 1 2024-09-20T19:37:00.594 INFO:tasks.rados.rados.0.smithi042.stdout:3340: writing smithi04268048-44 from 1669928 to 2334648 tid 2 2024-09-20T19:37:00.597 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:00.601 INFO:tasks.rados.rados.0.smithi042.stdout:3340: writing smithi04268048-44 from 3103254 to 3311243 tid 3 2024-09-20T19:37:00.601 INFO:tasks.rados.rados.0.smithi042.stdout:3341: write oid 21 current snap is 377 2024-09-20T19:37:00.601 INFO:tasks.rados.rados.0.smithi042.stdout:3340: finishing write tid 1 to smithi04268048-44 2024-09-20T19:37:00.601 INFO:tasks.rados.rados.0.smithi042.stdout:3341: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-20T19:37:00.606 INFO:tasks.rados.rados.0.smithi042.stdout:3341: writing smithi04268048-21 from 528050 to 1033703 tid 1 2024-09-20T19:37:00.613 INFO:tasks.rados.rados.0.smithi042.stdout:3341: writing smithi04268048-21 from 1612057 to 2136788 tid 2 2024-09-20T19:37:00.621 INFO:tasks.rados.rados.0.smithi042.stdout:3341: writing smithi04268048-21 from 2614324 to 3181727 tid 3 2024-09-20T19:37:00.621 INFO:tasks.rados.rados.0.smithi042.stdout:3342: read oid 29 snap -1 2024-09-20T19:37:00.621 INFO:tasks.rados.rados.0.smithi042.stdout:3342: expect deleted 2024-09-20T19:37:00.621 INFO:tasks.rados.rados.0.smithi042.stdout:3343: delete oid 7 current snap is 377 2024-09-20T19:37:00.621 INFO:tasks.rados.rados.0.smithi042.stdout:3340: finishing write tid 2 to smithi04268048-44 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: finishing write tid 3 to smithi04268048-44 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: finishing write tid 4 to smithi04268048-44 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: finishing write tid 5 to smithi04268048-44 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: oid 44 updating version 0 to 1229 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: oid 44 updating version 1229 to 1231 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: oid 44 version 1231 is already newer than 1230 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: oid 44 version 1231 is already newer than 1228 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1231 (ObjNum 1149 snap 377 seq_num 1149) dirty exists 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3340: left oid 44 (ObjNum 1149 snap 377 seq_num 1149) 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3341: finishing write tid 1 to smithi04268048-21 2024-09-20T19:37:00.622 INFO:tasks.rados.rados.0.smithi042.stdout:3341: finishing write tid 2 to smithi04268048-21 2024-09-20T19:37:00.625 INFO:tasks.rados.rados.0.smithi042.stdout:3340: done (3 left) 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3342: done (2 left) 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3343: done (1 left) 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3344: read oid 2 snap -1 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3344: expect deleted 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3345: read oid 41 snap 319 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3345: expect deleted 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3346: rollback oid 17 current snap is 377 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 17 to 324 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3347: read oid 11 snap 319 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3347: expect (ObjNum 937 snap 314 seq_num 937) 2024-09-20T19:37:00.626 INFO:tasks.rados.rados.0.smithi042.stdout:3348: delete oid 48 current snap is 377 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: finishing write tid 3 to smithi04268048-21 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: finishing write tid 4 to smithi04268048-21 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: finishing write tid 5 to smithi04268048-21 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: oid 21 updating version 0 to 1487 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: oid 21 version 1487 is already newer than 1486 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: oid 21 updating version 1487 to 1488 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: oid 21 updating version 1488 to 1489 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1489 (ObjNum 1150 snap 377 seq_num 1150) dirty exists 2024-09-20T19:37:00.627 INFO:tasks.rados.rados.0.smithi042.stdout:3341: left oid 21 (ObjNum 1150 snap 377 seq_num 1150) 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3341: done (5 left) 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3344: done (4 left) 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3348: done (3 left) 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3349: read oid 16 snap -1 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3349: expect (ObjNum 1133 snap 374 seq_num 1133) 2024-09-20T19:37:00.628 INFO:tasks.rados.rados.0.smithi042.stdout:3350: snap_remove snap 369 2024-09-20T19:37:00.668 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:37:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:37:00.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:00.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:00.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:00.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:01.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:01.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:01.213 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:37:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:37:01.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:01.232 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:37:01.232 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:37:01.232 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:01.249 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:37:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:37:01.266 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:37:01.266 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:37:01.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:37:01.287 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:37:01.308 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:37:01.308 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:37:01.329 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:37:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:37:01.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:01.352 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:37:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:37:01.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:01.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:01.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:01.596 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:01.606 INFO:tasks.rados.rados.0.smithi042.stdout:3350: done (4 left) 2024-09-20T19:37:01.606 INFO:tasks.rados.rados.0.smithi042.stdout:3351: read oid 10 snap -1 2024-09-20T19:37:01.606 INFO:tasks.rados.rados.0.smithi042.stdout:3351: expect (ObjNum 1138 snap 375 seq_num 1138) 2024-09-20T19:37:01.606 INFO:tasks.rados.rados.0.smithi042.stdout:3352: read oid 47 snap 324 2024-09-20T19:37:01.606 INFO:tasks.rados.rados.0.smithi042.stdout:3352: expect deleted 2024-09-20T19:37:01.607 INFO:tasks.rados.rados.0.smithi042.stdout:3353: write (excl) oid 36 current snap is 377 2024-09-20T19:37:01.607 INFO:tasks.rados.rados.0.smithi042.stdout:3353: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2024-09-20T19:37:01.610 INFO:tasks.rados.rados.0.smithi042.stdout:3353: writing smithi04268048-36 from 578471 to 1309871 tid 1 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.612 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[32075]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.613 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:01 smithi042 bash[22706]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.621 INFO:tasks.rados.rados.0.smithi042.stdout:3353: writing smithi04268048-36 from 1787503 to 2459561 tid 2 2024-09-20T19:37:01.626 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3353: writing smithi04268048-36 from 3096593 to 3378471 tid 3 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3354: copy_from oid 8 from oid 15 current snap is 377 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3346: finishing rollback tid 1 to smithi04268048-17 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3346: finishing rollback tid 2 to smithi04268048-17 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:37:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3349: expect (ObjNum 1133 snap 374 seq_num 1133) 2024-09-20T19:37:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:01.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:01.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: audit 2024-09-20T19:36:59.830732+0000 mgr.y (mgr.24491) 714 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: cluster 2024-09-20T19:37:00.442862+0000 mgr.y (mgr.24491) 715 : cluster [DBG] pgmap v1365: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 409 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: cluster 2024-09-20T19:37:00.577632+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:01 smithi162 bash[24557]: audit 2024-09-20T19:37:01.193987+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.42:0/2261674258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:01.847 INFO:tasks.rados.rados.0.smithi042.stdout:3347: expect (ObjNum 937 snap 314 seq_num 937) 2024-09-20T19:37:01.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:01.892 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:01.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:01.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:02.019 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:02.113 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2764891 2024-09-20T19:37:02.121 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.141 INFO:tasks.rados.rados.0.smithi042.stdout:3353: finishing write tid 1 to smithi04268048-36 2024-09-20T19:37:02.141 INFO:tasks.rados.rados.0.smithi042.stdout:3351: expect (ObjNum 1138 snap 375 seq_num 1138) 2024-09-20T19:37:02.146 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:02.166 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 330681 2024-09-20T19:37:02.201 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1780758 2024-09-20T19:37:02.221 INFO:tasks.rados.rados.0.smithi042.stdout:3353: finishing write tid 2 to smithi04268048-36 2024-09-20T19:37:02.221 INFO:tasks.rados.rados.0.smithi042.stdout:3353: finishing write tid 3 to smithi04268048-36 2024-09-20T19:37:02.221 INFO:tasks.rados.rados.0.smithi042.stdout:3353: finishing write tid 4 to smithi04268048-36 2024-09-20T19:37:02.221 INFO:tasks.rados.rados.0.smithi042.stdout:3353: finishing write tid 5 to smithi04268048-36 2024-09-20T19:37:02.221 INFO:tasks.rados.rados.0.smithi042.stdout:3353: oid 36 updating version 0 to 1158 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3353: oid 36 version 1158 is already newer than 1157 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3353: oid 36 version 1158 is already newer than 1156 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3353: oid 36 updating version 1158 to 1159 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 1159 (ObjNum 1153 snap 377 seq_num 1153) dirty exists 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3353: left oid 36 (ObjNum 1153 snap 377 seq_num 1153) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3345: done (7 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3346: done (6 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3347: done (5 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3349: done (4 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3351: done (3 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3352: done (2 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3353: done (1 left) 2024-09-20T19:37:02.222 INFO:tasks.rados.rados.0.smithi042.stdout:3355: snap_create 2024-09-20T19:37:02.223 INFO:tasks.rados.rados.0.smithi042.stdout:3354: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:37:02.223 INFO:tasks.rados.rados.0.smithi042.stdout:3354: finishing copy_from to smithi04268048-8 2024-09-20T19:37:02.223 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1147 (ObjNum 970 snap 321 seq_num 970) dirty exists 2024-09-20T19:37:02.249 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.272 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:02.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:02.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:02.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:02.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:02.606 INFO:tasks.rados.rados.0.smithi042.stdout:3354: done (1 left) 2024-09-20T19:37:02.606 INFO:tasks.rados.rados.0.smithi042.stdout:3355: done (0 left) 2024-09-20T19:37:02.607 INFO:tasks.rados.rados.0.smithi042.stdout:3356: write oid 26 current snap is 378 2024-09-20T19:37:02.607 INFO:tasks.rados.rados.0.smithi042.stdout:3356: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-09-20T19:37:02.623 INFO:tasks.rados.rados.0.smithi042.stdout:3356: writing smithi04268048-26 from 595278 to 1001927 tid 1 2024-09-20T19:37:02.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.638 INFO:tasks.rados.rados.0.smithi042.stdout:3356: writing smithi04268048-26 from 1729632 to 2261701 tid 2 2024-09-20T19:37:02.644 INFO:tasks.rados.rados.0.smithi042.stdout:3356: writing smithi04268048-26 from 3007663 to 3395278 tid 3 2024-09-20T19:37:02.644 INFO:tasks.rados.rados.0.smithi042.stdout:3357: write oid 45 current snap is 378 2024-09-20T19:37:02.644 INFO:tasks.rados.rados.0.smithi042.stdout:3356: finishing write tid 1 to smithi04268048-26 2024-09-20T19:37:02.644 INFO:tasks.rados.rados.0.smithi042.stdout:3357: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-20T19:37:02.650 INFO:tasks.rados.rados.0.smithi042.stdout:3357: writing smithi04268048-45 from 612085 to 1093983 tid 1 2024-09-20T19:37:02.659 INFO:tasks.rados.rados.0.smithi042.stdout:3357: writing smithi04268048-45 from 1788114 to 2463841 tid 2 2024-09-20T19:37:02.665 INFO:tasks.rados.rados.0.smithi042.stdout:3357: writing smithi04268048-45 from 3035086 to 3412085 tid 3 2024-09-20T19:37:02.665 INFO:tasks.rados.rados.0.smithi042.stdout:3358: snap_remove snap 375 2024-09-20T19:37:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:02.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.792 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:02 smithi042 bash[22706]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.792 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:02 smithi042 bash[22706]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.793 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:02 smithi042 bash[32075]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.793 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:02 smithi042 bash[32075]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.794 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:02.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:02 smithi162 bash[24557]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:02 smithi162 bash[24557]: cluster 2024-09-20T19:37:01.594797+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2024-09-20T19:37:02.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:02.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:02.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:03.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:03.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:03.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:03.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:03.309 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:03.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:03.434 INFO:tasks.rados.rados.0.smithi042.stdout:3358: done (2 left) 2024-09-20T19:37:03.434 INFO:tasks.rados.rados.0.smithi042.stdout:3359: read oid 20 snap -1 2024-09-20T19:37:03.434 INFO:tasks.rados.rados.0.smithi042.stdout:3359: expect (ObjNum 1146 snap 376 seq_num 1146) 2024-09-20T19:37:03.434 INFO:tasks.rados.rados.0.smithi042.stdout:3360: snap_remove snap 376 2024-09-20T19:37:03.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:03.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:03.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:37:03.614 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:37:03.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-20T19:37:03.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:03.693 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:03 smithi162 bash[24557]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:03.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:03.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[32075]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:02.444552+0000 mgr.y (mgr.24491) 716 : cluster [DBG] pgmap v1368: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 419 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:02.605256+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.431449+0000 mon.a (mon.0) 2506 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: audit 2024-09-20T19:37:03.511072+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.42:0/3075460060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:03 smithi042 bash[22706]: audit 2024-09-20T19:37:03.511579+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:03.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:03.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:04.071 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.100 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:04.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.232 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:04.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:04.426 INFO:tasks.rados.rados.0.smithi042.stdout:3360: done (3 left) 2024-09-20T19:37:04.426 INFO:tasks.rados.rados.0.smithi042.stdout:3361: delete oid 24 current snap is 378 2024-09-20T19:37:04.426 INFO:tasks.rados.rados.0.smithi042.stdout:3356: finishing write tid 2 to smithi04268048-26 2024-09-20T19:37:04.426 INFO:tasks.rados.rados.0.smithi042.stdout:3356: finishing write tid 3 to smithi04268048-26 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: finishing write tid 4 to smithi04268048-26 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: finishing write tid 5 to smithi04268048-26 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: oid 26 updating version 0 to 1258 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: oid 26 updating version 1258 to 1260 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: oid 26 version 1260 is already newer than 1259 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: oid 26 updating version 1260 to 1261 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1261 (ObjNum 1154 snap 378 seq_num 1154) dirty exists 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3356: left oid 26 (ObjNum 1154 snap 378 seq_num 1154) 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: finishing write tid 1 to smithi04268048-45 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: finishing write tid 2 to smithi04268048-45 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: finishing write tid 3 to smithi04268048-45 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: finishing write tid 4 to smithi04268048-45 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: finishing write tid 5 to smithi04268048-45 2024-09-20T19:37:04.427 INFO:tasks.rados.rados.0.smithi042.stdout:3357: oid 45 updating version 0 to 1555 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:3357: oid 45 version 1555 is already newer than 1553 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:3357: oid 45 version 1555 is already newer than 1554 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:3357: oid 45 version 1555 is already newer than 1552 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1555 (ObjNum 1155 snap 378 seq_num 1155) dirty exists 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:3357: left oid 45 (ObjNum 1155 snap 378 seq_num 1155) 2024-09-20T19:37:04.428 INFO:tasks.rados.rados.0.smithi042.stdout:3359: expect (ObjNum 1146 snap 376 seq_num 1146) 2024-09-20T19:37:04.428 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:37:04.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:04.550 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1055126 2024-09-20T19:37:04.589 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:3356: done (3 left) 2024-09-20T19:37:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:3357: done (2 left) 2024-09-20T19:37:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:3359: done (1 left) 2024-09-20T19:37:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:3361: done (0 left) 2024-09-20T19:37:04.593 INFO:tasks.rados.rados.0.smithi042.stdout:3362: snap_create 2024-09-20T19:37:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:04.714 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.726 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.727 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.728 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:04.728 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:04 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:04.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:04.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:04.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:04.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.417517+0000 mon.a (mon.0) 2508 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: audit 2024-09-20T19:37:04.419816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:04.992 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:04 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.424291+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2024-09-20T19:37:05.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:05.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:05.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:05.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:05.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:05.444 INFO:tasks.rados.rados.0.smithi042.stdout:3362: done (0 left) 2024-09-20T19:37:05.445 INFO:tasks.rados.rados.0.smithi042.stdout:3363: read oid 16 snap -1 2024-09-20T19:37:05.445 INFO:tasks.rados.rados.0.smithi042.stdout:3363: expect (ObjNum 1133 snap 374 seq_num 1133) 2024-09-20T19:37:05.445 INFO:tasks.rados.rados.0.smithi042.stdout:3364: delete oid 48 current snap is 379 2024-09-20T19:37:05.451 INFO:tasks.rados.rados.0.smithi042.stdout:3364: done (1 left) 2024-09-20T19:37:05.451 INFO:tasks.rados.rados.0.smithi042.stdout:3365: write oid 37 current snap is 379 2024-09-20T19:37:05.451 INFO:tasks.rados.rados.0.smithi042.stdout:3365: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-20T19:37:05.464 INFO:tasks.rados.rados.0.smithi042.stdout:3365: writing smithi04268048-37 from 662506 to 1086504 tid 1 2024-09-20T19:37:05.472 INFO:tasks.rados.rados.0.smithi042.stdout:3365: writing smithi04268048-37 from 1563560 to 2102967 tid 2 2024-09-20T19:37:05.481 INFO:tasks.rados.rados.0.smithi042.stdout:3365: writing smithi04268048-37 from 2717355 to 3207640 tid 3 2024-09-20T19:37:05.481 INFO:tasks.rados.rados.0.smithi042.stdout:3366: read oid 12 snap -1 2024-09-20T19:37:05.481 INFO:tasks.rados.rados.0.smithi042.stdout:3366: expect deleted 2024-09-20T19:37:05.481 INFO:tasks.rados.rados.0.smithi042.stdout:3367: read oid 7 snap -1 2024-09-20T19:37:05.482 INFO:tasks.rados.rados.0.smithi042.stdout:3367: expect deleted 2024-09-20T19:37:05.482 INFO:tasks.rados.rados.0.smithi042.stdout:3368: write (excl) oid 15 current snap is 379 2024-09-20T19:37:05.482 INFO:tasks.rados.rados.0.smithi042.stdout:3368: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-20T19:37:05.490 INFO:tasks.rados.rados.0.smithi042.stdout:3368: writing smithi04268048-15 from 679313 to 1178560 tid 1 2024-09-20T19:37:05.499 INFO:tasks.rados.rados.0.smithi042.stdout:3368: writing smithi04268048-15 from 1622042 to 2305107 tid 2 2024-09-20T19:37:05.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.508 INFO:tasks.rados.rados.0.smithi042.stdout:3368: writing smithi04268048-15 from 2744778 to 3446335 tid 3 2024-09-20T19:37:05.508 INFO:tasks.rados.rados.0.smithi042.stdout:3369: rollback oid 6 current snap is 379 2024-09-20T19:37:05.508 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 319 2024-09-20T19:37:05.508 INFO:tasks.rados.rados.0.smithi042.stdout:3363: expect (ObjNum 1133 snap 374 seq_num 1133) 2024-09-20T19:37:05.528 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:05.586 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:05.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:05.656 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:05.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:37:05.742 INFO:tasks.rados.rados.0.smithi042.stdout:3365: finishing write tid 1 to smithi04268048-37 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: finishing write tid 2 to smithi04268048-37 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: finishing write tid 3 to smithi04268048-37 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: finishing write tid 4 to smithi04268048-37 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: finishing write tid 5 to smithi04268048-37 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: oid 37 updating version 0 to 1162 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: oid 37 version 1162 is already newer than 1161 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: oid 37 version 1162 is already newer than 1160 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: oid 37 version 1162 is already newer than 1159 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1162 (ObjNum 1158 snap 379 seq_num 1158) dirty exists 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3365: left oid 37 (ObjNum 1158 snap 379 seq_num 1158) 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3368: finishing write tid 1 to smithi04268048-15 2024-09-20T19:37:05.743 INFO:tasks.rados.rados.0.smithi042.stdout:3368: finishing write tid 2 to smithi04268048-15 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3369: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3363: done (5 left) 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3365: done (4 left) 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3366: done (3 left) 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3367: done (2 left) 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3370: read oid 18 snap -1 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3370: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3371: delete oid 45 current snap is 379 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3369: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1237 (ObjNum 177 snap 0 seq_num 1415591568) dirty dne 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3368: finishing write tid 3 to smithi04268048-15 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3368: finishing write tid 4 to smithi04268048-15 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3368: finishing write tid 5 to smithi04268048-15 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3368: oid 15 updating version 0 to 1149 2024-09-20T19:37:05.744 INFO:tasks.rados.rados.0.smithi042.stdout:3368: oid 15 updating version 1149 to 1150 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:3368: oid 15 updating version 1150 to 1151 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:3368: oid 15 version 1151 is already newer than 1148 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1151 (ObjNum 1159 snap 379 seq_num 1159) dirty exists 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:3368: left oid 15 (ObjNum 1159 snap 379 seq_num 1159) 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:3370: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:37:05.745 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 406395 2024-09-20T19:37:05.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:05 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.788 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:05.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:05.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.735756+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:03.738140+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:04.445255+0000 mgr.y (mgr.24491) 717 : cluster [DBG] pgmap v1372: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 4.3 MiB/s wr, 28 op/s 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:05 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.439804+0000 mon.a (mon.0) 2511 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2024-09-20T19:37:05.824 INFO:tasks.rados.rados.0.smithi042.stdout:3368: done (3 left) 2024-09-20T19:37:05.825 INFO:tasks.rados.rados.0.smithi042.stdout:3369: done (2 left) 2024-09-20T19:37:05.825 INFO:tasks.rados.rados.0.smithi042.stdout:3370: done (1 left) 2024-09-20T19:37:05.825 INFO:tasks.rados.rados.0.smithi042.stdout:3371: done (0 left) 2024-09-20T19:37:05.825 INFO:tasks.rados.rados.0.smithi042.stdout:3372: snap_remove snap 368 2024-09-20T19:37:05.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:05.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:05.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:06.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:06.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:06.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:06.266 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:06.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:06.381 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:06.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:06.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:06.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:06.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:06.634 INFO:tasks.rados.rados.0.smithi042.stdout:3372: done (0 left) 2024-09-20T19:37:06.635 INFO:tasks.rados.rados.0.smithi042.stdout:3373: write (excl) oid 25 current snap is 379 2024-09-20T19:37:06.635 INFO:tasks.rados.rados.0.smithi042.stdout:3373: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-09-20T19:37:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[22706]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.647 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:06 smithi042 bash[32075]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.649 INFO:tasks.rados.rados.0.smithi042.stdout:3373: writing smithi04268048-25 from 712927 to 1362672 tid 1 2024-09-20T19:37:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:06.662 INFO:tasks.rados.rados.0.smithi042.stdout:3373: writing smithi04268048-25 from 2139006 to 2825740 tid 2 2024-09-20T19:37:06.662 INFO:tasks.rados.rados.0.smithi042.stdout:3374: rollback oid 24 current snap is 379 2024-09-20T19:37:06.663 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 378 2024-09-20T19:37:06.663 INFO:tasks.rados.rados.0.smithi042.stdout:3373: finishing write tid 1 to smithi04268048-25 2024-09-20T19:37:06.663 INFO:tasks.rados.rados.0.smithi042.stdout:3375: delete oid 16 current snap is 379 2024-09-20T19:37:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:06.667 INFO:tasks.rados.rados.0.smithi042.stdout:3373: finishing write tid 2 to smithi04268048-25 2024-09-20T19:37:06.667 INFO:tasks.rados.rados.0.smithi042.stdout:3373: finishing write tid 3 to smithi04268048-25 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3374: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1429 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: finishing write tid 4 to smithi04268048-25 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: oid 25 updating version 0 to 1072 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: oid 25 version 1072 is already newer than 1071 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: oid 25 version 1072 is already newer than 1070 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 1072 (ObjNum 1161 snap 379 seq_num 1161) dirty exists 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: left oid 25 (ObjNum 1161 snap 379 seq_num 1161) 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3373: done (2 left) 2024-09-20T19:37:06.668 INFO:tasks.rados.rados.0.smithi042.stdout:3374: done (1 left) 2024-09-20T19:37:06.669 INFO:tasks.rados.rados.0.smithi042.stdout:3375: done (0 left) 2024-09-20T19:37:06.669 INFO:tasks.rados.rados.0.smithi042.stdout:3376: snap_remove snap 349 2024-09-20T19:37:06.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:06.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:06 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:06 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.019800+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:06.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:06 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:06 smithi162 bash[24557]: cluster 2024-09-20T19:37:05.022776+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:06.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:06.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:06.923 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:06.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:07.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:07.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:07.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:07.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:07.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:07.412 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:07.440 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:07.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:07.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:37:07.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:07.619 INFO:tasks.rados.rados.0.smithi042.stdout:3376: done (0 left) 2024-09-20T19:37:07.619 INFO:tasks.rados.rados.0.smithi042.stdout:3377: delete oid 23 current snap is 379 2024-09-20T19:37:07.643 INFO:tasks.rados.rados.0.smithi042.stdout:3377: done (0 left) 2024-09-20T19:37:07.643 INFO:tasks.rados.rados.0.smithi042.stdout:3378: snap_create 2024-09-20T19:37:07.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[32075]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[32075]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[32075]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[32075]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[22706]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[22706]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[22706]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:07 smithi042 bash[22706]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:07.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:07.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:07 smithi162 bash[24557]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:07 smithi162 bash[24557]: cluster 2024-09-20T19:37:06.445974+0000 mgr.y (mgr.24491) 718 : cluster [DBG] pgmap v1374: 178 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 164 active+clean; 424 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:07.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:07 smithi162 bash[24557]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:07 smithi162 bash[24557]: cluster 2024-09-20T19:37:06.620565+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2024-09-20T19:37:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:07.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:07.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:07.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:08.033 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:08.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:08.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:08.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:08.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:08.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:08.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:08.449 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:08.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:08.635 INFO:tasks.rados.rados.0.smithi042.stdout:3378: done (0 left) 2024-09-20T19:37:08.636 INFO:tasks.rados.rados.0.smithi042.stdout:3379: snap_remove snap 367 2024-09-20T19:37:08.708 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:08.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:08 smithi042 bash[22706]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:08 smithi042 bash[22706]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:08 smithi042 bash[32075]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:08 smithi042 bash[32075]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.735 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:08.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:08 smithi162 bash[24557]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:08 smithi162 bash[24557]: cluster 2024-09-20T19:37:07.616552+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2024-09-20T19:37:08.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:08.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:08.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:09.051 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:09.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:09.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:09.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:09.254 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:09.357 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:09.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:09.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:09.630 INFO:tasks.rados.rados.0.smithi042.stdout:3379: done (0 left) 2024-09-20T19:37:09.630 INFO:tasks.rados.rados.0.smithi042.stdout:3380: copy_from oid 37 from oid 5 current snap is 380 2024-09-20T19:37:09.634 INFO:tasks.rados.rados.0.smithi042.stdout:3381: read oid 12 snap -1 2024-09-20T19:37:09.634 INFO:tasks.rados.rados.0.smithi042.stdout:3381: expect deleted 2024-09-20T19:37:09.635 INFO:tasks.rados.rados.0.smithi042.stdout:3382: write (excl) oid 18 current snap is 380 2024-09-20T19:37:09.635 INFO:tasks.rados.rados.0.smithi042.stdout:3382: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-20T19:37:09.642 INFO:tasks.rados.rados.0.smithi042.stdout:3382: writing smithi04268048-18 from 763348 to 1355193 tid 1 2024-09-20T19:37:09.648 INFO:tasks.rados.rados.0.smithi042.stdout:3382: writing smithi04268048-18 from 1914452 to 2348513 tid 2 2024-09-20T19:37:09.653 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:09.656 INFO:tasks.rados.rados.0.smithi042.stdout:3382: writing smithi04268048-18 from 2881893 to 3556163 tid 3 2024-09-20T19:37:09.656 INFO:tasks.rados.rados.0.smithi042.stdout:3383: copy_from oid 16 from oid 47 current snap is 380 2024-09-20T19:37:09.656 INFO:tasks.rados.rados.0.smithi042.stdout:3381: done (3 left) 2024-09-20T19:37:09.657 INFO:tasks.rados.rados.0.smithi042.stdout:3384: snap_remove snap 355 2024-09-20T19:37:09.717 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.718 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.718 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.718 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.718 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:09.718 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[22706]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:09.756 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:09.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:09.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:09.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:09.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:09 smithi162 bash[24557]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:09.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:09.930 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[46409]: debug 2024-09-20T19:37:09.666+0000 7f06e3068640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 171 after trimming was started 2024-09-20T19:37:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:08.447037+0000 mgr.y (mgr.24491) 719 : cluster [DBG] pgmap v1377: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 422 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-20T19:37:09.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:08.631102+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2024-09-20T19:37:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:09 smithi042 bash[32075]: cluster 2024-09-20T19:37:09.628756+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2024-09-20T19:37:10.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:10.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:10.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:10.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:10.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:10.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:10.300 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:10.403 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:10.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:10.664 INFO:tasks.rados.rados.0.smithi042.stdout:3384: done (3 left) 2024-09-20T19:37:10.665 INFO:tasks.rados.rados.0.smithi042.stdout:3385: copy_from oid 8 from oid 48 current snap is 380 2024-09-20T19:37:10.665 INFO:tasks.rados.rados.0.smithi042.stdout:3386: write oid 28 current snap is 380 2024-09-20T19:37:10.665 INFO:tasks.rados.rados.0.smithi042.stdout:3386: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-20T19:37:10.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:10.673 INFO:tasks.rados.rados.0.smithi042.stdout:3386: writing smithi04268048-28 from 780155 to 1447249 tid 1 2024-09-20T19:37:10.680 INFO:tasks.rados.rados.0.smithi042.stdout:3386: writing smithi04268048-28 from 1972934 to 2667006 tid 2 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3386: writing smithi04268048-28 from 3309316 to 3580155 tid 3 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3387: copy_from oid 36 from oid 9 current snap is 380 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3383: finishing copy_from to smithi04268048-16 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3383: got expected ENOENT (src dne) 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3383: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3382: finishing write tid 1 to smithi04268048-18 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3382: finishing write tid 2 to smithi04268048-18 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3382: finishing write tid 3 to smithi04268048-18 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3382: finishing write tid 4 to smithi04268048-18 2024-09-20T19:37:10.687 INFO:tasks.rados.rados.0.smithi042.stdout:3382: finishing write tid 5 to smithi04268048-18 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3382: oid 18 updating version 0 to 1503 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3382: oid 18 updating version 1503 to 1504 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3382: oid 18 version 1504 is already newer than 1502 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3382: oid 18 version 1504 is already newer than 1501 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1504 (ObjNum 1164 snap 380 seq_num 1164) dirty exists 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3382: left oid 18 (ObjNum 1164 snap 380 seq_num 1164) 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3380: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3380: finishing copy_from to smithi04268048-37 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1128 (ObjNum 1097 snap 364 seq_num 1097) dirty exists 2024-09-20T19:37:10.688 INFO:tasks.rados.rados.0.smithi042.stdout:3380: done (5 left) 2024-09-20T19:37:10.689 INFO:tasks.rados.rados.0.smithi042.stdout:3382: done (4 left) 2024-09-20T19:37:10.689 INFO:tasks.rados.rados.0.smithi042.stdout:3383: done (3 left) 2024-09-20T19:37:10.689 INFO:tasks.rados.rados.0.smithi042.stdout:3388: snap_create 2024-09-20T19:37:10.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:10.694 INFO:tasks.rados.rados.0.smithi042.stdout:3385: finishing copy_from to smithi04268048-8 2024-09-20T19:37:10.694 INFO:tasks.rados.rados.0.smithi042.stdout:3385: got expected ENOENT (src dne) 2024-09-20T19:37:10.695 INFO:tasks.rados.rados.0.smithi042.stdout:3386: finishing write tid 1 to smithi04268048-28 2024-09-20T19:37:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:10.699 INFO:tasks.rados.rados.0.smithi042.stdout:3385: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:37:10.699 INFO:tasks.rados.rados.0.smithi042.stdout:3386: finishing write tid 2 to smithi04268048-28 2024-09-20T19:37:10.700 INFO:tasks.rados.rados.0.smithi042.stdout:3386: finishing write tid 3 to smithi04268048-28 2024-09-20T19:37:10.700 INFO:tasks.rados.rados.0.smithi042.stdout:3386: finishing write tid 4 to smithi04268048-28 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: finishing write tid 5 to smithi04268048-28 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: oid 28 updating version 0 to 1183 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: oid 28 updating version 1183 to 1185 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: oid 28 version 1185 is already newer than 1182 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: oid 28 version 1185 is already newer than 1184 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1185 (ObjNum 1165 snap 380 seq_num 1165) dirty exists 2024-09-20T19:37:10.701 INFO:tasks.rados.rados.0.smithi042.stdout:3386: left oid 28 (ObjNum 1165 snap 380 seq_num 1165) 2024-09-20T19:37:10.709 INFO:tasks.rados.rados.0.smithi042.stdout:3387: finishing copy_from racing read to smithi04268048-36 2024-09-20T19:37:10.751 INFO:tasks.rados.rados.0.smithi042.stdout:3387: finishing copy_from to smithi04268048-36 2024-09-20T19:37:10.752 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 36 v 1095 (ObjNum 1118 snap 368 seq_num 1118) dirty exists 2024-09-20T19:37:10.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:10.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:10.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:10.888 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:37:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "size": 20, 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 600, 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [ 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4516 7.2 7:47f606bc:::smithi04268048-22:head [copy-from ver 712] snapc c4=[c4,c3,c0,b3,b2,aa] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:58.408670+0000", 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 372.475962224, 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.069963306, 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4516 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408670+0000", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408670+0000", 2024-09-20T19:37:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408672+0000", 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4849999999999999e-06 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408682+0000", 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.9920000000000002e-06 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408694+0000", 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1273e-05 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408705+0000", 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1114000000000001e-05 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.427441+0000", 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.018736613999999999 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.427448+0000", 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.2690000000000002e-06 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.421157+0000", 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.99370881700000002 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:37:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.421363+0000", 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00020555700000000001 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.454440+0000", 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.033077370000000002 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.454629+0000", 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018905699999999999 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.455828+0000", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0011988560000000001 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.455971+0000", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014282800000000001 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 1,2", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.467267+0000", 2024-09-20T19:37:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011295619999999999 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.478238+0000", 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010971257999999999 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.478308+0000", 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0629000000000004e-05 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.478331+0000", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2860000000000001e-05 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.478414+0000", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.2369999999999999e-05 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.478633+0000", 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000219337 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4517 7.2 7:47f606bc:::smithi04268048-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e550)", 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:30:58.408718+0000", 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 372.475913897, 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0643016140000001, 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4517 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408718+0000", 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408718+0000", 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408719+0000", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8100000000000003e-07 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408722+0000", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.9380000000000001e-06 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408728+0000", 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.9410000000000004e-06 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.408733+0000", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5550000000000002e-06 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.427466+0000", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.018732494999999998 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for peered", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:58.427470+0000", 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.4229999999999998e-06 2024-09-20T19:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.421581+0000", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.99411129399999998 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for missing object", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.421622+0000", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.0503999999999997e-05 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.454691+0000", 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.033069686000000001 2024-09-20T19:37:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.454800+0000", 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010833899999999999 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.456236+0000", 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0014359399999999999 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:37:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.456287+0000", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1555999999999999e-05 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.472728+0000", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016440753999999998 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.472822+0000", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.3715000000000007e-05 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.904 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:30:59.473020+0000", 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00019789300000000001 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4739 7.3 7:c0f7e14f:::smithi04268048-43:head [copy-from ver 712] snapc d0=[d0,cf,ce,cc] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:31:24.260695+0000", 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 346.623936555, 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.037932933000000002, 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.905 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4739 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260695+0000", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260695+0000", 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260701+0000", 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7359999999999998e-06 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260722+0000", 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.0463e-05 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.907 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260765+0000", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3207000000000002e-05 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260801+0000", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6446000000000003e-05 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.261010+0000", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000208712 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:37:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.261035+0000", 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4845999999999999e-05 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.267719+0000", 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0066839250000000003 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.267966+0000", 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000247314 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.909 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,4", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.280432+0000", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012466122 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.297656+0000", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017223378000000001 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.297750+0000", 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.4374000000000001e-05 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.298482+0000", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00073225999999999999 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.298559+0000", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.6829999999999995e-05 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.298628+0000", 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9319999999999994e-05 2024-09-20T19:37:10.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:4740 7.3 7:c0f7e14f:::smithi04268048-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e576)", 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:31:24.260916+0000", 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 346.62371585, 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.036628579000000001, 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 4740 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260916+0000", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260916+0000", 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260919+0000", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2459999999999999e-06 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260931+0000", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.1707e-05 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260960+0000", 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.898e-05 2024-09-20T19:37:10.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.260985+0000", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.4626000000000001e-05 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.261214+0000", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022967500000000001 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.261233+0000", 2024-09-20T19:37:10.915 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8244e-05 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.268359+0000", 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0071267470000000001 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.268481+0000", 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000121619 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.916 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.297356+0000", 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028874613 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.297446+0000", 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.0383999999999994e-05 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:31:24.297545+0000", 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8738000000000002e-05 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.917 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5211 7.a 7:5fdc189b:::smithi04268048-19:head [copy-from ver 719] snapc eb=[ea,e9,e7,e5,e2,e1,e0,df,dc,da,d6,cf,ce] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:09.348564+0000", 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 301.53606777599998, 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017692244999999999, 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5211 2024-09-20T19:37:10.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348564+0000", 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348564+0000", 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.919 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348568+0000", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9899999999999999e-06 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348581+0000", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2551999999999999e-05 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348615+0000", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4647999999999998e-05 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.920 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348632+0000", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.6562000000000002e-05 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.348831+0000", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000199163 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.349029+0000", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000197646 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.921 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.354803+0000", 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0057739289999999997 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.364625+0000", 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0098227339999999996 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.365596+0000", 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00097086299999999998 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.922 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.366185+0000", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00058844300000000002 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.366220+0000", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.4863000000000001e-05 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:09.366256+0000", 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6851999999999999e-05 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5369 7.3 7:c0f7e14f:::smithi04268048-43:head [copy-from ver 728] snapc f7=[f7,f6,f5,f4,f3,f0,ef,ed,ec,e7,e5,e1,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e642)", 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:30.387719+0000", 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 280.496912571, 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028714292999999998, 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5369 2024-09-20T19:37:10.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387719+0000", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387719+0000", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387724+0000", 2024-09-20T19:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.2259999999999999e-06 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387738+0000", 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3925e-05 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387764+0000", 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6639e-05 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387777+0000", 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2369e-05 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387857+0000", 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.0828000000000006e-05 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387864+0000", 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.6290000000000004e-06 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.927 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.391313+0000", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0034485000000000002 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.391608+0000", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029593799999999998 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,4", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.406323+0000", 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.014714272 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.928 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.414990+0000", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0086673670000000005 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.415385+0000", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00039460299999999999 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.416327+0000", 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00094183600000000004 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.929 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.416378+0000", 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1758999999999997e-05 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.416434+0000", 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5402000000000003e-05 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.930 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5370 7.3 7:c0f7e14f:::smithi04268048-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e642)", 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:30.387804+0000", 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 280.49682819899999, 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.024006853000000002, 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5370 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.931 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387804+0000", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387804+0000", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387805+0000", 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8599999999999997e-07 2024-09-20T19:37:10.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387808+0000", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.6049999999999998e-06 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387815+0000", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4030000000000004e-06 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387820+0000", 2024-09-20T19:37:10.933 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.22e-06 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.387984+0000", 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00016430199999999999 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.388013+0000", 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8782999999999999e-05 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.934 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.391958+0000", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0039453589999999998 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for blocked object", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.391993+0000", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.5165e-05 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.411414+0000", 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01942023 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.411593+0000", 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00017932899999999999 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:30.411811+0000", 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00021767100000000001 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.936 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5401 7.14 7:2924b69c:::smithi04268048-21:head [copy-from ver 757] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,ec,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e644)", 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:32.614463+0000", 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 278.27016918300001, 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.017341358000000001, 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5401 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.937 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614463+0000", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614463+0000", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614467+0000", 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.924e-06 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614479+0000", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2622999999999999e-05 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614511+0000", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.1466e-05 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614529+0000", 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8122999999999999e-05 2024-09-20T19:37:10.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614613+0000", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.4355000000000001e-05 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.614722+0000", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000109109 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.625664+0000", 2024-09-20T19:37:10.940 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010941378 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.631065+0000", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0054012499999999998 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.631135+0000", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9725000000000001e-05 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.631663+0000", 2024-09-20T19:37:10.941 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00052829099999999996 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.631742+0000", 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.9195000000000003e-05 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:32.631804+0000", 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1919000000000001e-05 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.942 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5404 7.a 7:5a4763d9:::smithi04268048-28:head [append 0~0] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.286504+0000", 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 277.59812765599997, 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020359245000000002, 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5404 2024-09-20T19:37:10.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286504+0000", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286504+0000", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286508+0000", 2024-09-20T19:37:10.944 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.9740000000000004e-06 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286519+0000", 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0375e-05 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286545+0000", 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6393999999999998e-05 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.945 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286558+0000", 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2714999999999999e-05 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286926+0000", 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000368654 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op must wait for map", 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286934+0000", 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.8180000000000004e-06 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.946 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295249+0000", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0083143409999999994 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295394+0000", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00014557400000000001 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.295617+0000", 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022250699999999999 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.947 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.304725+0000", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0091085809999999993 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.304850+0000", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012467899999999999 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.305002+0000", 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000151593 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.948 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.306676+0000", 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0016742980000000001 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.306864+0000", 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018774199999999999 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5405 7.a 7:5a4763d9:::smithi04268048-28:head [rollback f0] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:37:10.949 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.286589+0000", 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 277.59804288100003, 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.030939399999999999, 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5405 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.950 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286589+0000", 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286589+0000", 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286590+0000", 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.0100000000000001e-06 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.951 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286595+0000", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.6059999999999996e-06 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286605+0000", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.8670000000000006e-06 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286611+0000", 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.8920000000000003e-06 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.952 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287001+0000", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00038925199999999998 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287006+0000", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5330000000000002e-06 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.298939+0000", 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.011932745 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.953 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.299214+0000", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027521099999999998 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.299860+0000", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00064625200000000005 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.306912+0000", 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070516670000000002 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317218+0000", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.01030582 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317340+0000", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00012194299999999999 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317433+0000", 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.2682000000000005e-05 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317528+0000", 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.5920000000000003e-05 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5406 7.a 7:5a4763d9:::smithi04268048-28:head [append 0~0] snapc f8=[f8,f7,f6,f5,f4,f3,f0,ef,ed,e7,e5,e0,da,d6,cf] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:33.286621+0000", 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 277.59801105399998, 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.031405578000000003, 2024-09-20T19:37:10.956 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5406 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286621+0000", 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.957 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286621+0000", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286622+0000", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.0900000000000001e-07 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286625+0000", 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0199999999999999e-06 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286630+0000", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4729999999999999e-06 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.286636+0000", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.6250000000000004e-06 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287026+0000", 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00039035099999999999 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting_for_map not empty", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.287031+0000", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1410000000000004e-06 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.307012+0000", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019980736999999998 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.307532+0000", 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00052006999999999997 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.960 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.308025+0000", 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00049298500000000004 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317285+0000", 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0092600640000000001 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.961 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317580+0000", 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00029539900000000001 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317899+0000", 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00031812799999999999 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.317971+0000", 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.2648e-05 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.962 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:33.318027+0000", 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.5228e-05 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(client.14904.0:5616 7.0 e664/550 7:060c6a05:::smithi04268048-38:head v 664'762, mlcod=663'759)", 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:52.321788+0000", 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 258.56284408900001, 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.020208707999999999, 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.963 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.321788+0000", 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.321788+0000", 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.321789+0000", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.057e-06 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.328889+0000", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0070997109999999999 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.328947+0000", 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.8816000000000001e-05 2024-09-20T19:37:10.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.328973+0000", 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5380000000000001e-05 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.329212+0000", 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000239158 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.329277+0000", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4496e-05 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.341843+0000", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012566361 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:52.341997+0000", 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00015372899999999999 2024-09-20T19:37:10.967 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop_reply(osd.3.0:169 7.a e665/550 ondisk, result = 0)", 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:53.312111+0000", 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 257.572520994, 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.145353806, 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.968 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.312111+0000", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.312111+0000", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.312114+0000", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.36e-06 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.969 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.312126+0000", 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2046e-05 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.456029+0000", 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.14390240700000001 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.456055+0000", 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.6267000000000001e-05 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.970 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457432+0000", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0013765089999999999 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457442+0000", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.039e-05 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457465+0000", 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.2827e-05 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.971 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop_reply(osd.3.0:169 7.a e665/550 ondisk, result = 0)", 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:32:53.313014+0000", 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 257.57161785400001, 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.14437776699999999, 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.313014+0000", 2024-09-20T19:37:10.972 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.313014+0000", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.313017+0000", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.7760000000000002e-06 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.313026+0000", 2024-09-20T19:37:10.973 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.8170000000000005e-06 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.456026+0000", 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.142999981 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.456051+0000", 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5029e-05 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.974 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457348+0000", 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001297641 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457364+0000", 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.5305999999999999e-05 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:32:53.457392+0000", 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.8217000000000001e-05 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.975 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.4.0:194 7.1 e673/599 7:8715d6d2:::smithi04268048-16:105 v 673'572, mlcod=672'567)", 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:01.350591+0000", 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 249.534041242, 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016376499999999999, 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.350591+0000", 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.350591+0000", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.350593+0000", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.3460000000000001e-06 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.350625+0000", 2024-09-20T19:37:10.977 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.2233999999999997e-05 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.364381+0000", 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.013755972 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.364412+0000", 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.0669000000000001e-05 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.978 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.365083+0000", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00067142600000000003 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.365132+0000", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.8585999999999999e-05 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.366892+0000", 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001759573 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:01.366967+0000", 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 7.5693999999999997e-05 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5922 7.14 7:2a17ba7b:::smithi04268048-18:head [write 497705~577706,stat] snapc 10d=[10c,108,106,103,fd,fc,fb] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:23.331644+0000", 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 227.55298799100001, 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.028413121, 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.980 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5922 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.331644+0000", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.331644+0000", 2024-09-20T19:37:10.981 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.331648+0000", 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 4.3069999999999996e-06 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.331920+0000", 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00027146100000000002 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.982 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.333127+0000", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001207074 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.333139+0000", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.2476999999999999e-05 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.339208+0000", 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060689070000000001 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.339379+0000", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000171239 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.340004+0000", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00062501400000000002 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.352685+0000", 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.012680589000000001 2024-09-20T19:37:10.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.353873+0000", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001187465 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.359904+0000", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0060315519999999999 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.359987+0000", 2024-09-20T19:37:10.985 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.3051000000000002e-05 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.360057+0000", 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9985000000000002e-05 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5923 7.14 7:2a17ba7b:::smithi04268048-18:head [write 1642001~480939] snapc 10d=[10c,108,106,103,fd,fc,fb] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:23.335723+0000", 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 227.54890879600001, 2024-09-20T19:37:10.986 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.024525088, 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5923 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335723+0000", 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335723+0000", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335725+0000", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.1069999999999999e-06 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335849+0000", 2024-09-20T19:37:10.988 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000123468 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335862+0000", 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.3288000000000001e-05 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.335872+0000", 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.6970000000000005e-06 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.989 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.346358+0000", 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.010485763 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.346585+0000", 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00022740600000000001 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.346972+0000", 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00038736599999999999 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.352787+0000", 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0058149660000000004 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.353880+0000", 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0010929010000000001 2024-09-20T19:37:10.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.360142+0000", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0062614070000000001 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.360211+0000", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9429000000000002e-05 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.360248+0000", 2024-09-20T19:37:10.995 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.7289999999999997e-05 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5925 7.14 7:2a17ba7b:::smithi04268048-18:head [setxattr _header (58) in=7b,truncate 3697705] snapc 10d=[10c,108,106,103,fd,fc,fb] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:23.344332+0000", 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 227.540299527, 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019303358999999999, 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:10.996 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5925 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344332+0000", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:10.997 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344332+0000", 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344333+0000", 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.4e-07 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344342+0000", 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.7579999999999998e-06 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.998 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344348+0000", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.2770000000000002e-06 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344354+0000", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.7699999999999998e-06 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.352822+0000", 2024-09-20T19:37:10.999 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0084677669999999993 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.352939+0000", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000117235 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for subops from 2,5", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.353126+0000", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00018671599999999999 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "op_commit", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.354159+0000", 2024-09-20T19:37:11.000 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.001032956 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.360495+0000", 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.006336337 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "sub_op_commit_rec", 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363427+0000", 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0029317660000000001 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:11.001 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363531+0000", 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.00010446300000000001 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363636+0000", 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000104474 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_op(client.14904.0:5926 7.14 7:2a17ba7b:::smithi04268048-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e695)", 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:23.344361+0000", 2024-09-20T19:37:11.002 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 227.540270894, 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.019449724000000002, 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "started", 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "client_info": { 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "client": "client.14904", 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "client_addr": "v1:172.21.15.42:0/1545680571", 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "tid": 5926 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344361+0000", 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:11.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344361+0000", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344362+0000", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4499999999999997e-07 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344364+0000", 2024-09-20T19:37:11.004 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 2.5359999999999999e-06 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344370+0000", 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.1879999999999999e-06 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.344376+0000", 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.4820000000000002e-06 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:11.005 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.353757+0000", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0093809380000000001 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "waiting for rw locks", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.353849+0000", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.2223000000000001e-05 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363677+0000", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0098275859999999993 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:11.006 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363742+0000", 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.4976999999999998e-05 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:23.363811+0000", 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 6.9249000000000001e-05 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "description": "osd_repop(osd.6.0:1304 7.7 e698/550 7:e18d1735:::smithi04268048-41:f8 v 698'447, mlcod=694'444)", 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "initiated_at": "2024-09-20T19:33:26.394642+0000", 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 224.489989608, 2024-09-20T19:37:11.007 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.016894163, 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "continuous": false, 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "type_data": { 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "events": [ 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "initiated", 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.394642+0000", 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "header_read", 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.394642+0000", 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0 2024-09-20T19:37:11.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "throttled", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.394643+0000", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 5.1399999999999997e-07 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "all_read", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.394651+0000", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 8.5089999999999993e-06 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "dispatched", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.401607+0000", 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0069552850000000003 2024-09-20T19:37:11.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "queued_for_pg", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.401626+0000", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 1.8994000000000001e-05 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "reached_pg", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.401726+0000", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.000100653 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "started", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.401765+0000", 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 3.8998000000000002e-05 2024-09-20T19:37:11.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "commit_sent", 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.411445+0000", 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 0.0096799599999999996 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "event": "done", 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "time": "2024-09-20T19:33:26.411537+0000", 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: "duration": 9.1249999999999995e-05 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:11.011 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-20T19:37:11.012 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-20T19:37:11.012 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:37:11.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:11.140 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:11.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:11.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:11.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:11.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:11.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:11.665 INFO:tasks.rados.rados.0.smithi042.stdout:3385: done (3 left) 2024-09-20T19:37:11.666 INFO:tasks.rados.rados.0.smithi042.stdout:3386: done (2 left) 2024-09-20T19:37:11.666 INFO:tasks.rados.rados.0.smithi042.stdout:3387: done (1 left) 2024-09-20T19:37:11.666 INFO:tasks.rados.rados.0.smithi042.stdout:3388: done (0 left) 2024-09-20T19:37:11.666 INFO:tasks.rados.rados.0.smithi042.stdout:3389: snap_remove snap 378 2024-09-20T19:37:11.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:11.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:11.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:11.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:11.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[32075]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:11 smithi042 bash[22706]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:11.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:12.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:12.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: audit 2024-09-20T19:37:09.842005+0000 mgr.y (mgr.24491) 720 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:12.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:12.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: cluster 2024-09-20T19:37:10.447773+0000 mgr.y (mgr.24491) 721 : cluster [DBG] pgmap v1380: 178 pgs: 7 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 163 active+clean; 413 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-20T19:37:12.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:12.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:11 smithi162 bash[24557]: cluster 2024-09-20T19:37:10.661011+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2024-09-20T19:37:12.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:12.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:12.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:12.322 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:12.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:12.481 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:12.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:12.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:12.673 INFO:tasks.rados.rados.0.smithi042.stdout:3389: done (0 left) 2024-09-20T19:37:12.673 INFO:tasks.rados.rados.0.smithi042.stdout:3390: read oid 21 snap -1 2024-09-20T19:37:12.673 INFO:tasks.rados.rados.0.smithi042.stdout:3390: expect (ObjNum 1150 snap 377 seq_num 1150) 2024-09-20T19:37:12.673 INFO:tasks.rados.rados.0.smithi042.stdout:3391: write oid 41 current snap is 381 2024-09-20T19:37:12.673 INFO:tasks.rados.rados.0.smithi042.stdout:3391: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-09-20T19:37:12.693 INFO:tasks.rados.rados.0.smithi042.stdout:3391: writing smithi04268048-41 from 796962 to 1539305 tid 1 2024-09-20T19:37:12.706 INFO:tasks.rados.rados.0.smithi042.stdout:3391: writing smithi04268048-41 from 2031416 to 2469146 tid 2 2024-09-20T19:37:12.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:12.717 INFO:tasks.rados.rados.0.smithi042.stdout:3391: writing smithi04268048-41 from 2936739 to 3349906 tid 3 2024-09-20T19:37:12.717 INFO:tasks.rados.rados.0.smithi042.stdout:3392: snap_remove snap 380 2024-09-20T19:37:12.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:12 smithi162 bash[24557]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:12.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:12 smithi162 bash[24557]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:12.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:12.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:12.885 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:12.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:12.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:12.997 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:12 smithi042 bash[22706]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:12.997 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:12 smithi042 bash[22706]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:12.998 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:12 smithi042 bash[32075]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:12.998 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:12 smithi042 bash[32075]: cluster 2024-09-20T19:37:11.663012+0000 mon.a (mon.0) 2517 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2024-09-20T19:37:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:13.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:13.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:13.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:13.432 INFO:tasks.rados.rados.0.smithi042.stdout:3392: done (2 left) 2024-09-20T19:37:13.432 INFO:tasks.rados.rados.0.smithi042.stdout:3393: write (excl) oid 45 current snap is 381 2024-09-20T19:37:13.432 INFO:tasks.rados.rados.0.smithi042.stdout:3393: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-20T19:37:13.442 INFO:tasks.rados.rados.0.smithi042.stdout:3393: writing smithi04268048-45 from 413769 to 831361 tid 1 2024-09-20T19:37:13.460 INFO:tasks.rados.rados.0.smithi042.stdout:3393: writing smithi04268048-45 from 1573545 to 2271286 tid 2 2024-09-20T19:37:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3393: writing smithi04268048-45 from 2847809 to 3472248 tid 3 2024-09-20T19:37:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3394: delete oid 10 current snap is 381 2024-09-20T19:37:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3391: finishing write tid 1 to smithi04268048-41 2024-09-20T19:37:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3391: finishing write tid 2 to smithi04268048-41 2024-09-20T19:37:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3390: expect (ObjNum 1150 snap 377 seq_num 1150) 2024-09-20T19:37:13.502 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2239221 2024-09-20T19:37:13.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:13.534 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:13.538 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:13.623 INFO:tasks.rados.rados.0.smithi042.stdout:3391: finishing write tid 3 to smithi04268048-41 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: finishing write tid 4 to smithi04268048-41 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: finishing write tid 5 to smithi04268048-41 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: oid 41 updating version 0 to 1241 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: oid 41 updating version 1241 to 1242 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: oid 41 updating version 1242 to 1243 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: oid 41 version 1243 is already newer than 1240 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1243 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: left oid 41 (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3390: done (3 left) 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3391: done (2 left) 2024-09-20T19:37:13.624 INFO:tasks.rados.rados.0.smithi042.stdout:3394: done (1 left) 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3395: delete oid 30 current snap is 381 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: finishing write tid 1 to smithi04268048-45 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: finishing write tid 2 to smithi04268048-45 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: finishing write tid 3 to smithi04268048-45 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: finishing write tid 4 to smithi04268048-45 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: finishing write tid 5 to smithi04268048-45 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: oid 45 updating version 0 to 1574 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: oid 45 updating version 1574 to 1577 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: oid 45 version 1577 is already newer than 1576 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: oid 45 version 1577 is already newer than 1575 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1577 (ObjNum 1167 snap 381 seq_num 1167) dirty exists 2024-09-20T19:37:13.625 INFO:tasks.rados.rados.0.smithi042.stdout:3393: left oid 45 (ObjNum 1167 snap 381 seq_num 1167) 2024-09-20T19:37:13.628 INFO:tasks.rados.rados.0.smithi042.stdout:3393: done (1 left) 2024-09-20T19:37:13.628 INFO:tasks.rados.rados.0.smithi042.stdout:3395: done (0 left) 2024-09-20T19:37:13.628 INFO:tasks.rados.rados.0.smithi042.stdout:3396: write (excl) oid 48 current snap is 381 2024-09-20T19:37:13.628 INFO:tasks.rados.rados.0.smithi042.stdout:3396: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-20T19:37:13.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.644 INFO:tasks.rados.rados.0.smithi042.stdout:3396: writing smithi04268048-48 from 464190 to 1107529 tid 1 2024-09-20T19:37:13.658 INFO:tasks.rados.rados.0.smithi042.stdout:3396: writing smithi04268048-48 from 1748991 to 2194059 tid 2 2024-09-20T19:37:13.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:3396: writing smithi04268048-48 from 2930078 to 3504686 tid 3 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:3397: rollback oid 42 current snap is 381 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 363 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:3396: finishing write tid 1 to smithi04268048-48 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:3398: snap_create 2024-09-20T19:37:13.672 INFO:tasks.rados.rados.0.smithi042.stdout:3396: finishing write tid 2 to smithi04268048-48 2024-09-20T19:37:13.676 INFO:tasks.rados.rados.0.smithi042.stdout:3397: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:37:13.676 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1264 (ObjNum 33 snap 0 seq_num 1415525168) dirty dne 2024-09-20T19:37:13.676 INFO:tasks.rados.rados.0.smithi042.stdout:3396: finishing write tid 3 to smithi04268048-48 2024-09-20T19:37:13.676 INFO:tasks.rados.rados.0.smithi042.stdout:3396: finishing write tid 4 to smithi04268048-48 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: finishing write tid 5 to smithi04268048-48 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: oid 48 updating version 0 to 1079 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: oid 48 updating version 1079 to 1080 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: oid 48 version 1080 is already newer than 1078 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: oid 48 version 1080 is already newer than 1077 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1080 (ObjNum 1170 snap 381 seq_num 1170) dirty exists 2024-09-20T19:37:13.677 INFO:tasks.rados.rados.0.smithi042.stdout:3396: left oid 48 (ObjNum 1170 snap 381 seq_num 1170) 2024-09-20T19:37:13.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:13.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:13.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[22706]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:13.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:13 smithi042 bash[32075]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:13.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:13.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T19:37:14.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:14.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:12.449582+0000 mgr.y (mgr.24491) 722 : cluster [DBG] pgmap v1383: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 399 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:12.670408+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: cluster 2024-09-20T19:37:13.430139+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:14.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:13 smithi162 bash[24557]: audit 2024-09-20T19:37:13.447140+0000 mon.a (mon.0) 2520 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:14.061 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:37:14.159 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:37:14.160 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:37:14.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:14.189 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:14.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:14.314 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:14.317 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:37:14.318 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:37:14.336 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:37:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:37:14.354 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:37:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:37:14.371 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:37:14.371 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:37:14.393 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:37:14.394 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:37:14.400 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:14.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:14.418 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:37:14.418 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3396: done (2 left) 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3397: done (1 left) 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3398: done (0 left) 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3399: copy_from oid 21 from oid 19 current snap is 382 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3400: read oid 38 snap 381 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3400: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3401: read oid 7 snap 381 2024-09-20T19:37:14.439 INFO:tasks.rados.rados.0.smithi042.stdout:3401: expect deleted 2024-09-20T19:37:14.440 INFO:tasks.rados.rados.0.smithi042.stdout:3402: snap_remove snap 357 2024-09-20T19:37:14.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:14.444 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:37:14.444 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:37:14.469 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:37:14.469 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:37:14.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:14.563 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:14.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:14.691 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:14.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[22706]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:14.771 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:14.772 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:14 smithi042 bash[32075]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:14.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:14.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:14.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:14.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:15.053 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:15.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:13.960827+0000 mon.a (mon.0) 2521 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.291961+0000 mon.a (mon.0) 2522 : audit [DBG] from='client.? v1:172.21.15.42:0/1187490690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: cluster 2024-09-20T19:37:14.437236+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.689682+0000 mon.a (mon.0) 2524 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:37:15.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.690494+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:37:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:14 smithi162 bash[24557]: audit 2024-09-20T19:37:14.695804+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:15.076 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:15.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:15.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:15.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:15.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:15.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:15.465 INFO:tasks.rados.rados.0.smithi042.stdout:3402: done (3 left) 2024-09-20T19:37:15.466 INFO:tasks.rados.rados.0.smithi042.stdout:3403: write (excl) oid 37 current snap is 382 2024-09-20T19:37:15.466 INFO:tasks.rados.rados.0.smithi042.stdout:3403: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2024-09-20T19:37:15.467 INFO:tasks.rados.rados.0.smithi042.stdout:3403: writing smithi04268048-37 from 480997 to 915938 tid 1 2024-09-20T19:37:15.470 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:15.474 INFO:tasks.rados.rados.0.smithi042.stdout:3403: writing smithi04268048-37 from 1407473 to 2112552 tid 2 2024-09-20T19:37:15.482 INFO:tasks.rados.rados.0.smithi042.stdout:3403: writing smithi04268048-37 from 2557501 to 3343381 tid 3 2024-09-20T19:37:15.482 INFO:tasks.rados.rados.0.smithi042.stdout:3404: snap_create 2024-09-20T19:37:15.482 INFO:tasks.rados.rados.0.smithi042.stdout:3400: expect (ObjNum 904 snap 298 seq_num 904) 2024-09-20T19:37:15.482 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:15.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:15.632 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:15.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3399: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3399: finishing copy_from to smithi04268048-21 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 981 (ObjNum 1023 snap 336 seq_num 1023) dirty exists 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: finishing write tid 1 to smithi04268048-37 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: finishing write tid 2 to smithi04268048-37 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: finishing write tid 3 to smithi04268048-37 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: finishing write tid 4 to smithi04268048-37 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: finishing write tid 5 to smithi04268048-37 2024-09-20T19:37:15.707 INFO:tasks.rados.rados.0.smithi042.stdout:3403: oid 37 updating version 0 to 1166 2024-09-20T19:37:15.708 INFO:tasks.rados.rados.0.smithi042.stdout:3403: oid 37 version 1166 is already newer than 1165 2024-09-20T19:37:15.708 INFO:tasks.rados.rados.0.smithi042.stdout:3403: oid 37 version 1166 is already newer than 1163 2024-09-20T19:37:15.708 INFO:tasks.rados.rados.0.smithi042.stdout:3403: oid 37 version 1166 is already newer than 1164 2024-09-20T19:37:15.708 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1166 (ObjNum 1171 snap 382 seq_num 1171) dirty exists 2024-09-20T19:37:15.708 INFO:tasks.rados.rados.0.smithi042.stdout:3403: left oid 37 (ObjNum 1171 snap 382 seq_num 1171) 2024-09-20T19:37:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:15.723 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[32075]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:15.723 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[32075]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[32075]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[32075]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[22706]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[22706]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[22706]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:15.724 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:15 smithi042 bash[22706]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:15.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:15.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:15.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:15.985 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:15.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:16.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:15 smithi162 bash[24557]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:16.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:15 smithi162 bash[24557]: cluster 2024-09-20T19:37:14.450309+0000 mgr.y (mgr.24491) 723 : cluster [DBG] pgmap v1387: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2024-09-20T19:37:16.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:15 smithi162 bash[24557]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:16.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:15 smithi162 bash[24557]: cluster 2024-09-20T19:37:15.450547+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2024-09-20T19:37:16.088 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:16.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:16.351 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:16.380 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:16.459 INFO:tasks.rados.rados.0.smithi042.stdout:3399: done (4 left) 2024-09-20T19:37:16.459 INFO:tasks.rados.rados.0.smithi042.stdout:3400: done (3 left) 2024-09-20T19:37:16.459 INFO:tasks.rados.rados.0.smithi042.stdout:3401: done (2 left) 2024-09-20T19:37:16.459 INFO:tasks.rados.rados.0.smithi042.stdout:3403: done (1 left) 2024-09-20T19:37:16.459 INFO:tasks.rados.rados.0.smithi042.stdout:3404: done (0 left) 2024-09-20T19:37:16.460 INFO:tasks.rados.rados.0.smithi042.stdout:3405: snap_remove snap 381 2024-09-20T19:37:16.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:16.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:16.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:16.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:16.681 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:16.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:16.772 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:16.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:16.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:37:16.957 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:37:16.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-20T19:37:17.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:17.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:17.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:17.158 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:17.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:17.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:17.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:17.473 INFO:tasks.rados.rados.0.smithi042.stdout:3405: done (0 left) 2024-09-20T19:37:17.473 INFO:tasks.rados.rados.0.smithi042.stdout:3406: read oid 2 snap -1 2024-09-20T19:37:17.473 INFO:tasks.rados.rados.0.smithi042.stdout:3406: expect deleted 2024-09-20T19:37:17.473 INFO:tasks.rados.rados.0.smithi042.stdout:3407: rollback oid 16 current snap is 383 2024-09-20T19:37:17.474 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 324 2024-09-20T19:37:17.474 INFO:tasks.rados.rados.0.smithi042.stdout:3408: read oid 28 snap -1 2024-09-20T19:37:17.474 INFO:tasks.rados.rados.0.smithi042.stdout:3408: expect (ObjNum 1165 snap 380 seq_num 1165) 2024-09-20T19:37:17.474 INFO:tasks.rados.rados.0.smithi042.stdout:3409: snap_create 2024-09-20T19:37:17.480 INFO:tasks.rados.rados.0.smithi042.stdout:3407: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:37:17.484 INFO:tasks.rados.rados.0.smithi042.stdout:3407: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:37:17.484 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1438 (ObjNum 967 snap 320 seq_num 967) dirty exists 2024-09-20T19:37:17.485 INFO:tasks.rados.rados.0.smithi042.stdout:3408: expect (ObjNum 1165 snap 380 seq_num 1165) 2024-09-20T19:37:17.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:17 smithi162 bash[24557]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:17 smithi162 bash[24557]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:17 smithi162 bash[24557]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:17 smithi162 bash[24557]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:17.642 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3339261 2024-09-20T19:37:17.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:17.673 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:17.675 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1764815 2024-09-20T19:37:17.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:17.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:17.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[22706]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[22706]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[22706]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.912 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[22706]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[32075]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[32075]: cluster 2024-09-20T19:37:16.450841+0000 mgr.y (mgr.24491) 724 : cluster [DBG] pgmap v1390: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 403 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 270 B/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-20T19:37:17.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[32075]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.913 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:17 smithi042 bash[32075]: cluster 2024-09-20T19:37:16.457986+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2024-09-20T19:37:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:17.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:18.031 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:18.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:18.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:18.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:18.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:18.466 INFO:tasks.rados.rados.0.smithi042.stdout:3406: done (3 left) 2024-09-20T19:37:18.466 INFO:tasks.rados.rados.0.smithi042.stdout:3407: done (2 left) 2024-09-20T19:37:18.466 INFO:tasks.rados.rados.0.smithi042.stdout:3408: done (1 left) 2024-09-20T19:37:18.466 INFO:tasks.rados.rados.0.smithi042.stdout:3409: done (0 left) 2024-09-20T19:37:18.466 INFO:tasks.rados.rados.0.smithi042.stdout:3410: snap_create 2024-09-20T19:37:18.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:18 smithi162 bash[24557]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:18 smithi162 bash[24557]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.569 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:18.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:18 smithi042 bash[32075]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:18 smithi042 bash[32075]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:18 smithi042 bash[22706]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:18 smithi042 bash[22706]: cluster 2024-09-20T19:37:17.469355+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2024-09-20T19:37:18.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:18.805 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:18.831 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:18.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.961 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:18.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:19.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:19.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:19.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:19.348 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:19.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:19.474 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:37:19.476 INFO:tasks.rados.rados.0.smithi042.stdout:3410: done (0 left) 2024-09-20T19:37:19.476 INFO:tasks.rados.rados.0.smithi042.stdout:3411: snap_remove snap 377 2024-09-20T19:37:19.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:19.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:37:19.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:19.625 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:19.727 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:19.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[22706]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.741 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.741 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.741 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.741 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:19 smithi042 bash[32075]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:18.451391+0000 mgr.y (mgr.24491) 725 : cluster [DBG] pgmap v1392: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 392 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.8 MiB/s wr, 15 op/s 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:18.465118+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:18.566461+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.42:0/327253657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:18.566815+0000 mon.a (mon.0) 2531 : audit [INF] from='client.25997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:37:19.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:19.461353+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: audit 2024-09-20T19:37:19.463470+0000 mon.a (mon.0) 2533 : audit [INF] from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:37:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[24557]: cluster 2024-09-20T19:37:19.471365+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2024-09-20T19:37:19.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:19.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:20.004 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:20.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:20.119 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:20.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:20.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:20.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:20.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:20.394 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:20.471 INFO:tasks.rados.rados.0.smithi042.stdout:3411: done (0 left) 2024-09-20T19:37:20.472 INFO:tasks.rados.rados.0.smithi042.stdout:3412: read oid 48 snap -1 2024-09-20T19:37:20.472 INFO:tasks.rados.rados.0.smithi042.stdout:3412: expect (ObjNum 1170 snap 381 seq_num 1170) 2024-09-20T19:37:20.472 INFO:tasks.rados.rados.0.smithi042.stdout:3413: rollback oid 3 current snap is 385 2024-09-20T19:37:20.472 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 324 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3414: rollback oid 16 current snap is 385 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 363 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3415: read oid 8 snap 384 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3415: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3416: read oid 23 snap -1 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3416: expect deleted 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3417: rollback oid 7 current snap is 385 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 363 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3418: copy_from oid 42 from oid 17 current snap is 385 2024-09-20T19:37:20.473 INFO:tasks.rados.rados.0.smithi042.stdout:3419: rollback oid 19 current snap is 385 2024-09-20T19:37:20.474 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 19 to 319 2024-09-20T19:37:20.474 INFO:tasks.rados.rados.0.smithi042.stdout:3420: snap_remove snap 331 2024-09-20T19:37:20.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:20.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:20.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:20.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:20.784 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:20.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:20.916 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:20.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:21.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:21.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:21.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:21.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:21.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:21.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:21.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:21.436 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:21.483 INFO:tasks.rados.rados.0.smithi042.stdout:3420: done (8 left) 2024-09-20T19:37:21.483 INFO:tasks.rados.rados.0.smithi042.stdout:3421: snap_remove snap 383 2024-09-20T19:37:21.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.551 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.552 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:21 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:21.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: audit 2024-09-20T19:37:19.853425+0000 mgr.y (mgr.24491) 726 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.192932+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.195448+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.228978+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.230869+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.451926+0000 mgr.y (mgr.24491) 727 : cluster [DBG] pgmap v1395: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 386 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.469941+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.779414+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub starts 2024-09-20T19:37:21.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.685 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:21 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.784053+0000 osd.6 (osd.6) 205 : cluster [DBG] 4.a scrub ok 2024-09-20T19:37:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:21.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:21.828 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:21.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:21.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:21.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:21.974 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:21.974 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:22.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:22.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:22.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:22.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:22.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:22.485 INFO:tasks.rados.rados.0.smithi042.stdout:3421: done (8 left) 2024-09-20T19:37:22.485 INFO:tasks.rados.rados.0.smithi042.stdout:3422: copy_from oid 20 from oid 12 current snap is 385 2024-09-20T19:37:22.488 INFO:tasks.rados.rados.0.smithi042.stdout:3412: expect (ObjNum 1170 snap 381 seq_num 1170) 2024-09-20T19:37:22.495 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:22.529 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:22.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.592 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.593 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:22 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:3419: finishing rollback tid 0 to smithi04268048-19 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:3419: finishing rollback tid 1 to smithi04268048-19 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1187 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:3414: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:3414: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1440 (ObjNum 145 snap 0 seq_num 1415626576) dirty dne 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:3417: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:37:22.599 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1257 (ObjNum 129 snap 0 seq_num 1415928424) dirty dne 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3413: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3413: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3413: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 970 (ObjNum 949 snap 315 seq_num 949) dirty exists 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3412: done (8 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3413: done (7 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3414: done (6 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3416: done (5 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3417: done (4 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3419: done (3 left) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3423: delete oid 13 current snap is 385 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:3415: expect (ObjNum 970 snap 321 seq_num 970) 2024-09-20T19:37:22.600 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3418: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3418: finishing copy_from to smithi04268048-42 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3422: finishing copy_from to smithi04268048-20 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3422: got expected ENOENT (src dne) 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3422: finishing copy_from racing read to smithi04268048-20 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3415: done (3 left) 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3418: done (2 left) 2024-09-20T19:37:22.696 INFO:tasks.rados.rados.0.smithi042.stdout:3422: done (1 left) 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3423: done (0 left) 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3424: copy_from oid 16 from oid 41 current snap is 385 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3425: read oid 4 snap -1 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3425: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3426: rollback oid 5 current snap is 385 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 5 to 319 2024-09-20T19:37:22.697 INFO:tasks.rados.rados.0.smithi042.stdout:3427: snap_remove snap 324 2024-09-20T19:37:22.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:22.749 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:22.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.589366+0000 osd.2 (osd.2) 285 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:37:22.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.590428+0000 osd.2 (osd.2) 286 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:37:22.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.941708+0000 osd.1 (osd.1) 319 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:20.943768+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.153673+0000 osd.0 (osd.0) 99 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.156004+0000 osd.0 (osd.0) 100 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.479340+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.731132+0000 osd.6 (osd.6) 206 : cluster [DBG] 6.c scrub starts 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.733735+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.c scrub ok 2024-09-20T19:37:22.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.846956+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:22 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.848488+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:22.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:22.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:22.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:22.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:23.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:23.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.138 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:23.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:23.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:23.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:23.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:23.434 INFO:tasks.rados.rados.0.smithi042.stdout:3427: done (3 left) 2024-09-20T19:37:23.434 INFO:tasks.rados.rados.0.smithi042.stdout:3428: read oid 29 snap -1 2024-09-20T19:37:23.434 INFO:tasks.rados.rados.0.smithi042.stdout:3428: expect deleted 2024-09-20T19:37:23.435 INFO:tasks.rados.rados.0.smithi042.stdout:3429: copy_from oid 9 from oid 50 current snap is 385 2024-09-20T19:37:23.435 INFO:tasks.rados.rados.0.smithi042.stdout:3430: write oid 28 current snap is 385 2024-09-20T19:37:23.435 INFO:tasks.rados.rados.0.smithi042.stdout:3430: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-20T19:37:23.439 INFO:tasks.rados.rados.0.smithi042.stdout:3430: writing smithi04268048-28 from 514611 to 1100050 tid 1 2024-09-20T19:37:23.445 INFO:tasks.rados.rados.0.smithi042.stdout:3430: writing smithi04268048-28 from 1524437 to 2116832 tid 2 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3430: writing smithi04268048-28 from 2895994 to 3304418 tid 3 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3431: snap_create 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3426: finishing rollback tid 0 to smithi04268048-5 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3426: finishing rollback tid 1 to smithi04268048-5 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3426: finishing rollback tid 2 to smithi04268048-5 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1154 (ObjNum 834 snap 276 seq_num 834) dirty exists 2024-09-20T19:37:23.452 INFO:tasks.rados.rados.0.smithi042.stdout:3425: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:23.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:23.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:23.551 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:23.603 INFO:tasks.rados.rados.0.smithi042.stdout:3424: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3424: finishing copy_from to smithi04268048-16 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1243 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: finishing write tid 1 to smithi04268048-28 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: finishing write tid 2 to smithi04268048-28 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: finishing write tid 3 to smithi04268048-28 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: finishing write tid 4 to smithi04268048-28 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3429: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: finishing write tid 5 to smithi04268048-28 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: oid 28 updating version 0 to 1191 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: oid 28 version 1191 is already newer than 1189 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: oid 28 version 1191 is already newer than 1188 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:3430: oid 28 version 1191 is already newer than 1190 2024-09-20T19:37:23.604 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1191 (ObjNum 1173 snap 385 seq_num 1173) dirty exists 2024-09-20T19:37:23.605 INFO:tasks.rados.rados.0.smithi042.stdout:3430: left oid 28 (ObjNum 1173 snap 385 seq_num 1173) 2024-09-20T19:37:23.605 INFO:tasks.rados.rados.0.smithi042.stdout:3429: finishing copy_from to smithi04268048-9 2024-09-20T19:37:23.605 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 967 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-20T19:37:23.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.643 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:23 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.658 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:23.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.783 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.609515+0000 osd.2 (osd.2) 287 : cluster [DBG] 6.f scrub starts 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:21.611758+0000 osd.2 (osd.2) 288 : cluster [DBG] 6.f scrub ok 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.452891+0000 mgr.y (mgr.24491) 728 : cluster [DBG] pgmap v1398: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 367 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 759 KiB/s wr, 7 op/s 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.481839+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.661926+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e scrub starts 2024-09-20T19:37:23.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.663697+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.e scrub ok 2024-09-20T19:37:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:23 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.430939+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2024-09-20T19:37:23.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.915 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:23.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:24.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:24.019 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:24.045 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:24.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:24.170 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:37:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:37:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:37:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:37:24.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.300 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:37:24.300 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:37:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:24.401 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:24.436 INFO:tasks.rados.rados.0.smithi042.stdout:3424: done (6 left) 2024-09-20T19:37:24.436 INFO:tasks.rados.rados.0.smithi042.stdout:3425: done (5 left) 2024-09-20T19:37:24.436 INFO:tasks.rados.rados.0.smithi042.stdout:3426: done (4 left) 2024-09-20T19:37:24.436 INFO:tasks.rados.rados.0.smithi042.stdout:3428: done (3 left) 2024-09-20T19:37:24.437 INFO:tasks.rados.rados.0.smithi042.stdout:3429: done (2 left) 2024-09-20T19:37:24.437 INFO:tasks.rados.rados.0.smithi042.stdout:3430: done (1 left) 2024-09-20T19:37:24.437 INFO:tasks.rados.rados.0.smithi042.stdout:3431: done (0 left) 2024-09-20T19:37:24.437 INFO:tasks.rados.rados.0.smithi042.stdout:3432: snap_remove snap 363 2024-09-20T19:37:24.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.545 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:24 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.977196+0000 osd.1 (osd.1) 321 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:22.980126+0000 osd.1 (osd.1) 322 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.117468+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.121272+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.686890+0000 osd.4 (osd.4) 149 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:37:24.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.690400+0000 osd.4 (osd.4) 150 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:37:24.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.557 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:24 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.433562+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2024-09-20T19:37:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:24.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:24.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:24.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:24.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:24.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:25.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:25.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:25.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:25.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:25.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:25.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:25.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3432: done (0 left) 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3433: read oid 49 snap -1 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3433: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3434: read oid 27 snap 384 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3434: expect (ObjNum 1123 snap 368 seq_num 1123) 2024-09-20T19:37:25.451 INFO:tasks.rados.rados.0.smithi042.stdout:3435: snap_remove snap 319 2024-09-20T19:37:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:25.571 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:25.599 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:25.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:25.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:25.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:25 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:25.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.930459+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:23.932200+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.116598+0000 osd.0 (osd.0) 103 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.118304+0000 osd.0 (osd.0) 104 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.453443+0000 mgr.y (mgr.24491) 729 : cluster [DBG] pgmap v1402: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.652039+0000 osd.4 (osd.4) 151 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.658550+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:25 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.445566+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2024-09-20T19:37:25.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:25.992 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:25.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:26.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:26.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:26.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:26.257 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:26.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:26.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:26.450 INFO:tasks.rados.rados.0.smithi042.stdout:3435: done (2 left) 2024-09-20T19:37:26.450 INFO:tasks.rados.rados.0.smithi042.stdout:3436: snap_create 2024-09-20T19:37:26.450 INFO:tasks.rados.rados.0.smithi042.stdout:3433: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:37:26.450 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:26.489 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:26.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:26.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:26.561 INFO:tasks.rados.rados.0.smithi042.stdout:3434: expect (ObjNum 1123 snap 368 seq_num 1123) 2024-09-20T19:37:26.569 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:26.591 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2281719 2024-09-20T19:37:26.610 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2241263 2024-09-20T19:37:26.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:26.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:26 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:26.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:26.779 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:26.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.598114+0000 osd.2 (osd.2) 289 : cluster [DBG] 7.a scrub starts 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:24.599910+0000 osd.2 (osd.2) 290 : cluster [DBG] 7.a scrub ok 2024-09-20T19:37:26.888 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.133378+0000 osd.0 (osd.0) 105 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.135636+0000 osd.0 (osd.0) 106 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.728769+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.730220+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:26 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.447736+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2024-09-20T19:37:26.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:26.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:27.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:27.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.162 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:27.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:27.293 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:27.395 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3433: done (2 left) 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3434: done (1 left) 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3436: done (0 left) 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3437: read oid 1 snap -1 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3437: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3438: rollback oid 9 current snap is 387 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 385 2024-09-20T19:37:27.454 INFO:tasks.rados.rados.0.smithi042.stdout:3439: snap_remove snap 379 2024-09-20T19:37:27.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:27.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:27.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.669 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.670 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.671 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:27 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.682 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:37:27.696 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:37:27.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-20T19:37:27.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:27.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.568570+0000 osd.2 (osd.2) 291 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:37:27.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.575971+0000 osd.2 (osd.2) 292 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:37:27.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.977294+0000 osd.1 (osd.1) 325 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:25.979332+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.453984+0000 mgr.y (mgr.24491) 730 : cluster [DBG] pgmap v1405: 178 pgs: 10 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 156 active+clean; 358 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 390 KiB/s wr, 4 op/s 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:27 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.452316+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2024-09-20T19:37:27.801 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T19:37:27.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:27.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:27.938 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:27.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:28.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:28.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:28.155 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:37:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:37:28.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:28.175 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:37:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:37:28.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:28.192 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:37:28.192 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:37:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:28.209 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:37:28.209 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:37:28.230 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:37:28.230 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:37:28.252 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:37:28.252 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:37:28.273 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:37:28.274 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:37:28.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:28.296 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:37:28.296 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:37:28.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:28.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:28.445 INFO:tasks.rados.rados.0.smithi042.stdout:3439: done (2 left) 2024-09-20T19:37:28.445 INFO:tasks.rados.rados.0.smithi042.stdout:3440: write (excl) oid 46 current snap is 387 2024-09-20T19:37:28.445 INFO:tasks.rados.rados.0.smithi042.stdout:3440: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-20T19:37:28.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:28.466 INFO:tasks.rados.rados.0.smithi042.stdout:3440: writing smithi04268048-46 from 531418 to 1192106 tid 1 2024-09-20T19:37:28.489 INFO:tasks.rados.rados.0.smithi042.stdout:3440: writing smithi04268048-46 from 1982919 to 2435325 tid 2 2024-09-20T19:37:28.504 INFO:tasks.rados.rados.0.smithi042.stdout:3440: writing smithi04268048-46 from 3039770 to 3731418 tid 3 2024-09-20T19:37:28.504 INFO:tasks.rados.rados.0.smithi042.stdout:3441: copy_from oid 42 from oid 5 current snap is 387 2024-09-20T19:37:28.504 INFO:tasks.rados.rados.0.smithi042.stdout:3437: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:37:28.504 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:28.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:28.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.565 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.565 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.565 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.565 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:28 smithi162 bash[24557]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:28.645 INFO:tasks.rados.rados.0.smithi042.stdout:3438: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:37:28.645 INFO:tasks.rados.rados.0.smithi042.stdout:3438: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:37:28.645 INFO:tasks.rados.rados.0.smithi042.stdout:3438: finishing rollback tid 2 to smithi04268048-9 2024-09-20T19:37:28.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1142 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-20T19:37:28.645 INFO:tasks.rados.rados.0.smithi042.stdout:3440: finishing write tid 1 to smithi04268048-46 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: finishing write tid 2 to smithi04268048-46 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: finishing write tid 3 to smithi04268048-46 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: finishing write tid 4 to smithi04268048-46 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: finishing write tid 5 to smithi04268048-46 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: oid 46 updating version 0 to 1248 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: oid 46 updating version 1248 to 1249 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: oid 46 version 1249 is already newer than 1247 2024-09-20T19:37:28.646 INFO:tasks.rados.rados.0.smithi042.stdout:3440: oid 46 updating version 1249 to 1250 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1250 (ObjNum 1174 snap 387 seq_num 1174) dirty exists 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:3440: left oid 46 (ObjNum 1174 snap 387 seq_num 1174) 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:3437: done (3 left) 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:3438: done (2 left) 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:3440: done (1 left) 2024-09-20T19:37:28.647 INFO:tasks.rados.rados.0.smithi042.stdout:3442: snap_remove snap 382 2024-09-20T19:37:28.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:28.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[22706]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.981298+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:26.984363+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.631133+0000 osd.4 (osd.4) 153 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:27.633071+0000 osd.4 (osd.4) 154 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: audit 2024-09-20T19:37:28.136141+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.42:0/1803987365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.438274+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2024-09-20T19:37:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:28 smithi042 bash[32075]: audit 2024-09-20T19:37:28.492445+0000 mon.a (mon.0) 2544 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:28.705 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:28.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:28.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:28.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:28.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:28.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:29.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:29.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:29.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:29.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:29.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3442: done (1 left) 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3443: read oid 26 snap -1 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3443: expect (ObjNum 1154 snap 378 seq_num 1154) 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3444: rollback oid 22 current snap is 387 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 384 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3441: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3441: finishing copy_from to smithi04268048-42 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1154 (ObjNum 834 snap 276 seq_num 834) dirty exists 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3441: done (2 left) 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3445: read oid 1 snap 384 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3445: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:37:29.513 INFO:tasks.rados.rados.0.smithi042.stdout:3446: delete oid 37 current snap is 387 2024-09-20T19:37:29.524 INFO:tasks.rados.rados.0.smithi042.stdout:3444: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:37:29.524 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 0 (ObjNum 193 snap 0 seq_num 1415638240) dirty dne 2024-09-20T19:37:29.536 INFO:tasks.rados.rados.0.smithi042.stdout:3444: done (3 left) 2024-09-20T19:37:29.537 INFO:tasks.rados.rados.0.smithi042.stdout:3446: done (2 left) 2024-09-20T19:37:29.537 INFO:tasks.rados.rados.0.smithi042.stdout:3447: read oid 16 snap -1 2024-09-20T19:37:29.537 INFO:tasks.rados.rados.0.smithi042.stdout:3447: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:37:29.537 INFO:tasks.rados.rados.0.smithi042.stdout:3448: write oid 20 current snap is 387 2024-09-20T19:37:29.537 INFO:tasks.rados.rados.0.smithi042.stdout:3448: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-20T19:37:29.543 INFO:tasks.rados.rados.0.smithi042.stdout:3448: writing smithi04268048-20 from 565032 to 976218 tid 1 2024-09-20T19:37:29.549 INFO:tasks.rados.rados.0.smithi042.stdout:3448: writing smithi04268048-20 from 1583530 to 2039605 tid 2 2024-09-20T19:37:29.558 INFO:tasks.rados.rados.0.smithi042.stdout:3448: writing smithi04268048-20 from 2578263 to 3336856 tid 3 2024-09-20T19:37:29.558 INFO:tasks.rados.rados.0.smithi042.stdout:3449: write oid 48 current snap is 387 2024-09-20T19:37:29.558 INFO:tasks.rados.rados.0.smithi042.stdout:3449: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-20T19:37:29.564 INFO:tasks.rados.rados.0.smithi042.stdout:3449: writing smithi04268048-48 from 581839 to 1068274 tid 1 2024-09-20T19:37:29.571 INFO:tasks.rados.rados.0.smithi042.stdout:3449: writing smithi04268048-48 from 1642012 to 2241745 tid 2 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3449: writing smithi04268048-48 from 3005686 to 3575551 tid 3 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3450: read oid 11 snap -1 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3450: expect (ObjNum 1147 snap 376 seq_num 1147) 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3451: read oid 40 snap 385 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3451: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:3452: rollback oid 42 current snap is 387 2024-09-20T19:37:29.579 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 384 2024-09-20T19:37:29.580 INFO:tasks.rados.rados.0.smithi042.stdout:3445: expect (ObjNum 1089 snap 362 seq_num 1089) 2024-09-20T19:37:29.580 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 359501 2024-09-20T19:37:29.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:29.654 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1478699 2024-09-20T19:37:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:29 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.707 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1577286 2024-09-20T19:37:29.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:29.727 INFO:tasks.rados.rados.0.smithi042.stdout:3447: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:37:29.738 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2600342 2024-09-20T19:37:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:29.793 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:29.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.454832+0000 mgr.y (mgr.24491) 731 : cluster [DBG] pgmap v1408: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 285 MiB data, 1004 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.706623+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:37:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.709098+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:37:29.832 INFO:tasks.rados.rados.0.smithi042.stdout:3443: expect (ObjNum 1154 snap 378 seq_num 1154) 2024-09-20T19:37:29.833 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:29.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.873 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:29.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:29.895 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:29.957 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:29.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:30.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3448: finishing write tid 1 to smithi04268048-20 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3448: finishing write tid 2 to smithi04268048-20 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3443: done (7 left) 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3445: done (6 left) 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3447: done (5 left) 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3453: write oid 47 current snap is 387 2024-09-20T19:37:30.020 INFO:tasks.rados.rados.0.smithi042.stdout:3448: finishing write tid 3 to smithi04268048-20 2024-09-20T19:37:30.021 INFO:tasks.rados.rados.0.smithi042.stdout:3453: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-20T19:37:30.026 INFO:tasks.rados.rados.0.smithi042.stdout:3453: writing smithi04268048-47 from 598646 to 1160330 tid 1 2024-09-20T19:37:30.032 INFO:tasks.rados.rados.0.smithi042.stdout:3453: writing smithi04268048-47 from 1700494 to 2160238 tid 2 2024-09-20T19:37:30.039 INFO:tasks.rados.rados.0.smithi042.stdout:3453: writing smithi04268048-47 from 2633109 to 3414246 tid 3 2024-09-20T19:37:30.039 INFO:tasks.rados.rados.0.smithi042.stdout:3454: snap_remove snap 386 2024-09-20T19:37:30.104 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:30.129 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:30.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:30.243 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:30.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:30.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:30.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:30.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:30.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:30.531 INFO:tasks.rados.rados.0.smithi042.stdout:3454: done (6 left) 2024-09-20T19:37:30.531 INFO:tasks.rados.rados.0.smithi042.stdout:3455: delete oid 36 current snap is 387 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: finishing write tid 4 to smithi04268048-20 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: finishing write tid 5 to smithi04268048-20 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: oid 20 updating version 0 to 1236 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: oid 20 version 1236 is already newer than 1235 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: oid 20 version 1236 is already newer than 1234 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: oid 20 updating version 1236 to 1237 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1237 (ObjNum 1176 snap 387 seq_num 1176) dirty exists 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3448: left oid 20 (ObjNum 1176 snap 387 seq_num 1176) 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3449: finishing write tid 1 to smithi04268048-48 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3449: finishing write tid 2 to smithi04268048-48 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:3451: expect (ObjNum 897 snap 295 seq_num 897) 2024-09-20T19:37:30.532 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:30.601 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 927595 2024-09-20T19:37:30.623 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: finishing write tid 3 to smithi04268048-48 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: finishing write tid 4 to smithi04268048-48 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: finishing write tid 5 to smithi04268048-48 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: oid 48 updating version 0 to 1101 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: oid 48 version 1101 is already newer than 1099 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: oid 48 version 1101 is already newer than 1098 2024-09-20T19:37:30.624 INFO:tasks.rados.rados.0.smithi042.stdout:3449: oid 48 version 1101 is already newer than 1100 2024-09-20T19:37:30.625 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1101 (ObjNum 1177 snap 387 seq_num 1177) dirty exists 2024-09-20T19:37:30.625 INFO:tasks.rados.rados.0.smithi042.stdout:3449: left oid 48 (ObjNum 1177 snap 387 seq_num 1177) 2024-09-20T19:37:30.625 INFO:tasks.rados.rados.0.smithi042.stdout:3450: expect (ObjNum 1147 snap 376 seq_num 1147) 2024-09-20T19:37:30.650 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:30.650 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2818889 2024-09-20T19:37:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:30.661 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:30.718 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1162174 2024-09-20T19:37:30.752 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3452: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3452: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1288 (ObjNum 129 snap 0 seq_num 1415213920) dirty dne 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3448: done (6 left) 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3449: done (5 left) 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3450: done (4 left) 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3451: done (3 left) 2024-09-20T19:37:30.755 INFO:tasks.rados.rados.0.smithi042.stdout:3452: done (2 left) 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3455: done (1 left) 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3456: delete oid 39 current snap is 387 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: finishing write tid 1 to smithi04268048-47 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: finishing write tid 2 to smithi04268048-47 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: finishing write tid 3 to smithi04268048-47 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: finishing write tid 4 to smithi04268048-47 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: finishing write tid 5 to smithi04268048-47 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: oid 47 updating version 0 to 1289 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: oid 47 updating version 1289 to 1290 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: oid 47 updating version 1290 to 1291 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:3453: oid 47 updating version 1291 to 1292 2024-09-20T19:37:30.756 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 1292 (ObjNum 1178 snap 387 seq_num 1178) dirty exists 2024-09-20T19:37:30.757 INFO:tasks.rados.rados.0.smithi042.stdout:3453: left oid 47 (ObjNum 1178 snap 387 seq_num 1178) 2024-09-20T19:37:30.758 INFO:tasks.rados.rados.0.smithi042.stdout:3453: done (1 left) 2024-09-20T19:37:30.758 INFO:tasks.rados.rados.0.smithi042.stdout:3456: done (0 left) 2024-09-20T19:37:30.758 INFO:tasks.rados.rados.0.smithi042.stdout:3457: snap_remove snap 385 2024-09-20T19:37:30.764 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.764 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.764 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.764 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.766 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:30 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.777 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:30.798 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.959850+0000 osd.1 (osd.1) 329 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:37:30.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:28.963637+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:37:30.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:30 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.509863+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2024-09-20T19:37:30.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:30.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:31.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:31.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:31.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:31.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:31.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:31.295 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:31.308 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:37:31.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:31.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:31.527 INFO:tasks.rados.rados.0.smithi042.stdout:3457: done (0 left) 2024-09-20T19:37:31.527 INFO:tasks.rados.rados.0.smithi042.stdout:3458: snap_remove snap 384 2024-09-20T19:37:31.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:31.657 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[32075]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:31 smithi042 bash[22706]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.680 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:31.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.545806+0000 osd.2 (osd.2) 293 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:29.555243+0000 osd.2 (osd.2) 294 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: audit 2024-09-20T19:37:29.863024+0000 mgr.y (mgr.24491) 732 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.455575+0000 mgr.y (mgr.24491) 733 : cluster [DBG] pgmap v1410: 178 pgs: 10 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 155 active+clean; 261 MiB data, 977 MiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T19:37:31.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.527395+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2024-09-20T19:37:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.641929+0000 osd.7 (osd.7) 213 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:37:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:31 smithi162 bash[24557]: cluster 2024-09-20T19:37:30.645953+0000 osd.7 (osd.7) 214 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:37:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:31.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:31.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:32.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:32.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:32.177 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:32.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:32.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:32.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:32.457 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3458: done (0 left) 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3459: rollback oid 20 current snap is 387 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3459: done (0 left) 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3460: rollback oid 17 current snap is 387 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3460: done (0 left) 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3461: write oid 11 current snap is 387 2024-09-20T19:37:32.535 INFO:tasks.rados.rados.0.smithi042.stdout:3461: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-20T19:37:32.541 INFO:tasks.rados.rados.0.smithi042.stdout:3461: writing smithi04268048-11 from 649067 to 1152851 tid 1 2024-09-20T19:37:32.549 INFO:tasks.rados.rados.0.smithi042.stdout:3461: writing smithi04268048-11 from 1875940 to 2483011 tid 2 2024-09-20T19:37:32.557 INFO:tasks.rados.rados.0.smithi042.stdout:3461: writing smithi04268048-11 from 3115378 to 3846684 tid 3 2024-09-20T19:37:32.558 INFO:tasks.rados.rados.0.smithi042.stdout:3462: delete oid 47 current snap is 387 2024-09-20T19:37:32.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:32.565 INFO:tasks.rados.rados.0.smithi042.stdout:3461: finishing write tid 1 to smithi04268048-11 2024-09-20T19:37:32.568 INFO:tasks.rados.rados.0.smithi042.stdout:3461: finishing write tid 2 to smithi04268048-11 2024-09-20T19:37:32.569 INFO:tasks.rados.rados.0.smithi042.stdout:3461: finishing write tid 3 to smithi04268048-11 2024-09-20T19:37:32.569 INFO:tasks.rados.rados.0.smithi042.stdout:3461: finishing write tid 4 to smithi04268048-11 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: finishing write tid 5 to smithi04268048-11 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: oid 11 updating version 0 to 1146 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: oid 11 version 1146 is already newer than 1144 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: oid 11 version 1146 is already newer than 1143 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: oid 11 version 1146 is already newer than 1145 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1146 (ObjNum 1181 snap 387 seq_num 1181) dirty exists 2024-09-20T19:37:32.570 INFO:tasks.rados.rados.0.smithi042.stdout:3461: left oid 11 (ObjNum 1181 snap 387 seq_num 1181) 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3461: done (1 left) 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3462: done (0 left) 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3463: copy_from oid 41 from oid 22 current snap is 387 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3464: rollback oid 44 current snap is 387 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3464: done (1 left) 2024-09-20T19:37:32.571 INFO:tasks.rados.rados.0.smithi042.stdout:3465: delete oid 10 current snap is 387 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:32 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.574 INFO:tasks.rados.rados.0.smithi042.stdout:3465: done (1 left) 2024-09-20T19:37:32.575 INFO:tasks.rados.rados.0.smithi042.stdout:3466: write (excl) oid 8 current snap is 387 2024-09-20T19:37:32.575 INFO:tasks.rados.rados.0.smithi042.stdout:3466: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2024-09-20T19:37:32.582 INFO:tasks.rados.rados.0.smithi042.stdout:3466: writing smithi04268048-8 from 699488 to 1429019 tid 1 2024-09-20T19:37:32.588 INFO:tasks.rados.rados.0.smithi042.stdout:3466: writing smithi04268048-8 from 1935033 to 2405784 tid 2 2024-09-20T19:37:32.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:32.596 INFO:tasks.rados.rados.0.smithi042.stdout:3466: writing smithi04268048-8 from 3081294 to 3762769 tid 3 2024-09-20T19:37:32.596 INFO:tasks.rados.rados.0.smithi042.stdout:3467: write (excl) oid 29 current snap is 387 2024-09-20T19:37:32.596 INFO:tasks.rados.rados.0.smithi042.stdout:3467: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-09-20T19:37:32.602 INFO:tasks.rados.rados.0.smithi042.stdout:3467: writing smithi04268048-29 from 716295 to 1121075 tid 1 2024-09-20T19:37:32.608 INFO:tasks.rados.rados.0.smithi042.stdout:3467: writing smithi04268048-29 from 1593515 to 2207924 tid 2 2024-09-20T19:37:32.614 INFO:tasks.rados.rados.0.smithi042.stdout:3467: writing smithi04268048-29 from 2708717 to 3201464 tid 3 2024-09-20T19:37:32.618 INFO:tasks.rados.rados.0.smithi042.stdout:3467: writing smithi04268048-29 from 3640519 to 3916295 tid 4 2024-09-20T19:37:32.618 INFO:tasks.rados.rados.0.smithi042.stdout:3468: delete oid 21 current snap is 387 2024-09-20T19:37:32.618 INFO:tasks.rados.rados.0.smithi042.stdout:3463: finishing copy_from to smithi04268048-41 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3463: got expected ENOENT (src dne) 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3463: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: finishing write tid 1 to smithi04268048-8 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: finishing write tid 2 to smithi04268048-8 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: finishing write tid 3 to smithi04268048-8 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: finishing write tid 4 to smithi04268048-8 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: finishing write tid 5 to smithi04268048-8 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: oid 8 updating version 0 to 1240 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: oid 8 version 1240 is already newer than 1238 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: oid 8 version 1240 is already newer than 1239 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: oid 8 updating version 1240 to 1241 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1241 (ObjNum 1184 snap 387 seq_num 1184) dirty exists 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3466: left oid 8 (ObjNum 1184 snap 387 seq_num 1184) 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 1 to smithi04268048-29 2024-09-20T19:37:32.619 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 2 to smithi04268048-29 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 3 to smithi04268048-29 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3463: done (3 left) 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3466: done (2 left) 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3468: done (1 left) 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3469: write (excl) oid 37 current snap is 387 2024-09-20T19:37:32.620 INFO:tasks.rados.rados.0.smithi042.stdout:3469: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-20T19:37:32.628 INFO:tasks.rados.rados.0.smithi042.stdout:3469: writing smithi04268048-37 from 749909 to 1421540 tid 1 2024-09-20T19:37:32.635 INFO:tasks.rados.rados.0.smithi042.stdout:3469: writing smithi04268048-37 from 2110479 to 2728557 tid 2 2024-09-20T19:37:32.641 INFO:tasks.rados.rados.0.smithi042.stdout:3469: writing smithi04268048-37 from 3163563 to 3678854 tid 3 2024-09-20T19:37:32.641 INFO:tasks.rados.rados.0.smithi042.stdout:3470: rollback oid 45 current snap is 387 2024-09-20T19:37:32.641 INFO:tasks.rados.rados.0.smithi042.stdout:3470: done (2 left) 2024-09-20T19:37:32.641 INFO:tasks.rados.rados.0.smithi042.stdout:3471: write oid 42 current snap is 387 2024-09-20T19:37:32.641 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 4 to smithi04268048-29 2024-09-20T19:37:32.642 INFO:tasks.rados.rados.0.smithi042.stdout:3471: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-09-20T19:37:32.650 INFO:tasks.rados.rados.0.smithi042.stdout:3471: writing smithi04268048-42 from 766716 to 1513596 tid 1 2024-09-20T19:37:32.658 INFO:tasks.rados.rados.0.smithi042.stdout:3471: writing smithi04268048-42 from 2168961 to 2930697 tid 2 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3471: writing smithi04268048-42 from 3474633 to 3917549 tid 3 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3472: read oid 20 snap -1 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3472: expect (ObjNum 1176 snap 387 seq_num 1176) 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3473: write oid 10 current snap is 387 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 5 to smithi04268048-29 2024-09-20T19:37:32.664 INFO:tasks.rados.rados.0.smithi042.stdout:3473: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-09-20T19:37:32.671 INFO:tasks.rados.rados.0.smithi042.stdout:3473: writing smithi04268048-10 from 783523 to 1205652 tid 1 2024-09-20T19:37:32.678 INFO:tasks.rados.rados.0.smithi042.stdout:3473: writing smithi04268048-10 from 1827443 to 2449190 tid 2 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:31.525310+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.480250+0000 osd.2 (osd.2) 295 : cluster [DBG] 2.e scrub starts 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:32 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.482366+0000 osd.2 (osd.2) 296 : cluster [DBG] 2.e scrub ok 2024-09-20T19:37:32.687 INFO:tasks.rados.rados.0.smithi042.stdout:3473: writing smithi04268048-10 from 3218409 to 3872597 tid 3 2024-09-20T19:37:32.687 INFO:tasks.rados.rados.0.smithi042.stdout:3474: write (excl) oid 39 current snap is 387 2024-09-20T19:37:32.687 INFO:tasks.rados.rados.0.smithi042.stdout:3474: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-20T19:37:32.692 INFO:tasks.rados.rados.0.smithi042.stdout:3474: writing smithi04268048-39 from 400330 to 897708 tid 1 2024-09-20T19:37:32.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:32.699 INFO:tasks.rados.rados.0.smithi042.stdout:3474: writing smithi04268048-39 from 1485925 to 2000330 tid 2 2024-09-20T19:37:32.699 INFO:tasks.rados.rados.0.smithi042.stdout:3475: write oid 48 current snap is 387 2024-09-20T19:37:32.699 INFO:tasks.rados.rados.0.smithi042.stdout:3467: finishing write tid 6 to smithi04268048-29 2024-09-20T19:37:32.699 INFO:tasks.rados.rados.0.smithi042.stdout:3467: oid 29 updating version 0 to 964 2024-09-20T19:37:32.699 INFO:tasks.rados.rados.0.smithi042.stdout:3467: oid 29 version 964 is already newer than 961 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:3467: oid 29 version 964 is already newer than 962 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:3467: oid 29 version 964 is already newer than 960 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:3467: oid 29 version 964 is already newer than 963 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 964 (ObjNum 1185 snap 387 seq_num 1185) dirty exists 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:3467: left oid 29 (ObjNum 1185 snap 387 seq_num 1185) 2024-09-20T19:37:32.700 INFO:tasks.rados.rados.0.smithi042.stdout:3469: finishing write tid 1 to smithi04268048-37 2024-09-20T19:37:32.701 INFO:tasks.rados.rados.0.smithi042.stdout:3475: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-20T19:37:32.704 INFO:tasks.rados.rados.0.smithi042.stdout:3475: writing smithi04268048-48 from 417137 to 989764 tid 1 2024-09-20T19:37:32.711 INFO:tasks.rados.rados.0.smithi042.stdout:3475: writing smithi04268048-48 from 1428054 to 2017137 tid 2 2024-09-20T19:37:32.711 INFO:tasks.rados.rados.0.smithi042.stdout:3467: done (6 left) 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3476: write (excl) oid 45 current snap is 387 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: finishing write tid 2 to smithi04268048-37 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: finishing write tid 3 to smithi04268048-37 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: finishing write tid 4 to smithi04268048-37 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: finishing write tid 5 to smithi04268048-37 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: oid 37 updating version 0 to 1169 2024-09-20T19:37:32.712 INFO:tasks.rados.rados.0.smithi042.stdout:3469: oid 37 version 1169 is already newer than 1168 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3469: oid 37 updating version 1169 to 1171 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3469: oid 37 version 1171 is already newer than 1170 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1171 (ObjNum 1187 snap 387 seq_num 1187) dirty exists 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3469: left oid 37 (ObjNum 1187 snap 387 seq_num 1187) 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3471: finishing write tid 1 to smithi04268048-42 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3471: finishing write tid 2 to smithi04268048-42 2024-09-20T19:37:32.713 INFO:tasks.rados.rados.0.smithi042.stdout:3476: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-20T19:37:32.718 INFO:tasks.rados.rados.0.smithi042.stdout:3476: writing smithi04268048-45 from 433944 to 1081820 tid 1 2024-09-20T19:37:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3476: writing smithi04268048-45 from 1486536 to 2033944 tid 2 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3469: done (6 left) 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3477: read oid 32 snap -1 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3477: expect (ObjNum 1128 snap 370 seq_num 1128) 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3478: snap_create 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: finishing write tid 3 to smithi04268048-42 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: finishing write tid 4 to smithi04268048-42 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: finishing write tid 5 to smithi04268048-42 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: oid 42 updating version 0 to 1301 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: oid 42 version 1301 is already newer than 1298 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: oid 42 version 1301 is already newer than 1299 2024-09-20T19:37:32.725 INFO:tasks.rados.rados.0.smithi042.stdout:3471: oid 42 version 1301 is already newer than 1300 2024-09-20T19:37:32.726 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1301 (ObjNum 1188 snap 387 seq_num 1188) dirty exists 2024-09-20T19:37:32.726 INFO:tasks.rados.rados.0.smithi042.stdout:3471: left oid 42 (ObjNum 1188 snap 387 seq_num 1188) 2024-09-20T19:37:32.726 INFO:tasks.rados.rados.0.smithi042.stdout:3473: finishing write tid 1 to smithi04268048-10 2024-09-20T19:37:32.726 INFO:tasks.rados.rados.0.smithi042.stdout:3472: expect (ObjNum 1176 snap 387 seq_num 1176) 2024-09-20T19:37:32.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:32.831 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:32.832 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:32.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:32.878 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: finishing write tid 2 to smithi04268048-10 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: finishing write tid 3 to smithi04268048-10 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: finishing write tid 4 to smithi04268048-10 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: finishing write tid 5 to smithi04268048-10 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: oid 10 updating version 0 to 968 2024-09-20T19:37:32.954 INFO:tasks.rados.rados.0.smithi042.stdout:3473: oid 10 version 968 is already newer than 965 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3473: oid 10 version 968 is already newer than 967 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3473: oid 10 version 968 is already newer than 966 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 968 (ObjNum 1189 snap 387 seq_num 1189) dirty exists 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3473: left oid 10 (ObjNum 1189 snap 387 seq_num 1189) 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: finishing write tid 1 to smithi04268048-39 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: finishing write tid 2 to smithi04268048-39 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: finishing write tid 3 to smithi04268048-39 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: finishing write tid 4 to smithi04268048-39 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: oid 39 updating version 0 to 1245 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: oid 39 updating version 1245 to 1246 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: oid 39 updating version 1246 to 1247 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1247 (ObjNum 1190 snap 387 seq_num 1190) dirty exists 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3474: left oid 39 (ObjNum 1190 snap 387 seq_num 1190) 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3475: finishing write tid 1 to smithi04268048-48 2024-09-20T19:37:32.955 INFO:tasks.rados.rados.0.smithi042.stdout:3475: finishing write tid 2 to smithi04268048-48 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: finishing write tid 3 to smithi04268048-48 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: finishing write tid 4 to smithi04268048-48 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: oid 48 updating version 0 to 1103 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: oid 48 updating version 1103 to 1104 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: oid 48 version 1104 is already newer than 1102 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1104 (ObjNum 1191 snap 387 seq_num 1191) dirty exists 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3475: left oid 48 (ObjNum 1191 snap 387 seq_num 1191) 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3476: finishing write tid 1 to smithi04268048-45 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:3477: expect (ObjNum 1128 snap 370 seq_num 1128) 2024-09-20T19:37:32.956 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:32.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:32.981 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:32.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:33.017 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:33.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:33.113 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2192248 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: finishing write tid 2 to smithi04268048-45 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: finishing write tid 3 to smithi04268048-45 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: finishing write tid 4 to smithi04268048-45 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: oid 45 updating version 0 to 1597 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: oid 45 version 1597 is already newer than 1596 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: oid 45 updating version 1597 to 1598 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1598 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:37:33.133 INFO:tasks.rados.rados.0.smithi042.stdout:3476: left oid 45 (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:37:33.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:33.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:33.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:33.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3471: done (7 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3472: done (6 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3473: done (5 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3474: done (4 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3475: done (3 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3476: done (2 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3477: done (1 left) 2024-09-20T19:37:33.443 INFO:tasks.rados.rados.0.smithi042.stdout:3478: done (0 left) 2024-09-20T19:37:33.444 INFO:tasks.rados.rados.0.smithi042.stdout:3479: write (excl) oid 44 current snap is 388 2024-09-20T19:37:33.444 INFO:tasks.rados.rados.0.smithi042.stdout:3479: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-20T19:37:33.446 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:37:33.449 INFO:tasks.rados.rados.0.smithi042.stdout:3479: writing smithi04268048-44 from 450751 to 1173876 tid 1 2024-09-20T19:37:33.455 INFO:tasks.rados.rados.0.smithi042.stdout:3479: writing smithi04268048-44 from 1945018 to 2050751 tid 2 2024-09-20T19:37:33.455 INFO:tasks.rados.rados.0.smithi042.stdout:3480: rollback oid 1 current snap is 388 2024-09-20T19:37:33.455 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 387 2024-09-20T19:37:33.455 INFO:tasks.rados.rados.0.smithi042.stdout:3481: copy_from oid 24 from oid 16 current snap is 388 2024-09-20T19:37:33.455 INFO:tasks.rados.rados.0.smithi042.stdout:3482: snap_create 2024-09-20T19:37:33.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:33.475 INFO:tasks.rados.rados.0.smithi042.stdout:3479: finishing write tid 1 to smithi04268048-44 2024-09-20T19:37:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:3479: finishing write tid 2 to smithi04268048-44 2024-09-20T19:37:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:3480: finishing rollback tid 0 to smithi04268048-1 2024-09-20T19:37:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:3480: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:37:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:3480: finishing rollback tid 2 to smithi04268048-1 2024-09-20T19:37:33.476 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1138 (ObjNum 1089 snap 362 seq_num 1089) dirty exists 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3481: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: finishing write tid 3 to smithi04268048-44 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: finishing write tid 4 to smithi04268048-44 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: oid 44 updating version 0 to 1255 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: oid 44 updating version 1255 to 1256 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: oid 44 version 1256 is already newer than 1254 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1256 (ObjNum 1193 snap 388 seq_num 1193) dirty exists 2024-09-20T19:37:33.477 INFO:tasks.rados.rados.0.smithi042.stdout:3479: left oid 44 (ObjNum 1193 snap 388 seq_num 1193) 2024-09-20T19:37:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:3481: finishing copy_from to smithi04268048-24 2024-09-20T19:37:33.480 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1243 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:37:33.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:33.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:33.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:33.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:33.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.739 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.740 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.742 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.743 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:33 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:33.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.456434+0000 mgr.y (mgr.24491) 734 : cluster [DBG] pgmap v1413: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 202 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 927 KiB/s wr, 16 op/s 2024-09-20T19:37:33.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.533249+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.584285+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.b scrub starts 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:32.585489+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.b scrub ok 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:32.747662+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.42:0/174386777' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:32.747942+0000 mon.a (mon.0) 2549 : audit [INF] from='client.26015 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.429414+0000 mon.a (mon.0) 2550 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.429467+0000 mon.a (mon.0) 2551 : cluster [INF] Cluster is now healthy 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: audit 2024-09-20T19:37:33.432085+0000 mon.a (mon.0) 2552 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:33.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:33 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.439710+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2024-09-20T19:37:33.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:33.885 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:33.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:33.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:34.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:34.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:34.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:34.247 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:34.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:34.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:34.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:34.446 INFO:tasks.rados.rados.0.smithi042.stdout:3479: done (3 left) 2024-09-20T19:37:34.447 INFO:tasks.rados.rados.0.smithi042.stdout:3480: done (2 left) 2024-09-20T19:37:34.447 INFO:tasks.rados.rados.0.smithi042.stdout:3481: done (1 left) 2024-09-20T19:37:34.447 INFO:tasks.rados.rados.0.smithi042.stdout:3482: done (0 left) 2024-09-20T19:37:34.447 INFO:tasks.rados.rados.0.smithi042.stdout:3483: delete oid 8 current snap is 389 2024-09-20T19:37:34.455 INFO:tasks.rados.rados.0.smithi042.stdout:3483: done (0 left) 2024-09-20T19:37:34.455 INFO:tasks.rados.rados.0.smithi042.stdout:3484: write (excl) oid 30 current snap is 389 2024-09-20T19:37:34.455 INFO:tasks.rados.rados.0.smithi042.stdout:3484: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-09-20T19:37:34.463 INFO:tasks.rados.rados.0.smithi042.stdout:3484: writing smithi04268048-30 from 484365 to 1074341 tid 1 2024-09-20T19:37:34.472 INFO:tasks.rados.rados.0.smithi042.stdout:3484: writing smithi04268048-30 from 1661982 to 2084365 tid 2 2024-09-20T19:37:34.472 INFO:tasks.rados.rados.0.smithi042.stdout:3485: snap_remove snap 388 2024-09-20T19:37:34.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:34.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:34.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:34.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:34.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:34 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:34.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:34.929 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:34.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.784790+0000 osd.4 (osd.4) 155 : cluster [DBG] 7.e scrub starts 2024-09-20T19:37:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:33.788155+0000 osd.4 (osd.4) 156 : cluster [DBG] 7.e scrub ok 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.441620+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.494380+0000 osd.2 (osd.2) 297 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:34 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.496743+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:37:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:34.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:35.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:35.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:35.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:35.192 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:35.296 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:35.322 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:35.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:35.455 INFO:tasks.rados.rados.0.smithi042.stdout:3485: done (1 left) 2024-09-20T19:37:35.455 INFO:tasks.rados.rados.0.smithi042.stdout:3486: snap_remove snap 387 2024-09-20T19:37:35.460 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:35.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:35.587 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:35.690 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.703 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:35 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.719 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:34.457179+0000 mgr.y (mgr.24491) 735 : cluster [DBG] pgmap v1417: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.9 MiB/s wr, 29 op/s 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.418914+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.420168+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:35 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.453241+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2024-09-20T19:37:35.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:35.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:35.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:35.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:35.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:36.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:36.107 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:37:36.107 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:37:36.107 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:37:36.108 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:37:36.108 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:37:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:no valid command found; 10 closest matches: 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:0 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:1 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:2 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:abort 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:assert 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:bench [] [] [] [] 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs debug_inject_read_zeros 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs files list 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:bluefs stats 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:bluestore allocator dump block 2024-09-20T19:37:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:Error EINVAL: invalid command 2024-09-20T19:37:36.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:36.231 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:36.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:36.455 INFO:tasks.rados.rados.0.smithi042.stdout:3486: done (1 left) 2024-09-20T19:37:36.455 INFO:tasks.rados.rados.0.smithi042.stdout:3487: delete oid 6 current snap is 389 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: finishing write tid 1 to smithi04268048-30 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: finishing write tid 2 to smithi04268048-30 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: finishing write tid 3 to smithi04268048-30 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: finishing write tid 4 to smithi04268048-30 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: oid 30 updating version 0 to 1565 2024-09-20T19:37:36.456 INFO:tasks.rados.rados.0.smithi042.stdout:3484: oid 30 version 1565 is already newer than 1564 2024-09-20T19:37:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:3484: oid 30 updating version 1565 to 1566 2024-09-20T19:37:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1566 (ObjNum 1195 snap 389 seq_num 1195) dirty exists 2024-09-20T19:37:36.457 INFO:tasks.rados.rados.0.smithi042.stdout:3484: left oid 30 (ObjNum 1195 snap 389 seq_num 1195) 2024-09-20T19:37:36.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:36.468 INFO:tasks.rados.rados.0.smithi042.stdout:3484: done (1 left) 2024-09-20T19:37:36.468 INFO:tasks.rados.rados.0.smithi042.stdout:3487: done (0 left) 2024-09-20T19:37:36.468 INFO:tasks.rados.rados.0.smithi042.stdout:3488: delete oid 1 current snap is 389 2024-09-20T19:37:36.479 INFO:tasks.rados.rados.0.smithi042.stdout:3488: done (0 left) 2024-09-20T19:37:36.479 INFO:tasks.rados.rados.0.smithi042.stdout:3489: read oid 4 snap -1 2024-09-20T19:37:36.479 INFO:tasks.rados.rados.0.smithi042.stdout:3489: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:36.479 INFO:tasks.rados.rados.0.smithi042.stdout:3490: copy_from oid 42 from oid 3 current snap is 389 2024-09-20T19:37:36.479 INFO:tasks.rados.rados.0.smithi042.stdout:3491: delete oid 47 current snap is 389 2024-09-20T19:37:36.489 INFO:tasks.rados.rados.0.smithi042.stdout:3489: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:36.489 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:36.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:36.591 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 850685 2024-09-20T19:37:36.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:36.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:36.680 INFO:tasks.rados.rados.0.smithi042.stdout:3490: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:37:36.680 INFO:tasks.rados.rados.0.smithi042.stdout:3490: finishing copy_from to smithi04268048-42 2024-09-20T19:37:36.680 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 970 (ObjNum 949 snap 315 seq_num 949) dirty exists 2024-09-20T19:37:36.680 INFO:tasks.rados.rados.0.smithi042.stdout:3489: done (2 left) 2024-09-20T19:37:36.680 INFO:tasks.rados.rados.0.smithi042.stdout:3490: done (1 left) 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3491: done (0 left) 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3492: read oid 12 snap -1 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3492: expect deleted 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3493: rollback oid 2 current snap is 389 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3493: done (1 left) 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3494: write (excl) oid 41 current snap is 389 2024-09-20T19:37:36.681 INFO:tasks.rados.rados.0.smithi042.stdout:3494: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-20T19:37:36.681 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:36.686 INFO:tasks.rados.rados.0.smithi042.stdout:3494: writing smithi04268048-41 from 551593 to 1042565 tid 1 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3494: writing smithi04268048-41 from 1779557 to 2151593 tid 2 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3495: read oid 2 snap -1 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3495: expect deleted 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3496: read oid 28 snap -1 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3496: expect (ObjNum 1173 snap 385 seq_num 1173) 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3497: read oid 4 snap -1 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3497: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3498: copy_from oid 5 from oid 17 current snap is 389 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3492: done (5 left) 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3499: read oid 6 snap -1 2024-09-20T19:37:36.693 INFO:tasks.rados.rados.0.smithi042.stdout:3499: expect deleted 2024-09-20T19:37:36.694 INFO:tasks.rados.rados.0.smithi042.stdout:3500: copy_from oid 3 from oid 45 current snap is 389 2024-09-20T19:37:36.694 INFO:tasks.rados.rados.0.smithi042.stdout:3501: snap_create 2024-09-20T19:37:36.694 INFO:tasks.rados.rados.0.smithi042.stdout:3494: finishing write tid 1 to smithi04268048-41 2024-09-20T19:37:36.699 INFO:tasks.rados.rados.0.smithi042.stdout:3494: finishing write tid 2 to smithi04268048-41 2024-09-20T19:37:36.700 INFO:tasks.rados.rados.0.smithi042.stdout:3494: finishing write tid 3 to smithi04268048-41 2024-09-20T19:37:36.700 INFO:tasks.rados.rados.0.smithi042.stdout:3494: finishing write tid 4 to smithi04268048-41 2024-09-20T19:37:36.701 INFO:tasks.rados.rados.0.smithi042.stdout:3494: oid 41 updating version 0 to 1248 2024-09-20T19:37:36.701 INFO:tasks.rados.rados.0.smithi042.stdout:3494: oid 41 updating version 1248 to 1250 2024-09-20T19:37:36.701 INFO:tasks.rados.rados.0.smithi042.stdout:3494: oid 41 version 1250 is already newer than 1249 2024-09-20T19:37:36.701 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1250 (ObjNum 1199 snap 389 seq_num 1199) dirty exists 2024-09-20T19:37:36.701 INFO:tasks.rados.rados.0.smithi042.stdout:3494: left oid 41 (ObjNum 1199 snap 389 seq_num 1199) 2024-09-20T19:37:36.704 INFO:tasks.rados.rados.0.smithi042.stdout:3497: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:36.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:36.754 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:36.907 INFO:tasks.rados.rados.0.smithi042.stdout:3500: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:37:36.907 INFO:tasks.rados.rados.0.smithi042.stdout:3500: finishing copy_from to smithi04268048-3 2024-09-20T19:37:36.907 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1598 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:37:36.907 INFO:tasks.rados.rados.0.smithi042.stdout:3498: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:37:36.907 INFO:tasks.rados.rados.0.smithi042.stdout:3496: expect (ObjNum 1173 snap 385 seq_num 1173) 2024-09-20T19:37:36.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:36.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:36.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:37.022 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1941796 2024-09-20T19:37:37.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:37.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:37.148 INFO:tasks.rados.rados.0.smithi042.stdout:3498: finishing copy_from to smithi04268048-5 2024-09-20T19:37:37.148 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:37:37.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:37.229 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:37.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:37.361 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:37.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:37.466 INFO:tasks.rados.rados.0.smithi042.stdout:3494: done (7 left) 2024-09-20T19:37:37.466 INFO:tasks.rados.rados.0.smithi042.stdout:3495: done (6 left) 2024-09-20T19:37:37.466 INFO:tasks.rados.rados.0.smithi042.stdout:3496: done (5 left) 2024-09-20T19:37:37.466 INFO:tasks.rados.rados.0.smithi042.stdout:3497: done (4 left) 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3498: done (3 left) 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3499: done (2 left) 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3500: done (1 left) 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3501: done (0 left) 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3502: write (excl) oid 28 current snap is 390 2024-09-20T19:37:37.467 INFO:tasks.rados.rados.0.smithi042.stdout:3502: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-20T19:37:37.472 INFO:tasks.rados.rados.0.smithi042.stdout:3502: writing smithi04268048-28 from 568400 to 1134621 tid 1 2024-09-20T19:37:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3502: writing smithi04268048-28 from 1838039 to 2168400 tid 2 2024-09-20T19:37:37.478 INFO:tasks.rados.rados.0.smithi042.stdout:3503: snap_remove snap 389 2024-09-20T19:37:37.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:37.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:37:37.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:37 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:37.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:37.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.939414+0000 osd.5 (osd.5) 287 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:35.945526+0000 osd.5 (osd.5) 288 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.452518+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.457822+0000 mgr.y (mgr.24491) 736 : cluster [DBG] pgmap v1420: 178 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 157 active+clean; 188 MiB data, 849 MiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.771552+0000 osd.4 (osd.4) 157 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.738 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:37 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.773208+0000 osd.4 (osd.4) 158 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:37:37.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T19:37:37.793 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T19:37:37.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T19:37:37.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:37.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:37.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:37.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:38.012 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:38.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:38.142 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:38.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.279 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:38.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:38.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:38.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:38.445 INFO:tasks.rados.rados.0.smithi042.stdout:3503: done (1 left) 2024-09-20T19:37:38.445 INFO:tasks.rados.rados.0.smithi042.stdout:3504: rollback oid 41 current snap is 390 2024-09-20T19:37:38.445 INFO:tasks.rados.rados.0.smithi042.stdout:3504: done (1 left) 2024-09-20T19:37:38.445 INFO:tasks.rados.rados.0.smithi042.stdout:3505: rollback oid 42 current snap is 390 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3505: done (1 left) 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3506: read oid 45 snap -1 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3506: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3507: delete oid 39 current snap is 390 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3502: finishing write tid 1 to smithi04268048-28 2024-09-20T19:37:38.446 INFO:tasks.rados.rados.0.smithi042.stdout:3502: finishing write tid 2 to smithi04268048-28 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: finishing write tid 3 to smithi04268048-28 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: finishing write tid 4 to smithi04268048-28 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: oid 28 updating version 0 to 1194 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: oid 28 version 1194 is already newer than 1192 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: oid 28 version 1194 is already newer than 1193 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1194 (ObjNum 1200 snap 390 seq_num 1200) dirty exists 2024-09-20T19:37:38.447 INFO:tasks.rados.rados.0.smithi042.stdout:3502: left oid 28 (ObjNum 1200 snap 390 seq_num 1200) 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3502: done (2 left) 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3507: done (1 left) 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3508: rollback oid 35 current snap is 390 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3508: done (1 left) 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3509: write oid 8 current snap is 390 2024-09-20T19:37:38.476 INFO:tasks.rados.rados.0.smithi042.stdout:3509: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-20T19:37:38.483 INFO:tasks.rados.rados.0.smithi042.stdout:3509: writing smithi04268048-8 from 602014 to 1035086 tid 1 2024-09-20T19:37:38.493 INFO:tasks.rados.rados.0.smithi042.stdout:3509: writing smithi04268048-8 from 1671356 to 2202014 tid 2 2024-09-20T19:37:38.494 INFO:tasks.rados.rados.0.smithi042.stdout:3510: delete oid 41 current snap is 390 2024-09-20T19:37:38.494 INFO:tasks.rados.rados.0.smithi042.stdout:3506: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:37:38.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:38.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:38.547 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1394150 2024-09-20T19:37:38.558 INFO:tasks.rados.rados.0.smithi042.stdout:3509: finishing write tid 1 to smithi04268048-8 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: finishing write tid 2 to smithi04268048-8 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: finishing write tid 3 to smithi04268048-8 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: finishing write tid 4 to smithi04268048-8 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: oid 8 updating version 0 to 1244 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: oid 8 version 1244 is already newer than 1243 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: oid 8 updating version 1244 to 1245 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1245 (ObjNum 1202 snap 390 seq_num 1202) dirty exists 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: left oid 8 (ObjNum 1202 snap 390 seq_num 1202) 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3506: done (2 left) 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3509: done (1 left) 2024-09-20T19:37:38.559 INFO:tasks.rados.rados.0.smithi042.stdout:3510: done (0 left) 2024-09-20T19:37:38.560 INFO:tasks.rados.rados.0.smithi042.stdout:3511: write (excl) oid 13 current snap is 390 2024-09-20T19:37:38.560 INFO:tasks.rados.rados.0.smithi042.stdout:3511: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-09-20T19:37:38.579 INFO:tasks.rados.rados.0.smithi042.stdout:3511: writing smithi04268048-13 from 635628 to 1219198 tid 1 2024-09-20T19:37:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3511: writing smithi04268048-13 from 1671967 to 2235628 tid 2 2024-09-20T19:37:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3512: write (excl) oid 26 current snap is 390 2024-09-20T19:37:38.596 INFO:tasks.rados.rados.0.smithi042.stdout:3512: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-20T19:37:38.613 INFO:tasks.rados.rados.0.smithi042.stdout:3512: writing smithi04268048-26 from 652435 to 1311254 tid 1 2024-09-20T19:37:38.623 INFO:tasks.rados.rados.0.smithi042.stdout:3512: writing smithi04268048-26 from 1730449 to 2252435 tid 2 2024-09-20T19:37:38.624 INFO:tasks.rados.rados.0.smithi042.stdout:3513: write (excl) oid 17 current snap is 390 2024-09-20T19:37:38.624 INFO:tasks.rados.rados.0.smithi042.stdout:3513: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-20T19:37:38.636 INFO:tasks.rados.rados.0.smithi042.stdout:3513: writing smithi04268048-17 from 669242 to 1403310 tid 1 2024-09-20T19:37:38.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3513: writing smithi04268048-17 from 2188931 to 2269242 tid 2 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3514: read oid 21 snap -1 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3514: expect deleted 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3515: delete oid 20 current snap is 390 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: finishing write tid 1 to smithi04268048-13 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: finishing write tid 2 to smithi04268048-13 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: finishing write tid 3 to smithi04268048-13 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: finishing write tid 4 to smithi04268048-13 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: oid 13 updating version 0 to 610 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: oid 13 updating version 610 to 611 2024-09-20T19:37:38.644 INFO:tasks.rados.rados.0.smithi042.stdout:3511: oid 13 updating version 611 to 612 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 612 (ObjNum 1204 snap 390 seq_num 1204) dirty exists 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3511: left oid 13 (ObjNum 1204 snap 390 seq_num 1204) 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: finishing write tid 1 to smithi04268048-26 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: finishing write tid 2 to smithi04268048-26 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: finishing write tid 3 to smithi04268048-26 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: finishing write tid 4 to smithi04268048-26 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: oid 26 updating version 0 to 1262 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: oid 26 updating version 1262 to 1263 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: oid 26 updating version 1263 to 1264 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 1264 (ObjNum 1205 snap 390 seq_num 1205) dirty exists 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3512: left oid 26 (ObjNum 1205 snap 390 seq_num 1205) 2024-09-20T19:37:38.645 INFO:tasks.rados.rados.0.smithi042.stdout:3513: finishing write tid 1 to smithi04268048-17 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3511: done (4 left) 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3512: done (3 left) 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3514: done (2 left) 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3515: done (1 left) 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3516: rollback oid 15 current snap is 390 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3516: done (1 left) 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3517: read oid 7 snap -1 2024-09-20T19:37:38.648 INFO:tasks.rados.rados.0.smithi042.stdout:3517: expect deleted 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3518: read oid 45 snap -1 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3518: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3519: read oid 33 snap -1 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3519: expect (ObjNum 1143 snap 375 seq_num 1143) 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3520: copy_from oid 19 from oid 12 current snap is 390 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3513: finishing write tid 2 to smithi04268048-17 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3521: write oid 9 current snap is 390 2024-09-20T19:37:38.649 INFO:tasks.rados.rados.0.smithi042.stdout:3521: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-20T19:37:38.659 INFO:tasks.rados.rados.0.smithi042.stdout:3521: writing smithi04268048-9 from 702856 to 1187422 tid 1 2024-09-20T19:37:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:38.668 INFO:tasks.rados.rados.0.smithi042.stdout:3521: writing smithi04268048-9 from 1789542 to 2302856 tid 2 2024-09-20T19:37:38.668 INFO:tasks.rados.rados.0.smithi042.stdout:3522: snap_create 2024-09-20T19:37:38.668 INFO:tasks.rados.rados.0.smithi042.stdout:3513: finishing write tid 3 to smithi04268048-17 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3513: finishing write tid 4 to smithi04268048-17 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3513: oid 17 updating version 0 to 1163 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3513: oid 17 updating version 1163 to 1164 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3513: oid 17 version 1164 is already newer than 1162 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1164 (ObjNum 1206 snap 390 seq_num 1206) dirty exists 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3513: left oid 17 (ObjNum 1206 snap 390 seq_num 1206) 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3520: finishing copy_from to smithi04268048-19 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3520: got expected ENOENT (src dne) 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3520: finishing copy_from racing read to smithi04268048-19 2024-09-20T19:37:38.669 INFO:tasks.rados.rados.0.smithi042.stdout:3518: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:37:38.681 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1886375 2024-09-20T19:37:38.724 INFO:tasks.rados.rados.0.smithi042.stdout:3519: expect (ObjNum 1143 snap 375 seq_num 1143) 2024-09-20T19:37:38.767 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[32075]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:38 smithi042 bash[22706]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.784 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 533449 2024-09-20T19:37:38.792 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.908766+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:36.910988+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.461403+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.476521+0000 osd.2 (osd.2) 301 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.478814+0000 osd.2 (osd.2) 302 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.775808+0000 osd.4 (osd.4) 159 : cluster [DBG] 4.e scrub starts 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:37.778160+0000 osd.4 (osd.4) 160 : cluster [DBG] 4.e scrub ok 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:38 smithi162 bash[24557]: cluster 2024-09-20T19:37:38.441176+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2024-09-20T19:37:38.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:38.904 INFO:tasks.rados.rados.0.smithi042.stdout:3521: finishing write tid 1 to smithi04268048-9 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: finishing write tid 2 to smithi04268048-9 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: finishing write tid 3 to smithi04268048-9 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: finishing write tid 4 to smithi04268048-9 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: oid 9 updating version 0 to 1152 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: oid 9 version 1152 is already newer than 1151 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: oid 9 updating version 1152 to 1153 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1153 (ObjNum 1208 snap 390 seq_num 1208) dirty exists 2024-09-20T19:37:38.905 INFO:tasks.rados.rados.0.smithi042.stdout:3521: left oid 9 (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:38.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:38.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:39.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:39.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:39.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:39.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:39.301 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:39.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:39.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:39.442 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3513: done (6 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3517: done (5 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3518: done (4 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3519: done (3 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3520: done (2 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3521: done (1 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3522: done (0 left) 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3523: write oid 10 current snap is 391 2024-09-20T19:37:39.467 INFO:tasks.rados.rados.0.smithi042.stdout:3523: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-20T19:37:39.475 INFO:tasks.rados.rados.0.smithi042.stdout:3523: writing smithi04268048-10 from 719663 to 1395831 tid 1 2024-09-20T19:37:39.480 INFO:tasks.rados.rados.0.smithi042.stdout:3523: writing smithi04268048-10 from 1964377 to 2319663 tid 2 2024-09-20T19:37:39.480 INFO:tasks.rados.rados.0.smithi042.stdout:3524: snap_remove snap 390 2024-09-20T19:37:39.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:39.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:39 smithi162 bash[24557]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:39 smithi162 bash[24557]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.575 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:39.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:39.704 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:39.807 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:39.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:39 smithi042 bash[32075]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:39 smithi042 bash[32075]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:39 smithi042 bash[22706]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:39 smithi042 bash[22706]: cluster 2024-09-20T19:37:38.458461+0000 mgr.y (mgr.24491) 737 : cluster [DBG] pgmap v1423: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 161 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 6.8 MiB/s rd, 6.4 MiB/s wr, 27 op/s 2024-09-20T19:37:39.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:39.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:39.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:40.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:40.091 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:40.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:40.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:40.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3524: done (1 left) 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3525: read oid 13 snap -1 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3525: expect (ObjNum 1204 snap 390 seq_num 1204) 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3526: read oid 46 snap -1 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3526: expect (ObjNum 1174 snap 387 seq_num 1174) 2024-09-20T19:37:40.489 INFO:tasks.rados.rados.0.smithi042.stdout:3527: read oid 35 snap -1 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3527: expect deleted 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3528: rollback oid 24 current snap is 391 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3528: done (4 left) 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3529: snap_create 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: finishing write tid 1 to smithi04268048-10 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: finishing write tid 2 to smithi04268048-10 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: finishing write tid 3 to smithi04268048-10 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: finishing write tid 4 to smithi04268048-10 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: oid 10 updating version 0 to 970 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: oid 10 updating version 970 to 972 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: oid 10 version 972 is already newer than 971 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 972 (ObjNum 1209 snap 391 seq_num 1209) dirty exists 2024-09-20T19:37:40.490 INFO:tasks.rados.rados.0.smithi042.stdout:3523: left oid 10 (ObjNum 1209 snap 391 seq_num 1209) 2024-09-20T19:37:40.505 INFO:tasks.rados.rados.0.smithi042.stdout:3526: expect (ObjNum 1174 snap 387 seq_num 1174) 2024-09-20T19:37:40.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:40.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:40.649 INFO:tasks.rados.rados.0.smithi042.stdout:3525: expect (ObjNum 1204 snap 390 seq_num 1204) 2024-09-20T19:37:40.672 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 128776 2024-09-20T19:37:40.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:40.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:40 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:40 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:40.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:40.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:40.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:40.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:40 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:40 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:40 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:40 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.465934+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2024-09-20T19:37:40.975 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:41.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.129 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:37:41.166 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:37:41.183 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:37:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:37:41.200 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:37:41.200 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:37:41.217 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:37:41.218 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:37:41.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.238 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:37:41.238 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:37:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:41.260 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:37:41.260 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:37:41.281 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:37:41.281 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:37:41.307 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:37:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:37:41.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:41.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:41.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.488 INFO:tasks.rados.rados.0.smithi042.stdout:3523: done (4 left) 2024-09-20T19:37:41.488 INFO:tasks.rados.rados.0.smithi042.stdout:3525: done (3 left) 2024-09-20T19:37:41.488 INFO:tasks.rados.rados.0.smithi042.stdout:3526: done (2 left) 2024-09-20T19:37:41.488 INFO:tasks.rados.rados.0.smithi042.stdout:3527: done (1 left) 2024-09-20T19:37:41.489 INFO:tasks.rados.rados.0.smithi042.stdout:3529: done (0 left) 2024-09-20T19:37:41.489 INFO:tasks.rados.rados.0.smithi042.stdout:3530: delete oid 46 current snap is 392 2024-09-20T19:37:41.500 INFO:tasks.rados.rados.0.smithi042.stdout:3530: done (0 left) 2024-09-20T19:37:41.500 INFO:tasks.rados.rados.0.smithi042.stdout:3531: delete oid 24 current snap is 392 2024-09-20T19:37:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:41.512 INFO:tasks.rados.rados.0.smithi042.stdout:3531: done (0 left) 2024-09-20T19:37:41.513 INFO:tasks.rados.rados.0.smithi042.stdout:3532: write oid 11 current snap is 392 2024-09-20T19:37:41.513 INFO:tasks.rados.rados.0.smithi042.stdout:3532: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-20T19:37:41.519 INFO:tasks.rados.rados.0.smithi042.stdout:3532: writing smithi04268048-11 from 770084 to 1271999 tid 1 2024-09-20T19:37:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3532: writing smithi04268048-11 from 2023470 to 2370084 tid 2 2024-09-20T19:37:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3533: write (excl) oid 1 current snap is 392 2024-09-20T19:37:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3532: finishing write tid 1 to smithi04268048-11 2024-09-20T19:37:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3533: seq_num 1213 ranges {786891=577164,2081952=304939} 2024-09-20T19:37:41.534 INFO:tasks.rados.rados.0.smithi042.stdout:3533: writing smithi04268048-1 from 786891 to 1364055 tid 1 2024-09-20T19:37:41.540 INFO:tasks.rados.rados.0.smithi042.stdout:3533: writing smithi04268048-1 from 2081952 to 2386891 tid 2 2024-09-20T19:37:41.540 INFO:tasks.rados.rados.0.smithi042.stdout:3534: read oid 9 snap -1 2024-09-20T19:37:41.540 INFO:tasks.rados.rados.0.smithi042.stdout:3534: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:41.540 INFO:tasks.rados.rados.0.smithi042.stdout:3535: delete oid 15 current snap is 392 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: finishing write tid 2 to smithi04268048-11 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: finishing write tid 3 to smithi04268048-11 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: finishing write tid 4 to smithi04268048-11 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: oid 11 updating version 0 to 1155 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: oid 11 updating version 1155 to 1157 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: oid 11 version 1157 is already newer than 1156 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1157 (ObjNum 1212 snap 392 seq_num 1212) dirty exists 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3532: left oid 11 (ObjNum 1212 snap 392 seq_num 1212) 2024-09-20T19:37:41.541 INFO:tasks.rados.rados.0.smithi042.stdout:3533: finishing write tid 1 to smithi04268048-1 2024-09-20T19:37:41.545 INFO:tasks.rados.rados.0.smithi042.stdout:3533: finishing write tid 2 to smithi04268048-1 2024-09-20T19:37:41.546 INFO:tasks.rados.rados.0.smithi042.stdout:3533: finishing write tid 3 to smithi04268048-1 2024-09-20T19:37:41.546 INFO:tasks.rados.rados.0.smithi042.stdout:3532: done (3 left) 2024-09-20T19:37:41.546 INFO:tasks.rados.rados.0.smithi042.stdout:3535: done (2 left) 2024-09-20T19:37:41.546 INFO:tasks.rados.rados.0.smithi042.stdout:3536: snap_create 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:3533: finishing write tid 4 to smithi04268048-1 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:3533: oid 1 updating version 0 to 1140 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:3533: oid 1 updating version 1140 to 1142 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:3533: oid 1 version 1142 is already newer than 1141 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1142 (ObjNum 1213 snap 392 seq_num 1213) dirty exists 2024-09-20T19:37:41.547 INFO:tasks.rados.rados.0.smithi042.stdout:3533: left oid 1 (ObjNum 1213 snap 392 seq_num 1213) 2024-09-20T19:37:41.554 INFO:tasks.rados.rados.0.smithi042.stdout:3534: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:41.586 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 268936 2024-09-20T19:37:41.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:41.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[22706]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:41 smithi042 bash[32075]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.512823+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:39.514388+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: audit 2024-09-20T19:37:39.874149+0000 mgr.y (mgr.24491) 738 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:40.459190+0000 mgr.y (mgr.24491) 739 : cluster [DBG] pgmap v1425: 178 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 159 active+clean; 155 MiB data, 790 MiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 6.9 MiB/s wr, 33 op/s 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: cluster 2024-09-20T19:37:40.486049+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:41 smithi162 bash[24557]: audit 2024-09-20T19:37:41.147221+0000 mon.a (mon.0) 2561 : audit [DBG] from='client.? v1:172.21.15.42:0/3718296030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:41.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:41.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:41.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:42.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:42.036 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:42.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:42.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:42.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:42.506 INFO:tasks.rados.rados.0.smithi042.stdout:3533: done (2 left) 2024-09-20T19:37:42.506 INFO:tasks.rados.rados.0.smithi042.stdout:3534: done (1 left) 2024-09-20T19:37:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3536: done (0 left) 2024-09-20T19:37:42.507 INFO:tasks.rados.rados.0.smithi042.stdout:3537: delete oid 25 current snap is 393 2024-09-20T19:37:42.515 INFO:tasks.rados.rados.0.smithi042.stdout:3537: done (0 left) 2024-09-20T19:37:42.515 INFO:tasks.rados.rados.0.smithi042.stdout:3538: read oid 34 snap -1 2024-09-20T19:37:42.515 INFO:tasks.rados.rados.0.smithi042.stdout:3538: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:37:42.516 INFO:tasks.rados.rados.0.smithi042.stdout:3539: write oid 23 current snap is 393 2024-09-20T19:37:42.516 INFO:tasks.rados.rados.0.smithi042.stdout:3539: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-20T19:37:42.520 INFO:tasks.rados.rados.0.smithi042.stdout:3539: writing smithi04268048-23 from 437312 to 840223 tid 1 2024-09-20T19:37:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3539: writing smithi04268048-23 from 1341045 to 2037794 tid 2 2024-09-20T19:37:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3540: write (excl) oid 13 current snap is 393 2024-09-20T19:37:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3540: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-09-20T19:37:42.532 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.534 INFO:tasks.rados.rados.0.smithi042.stdout:3540: writing smithi04268048-13 from 454119 to 1048632 tid 1 2024-09-20T19:37:42.539 INFO:tasks.rados.rados.0.smithi042.stdout:3540: writing smithi04268048-13 from 1515880 to 2072640 tid 2 2024-09-20T19:37:42.539 INFO:tasks.rados.rados.0.smithi042.stdout:3541: write (excl) oid 22 current snap is 393 2024-09-20T19:37:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3541: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-20T19:37:42.543 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:42 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.544 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:42 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:42 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:42 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.547 INFO:tasks.rados.rados.0.smithi042.stdout:3541: writing smithi04268048-22 from 470926 to 1140688 tid 1 2024-09-20T19:37:42.554 INFO:tasks.rados.rados.0.smithi042.stdout:3541: writing smithi04268048-22 from 1574362 to 2274780 tid 2 2024-09-20T19:37:42.554 INFO:tasks.rados.rados.0.smithi042.stdout:3542: snap_create 2024-09-20T19:37:42.554 INFO:tasks.rados.rados.0.smithi042.stdout:3538: expect (ObjNum 911 snap 299 seq_num 911) 2024-09-20T19:37:42.556 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1958884 2024-09-20T19:37:42.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:42.579 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:42.613 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:42.648 INFO:tasks.rados.rados.0.smithi042.stdout:3539: finishing write tid 1 to smithi04268048-23 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: finishing write tid 2 to smithi04268048-23 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: finishing write tid 3 to smithi04268048-23 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: finishing write tid 4 to smithi04268048-23 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: oid 23 updating version 0 to 1167 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: oid 23 version 1167 is already newer than 1165 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: oid 23 version 1167 is already newer than 1166 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1167 (ObjNum 1216 snap 393 seq_num 1216) dirty exists 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3539: left oid 23 (ObjNum 1216 snap 393 seq_num 1216) 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3540: finishing write tid 1 to smithi04268048-13 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3540: finishing write tid 2 to smithi04268048-13 2024-09-20T19:37:42.649 INFO:tasks.rados.rados.0.smithi042.stdout:3540: finishing write tid 3 to smithi04268048-13 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3540: finishing write tid 4 to smithi04268048-13 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3540: oid 13 updating version 0 to 613 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3540: oid 13 updating version 613 to 614 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3540: oid 13 updating version 614 to 615 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 615 (ObjNum 1217 snap 393 seq_num 1217) dirty exists 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3540: left oid 13 (ObjNum 1217 snap 393 seq_num 1217) 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: finishing write tid 1 to smithi04268048-22 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: finishing write tid 2 to smithi04268048-22 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: finishing write tid 3 to smithi04268048-22 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: finishing write tid 4 to smithi04268048-22 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: oid 22 updating version 0 to 1035 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: oid 22 updating version 1035 to 1036 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:3541: oid 22 updating version 1036 to 1037 2024-09-20T19:37:42.650 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1037 (ObjNum 1218 snap 393 seq_num 1218) dirty exists 2024-09-20T19:37:42.651 INFO:tasks.rados.rados.0.smithi042.stdout:3541: left oid 22 (ObjNum 1218 snap 393 seq_num 1218) 2024-09-20T19:37:42.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:42.787 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.794 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 5, 6, 7, 0, 4, 3] 2024-09-20T19:37:42.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T19:37:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:42 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:42 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.487123+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2024-09-20T19:37:42.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:42.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:42.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:43.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:43.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:43.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:43.199 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:43.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:43.329 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:43.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3538: done (4 left) 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3539: done (3 left) 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3540: done (2 left) 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3541: done (1 left) 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3542: done (0 left) 2024-09-20T19:37:43.444 INFO:tasks.rados.rados.0.smithi042.stdout:3543: copy_from oid 24 from oid 27 current snap is 394 2024-09-20T19:37:43.445 INFO:tasks.rados.rados.0.smithi042.stdout:3544: snap_remove snap 392 2024-09-20T19:37:43.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:43.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:43.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:43.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:43 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:43.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:43.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.508133+0000 osd.2 (osd.2) 305 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:41.511302+0000 osd.2 (osd.2) 306 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.135643+0000 osd.1 (osd.1) 331 : cluster [DBG] 7.2 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.138178+0000 osd.1 (osd.1) 332 : cluster [DBG] 7.2 scrub ok 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.460223+0000 mgr.y (mgr.24491) 740 : cluster [DBG] pgmap v1428: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 129 MiB data, 715 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.504638+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.724094+0000 osd.6 (osd.6) 214 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:37:43.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.727129+0000 osd.6 (osd.6) 215 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:37:43.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:43 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.441329+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2024-09-20T19:37:43.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:43.981 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:43.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:44.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:44.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:44.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:44.299 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:37:44.342 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:44.370 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:3544: done (1 left) 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:3545: rollback oid 45 current snap is 394 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 393 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:3543: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:3543: finishing copy_from to smithi04268048-24 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1254 (ObjNum 1123 snap 368 seq_num 1123) dirty exists 2024-09-20T19:37:44.447 INFO:tasks.rados.rados.0.smithi042.stdout:3543: done (1 left) 2024-09-20T19:37:44.448 INFO:tasks.rados.rados.0.smithi042.stdout:3546: rollback oid 18 current snap is 394 2024-09-20T19:37:44.448 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 18 to 391 2024-09-20T19:37:44.448 INFO:tasks.rados.rados.0.smithi042.stdout:3547: delete oid 6 current snap is 394 2024-09-20T19:37:44.459 INFO:tasks.rados.rados.0.smithi042.stdout:3546: finishing rollback tid 0 to smithi04268048-18 2024-09-20T19:37:44.459 INFO:tasks.rados.rados.0.smithi042.stdout:3546: finishing rollback tid 1 to smithi04268048-18 2024-09-20T19:37:44.459 INFO:tasks.rados.rados.0.smithi042.stdout:3546: finishing rollback tid 2 to smithi04268048-18 2024-09-20T19:37:44.460 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1570 (ObjNum 1164 snap 380 seq_num 1164) dirty exists 2024-09-20T19:37:44.462 INFO:tasks.rados.rados.0.smithi042.stdout:3545: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:37:44.462 INFO:tasks.rados.rados.0.smithi042.stdout:3545: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:37:44.469 INFO:tasks.rados.rados.0.smithi042.stdout:3545: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:37:44.469 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1602 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:37:44.470 INFO:tasks.rados.rados.0.smithi042.stdout:3545: done (2 left) 2024-09-20T19:37:44.471 INFO:tasks.rados.rados.0.smithi042.stdout:3546: done (1 left) 2024-09-20T19:37:44.471 INFO:tasks.rados.rados.0.smithi042.stdout:3547: done (0 left) 2024-09-20T19:37:44.471 INFO:tasks.rados.rados.0.smithi042.stdout:3548: snap_remove snap 393 2024-09-20T19:37:44.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:44.500 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.525 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.526 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.527 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:44 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:44.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:44.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:44.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.888095+0000 osd.5 (osd.5) 291 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:42.892107+0000 osd.5 (osd.5) 292 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: audit 2024-09-20T19:37:43.523920+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: audit 2024-09-20T19:37:43.525949+0000 mon.a (mon.0) 2566 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.846864+0000 osd.5 (osd.5) 293 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:43.848396+0000 osd.5 (osd.5) 294 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:44 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.443337+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2024-09-20T19:37:44.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:44.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:44.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:44.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:44.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:44.991 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:37:44.992 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:37:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:45.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:45.135 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:45.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:45.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:45.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:45.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:45.393 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:45.455 INFO:tasks.rados.rados.0.smithi042.stdout:3548: done (0 left) 2024-09-20T19:37:45.455 INFO:tasks.rados.rados.0.smithi042.stdout:3549: read oid 24 snap -1 2024-09-20T19:37:45.455 INFO:tasks.rados.rados.0.smithi042.stdout:3549: expect (ObjNum 1123 snap 368 seq_num 1123) 2024-09-20T19:37:45.455 INFO:tasks.rados.rados.0.smithi042.stdout:3550: delete oid 38 current snap is 394 2024-09-20T19:37:45.466 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:37:45.468 INFO:tasks.rados.rados.0.smithi042.stdout:3550: done (1 left) 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:3551: rollback oid 14 current snap is 394 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 14 to 391 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:3552: rollback oid 38 current snap is 394 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 38 to 391 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:3553: read oid 4 snap 391 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:3553: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:45.469 INFO:tasks.rados.rados.0.smithi042.stdout:3554: delete oid 22 current snap is 394 2024-09-20T19:37:45.472 INFO:tasks.rados.rados.0.smithi042.stdout:3552: finishing rollback tid 1 to smithi04268048-38 2024-09-20T19:37:45.473 INFO:tasks.rados.rados.0.smithi042.stdout:3552: finishing rollback tid 2 to smithi04268048-38 2024-09-20T19:37:45.473 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1158 (ObjNum 904 snap 298 seq_num 904) dirty exists 2024-09-20T19:37:45.481 INFO:tasks.rados.rados.0.smithi042.stdout:3552: done (4 left) 2024-09-20T19:37:45.481 INFO:tasks.rados.rados.0.smithi042.stdout:3554: done (3 left) 2024-09-20T19:37:45.481 INFO:tasks.rados.rados.0.smithi042.stdout:3555: snap_create 2024-09-20T19:37:45.481 INFO:tasks.rados.rados.0.smithi042.stdout:3551: finishing rollback tid 0 to smithi04268048-14 2024-09-20T19:37:45.481 INFO:tasks.rados.rados.0.smithi042.stdout:3551: finishing rollback tid 1 to smithi04268048-14 2024-09-20T19:37:45.483 INFO:tasks.rados.rados.0.smithi042.stdout:3551: finishing rollback tid 2 to smithi04268048-14 2024-09-20T19:37:45.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1163 (ObjNum 1141 snap 375 seq_num 1141) dirty exists 2024-09-20T19:37:45.483 INFO:tasks.rados.rados.0.smithi042.stdout:3549: expect (ObjNum 1123 snap 368 seq_num 1123) 2024-09-20T19:37:45.484 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:45.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:45.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:45.584 INFO:tasks.rados.rados.0.smithi042.stdout:3553: expect (ObjNum 962 snap 319 seq_num 962) 2024-09-20T19:37:45.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:45.627 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:45.680 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:45.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.757 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.758 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[32075]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:45 smithi042 bash[22706]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.772 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.460740+0000 mgr.y (mgr.24491) 741 : cluster [DBG] pgmap v1432: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.508716+0000 osd.2 (osd.2) 307 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.510890+0000 osd.2 (osd.2) 308 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.722135+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.725691+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.762038+0000 osd.6 (osd.6) 216 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:44.764596+0000 osd.6 (osd.6) 217 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:37:45.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.252909+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.42:0/3971322838' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.253361+0000 mon.a (mon.0) 2568 : audit [INF] from='client.26033 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:45.440144+0000 mon.a (mon.0) 2569 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: audit 2024-09-20T19:37:45.442222+0000 mon.a (mon.0) 2570 : audit [INF] from='client.26033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:45 smithi162 bash[24557]: cluster 2024-09-20T19:37:45.450922+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2024-09-20T19:37:45.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:45.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:46.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:46.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:46.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:46.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T19:37:46.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:46.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:46.400 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:46.413 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3549: done (3 left) 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3551: done (2 left) 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3553: done (1 left) 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3555: done (0 left) 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3556: read oid 39 snap -1 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3556: expect deleted 2024-09-20T19:37:46.466 INFO:tasks.rados.rados.0.smithi042.stdout:3557: delete oid 43 current snap is 395 2024-09-20T19:37:46.473 INFO:tasks.rados.rados.0.smithi042.stdout:3557: done (1 left) 2024-09-20T19:37:46.473 INFO:tasks.rados.rados.0.smithi042.stdout:3558: write (excl) oid 37 current snap is 395 2024-09-20T19:37:46.473 INFO:tasks.rados.rados.0.smithi042.stdout:3558: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-20T19:37:46.479 INFO:tasks.rados.rados.0.smithi042.stdout:3558: writing smithi04268048-37 from 554961 to 1200968 tid 1 2024-09-20T19:37:46.486 INFO:tasks.rados.rados.0.smithi042.stdout:3558: writing smithi04268048-37 from 1750419 to 2201833 tid 2 2024-09-20T19:37:46.486 INFO:tasks.rados.rados.0.smithi042.stdout:3559: read oid 15 snap -1 2024-09-20T19:37:46.486 INFO:tasks.rados.rados.0.smithi042.stdout:3559: expect deleted 2024-09-20T19:37:46.487 INFO:tasks.rados.rados.0.smithi042.stdout:3560: read oid 42 snap -1 2024-09-20T19:37:46.487 INFO:tasks.rados.rados.0.smithi042.stdout:3560: expect (ObjNum 949 snap 315 seq_num 949) 2024-09-20T19:37:46.487 INFO:tasks.rados.rados.0.smithi042.stdout:3561: write (excl) oid 21 current snap is 395 2024-09-20T19:37:46.487 INFO:tasks.rados.rados.0.smithi042.stdout:3561: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-20T19:37:46.492 INFO:tasks.rados.rados.0.smithi042.stdout:3561: writing smithi04268048-21 from 571768 to 1009377 tid 1 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3561: writing smithi04268048-21 from 1808901 to 2520326 tid 2 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3556: done (4 left) 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3562: read oid 41 snap -1 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3562: expect deleted 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3563: write (excl) oid 18 current snap is 395 2024-09-20T19:37:46.501 INFO:tasks.rados.rados.0.smithi042.stdout:3563: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-20T19:37:46.507 INFO:tasks.rados.rados.0.smithi042.stdout:3563: writing smithi04268048-18 from 588575 to 1101433 tid 1 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3563: writing smithi04268048-18 from 1867383 to 2322466 tid 2 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3564: delete oid 31 current snap is 395 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: finishing write tid 1 to smithi04268048-37 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: finishing write tid 2 to smithi04268048-37 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: finishing write tid 3 to smithi04268048-37 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: finishing write tid 4 to smithi04268048-37 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: oid 37 updating version 0 to 1172 2024-09-20T19:37:46.515 INFO:tasks.rados.rados.0.smithi042.stdout:3558: oid 37 updating version 1172 to 1174 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3558: oid 37 version 1174 is already newer than 1173 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1174 (ObjNum 1223 snap 395 seq_num 1223) dirty exists 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3558: left oid 37 (ObjNum 1223 snap 395 seq_num 1223) 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: finishing write tid 1 to smithi04268048-21 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: finishing write tid 2 to smithi04268048-21 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: finishing write tid 3 to smithi04268048-21 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: finishing write tid 4 to smithi04268048-21 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: oid 21 updating version 0 to 1574 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: oid 21 version 1574 is already newer than 1573 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: oid 21 updating version 1574 to 1575 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1575 (ObjNum 1224 snap 395 seq_num 1224) dirty exists 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3561: left oid 21 (ObjNum 1224 snap 395 seq_num 1224) 2024-09-20T19:37:46.516 INFO:tasks.rados.rados.0.smithi042.stdout:3560: expect (ObjNum 949 snap 315 seq_num 949) 2024-09-20T19:37:46.517 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:46.626 INFO:tasks.rados.rados.0.smithi042.stdout:3563: finishing write tid 1 to smithi04268048-18 2024-09-20T19:37:46.626 INFO:tasks.rados.rados.0.smithi042.stdout:3558: done (6 left) 2024-09-20T19:37:46.626 INFO:tasks.rados.rados.0.smithi042.stdout:3559: done (5 left) 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3560: done (4 left) 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3561: done (3 left) 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3562: done (2 left) 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3564: done (1 left) 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3565: write (excl) oid 11 current snap is 395 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3563: finishing write tid 2 to smithi04268048-18 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3563: finishing write tid 3 to smithi04268048-18 2024-09-20T19:37:46.627 INFO:tasks.rados.rados.0.smithi042.stdout:3565: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-20T19:37:46.633 INFO:tasks.rados.rados.0.smithi042.stdout:3565: writing smithi04268048-11 from 622189 to 1285545 tid 1 2024-09-20T19:37:46.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3565: writing smithi04268048-11 from 1984347 to 2443099 tid 2 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3566: snap_create 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3563: finishing write tid 4 to smithi04268048-18 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3563: oid 18 updating version 0 to 1577 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3563: oid 18 updating version 1577 to 1579 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3563: oid 18 version 1579 is already newer than 1578 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1579 (ObjNum 1225 snap 395 seq_num 1225) dirty exists 2024-09-20T19:37:46.641 INFO:tasks.rados.rados.0.smithi042.stdout:3563: left oid 18 (ObjNum 1225 snap 395 seq_num 1225) 2024-09-20T19:37:46.643 INFO:tasks.rados.rados.0.smithi042.stdout:3565: finishing write tid 1 to smithi04268048-11 2024-09-20T19:37:46.645 INFO:tasks.rados.rados.0.smithi042.stdout:3565: finishing write tid 2 to smithi04268048-11 2024-09-20T19:37:46.646 INFO:tasks.rados.rados.0.smithi042.stdout:3565: finishing write tid 3 to smithi04268048-11 2024-09-20T19:37:46.646 INFO:tasks.rados.rados.0.smithi042.stdout:3565: finishing write tid 4 to smithi04268048-11 2024-09-20T19:37:46.646 INFO:tasks.rados.rados.0.smithi042.stdout:3565: oid 11 updating version 0 to 1164 2024-09-20T19:37:46.646 INFO:tasks.rados.rados.0.smithi042.stdout:3565: oid 11 updating version 1164 to 1165 2024-09-20T19:37:46.647 INFO:tasks.rados.rados.0.smithi042.stdout:3565: oid 11 version 1165 is already newer than 1163 2024-09-20T19:37:46.647 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1165 (ObjNum 1227 snap 395 seq_num 1227) dirty exists 2024-09-20T19:37:46.647 INFO:tasks.rados.rados.0.smithi042.stdout:3565: left oid 11 (ObjNum 1227 snap 395 seq_num 1227) 2024-09-20T19:37:46.663 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:46.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:46.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:46.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:46.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:46.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:47.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:47.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:47.155 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:47.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:47.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:47.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:47.444 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:47.473 INFO:tasks.rados.rados.0.smithi042.stdout:3563: done (2 left) 2024-09-20T19:37:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:3565: done (1 left) 2024-09-20T19:37:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:3566: done (0 left) 2024-09-20T19:37:47.474 INFO:tasks.rados.rados.0.smithi042.stdout:3567: snap_remove snap 395 2024-09-20T19:37:47.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:47 smithi162 bash[24557]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:47 smithi162 bash[24557]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:47 smithi162 bash[24557]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:47 smithi162 bash[24557]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.577 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:47.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:47.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[22706]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[22706]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[22706]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[22706]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[32075]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[32075]: cluster 2024-09-20T19:37:46.461206+0000 mgr.y (mgr.24491) 742 : cluster [DBG] pgmap v1435: 178 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 165 active+clean; 122 MiB data, 675 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 908 KiB/s wr, 9 op/s 2024-09-20T19:37:47.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[32075]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:47 smithi042 bash[32075]: cluster 2024-09-20T19:37:46.462918+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2024-09-20T19:37:47.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:47.813 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.844 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:47.947 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:47.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:48.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.108 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:48.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:48.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:48.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:48.472 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.490 INFO:tasks.rados.rados.0.smithi042.stdout:3567: done (0 left) 2024-09-20T19:37:48.491 INFO:tasks.rados.rados.0.smithi042.stdout:3568: rollback oid 2 current snap is 396 2024-09-20T19:37:48.491 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 394 2024-09-20T19:37:48.491 INFO:tasks.rados.rados.0.smithi042.stdout:3569: read oid 29 snap -1 2024-09-20T19:37:48.491 INFO:tasks.rados.rados.0.smithi042.stdout:3569: expect (ObjNum 1185 snap 387 seq_num 1185) 2024-09-20T19:37:48.491 INFO:tasks.rados.rados.0.smithi042.stdout:3570: snap_create 2024-09-20T19:37:48.499 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:48.503 INFO:tasks.rados.rados.0.smithi042.stdout:3568: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:37:48.503 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:37:48.512 INFO:tasks.rados.rados.0.smithi042.stdout:3569: expect (ObjNum 1185 snap 387 seq_num 1185) 2024-09-20T19:37:48.556 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 24323 2024-09-20T19:37:48.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:48.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:48.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:48 smithi162 bash[24557]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:48 smithi162 bash[24557]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:48.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:48.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:37:48.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:48 smithi042 bash[32075]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.883 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:48 smithi042 bash[32075]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.883 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:48 smithi042 bash[22706]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.884 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:48 smithi042 bash[22706]: cluster 2024-09-20T19:37:47.472756+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2024-09-20T19:37:48.885 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:48.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:48.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:49.119 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.147 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:49.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:49.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:49.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3568: done (2 left) 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3569: done (1 left) 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3570: done (0 left) 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3571: read oid 16 snap -1 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3571: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:37:49.500 INFO:tasks.rados.rados.0.smithi042.stdout:3572: snap_remove snap 391 2024-09-20T19:37:49.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:49.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:49 smithi162 bash[24557]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:49 smithi162 bash[24557]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:49 smithi162 bash[24557]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:49 smithi162 bash[24557]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.634 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:49.767 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[22706]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[22706]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[22706]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[22706]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[32075]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[32075]: cluster 2024-09-20T19:37:48.461801+0000 mgr.y (mgr.24491) 743 : cluster [DBG] pgmap v1437: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 121 MiB data, 646 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[32075]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.779 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:49 smithi042 bash[32075]: cluster 2024-09-20T19:37:48.487263+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2024-09-20T19:37:49.795 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:49.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:49.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:49.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:50.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:50.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:50.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:50.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:37:50.188 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:50.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:50.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:50.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:50.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:50.511 INFO:tasks.rados.rados.0.smithi042.stdout:3572: done (1 left) 2024-09-20T19:37:50.511 INFO:tasks.rados.rados.0.smithi042.stdout:3573: copy_from oid 18 from oid 12 current snap is 397 2024-09-20T19:37:50.511 INFO:tasks.rados.rados.0.smithi042.stdout:3571: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:37:50.536 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3504513 2024-09-20T19:37:50.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:50.574 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3573: finishing copy_from to smithi04268048-18 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3573: got expected ENOENT (src dne) 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3573: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3571: done (1 left) 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3573: done (0 left) 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3574: rollback oid 30 current snap is 397 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 396 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3575: read oid 9 snap -1 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3575: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3576: read oid 43 snap 396 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3576: expect deleted 2024-09-20T19:37:50.628 INFO:tasks.rados.rados.0.smithi042.stdout:3577: snap_remove snap 394 2024-09-20T19:37:50.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:50.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:50 smithi162 bash[24557]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:50 smithi162 bash[24557]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:50.817 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:50 smithi042 bash[32075]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.817 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:50 smithi042 bash[32075]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:50 smithi042 bash[22706]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:50 smithi042 bash[22706]: cluster 2024-09-20T19:37:49.496620+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2024-09-20T19:37:50.833 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:50.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:50.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:51.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:51.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:51.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:51.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:51.505 INFO:tasks.rados.rados.0.smithi042.stdout:3577: done (3 left) 2024-09-20T19:37:51.505 INFO:tasks.rados.rados.0.smithi042.stdout:3578: write (excl) oid 22 current snap is 397 2024-09-20T19:37:51.505 INFO:tasks.rados.rados.0.smithi042.stdout:3578: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-20T19:37:51.514 INFO:tasks.rados.rados.0.smithi042.stdout:3578: writing smithi04268048-22 from 638996 to 1377601 tid 1 2024-09-20T19:37:51.521 INFO:tasks.rados.rados.0.smithi042.stdout:3578: writing smithi04268048-22 from 1926476 to 2638996 tid 2 2024-09-20T19:37:51.521 INFO:tasks.rados.rados.0.smithi042.stdout:3579: rollback oid 8 current snap is 397 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 8 to 396 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:3574: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:3574: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:3574: finishing rollback tid 2 to smithi04268048-30 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1585 (ObjNum 1195 snap 389 seq_num 1195) dirty exists 2024-09-20T19:37:51.522 INFO:tasks.rados.rados.0.smithi042.stdout:3575: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:51.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:51.591 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:51.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[22706]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.598 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:51 smithi042 bash[32075]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.608 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:51.626 INFO:tasks.rados.rados.0.smithi042.stdout:3578: finishing write tid 1 to smithi04268048-22 2024-09-20T19:37:51.626 INFO:tasks.rados.rados.0.smithi042.stdout:3578: finishing write tid 2 to smithi04268048-22 2024-09-20T19:37:51.626 INFO:tasks.rados.rados.0.smithi042.stdout:3578: finishing write tid 3 to smithi04268048-22 2024-09-20T19:37:51.626 INFO:tasks.rados.rados.0.smithi042.stdout:3578: finishing write tid 4 to smithi04268048-22 2024-09-20T19:37:51.626 INFO:tasks.rados.rados.0.smithi042.stdout:3578: oid 22 updating version 0 to 1039 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3578: oid 22 updating version 1039 to 1041 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3578: oid 22 version 1041 is already newer than 1040 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1041 (ObjNum 1228 snap 397 seq_num 1228) dirty exists 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3578: left oid 22 (ObjNum 1228 snap 397 seq_num 1228) 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3579: finishing rollback tid 0 to smithi04268048-8 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3579: finishing rollback tid 1 to smithi04268048-8 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3579: finishing rollback tid 2 to smithi04268048-8 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1249 (ObjNum 1202 snap 390 seq_num 1202) dirty exists 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3574: done (4 left) 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3575: done (3 left) 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3576: done (2 left) 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3578: done (1 left) 2024-09-20T19:37:51.627 INFO:tasks.rados.rados.0.smithi042.stdout:3579: done (0 left) 2024-09-20T19:37:51.628 INFO:tasks.rados.rados.0.smithi042.stdout:3580: write oid 17 current snap is 397 2024-09-20T19:37:51.628 INFO:tasks.rados.rados.0.smithi042.stdout:3580: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-20T19:37:51.633 INFO:tasks.rados.rados.0.smithi042.stdout:3580: writing smithi04268048-17 from 655803 to 1069657 tid 1 2024-09-20T19:37:51.639 INFO:tasks.rados.rados.0.smithi042.stdout:3580: writing smithi04268048-17 from 1584958 to 2047379 tid 2 2024-09-20T19:37:51.639 INFO:tasks.rados.rados.0.smithi042.stdout:3581: write (excl) oid 34 current snap is 397 2024-09-20T19:37:51.640 INFO:tasks.rados.rados.0.smithi042.stdout:3580: finishing write tid 1 to smithi04268048-17 2024-09-20T19:37:51.640 INFO:tasks.rados.rados.0.smithi042.stdout:3581: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-20T19:37:51.646 INFO:tasks.rados.rados.0.smithi042.stdout:3581: writing smithi04268048-34 from 672610 to 1161713 tid 1 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3581: writing smithi04268048-34 from 1643440 to 2365872 tid 2 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3582: snap_create 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: finishing write tid 2 to smithi04268048-17 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: finishing write tid 3 to smithi04268048-17 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: finishing write tid 4 to smithi04268048-17 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: oid 17 updating version 0 to 1170 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: oid 17 version 1170 is already newer than 1169 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: oid 17 version 1170 is already newer than 1168 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1170 (ObjNum 1229 snap 397 seq_num 1229) dirty exists 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3580: left oid 17 (ObjNum 1229 snap 397 seq_num 1229) 2024-09-20T19:37:51.653 INFO:tasks.rados.rados.0.smithi042.stdout:3581: finishing write tid 1 to smithi04268048-34 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: finishing write tid 2 to smithi04268048-34 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: finishing write tid 3 to smithi04268048-34 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: finishing write tid 4 to smithi04268048-34 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: oid 34 updating version 0 to 1239 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: oid 34 updating version 1239 to 1241 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:3581: oid 34 version 1241 is already newer than 1240 2024-09-20T19:37:51.658 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1241 (ObjNum 1230 snap 397 seq_num 1230) dirty exists 2024-09-20T19:37:51.659 INFO:tasks.rados.rados.0.smithi042.stdout:3581: left oid 34 (ObjNum 1230 snap 397 seq_num 1230) 2024-09-20T19:37:51.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: audit 2024-09-20T19:37:49.885406+0000 mgr.y (mgr.24491) 744 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:37:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: cluster 2024-09-20T19:37:50.462304+0000 mgr.y (mgr.24491) 745 : cluster [DBG] pgmap v1440: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 119 MiB data, 642 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-20T19:37:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:51 smithi162 bash[24557]: cluster 2024-09-20T19:37:50.507488+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2024-09-20T19:37:51.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:51.868 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:51.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:51.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:52.001 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:52.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:52.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:52.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:52.261 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:52.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:52.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:52.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:52.536 INFO:tasks.rados.rados.0.smithi042.stdout:3580: done (2 left) 2024-09-20T19:37:52.536 INFO:tasks.rados.rados.0.smithi042.stdout:3581: done (1 left) 2024-09-20T19:37:52.537 INFO:tasks.rados.rados.0.smithi042.stdout:3582: done (0 left) 2024-09-20T19:37:52.537 INFO:tasks.rados.rados.0.smithi042.stdout:3583: write (excl) oid 23 current snap is 398 2024-09-20T19:37:52.537 INFO:tasks.rados.rados.0.smithi042.stdout:3583: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-20T19:37:52.544 INFO:tasks.rados.rados.0.smithi042.stdout:3583: writing smithi04268048-23 from 689417 to 1253769 tid 1 2024-09-20T19:37:52.551 INFO:tasks.rados.rados.0.smithi042.stdout:3583: writing smithi04268048-23 from 1701922 to 2168012 tid 2 2024-09-20T19:37:52.551 INFO:tasks.rados.rados.0.smithi042.stdout:3584: write oid 22 current snap is 398 2024-09-20T19:37:52.551 INFO:tasks.rados.rados.0.smithi042.stdout:3584: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-20T19:37:52.559 INFO:tasks.rados.rados.0.smithi042.stdout:3584: writing smithi04268048-22 from 706224 to 1462178 tid 1 2024-09-20T19:37:52.567 INFO:tasks.rados.rados.0.smithi042.stdout:3584: writing smithi04268048-22 from 2160404 to 2706224 tid 2 2024-09-20T19:37:52.567 INFO:tasks.rados.rados.0.smithi042.stdout:3585: snap_create 2024-09-20T19:37:52.567 INFO:tasks.rados.rados.0.smithi042.stdout:3583: finishing write tid 1 to smithi04268048-23 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: finishing write tid 2 to smithi04268048-23 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: finishing write tid 3 to smithi04268048-23 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: finishing write tid 4 to smithi04268048-23 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: oid 23 updating version 0 to 1171 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: oid 23 updating version 1171 to 1172 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: oid 23 updating version 1172 to 1173 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1173 (ObjNum 1231 snap 398 seq_num 1231) dirty exists 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3583: left oid 23 (ObjNum 1231 snap 398 seq_num 1231) 2024-09-20T19:37:52.568 INFO:tasks.rados.rados.0.smithi042.stdout:3584: finishing write tid 1 to smithi04268048-22 2024-09-20T19:37:52.572 INFO:tasks.rados.rados.0.smithi042.stdout:3584: finishing write tid 2 to smithi04268048-22 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:3584: finishing write tid 3 to smithi04268048-22 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:3584: finishing write tid 4 to smithi04268048-22 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:3584: oid 22 updating version 0 to 1043 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:3584: oid 22 version 1043 is already newer than 1042 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:3584: oid 22 updating version 1043 to 1044 2024-09-20T19:37:52.573 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1044 (ObjNum 1232 snap 398 seq_num 1232) dirty exists 2024-09-20T19:37:52.574 INFO:tasks.rados.rados.0.smithi042.stdout:3584: left oid 22 (ObjNum 1232 snap 398 seq_num 1232) 2024-09-20T19:37:52.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:52.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:52 smithi162 bash[24557]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:52 smithi162 bash[24557]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:52.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:52.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:52.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:52.895 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:52 smithi042 bash[32075]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.896 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:52 smithi042 bash[32075]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:52 smithi042 bash[22706]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:52 smithi042 bash[22706]: cluster 2024-09-20T19:37:51.503782+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2024-09-20T19:37:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:52.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:53.011 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:53.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:53.163 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:53.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:53.292 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:53.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:53.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:53.427 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:37:53.454 INFO:tasks.rados.rados.0.smithi042.stdout:3583: done (2 left) 2024-09-20T19:37:53.454 INFO:tasks.rados.rados.0.smithi042.stdout:3584: done (1 left) 2024-09-20T19:37:53.455 INFO:tasks.rados.rados.0.smithi042.stdout:3585: done (0 left) 2024-09-20T19:37:53.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: write oid 40 current snap is 399 2024-09-20T19:37:53.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: seq_num 1233 ranges {723031=431203,1818886=469759} 2024-09-20T19:37:53.455 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T19:37:53.456 INFO:tasks.rados.rados.0.smithi042.stdout:3586: writing smithi04268048-40 from 723031 to 1154234 tid 1 2024-09-20T19:37:53.464 INFO:tasks.rados.rados.0.smithi042.stdout:3586: writing smithi04268048-40 from 1818886 to 2288645 tid 2 2024-09-20T19:37:53.464 INFO:tasks.rados.rados.0.smithi042.stdout:3587: snap_remove snap 396 2024-09-20T19:37:53.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:53.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:53.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:53.676 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:53.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:53.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.792 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[32075]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:53 smithi042 bash[22706]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:52.463915+0000 mgr.y (mgr.24491) 746 : cluster [DBG] pgmap v1443: 178 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 165 active+clean; 102 MiB data, 617 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 510 KiB/s wr, 11 op/s 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:52.534437+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.059336+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.42:0/217332771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.059735+0000 mon.a (mon.0) 2579 : audit [INF] from='client.26039 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: audit 2024-09-20T19:37:53.442780+0000 mon.a (mon.0) 2580 : audit [INF] from='client.26039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:37:53.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:53 smithi162 bash[24557]: cluster 2024-09-20T19:37:53.447035+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2024-09-20T19:37:53.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:53.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:53.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:53.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:54.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:54.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:54.198 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:54.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:54.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:54.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:54.454 INFO:tasks.rados.rados.0.smithi042.stdout:3587: done (1 left) 2024-09-20T19:37:54.454 INFO:tasks.rados.rados.0.smithi042.stdout:3588: rollback oid 44 current snap is 399 2024-09-20T19:37:54.454 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 398 2024-09-20T19:37:54.454 INFO:tasks.rados.rados.0.smithi042.stdout:3586: finishing write tid 1 to smithi04268048-40 2024-09-20T19:37:54.454 INFO:tasks.rados.rados.0.smithi042.stdout:3586: finishing write tid 2 to smithi04268048-40 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: finishing write tid 3 to smithi04268048-40 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: finishing write tid 4 to smithi04268048-40 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: oid 40 updating version 0 to 1063 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: oid 40 version 1063 is already newer than 1062 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: oid 40 version 1063 is already newer than 1061 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1063 (ObjNum 1233 snap 399 seq_num 1233) dirty exists 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: left oid 40 (ObjNum 1233 snap 399 seq_num 1233) 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3586: done (1 left) 2024-09-20T19:37:54.455 INFO:tasks.rados.rados.0.smithi042.stdout:3589: snap_remove snap 397 2024-09-20T19:37:54.464 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:54.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:54.596 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:54.699 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.727 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:54.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:54.961 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:54.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:55.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:55.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:55.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:55.248 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:55.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:55.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:3589: done (1 left) 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:3590: delete oid 4 current snap is 399 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:3588: finishing rollback tid 0 to smithi04268048-44 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:3588: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:3588: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:37:55.467 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1268 (ObjNum 1193 snap 388 seq_num 1193) dirty exists 2024-09-20T19:37:55.473 INFO:tasks.rados.rados.0.smithi042.stdout:3588: done (1 left) 2024-09-20T19:37:55.473 INFO:tasks.rados.rados.0.smithi042.stdout:3590: done (0 left) 2024-09-20T19:37:55.473 INFO:tasks.rados.rados.0.smithi042.stdout:3591: snap_remove snap 398 2024-09-20T19:37:55.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:55.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:55.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:55.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[32075]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[32075]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[32075]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[32075]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[22706]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[22706]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[22706]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:55 smithi042 bash[22706]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:55.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:55.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:55 smithi162 bash[24557]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:55 smithi162 bash[24557]: cluster 2024-09-20T19:37:54.452060+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2024-09-20T19:37:55.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:55 smithi162 bash[24557]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:55 smithi162 bash[24557]: cluster 2024-09-20T19:37:54.464564+0000 mgr.y (mgr.24491) 747 : cluster [DBG] pgmap v1447: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-20T19:37:55.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:55.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:55.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:55.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:55.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:56.016 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:56.107 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:37:56.107 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:37:56.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.124 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:37:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:37:56.143 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:37:56.143 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:37:56.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:56.160 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:37:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:37:56.179 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:37:56.179 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:37:56.198 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:37:56.199 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:37:56.220 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:37:56.221 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:37:56.242 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:37:56.242 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:37:56.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:56.276 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:56.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:56.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:37:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:37:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:3591: done (0 left) 2024-09-20T19:37:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:3592: copy_from oid 41 from oid 34 current snap is 399 2024-09-20T19:37:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:3593: read oid 31 snap -1 2024-09-20T19:37:56.494 INFO:tasks.rados.rados.0.smithi042.stdout:3593: expect deleted 2024-09-20T19:37:56.495 INFO:tasks.rados.rados.0.smithi042.stdout:3594: snap_create 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[22706]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.505 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.506 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:56 smithi042 bash[32075]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T19:37:56.507 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:56.511 INFO:tasks.rados.rados.0.smithi042.stdout:3592: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:37:56.514 INFO:tasks.rados.rados.0.smithi042.stdout:3592: finishing copy_from to smithi04268048-41 2024-09-20T19:37:56.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1241 (ObjNum 1230 snap 397 seq_num 1230) dirty exists 2024-09-20T19:37:56.529 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:56.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:56.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.463689+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2024-09-20T19:37:56.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.644 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.680431+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:37:56.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: cluster 2024-09-20T19:37:55.682440+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:37:56.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:56 smithi162 bash[24557]: audit 2024-09-20T19:37:56.086794+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.42:0/900977534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:37:56.659 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:56.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:56.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:56.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:56.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:57.022 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:57.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:57.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:57.282 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:57.339 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:57.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:57.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:57.475 INFO:tasks.rados.rados.0.smithi042.stdout:3592: done (2 left) 2024-09-20T19:37:57.475 INFO:tasks.rados.rados.0.smithi042.stdout:3593: done (1 left) 2024-09-20T19:37:57.475 INFO:tasks.rados.rados.0.smithi042.stdout:3594: done (0 left) 2024-09-20T19:37:57.475 INFO:tasks.rados.rados.0.smithi042.stdout:3595: write (excl) oid 48 current snap is 400 2024-09-20T19:37:57.475 INFO:tasks.rados.rados.0.smithi042.stdout:3595: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-09-20T19:37:57.482 INFO:tasks.rados.rados.0.smithi042.stdout:3595: writing smithi04268048-48 from 756645 to 1338346 tid 1 2024-09-20T19:37:57.489 INFO:tasks.rados.rados.0.smithi042.stdout:3595: writing smithi04268048-48 from 1935850 to 2409278 tid 2 2024-09-20T19:37:57.489 INFO:tasks.rados.rados.0.smithi042.stdout:3596: rollback oid 6 current snap is 400 2024-09-20T19:37:57.490 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 399 2024-09-20T19:37:57.490 INFO:tasks.rados.rados.0.smithi042.stdout:3597: snap_create 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: finishing write tid 1 to smithi04268048-48 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: finishing write tid 2 to smithi04268048-48 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: finishing write tid 3 to smithi04268048-48 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: finishing write tid 4 to smithi04268048-48 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: oid 48 updating version 0 to 1122 2024-09-20T19:37:57.497 INFO:tasks.rados.rados.0.smithi042.stdout:3595: oid 48 version 1122 is already newer than 1121 2024-09-20T19:37:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3595: oid 48 version 1122 is already newer than 1120 2024-09-20T19:37:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1122 (ObjNum 1235 snap 400 seq_num 1235) dirty exists 2024-09-20T19:37:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3595: left oid 48 (ObjNum 1235 snap 400 seq_num 1235) 2024-09-20T19:37:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:3596: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:37:57.498 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 0 (ObjNum 257 snap 0 seq_num 1415567424) dirty dne 2024-09-20T19:37:57.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[32075]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[32075]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[32075]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[32075]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[22706]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.634 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[22706]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[22706]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:57 smithi042 bash[22706]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:57.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:57 smithi162 bash[24557]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:57 smithi162 bash[24557]: cluster 2024-09-20T19:37:56.465325+0000 mgr.y (mgr.24491) 748 : cluster [DBG] pgmap v1449: 178 pgs: 7 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 162 active+clean; 96 MiB data, 602 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 672 KiB/s wr, 2 op/s 2024-09-20T19:37:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:57 smithi162 bash[24557]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:57 smithi162 bash[24557]: cluster 2024-09-20T19:37:56.475190+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2024-09-20T19:37:57.834 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:37:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:37:57.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:57.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:57.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:58.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:58.099 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:37:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:37:58.099 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:37:58.100 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:37:58.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:58.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:58.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:58.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:58.385 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:58.455 INFO:tasks.rados.rados.0.smithi042.stdout:3595: done (2 left) 2024-09-20T19:37:58.455 INFO:tasks.rados.rados.0.smithi042.stdout:3596: done (1 left) 2024-09-20T19:37:58.455 INFO:tasks.rados.rados.0.smithi042.stdout:3597: done (0 left) 2024-09-20T19:37:58.455 INFO:tasks.rados.rados.0.smithi042.stdout:3598: write (excl) oid 21 current snap is 401 2024-09-20T19:37:58.455 INFO:tasks.rados.rados.0.smithi042.stdout:3598: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-20T19:37:58.463 INFO:tasks.rados.rados.0.smithi042.stdout:3598: writing smithi04268048-21 from 773452 to 1430402 tid 1 2024-09-20T19:37:58.470 INFO:tasks.rados.rados.0.smithi042.stdout:3598: writing smithi04268048-21 from 1877979 to 2495065 tid 2 2024-09-20T19:37:58.471 INFO:tasks.rados.rados.0.smithi042.stdout:3599: write oid 38 current snap is 401 2024-09-20T19:37:58.471 INFO:tasks.rados.rados.0.smithi042.stdout:3599: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-09-20T19:37:58.480 INFO:tasks.rados.rados.0.smithi042.stdout:3599: writing smithi04268048-38 from 790259 to 1522458 tid 1 2024-09-20T19:37:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3599: writing smithi04268048-38 from 1936461 to 2413558 tid 2 2024-09-20T19:37:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3600: copy_from oid 37 from oid 5 current snap is 401 2024-09-20T19:37:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3601: write oid 27 current snap is 401 2024-09-20T19:37:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3601: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-20T19:37:58.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:58.492 INFO:tasks.rados.rados.0.smithi042.stdout:3601: writing smithi04268048-27 from 407066 to 814514 tid 1 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3601: writing smithi04268048-27 from 1594943 to 2215698 tid 2 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3602: delete oid 26 current snap is 401 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3598: finishing write tid 1 to smithi04268048-21 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3598: finishing write tid 2 to smithi04268048-21 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3598: finishing write tid 3 to smithi04268048-21 2024-09-20T19:37:58.501 INFO:tasks.rados.rados.0.smithi042.stdout:3598: finishing write tid 4 to smithi04268048-21 2024-09-20T19:37:58.502 INFO:tasks.rados.rados.0.smithi042.stdout:3598: oid 21 updating version 0 to 1593 2024-09-20T19:37:58.502 INFO:tasks.rados.rados.0.smithi042.stdout:3598: oid 21 updating version 1593 to 1595 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3598: oid 21 version 1595 is already newer than 1594 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 21 v 1595 (ObjNum 1236 snap 401 seq_num 1236) dirty exists 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3598: left oid 21 (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3599: finishing write tid 1 to smithi04268048-38 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3599: finishing write tid 2 to smithi04268048-38 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3599: finishing write tid 3 to smithi04268048-38 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3599: finishing write tid 4 to smithi04268048-38 2024-09-20T19:37:58.503 INFO:tasks.rados.rados.0.smithi042.stdout:3599: oid 38 updating version 0 to 1161 2024-09-20T19:37:58.504 INFO:tasks.rados.rados.0.smithi042.stdout:3599: oid 38 version 1161 is already newer than 1159 2024-09-20T19:37:58.504 INFO:tasks.rados.rados.0.smithi042.stdout:3599: oid 38 version 1161 is already newer than 1160 2024-09-20T19:37:58.504 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1161 (ObjNum 1237 snap 401 seq_num 1237) dirty exists 2024-09-20T19:37:58.504 INFO:tasks.rados.rados.0.smithi042.stdout:3599: left oid 38 (ObjNum 1237 snap 401 seq_num 1237) 2024-09-20T19:37:58.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:58 smithi162 bash[24557]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.507 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:58 smithi162 bash[24557]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.508 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:58 smithi162 bash[24557]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.508 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:58 smithi162 bash[24557]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.517 INFO:tasks.rados.rados.0.smithi042.stdout:3600: finishing copy_from racing read to smithi04268048-37 2024-09-20T19:37:58.518 INFO:tasks.rados.rados.0.smithi042.stdout:3601: finishing write tid 1 to smithi04268048-27 2024-09-20T19:37:58.518 INFO:tasks.rados.rados.0.smithi042.stdout:3601: finishing write tid 2 to smithi04268048-27 2024-09-20T19:37:58.518 INFO:tasks.rados.rados.0.smithi042.stdout:3601: finishing write tid 3 to smithi04268048-27 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:3601: finishing write tid 4 to smithi04268048-27 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:3601: oid 27 updating version 0 to 1318 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:3601: oid 27 version 1318 is already newer than 1317 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:3601: oid 27 updating version 1318 to 1319 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1319 (ObjNum 1238 snap 401 seq_num 1238) dirty exists 2024-09-20T19:37:58.519 INFO:tasks.rados.rados.0.smithi042.stdout:3601: left oid 27 (ObjNum 1238 snap 401 seq_num 1238) 2024-09-20T19:37:58.520 INFO:tasks.rados.rados.0.smithi042.stdout:3598: done (4 left) 2024-09-20T19:37:58.520 INFO:tasks.rados.rados.0.smithi042.stdout:3599: done (3 left) 2024-09-20T19:37:58.520 INFO:tasks.rados.rados.0.smithi042.stdout:3601: done (2 left) 2024-09-20T19:37:58.521 INFO:tasks.rados.rados.0.smithi042.stdout:3602: done (1 left) 2024-09-20T19:37:58.521 INFO:tasks.rados.rados.0.smithi042.stdout:3603: delete oid 43 current snap is 401 2024-09-20T19:37:58.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:58.521 INFO:tasks.rados.rados.0.smithi042.stdout:3603: done (1 left) 2024-09-20T19:37:58.522 INFO:tasks.rados.rados.0.smithi042.stdout:3604: snap_remove snap 399 2024-09-20T19:37:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:58.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[32075]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[32075]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[32075]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[32075]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[22706]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[22706]: cluster 2024-09-20T19:37:57.472834+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2024-09-20T19:37:58.626 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[22706]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:58 smithi042 bash[22706]: cluster 2024-09-20T19:37:58.449054+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2024-09-20T19:37:58.645 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:58.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:58.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:58.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:58.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:58.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:59.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:59.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:59.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:59.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:37:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:37:59.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:59.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:37:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:37:59.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:59.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:37:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3604: done (1 left) 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3605: read oid 9 snap -1 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3605: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3606: read oid 32 snap 400 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3606: expect (ObjNum 1128 snap 370 seq_num 1128) 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3607: rollback oid 25 current snap is 401 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 25 to 400 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3608: rollback oid 16 current snap is 401 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 16 to 400 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3609: write oid 35 current snap is 401 2024-09-20T19:37:59.499 INFO:tasks.rados.rados.0.smithi042.stdout:3609: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-20T19:37:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:37:59.503 INFO:tasks.rados.rados.0.smithi042.stdout:3609: writing smithi04268048-35 from 457487 to 1207035 tid 1 2024-09-20T19:37:59.510 INFO:tasks.rados.rados.0.smithi042.stdout:3609: writing smithi04268048-35 from 1770389 to 2254824 tid 2 2024-09-20T19:37:59.514 INFO:tasks.rados.rados.0.smithi042.stdout:3609: writing smithi04268048-35 from 2746287 to 2857487 tid 3 2024-09-20T19:37:59.514 INFO:tasks.rados.rados.0.smithi042.stdout:3610: delete oid 33 current snap is 401 2024-09-20T19:37:59.514 INFO:tasks.rados.rados.0.smithi042.stdout:3600: finishing copy_from to smithi04268048-37 2024-09-20T19:37:59.515 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:37:59.515 INFO:tasks.rados.rados.0.smithi042.stdout:3606: expect (ObjNum 1128 snap 370 seq_num 1128) 2024-09-20T19:37:59.525 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 965745 2024-09-20T19:37:59.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:59.550 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:37:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:37:59.634 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:59.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:59.670 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:37:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:37:59.697 INFO:tasks.rados.rados.0.smithi042.stdout:3608: finishing rollback tid 0 to smithi04268048-16 2024-09-20T19:37:59.697 INFO:tasks.rados.rados.0.smithi042.stdout:3607: finishing rollback tid 1 to smithi04268048-25 2024-09-20T19:37:59.697 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 0 (ObjNum 225 snap 0 seq_num 1415322064) dirty dne 2024-09-20T19:37:59.698 INFO:tasks.rados.rados.0.smithi042.stdout:3605: expect (ObjNum 1208 snap 390 seq_num 1208) 2024-09-20T19:37:59.698 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:37:59.747 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T19:37:59.773 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:59.790 INFO:tasks.rados.rados.0.smithi042.stdout:3608: finishing rollback tid 1 to smithi04268048-16 2024-09-20T19:37:59.790 INFO:tasks.rados.rados.0.smithi042.stdout:3608: finishing rollback tid 2 to smithi04268048-16 2024-09-20T19:37:59.790 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1444 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: finishing write tid 1 to smithi04268048-35 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: finishing write tid 2 to smithi04268048-35 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: finishing write tid 3 to smithi04268048-35 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: finishing write tid 4 to smithi04268048-35 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: finishing write tid 5 to smithi04268048-35 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: oid 35 updating version 0 to 1245 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: oid 35 version 1245 is already newer than 1244 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: oid 35 version 1245 is already newer than 1242 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: oid 35 version 1245 is already newer than 1243 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1245 (ObjNum 1241 snap 401 seq_num 1241) dirty exists 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3609: left oid 35 (ObjNum 1241 snap 401 seq_num 1241) 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3600: done (6 left) 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3605: done (5 left) 2024-09-20T19:37:59.791 INFO:tasks.rados.rados.0.smithi042.stdout:3606: done (4 left) 2024-09-20T19:37:59.792 INFO:tasks.rados.rados.0.smithi042.stdout:3607: done (3 left) 2024-09-20T19:37:59.792 INFO:tasks.rados.rados.0.smithi042.stdout:3608: done (2 left) 2024-09-20T19:37:59.792 INFO:tasks.rados.rados.0.smithi042.stdout:3609: done (1 left) 2024-09-20T19:37:59.792 INFO:tasks.rados.rados.0.smithi042.stdout:3610: done (0 left) 2024-09-20T19:37:59.792 INFO:tasks.rados.rados.0.smithi042.stdout:3611: delete oid 35 current snap is 401 2024-09-20T19:37:59.793 INFO:tasks.rados.rados.0.smithi042.stdout:3611: done (0 left) 2024-09-20T19:37:59.794 INFO:tasks.rados.rados.0.smithi042.stdout:3612: write oid 1 current snap is 401 2024-09-20T19:37:59.794 INFO:tasks.rados.rados.0.smithi042.stdout:3612: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-20T19:37:59.799 INFO:tasks.rados.rados.0.smithi042.stdout:3612: writing smithi04268048-1 from 507908 to 1083203 tid 1 2024-09-20T19:37:59.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.800 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.800 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.801 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.801 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[32075]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:37:59 smithi042 bash[22706]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:58.465878+0000 mgr.y (mgr.24491) 749 : cluster [DBG] pgmap v1453: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 91 MiB data, 587 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:58.560932+0000 mon.a (mon.0) 2587 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:58.563329+0000 mon.a (mon.0) 2588 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.337318+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.42:0/1030432804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.337856+0000 mon.a (mon.0) 2589 : audit [INF] from='client.26051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:59.482832+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: audit 2024-09-20T19:37:59.484738+0000 mon.a (mon.0) 2591 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[24557]: cluster 2024-09-20T19:37:59.491987+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2024-09-20T19:37:59.808 INFO:tasks.rados.rados.0.smithi042.stdout:3612: writing smithi04268048-1 from 1829482 to 2461244 tid 2 2024-09-20T19:37:59.808 INFO:tasks.rados.rados.0.smithi042.stdout:3613: read oid 22 snap -1 2024-09-20T19:37:59.808 INFO:tasks.rados.rados.0.smithi042.stdout:3613: expect (ObjNum 1232 snap 398 seq_num 1232) 2024-09-20T19:37:59.808 INFO:tasks.rados.rados.0.smithi042.stdout:3614: rollback oid 42 current snap is 401 2024-09-20T19:37:59.808 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 400 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3612: finishing write tid 1 to smithi04268048-1 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3615: rollback oid 3 current snap is 401 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 3 to 400 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3616: rollback oid 39 current snap is 401 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 39 to 400 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3617: read oid 7 snap -1 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3617: expect deleted 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3618: copy_from oid 35 from oid 38 current snap is 401 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3619: write (excl) oid 17 current snap is 401 2024-09-20T19:37:59.809 INFO:tasks.rados.rados.0.smithi042.stdout:3619: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-09-20T19:37:59.811 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:37:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:37:59.819 INFO:tasks.rados.rados.0.smithi042.stdout:3619: writing smithi04268048-17 from 524715 to 1175259 tid 1 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3619: writing smithi04268048-17 from 1887964 to 2379737 tid 2 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3620: read oid 20 snap -1 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3620: expect deleted 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3621: read oid 33 snap -1 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3621: expect deleted 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3622: write (excl) oid 9 current snap is 401 2024-09-20T19:37:59.827 INFO:tasks.rados.rados.0.smithi042.stdout:3622: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-20T19:37:59.834 INFO:tasks.rados.rados.0.smithi042.stdout:3622: writing smithi04268048-9 from 541522 to 1267315 tid 1 2024-09-20T19:37:59.842 INFO:tasks.rados.rados.0.smithi042.stdout:3622: writing smithi04268048-9 from 1946446 to 2581877 tid 2 2024-09-20T19:37:59.842 INFO:tasks.rados.rados.0.smithi042.stdout:3623: copy_from oid 12 from oid 4 current snap is 401 2024-09-20T19:37:59.842 INFO:tasks.rados.rados.0.smithi042.stdout:3614: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:37:59.842 INFO:tasks.rados.rados.0.smithi042.stdout:3618: finishing copy_from racing read to smithi04268048-35 2024-09-20T19:37:59.842 INFO:tasks.rados.rados.0.smithi042.stdout:3613: expect (ObjNum 1232 snap 398 seq_num 1232) 2024-09-20T19:37:59.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:59.938 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:37:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:59.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3614: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3614: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1325 (ObjNum 949 snap 315 seq_num 949) dirty exists 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3615: finishing rollback tid 0 to smithi04268048-3 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3615: finishing rollback tid 1 to smithi04268048-3 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3616: finishing rollback tid 1 to smithi04268048-39 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 0 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3615: finishing rollback tid 2 to smithi04268048-3 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1601 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3612: finishing write tid 2 to smithi04268048-1 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3612: finishing write tid 3 to smithi04268048-1 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3612: finishing write tid 4 to smithi04268048-1 2024-09-20T19:37:59.959 INFO:tasks.rados.rados.0.smithi042.stdout:3612: oid 1 updating version 0 to 1145 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3612: oid 1 version 1145 is already newer than 1144 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3612: oid 1 version 1145 is already newer than 1143 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1145 (ObjNum 1244 snap 401 seq_num 1244) dirty exists 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3612: left oid 1 (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3612: done (11 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3613: done (10 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3614: done (9 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3615: done (8 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3616: done (7 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3617: done (6 left) 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3624: write oid 50 current snap is 401 2024-09-20T19:37:59.960 INFO:tasks.rados.rados.0.smithi042.stdout:3624: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-09-20T19:37:59.964 INFO:tasks.rados.rados.0.smithi042.stdout:3624: writing smithi04268048-50 from 558329 to 1075724 tid 1 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3624: writing smithi04268048-50 from 1721281 to 2500370 tid 2 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3625: copy_from oid 22 from oid 14 current snap is 401 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3618: finishing copy_from to smithi04268048-35 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1161 (ObjNum 1237 snap 401 seq_num 1237) dirty exists 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3619: finishing write tid 1 to smithi04268048-17 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3619: finishing write tid 2 to smithi04268048-17 2024-09-20T19:37:59.974 INFO:tasks.rados.rados.0.smithi042.stdout:3619: finishing write tid 3 to smithi04268048-17 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:3619: finishing write tid 4 to smithi04268048-17 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:3619: oid 17 updating version 0 to 1176 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:3619: oid 17 version 1176 is already newer than 1174 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:3619: oid 17 version 1176 is already newer than 1175 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1176 (ObjNum 1245 snap 401 seq_num 1245) dirty exists 2024-09-20T19:37:59.975 INFO:tasks.rados.rados.0.smithi042.stdout:3619: left oid 17 (ObjNum 1245 snap 401 seq_num 1245) 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3622: finishing write tid 1 to smithi04268048-9 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3623: finishing copy_from to smithi04268048-12 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3623: got expected ENOENT (src dne) 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3623: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3622: finishing write tid 2 to smithi04268048-9 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3622: finishing write tid 3 to smithi04268048-9 2024-09-20T19:37:59.976 INFO:tasks.rados.rados.0.smithi042.stdout:3622: finishing write tid 4 to smithi04268048-9 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3622: oid 9 updating version 0 to 1173 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3622: oid 9 updating version 1173 to 1175 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3622: oid 9 version 1175 is already newer than 1174 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1175 (ObjNum 1246 snap 401 seq_num 1246) dirty exists 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3622: left oid 9 (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3618: done (7 left) 2024-09-20T19:37:59.977 INFO:tasks.rados.rados.0.smithi042.stdout:3619: done (6 left) 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3620: done (5 left) 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3621: done (4 left) 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3622: done (3 left) 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3623: done (2 left) 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3626: write (excl) oid 27 current snap is 401 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3624: finishing write tid 1 to smithi04268048-50 2024-09-20T19:37:59.978 INFO:tasks.rados.rados.0.smithi042.stdout:3626: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-09-20T19:37:59.979 INFO:tasks.rados.rados.0.smithi042.stdout:3626: writing smithi04268048-27 from 575136 to 1167780 tid 1 2024-09-20T19:37:59.986 INFO:tasks.rados.rados.0.smithi042.stdout:3626: writing smithi04268048-27 from 1663410 to 2302510 tid 2 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3626: writing smithi04268048-27 from 2705542 to 2975136 tid 3 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3627: snap_create 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: finishing write tid 2 to smithi04268048-50 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: finishing write tid 3 to smithi04268048-50 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: finishing write tid 4 to smithi04268048-50 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: oid 50 updating version 0 to 1178 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: oid 50 updating version 1178 to 1179 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: oid 50 version 1179 is already newer than 1177 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1179 (ObjNum 1247 snap 401 seq_num 1247) dirty exists 2024-09-20T19:37:59.991 INFO:tasks.rados.rados.0.smithi042.stdout:3624: left oid 50 (ObjNum 1247 snap 401 seq_num 1247) 2024-09-20T19:37:59.992 INFO:tasks.rados.rados.0.smithi042.stdout:3626: finishing write tid 1 to smithi04268048-27 2024-09-20T19:37:59.997 INFO:tasks.rados.rados.0.smithi042.stdout:3626: finishing write tid 2 to smithi04268048-27 2024-09-20T19:37:59.997 INFO:tasks.rados.rados.0.smithi042.stdout:3625: finishing copy_from racing read to smithi04268048-22 2024-09-20T19:37:59.997 INFO:tasks.rados.rados.0.smithi042.stdout:3626: finishing write tid 3 to smithi04268048-27 2024-09-20T19:37:59.998 INFO:tasks.rados.rados.0.smithi042.stdout:3626: finishing write tid 4 to smithi04268048-27 2024-09-20T19:37:59.999 INFO:tasks.rados.rados.0.smithi042.stdout:3626: finishing write tid 5 to smithi04268048-27 2024-09-20T19:37:59.999 INFO:tasks.rados.rados.0.smithi042.stdout:3626: oid 27 updating version 0 to 1329 2024-09-20T19:37:59.999 INFO:tasks.rados.rados.0.smithi042.stdout:3626: oid 27 version 1329 is already newer than 1326 2024-09-20T19:37:59.999 INFO:tasks.rados.rados.0.smithi042.stdout:3626: oid 27 version 1329 is already newer than 1328 2024-09-20T19:37:59.999 INFO:tasks.rados.rados.0.smithi042.stdout:3626: oid 27 version 1329 is already newer than 1327 2024-09-20T19:38:00.000 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1329 (ObjNum 1248 snap 401 seq_num 1248) dirty exists 2024-09-20T19:38:00.000 INFO:tasks.rados.rados.0.smithi042.stdout:3626: left oid 27 (ObjNum 1248 snap 401 seq_num 1248) 2024-09-20T19:38:00.000 INFO:tasks.rados.rados.0.smithi042.stdout:3625: finishing copy_from to smithi04268048-22 2024-09-20T19:38:00.000 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1163 (ObjNum 1141 snap 375 seq_num 1141) dirty exists 2024-09-20T19:38:00.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:00.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:00.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:00.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:00.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:37:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:00.426 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:00.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:00.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3624: done (3 left) 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3625: done (2 left) 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3626: done (1 left) 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3627: done (0 left) 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3628: write (excl) oid 39 current snap is 402 2024-09-20T19:38:00.577 INFO:tasks.rados.rados.0.smithi042.stdout:3628: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-20T19:38:00.586 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:00.586 INFO:tasks.rados.rados.0.smithi042.stdout:3628: writing smithi04268048-39 from 591943 to 1259836 tid 1 2024-09-20T19:38:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:00.594 INFO:tasks.rados.rados.0.smithi042.stdout:3628: writing smithi04268048-39 from 1721892 to 2504650 tid 2 2024-09-20T19:38:00.595 INFO:tasks.rados.rados.0.smithi042.stdout:3629: snap_remove snap 401 2024-09-20T19:38:00.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:00.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:00.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:00.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:00.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:00.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:01.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:38:01.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:01.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:01.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:01.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:01.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:01.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:01.471 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:01.497 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:01.580 INFO:tasks.rados.rados.0.smithi042.stdout:3629: done (1 left) 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3630: copy_from oid 13 from oid 17 current snap is 402 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: finishing write tid 1 to smithi04268048-39 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: finishing write tid 2 to smithi04268048-39 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: finishing write tid 3 to smithi04268048-39 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: finishing write tid 4 to smithi04268048-39 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: oid 39 updating version 0 to 1254 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: oid 39 updating version 1254 to 1255 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: oid 39 version 1255 is already newer than 1253 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1255 (ObjNum 1249 snap 402 seq_num 1249) dirty exists 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: left oid 39 (ObjNum 1249 snap 402 seq_num 1249) 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3628: done (1 left) 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:3631: rollback oid 43 current snap is 402 2024-09-20T19:38:01.581 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 400 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3632: copy_from oid 25 from oid 27 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3633: rollback oid 4 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 4 to 400 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3634: copy_from oid 6 from oid 48 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3635: rollback oid 31 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 400 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3636: rollback oid 33 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 400 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3637: rollback oid 12 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 400 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3638: write oid 37 current snap is 402 2024-09-20T19:38:01.582 INFO:tasks.rados.rados.0.smithi042.stdout:3638: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-20T19:38:01.589 INFO:tasks.rados.rados.0.smithi042.stdout:3638: writing smithi04268048-37 from 608750 to 1351892 tid 1 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:3638: writing smithi04268048-37 from 1780374 to 2423143 tid 2 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:3639: snap_create 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:3637: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 0 (ObjNum 817 snap 0 seq_num 173721922) dirty dne 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:3635: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:38:01.597 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 0 (ObjNum 257 snap 0 seq_num 1414866560) dirty dne 2024-09-20T19:38:01.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:01.601 INFO:tasks.rados.rados.0.smithi042.stdout:3633: finishing rollback tid 1 to smithi04268048-4 2024-09-20T19:38:01.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:01.601 INFO:tasks.rados.rados.0.smithi042.stdout:3631: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:38:01.601 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 0 (ObjNum 225 snap 0 seq_num 1415322064) dirty dne 2024-09-20T19:38:01.607 INFO:tasks.rados.rados.0.smithi042.stdout:3638: finishing write tid 1 to smithi04268048-37 2024-09-20T19:38:01.609 INFO:tasks.rados.rados.0.smithi042.stdout:3636: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:38:01.610 INFO:tasks.rados.rados.0.smithi042.stdout:3636: finishing rollback tid 2 to smithi04268048-33 2024-09-20T19:38:01.611 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1327 (ObjNum 1143 snap 375 seq_num 1143) dirty exists 2024-09-20T19:38:01.612 INFO:tasks.rados.rados.0.smithi042.stdout:3630: finishing copy_from racing read to smithi04268048-13 2024-09-20T19:38:01.613 INFO:tasks.rados.rados.0.smithi042.stdout:3638: finishing write tid 2 to smithi04268048-37 2024-09-20T19:38:01.613 INFO:tasks.rados.rados.0.smithi042.stdout:3638: finishing write tid 3 to smithi04268048-37 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:3638: finishing write tid 4 to smithi04268048-37 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:3638: oid 37 updating version 0 to 1177 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:3638: oid 37 version 1177 is already newer than 1175 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:3638: oid 37 version 1177 is already newer than 1176 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1177 (ObjNum 1250 snap 402 seq_num 1250) dirty exists 2024-09-20T19:38:01.614 INFO:tasks.rados.rados.0.smithi042.stdout:3638: left oid 37 (ObjNum 1250 snap 402 seq_num 1250) 2024-09-20T19:38:01.618 INFO:tasks.rados.rados.0.smithi042.stdout:3634: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:38:01.619 INFO:tasks.rados.rados.0.smithi042.stdout:3632: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:38:01.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:01.627 INFO:tasks.rados.rados.0.smithi042.stdout:3634: finishing copy_from to smithi04268048-6 2024-09-20T19:38:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1122 (ObjNum 1235 snap 400 seq_num 1235) dirty exists 2024-09-20T19:38:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3630: finishing copy_from to smithi04268048-13 2024-09-20T19:38:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 1176 (ObjNum 1245 snap 401 seq_num 1245) dirty exists 2024-09-20T19:38:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:3632: finishing copy_from to smithi04268048-25 2024-09-20T19:38:01.628 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 1329 (ObjNum 1248 snap 401 seq_num 1248) dirty exists 2024-09-20T19:38:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:01.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:01.750 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:01.853 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[32075]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:01.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:01.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:01.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:01.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:01.866 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:01 smithi042 bash[22706]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:01.881 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:01.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:01.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.013 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:02.055 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:02.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: audit 2024-09-20T19:37:59.896592+0000 mgr.y (mgr.24491) 750 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:02.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:02.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: cluster 2024-09-20T19:38:00.466497+0000 mgr.y (mgr.24491) 751 : cluster [DBG] pgmap v1455: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 90 MiB data, 590 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-20T19:38:02.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:02.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:01 smithi162 bash[24557]: cluster 2024-09-20T19:38:00.573795+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2024-09-20T19:38:02.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:02.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:02.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:02.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.408 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:02.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:02.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3630: done (9 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3631: done (8 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3632: done (7 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3633: done (6 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3634: done (5 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3635: done (4 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3636: done (3 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3637: done (2 left) 2024-09-20T19:38:02.594 INFO:tasks.rados.rados.0.smithi042.stdout:3638: done (1 left) 2024-09-20T19:38:02.595 INFO:tasks.rados.rados.0.smithi042.stdout:3639: done (0 left) 2024-09-20T19:38:02.595 INFO:tasks.rados.rados.0.smithi042.stdout:3640: snap_remove snap 400 2024-09-20T19:38:02.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:02.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:02 smithi042 bash[32075]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.652 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:02 smithi042 bash[32075]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.652 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:02 smithi042 bash[22706]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.652 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:02 smithi042 bash[22706]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.669 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:02.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:02.795 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:02.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:02 smithi162 bash[24557]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:02 smithi162 bash[24557]: cluster 2024-09-20T19:38:01.576930+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2024-09-20T19:38:02.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:02.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:03.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:03.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:03.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:03.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:03.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:03.444 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:03.459 INFO:tasks.rados.rados.0.smithi042.stdout:3640: done (0 left) 2024-09-20T19:38:03.459 INFO:tasks.rados.rados.0.smithi042.stdout:3641: write oid 7 current snap is 403 2024-09-20T19:38:03.459 INFO:tasks.rados.rados.0.smithi042.stdout:3641: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-20T19:38:03.467 INFO:tasks.rados.rados.0.smithi042.stdout:3641: writing smithi04268048-7 from 625557 to 1043948 tid 1 2024-09-20T19:38:03.480 INFO:tasks.rados.rados.0.smithi042.stdout:3641: writing smithi04268048-7 from 1838856 to 2625283 tid 2 2024-09-20T19:38:03.480 INFO:tasks.rados.rados.0.smithi042.stdout:3642: write oid 39 current snap is 403 2024-09-20T19:38:03.480 INFO:tasks.rados.rados.0.smithi042.stdout:3642: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-20T19:38:03.489 INFO:tasks.rados.rados.0.smithi042.stdout:3642: writing smithi04268048-39 from 642364 to 1136004 tid 1 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3642: writing smithi04268048-39 from 1897338 to 2543776 tid 2 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3643: snap_create 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: finishing write tid 1 to smithi04268048-7 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: finishing write tid 2 to smithi04268048-7 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: finishing write tid 3 to smithi04268048-7 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: finishing write tid 4 to smithi04268048-7 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: oid 7 updating version 0 to 1267 2024-09-20T19:38:03.500 INFO:tasks.rados.rados.0.smithi042.stdout:3641: oid 7 updating version 1267 to 1268 2024-09-20T19:38:03.501 INFO:tasks.rados.rados.0.smithi042.stdout:3641: oid 7 version 1268 is already newer than 1266 2024-09-20T19:38:03.501 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1268 (ObjNum 1251 snap 403 seq_num 1251) dirty exists 2024-09-20T19:38:03.501 INFO:tasks.rados.rados.0.smithi042.stdout:3641: left oid 7 (ObjNum 1251 snap 403 seq_num 1251) 2024-09-20T19:38:03.501 INFO:tasks.rados.rados.0.smithi042.stdout:3642: finishing write tid 1 to smithi04268048-39 2024-09-20T19:38:03.505 INFO:tasks.rados.rados.0.smithi042.stdout:3642: finishing write tid 2 to smithi04268048-39 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: finishing write tid 3 to smithi04268048-39 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: finishing write tid 4 to smithi04268048-39 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: oid 39 updating version 0 to 1256 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: oid 39 updating version 1256 to 1258 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: oid 39 version 1258 is already newer than 1257 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1258 (ObjNum 1252 snap 403 seq_num 1252) dirty exists 2024-09-20T19:38:03.506 INFO:tasks.rados.rados.0.smithi042.stdout:3642: left oid 39 (ObjNum 1252 snap 403 seq_num 1252) 2024-09-20T19:38:03.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:03.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:03.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:03.699 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:03.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:03 smithi162 bash[24557]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:03.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[32075]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.467691+0000 mgr.y (mgr.24491) 752 : cluster [DBG] pgmap v1458: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 111 MiB data, 641 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 23 op/s 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.591715+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.640799+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:02.642435+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:03 smithi042 bash[22706]: cluster 2024-09-20T19:38:03.456639+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2024-09-20T19:38:03.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:03.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:04.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:04.091 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:04.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:04.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:04.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:04.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:04.466 INFO:tasks.rados.rados.0.smithi042.stdout:3641: done (2 left) 2024-09-20T19:38:04.466 INFO:tasks.rados.rados.0.smithi042.stdout:3642: done (1 left) 2024-09-20T19:38:04.466 INFO:tasks.rados.rados.0.smithi042.stdout:3643: done (0 left) 2024-09-20T19:38:04.466 INFO:tasks.rados.rados.0.smithi042.stdout:3644: write oid 40 current snap is 404 2024-09-20T19:38:04.466 INFO:tasks.rados.rados.0.smithi042.stdout:3644: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-09-20T19:38:04.473 INFO:tasks.rados.rados.0.smithi042.stdout:3644: writing smithi04268048-40 from 659171 to 1228060 tid 1 2024-09-20T19:38:04.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:04.486 INFO:tasks.rados.rados.0.smithi042.stdout:3644: writing smithi04268048-40 from 1839467 to 2629563 tid 2 2024-09-20T19:38:04.486 INFO:tasks.rados.rados.0.smithi042.stdout:3645: write (excl) oid 34 current snap is 404 2024-09-20T19:38:04.486 INFO:tasks.rados.rados.0.smithi042.stdout:3645: seq_num 1254 ranges {675978=644138,1897949=650107} 2024-09-20T19:38:04.497 INFO:tasks.rados.rados.0.smithi042.stdout:3645: writing smithi04268048-34 from 675978 to 1320116 tid 1 2024-09-20T19:38:04.508 INFO:tasks.rados.rados.0.smithi042.stdout:3645: writing smithi04268048-34 from 1897949 to 2548056 tid 2 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3646: read oid 28 snap -1 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3646: expect (ObjNum 1200 snap 390 seq_num 1200) 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3647: copy_from oid 11 from oid 22 current snap is 404 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: finishing write tid 1 to smithi04268048-40 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: finishing write tid 2 to smithi04268048-40 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: finishing write tid 3 to smithi04268048-40 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: finishing write tid 4 to smithi04268048-40 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: oid 40 updating version 0 to 1065 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: oid 40 updating version 1065 to 1066 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: oid 40 version 1066 is already newer than 1064 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1066 (ObjNum 1253 snap 404 seq_num 1253) dirty exists 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3644: left oid 40 (ObjNum 1253 snap 404 seq_num 1253) 2024-09-20T19:38:04.509 INFO:tasks.rados.rados.0.smithi042.stdout:3645: finishing write tid 1 to smithi04268048-34 2024-09-20T19:38:04.510 INFO:tasks.rados.rados.0.smithi042.stdout:3644: done (3 left) 2024-09-20T19:38:04.510 INFO:tasks.rados.rados.0.smithi042.stdout:3648: write (excl) oid 3 current snap is 404 2024-09-20T19:38:04.510 INFO:tasks.rados.rados.0.smithi042.stdout:3648: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-20T19:38:04.519 INFO:tasks.rados.rados.0.smithi042.stdout:3648: writing smithi04268048-3 from 692785 to 1128525 tid 1 2024-09-20T19:38:04.532 INFO:tasks.rados.rados.0.smithi042.stdout:3648: writing smithi04268048-3 from 1672784 to 2466549 tid 2 2024-09-20T19:38:04.538 INFO:tasks.rados.rados.0.smithi042.stdout:3648: writing smithi04268048-3 from 2897503 to 3092785 tid 3 2024-09-20T19:38:04.538 INFO:tasks.rados.rados.0.smithi042.stdout:3649: read oid 49 snap -1 2024-09-20T19:38:04.538 INFO:tasks.rados.rados.0.smithi042.stdout:3649: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:38:04.538 INFO:tasks.rados.rados.0.smithi042.stdout:3650: snap_remove snap 403 2024-09-20T19:38:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:04.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:04.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:04.714 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:04.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:04.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:04.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:04.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:05.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:05.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:05.131 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:05.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:05.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:05.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:05.392 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:05.474 INFO:tasks.rados.rados.0.smithi042.stdout:3650: done (5 left) 2024-09-20T19:38:05.474 INFO:tasks.rados.rados.0.smithi042.stdout:3651: read oid 44 snap 402 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:3651: expect (ObjNum 1193 snap 388 seq_num 1193) 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:3652: rollback oid 15 current snap is 404 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 402 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:3653: rollback oid 6 current snap is 404 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 6 to 402 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:3654: write (excl) oid 24 current snap is 404 2024-09-20T19:38:05.475 INFO:tasks.rados.rados.0.smithi042.stdout:3654: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-20T19:38:05.482 INFO:tasks.rados.rados.0.smithi042.stdout:3654: writing smithi04268048-24 from 709592 to 1220581 tid 1 2024-09-20T19:38:05.490 INFO:tasks.rados.rados.0.smithi042.stdout:3654: writing smithi04268048-24 from 1731266 to 2385042 tid 2 2024-09-20T19:38:05.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:05.496 INFO:tasks.rados.rados.0.smithi042.stdout:3654: writing smithi04268048-24 from 3041279 to 3109592 tid 3 2024-09-20T19:38:05.496 INFO:tasks.rados.rados.0.smithi042.stdout:3655: read oid 46 snap -1 2024-09-20T19:38:05.496 INFO:tasks.rados.rados.0.smithi042.stdout:3655: expect deleted 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3656: read oid 48 snap 402 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3656: expect (ObjNum 1235 snap 400 seq_num 1235) 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3657: read oid 13 snap 402 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3657: expect (ObjNum 1245 snap 401 seq_num 1245) 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3658: rollback oid 50 current snap is 404 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 402 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:3646: expect (ObjNum 1200 snap 390 seq_num 1200) 2024-09-20T19:38:05.497 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:05.522 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:05.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:05 smithi162 bash[24557]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:05 smithi162 bash[24557]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:05 smithi162 bash[24557]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:05 smithi162 bash[24557]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.561 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 311434 2024-09-20T19:38:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:05.612 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:05.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:05.644 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: finishing write tid 2 to smithi04268048-34 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: finishing write tid 3 to smithi04268048-34 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: finishing write tid 4 to smithi04268048-34 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: oid 34 updating version 0 to 1247 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: oid 34 updating version 1247 to 1248 2024-09-20T19:38:05.669 INFO:tasks.rados.rados.0.smithi042.stdout:3645: oid 34 updating version 1248 to 1249 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1249 (ObjNum 1254 snap 404 seq_num 1254) dirty exists 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3645: left oid 34 (ObjNum 1254 snap 404 seq_num 1254) 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3647: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3648: finishing write tid 1 to smithi04268048-3 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3645: done (12 left) 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3646: done (11 left) 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3659: read oid 10 snap -1 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3659: expect (ObjNum 1209 snap 391 seq_num 1209) 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3660: rollback oid 26 current snap is 404 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 26 to 402 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3647: finishing copy_from to smithi04268048-11 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1163 (ObjNum 1141 snap 375 seq_num 1141) dirty exists 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3648: finishing write tid 2 to smithi04268048-3 2024-09-20T19:38:05.670 INFO:tasks.rados.rados.0.smithi042.stdout:3648: finishing write tid 3 to smithi04268048-3 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: finishing write tid 4 to smithi04268048-3 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: finishing write tid 5 to smithi04268048-3 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: oid 3 updating version 0 to 1604 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: oid 3 version 1604 is already newer than 1603 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: oid 3 version 1604 is already newer than 1602 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: oid 3 updating version 1604 to 1605 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1605 (ObjNum 1255 snap 404 seq_num 1255) dirty exists 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3648: left oid 3 (ObjNum 1255 snap 404 seq_num 1255) 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:3649: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:38:05.671 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:05.702 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:05.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:05.756 INFO:tasks.rados.rados.0.smithi042.stdout:3652: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:38:05.756 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 0 (ObjNum 129 snap 0 seq_num 1415481712) dirty dne 2024-09-20T19:38:05.756 INFO:tasks.rados.rados.0.smithi042.stdout:3651: expect (ObjNum 1193 snap 388 seq_num 1193) 2024-09-20T19:38:05.759 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1576227 2024-09-20T19:38:05.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:05.780 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:05.845 INFO:tasks.rados.rados.0.smithi042.stdout:3656: expect (ObjNum 1235 snap 400 seq_num 1235) 2024-09-20T19:38:05.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:05.880 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 659657 2024-09-20T19:38:05.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[32075]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[32075]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[32075]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[32075]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[22706]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[22706]: cluster 2024-09-20T19:38:04.459196+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[22706]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:05 smithi042 bash[22706]: cluster 2024-09-20T19:38:04.468406+0000 mgr.y (mgr.24491) 753 : cluster [DBG] pgmap v1462: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 8.1 MiB/s wr, 29 op/s 2024-09-20T19:38:05.904 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:05.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:05.938 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2750815 2024-09-20T19:38:05.943 INFO:tasks.rados.rados.0.smithi042.stdout:3657: expect (ObjNum 1245 snap 401 seq_num 1245) 2024-09-20T19:38:05.958 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1328405 2024-09-20T19:38:06.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:06.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:06.080 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 627307 2024-09-20T19:38:06.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:06.139 INFO:tasks.rados.rados.0.smithi042.stdout:3653: finishing rollback tid 0 to smithi04268048-6 2024-09-20T19:38:06.139 INFO:tasks.rados.rados.0.smithi042.stdout:3653: finishing rollback tid 1 to smithi04268048-6 2024-09-20T19:38:06.139 INFO:tasks.rados.rados.0.smithi042.stdout:3647: done (12 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3648: done (11 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3649: done (10 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3651: done (9 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3652: done (8 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3655: done (7 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3656: done (6 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3657: done (5 left) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3661: read oid 48 snap -1 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3661: expect (ObjNum 1235 snap 400 seq_num 1235) 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3662: snap_create 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3653: finishing rollback tid 2 to smithi04268048-6 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1252 (ObjNum 1235 snap 400 seq_num 1235) dirty exists 2024-09-20T19:38:06.140 INFO:tasks.rados.rados.0.smithi042.stdout:3658: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3658: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3658: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1189 (ObjNum 1247 snap 401 seq_num 1247) dirty exists 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: finishing write tid 1 to smithi04268048-24 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: finishing write tid 2 to smithi04268048-24 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: finishing write tid 3 to smithi04268048-24 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: finishing write tid 4 to smithi04268048-24 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: finishing write tid 5 to smithi04268048-24 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: oid 24 updating version 0 to 1447 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: oid 24 version 1447 is already newer than 1446 2024-09-20T19:38:06.141 INFO:tasks.rados.rados.0.smithi042.stdout:3654: oid 24 updating version 1447 to 1448 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:3654: oid 24 version 1448 is already newer than 1445 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1448 (ObjNum 1256 snap 404 seq_num 1256) dirty exists 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:3654: left oid 24 (ObjNum 1256 snap 404 seq_num 1256) 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:3660: finishing rollback tid 1 to smithi04268048-26 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 26 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:06.142 INFO:tasks.rados.rados.0.smithi042.stdout:3659: expect (ObjNum 1209 snap 391 seq_num 1209) 2024-09-20T19:38:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:06.174 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:06.232 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2137945 2024-09-20T19:38:06.234 INFO:tasks.rados.rados.0.smithi042.stdout:3661: expect (ObjNum 1235 snap 400 seq_num 1235) 2024-09-20T19:38:06.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:06.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:06.316 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:06.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:06.407 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3653: done (6 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3654: done (5 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3658: done (4 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3659: done (3 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3660: done (2 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3661: done (1 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3662: done (0 left) 2024-09-20T19:38:06.473 INFO:tasks.rados.rados.0.smithi042.stdout:3663: snap_remove snap 402 2024-09-20T19:38:06.526 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:06.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:06 smithi042 bash[22706]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.537 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:06 smithi042 bash[22706]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.537 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:06 smithi042 bash[32075]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.538 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:06 smithi042 bash[32075]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:06.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:06 smithi162 bash[24557]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:06 smithi162 bash[24557]: cluster 2024-09-20T19:38:05.468873+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2024-09-20T19:38:06.614 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:06.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:06.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:06.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:06.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:06.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:06.934 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:06.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:07.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:07.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:07.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:07.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:07.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:07.477 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T19:38:07.481 INFO:tasks.rados.rados.0.smithi042.stdout:3663: done (0 left) 2024-09-20T19:38:07.482 INFO:tasks.rados.rados.0.smithi042.stdout:3664: read oid 38 snap -1 2024-09-20T19:38:07.482 INFO:tasks.rados.rados.0.smithi042.stdout:3664: expect (ObjNum 1237 snap 401 seq_num 1237) 2024-09-20T19:38:07.482 INFO:tasks.rados.rados.0.smithi042.stdout:3665: snap_create 2024-09-20T19:38:07.496 INFO:tasks.rados.rados.0.smithi042.stdout:3664: expect (ObjNum 1237 snap 401 seq_num 1237) 2024-09-20T19:38:07.557 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:07.575 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:07.651 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 295624 2024-09-20T19:38:07.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:07.701 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:07.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:07 smithi162 bash[24557]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:07.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[32075]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: cluster 2024-09-20T19:38:06.468904+0000 mgr.y (mgr.24491) 754 : cluster [DBG] pgmap v1465: 178 pgs: 178 active+clean; 120 MiB data, 660 MiB used, 625 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 6 op/s 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: cluster 2024-09-20T19:38:06.471491+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2024-09-20T19:38:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: audit 2024-09-20T19:38:07.122495+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.42:0/2950672608' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:07 smithi042 bash[22706]: audit 2024-09-20T19:38:07.122961+0000 mon.a (mon.0) 2600 : audit [INF] from='client.26057 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:38:07.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:07.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:07.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:08.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:08.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:08.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:08.224 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:08.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:08.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:08.458 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:08.467 INFO:tasks.rados.rados.0.smithi042.stdout:3664: done (1 left) 2024-09-20T19:38:08.467 INFO:tasks.rados.rados.0.smithi042.stdout:3665: done (0 left) 2024-09-20T19:38:08.467 INFO:tasks.rados.rados.0.smithi042.stdout:3666: read oid 31 snap -1 2024-09-20T19:38:08.468 INFO:tasks.rados.rados.0.smithi042.stdout:3666: expect deleted 2024-09-20T19:38:08.468 INFO:tasks.rados.rados.0.smithi042.stdout:3667: delete oid 44 current snap is 406 2024-09-20T19:38:08.471 INFO:tasks.rados.rados.0.smithi042.stdout:3667: done (1 left) 2024-09-20T19:38:08.471 INFO:tasks.rados.rados.0.smithi042.stdout:3668: copy_from oid 41 from oid 5 current snap is 406 2024-09-20T19:38:08.471 INFO:tasks.rados.rados.0.smithi042.stdout:3669: read oid 27 snap -1 2024-09-20T19:38:08.471 INFO:tasks.rados.rados.0.smithi042.stdout:3669: expect (ObjNum 1248 snap 401 seq_num 1248) 2024-09-20T19:38:08.472 INFO:tasks.rados.rados.0.smithi042.stdout:3670: delete oid 17 current snap is 406 2024-09-20T19:38:08.480 INFO:tasks.rados.rados.0.smithi042.stdout:3666: done (3 left) 2024-09-20T19:38:08.480 INFO:tasks.rados.rados.0.smithi042.stdout:3670: done (2 left) 2024-09-20T19:38:08.481 INFO:tasks.rados.rados.0.smithi042.stdout:3671: read oid 23 snap -1 2024-09-20T19:38:08.481 INFO:tasks.rados.rados.0.smithi042.stdout:3671: expect (ObjNum 1231 snap 398 seq_num 1231) 2024-09-20T19:38:08.481 INFO:tasks.rados.rados.0.smithi042.stdout:3672: read oid 49 snap -1 2024-09-20T19:38:08.481 INFO:tasks.rados.rados.0.smithi042.stdout:3672: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:38:08.481 INFO:tasks.rados.rados.0.smithi042.stdout:3673: delete oid 39 current snap is 406 2024-09-20T19:38:08.485 INFO:tasks.rados.rados.0.smithi042.stdout:3673: done (4 left) 2024-09-20T19:38:08.485 INFO:tasks.rados.rados.0.smithi042.stdout:3674: read oid 17 snap -1 2024-09-20T19:38:08.485 INFO:tasks.rados.rados.0.smithi042.stdout:3674: expect deleted 2024-09-20T19:38:08.486 INFO:tasks.rados.rados.0.smithi042.stdout:3675: snap_create 2024-09-20T19:38:08.491 INFO:tasks.rados.rados.0.smithi042.stdout:3671: expect (ObjNum 1231 snap 398 seq_num 1231) 2024-09-20T19:38:08.491 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.502 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:08.570 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:08.609 INFO:tasks.rados.rados.0.smithi042.stdout:3672: expect (ObjNum 1132 snap 373 seq_num 1132) 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:08 smithi162 bash[24557]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:08.640 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.673 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.705 INFO:tasks.rados.rados.0.smithi042.stdout:3668: finishing copy_from racing read to smithi04268048-41 2024-09-20T19:38:08.705 INFO:tasks.rados.rados.0.smithi042.stdout:3669: expect (ObjNum 1248 snap 401 seq_num 1248) 2024-09-20T19:38:08.705 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.734 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:08.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.744 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[32075]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: audit 2024-09-20T19:38:07.471316+0000 mon.a (mon.0) 2601 : audit [INF] from='client.26057 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: cluster 2024-09-20T19:38:07.479548+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.745 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:08 smithi042 bash[22706]: cluster 2024-09-20T19:38:08.459627+0000 mon.a (mon.0) 2603 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2024-09-20T19:38:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:08.799 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:08.847 INFO:tasks.rados.rados.0.smithi042.stdout:3668: finishing copy_from to smithi04268048-41 2024-09-20T19:38:08.847 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:38:08.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:08.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:08.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:09.114 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:09.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:09.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:09.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:09.465 INFO:tasks.rados.rados.0.smithi042.stdout:3668: done (5 left) 2024-09-20T19:38:09.466 INFO:tasks.rados.rados.0.smithi042.stdout:3669: done (4 left) 2024-09-20T19:38:09.466 INFO:tasks.rados.rados.0.smithi042.stdout:3671: done (3 left) 2024-09-20T19:38:09.466 INFO:tasks.rados.rados.0.smithi042.stdout:3672: done (2 left) 2024-09-20T19:38:09.466 INFO:tasks.rados.rados.0.smithi042.stdout:3674: done (1 left) 2024-09-20T19:38:09.466 INFO:tasks.rados.rados.0.smithi042.stdout:3675: done (0 left) 2024-09-20T19:38:09.467 INFO:tasks.rados.rados.0.smithi042.stdout:3676: copy_from oid 25 from oid 7 current snap is 407 2024-09-20T19:38:09.467 INFO:tasks.rados.rados.0.smithi042.stdout:3677: write oid 4 current snap is 407 2024-09-20T19:38:09.467 INFO:tasks.rados.rados.0.smithi042.stdout:3677: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-20T19:38:09.485 INFO:tasks.rados.rados.0.smithi042.stdout:3677: writing smithi04268048-4 from 776820 to 1188805 tid 1 2024-09-20T19:38:09.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:09.509 INFO:tasks.rados.rados.0.smithi042.stdout:3677: writing smithi04268048-4 from 1848841 to 2393602 tid 2 2024-09-20T19:38:09.519 INFO:tasks.rados.rados.0.smithi042.stdout:3677: writing smithi04268048-4 from 2918265 to 3176820 tid 3 2024-09-20T19:38:09.520 INFO:tasks.rados.rados.0.smithi042.stdout:3678: write (excl) oid 49 current snap is 407 2024-09-20T19:38:09.520 INFO:tasks.rados.rados.0.smithi042.stdout:3678: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-20T19:38:09.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:09.533 INFO:tasks.rados.rados.0.smithi042.stdout:3678: writing smithi04268048-49 from 793627 to 1280861 tid 1 2024-09-20T19:38:09.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[22706]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[22706]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.541 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.543 INFO:tasks.rados.rados.0.smithi042.stdout:3678: writing smithi04268048-49 from 1790970 to 2195742 tid 2 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3678: writing smithi04268048-49 from 2829335 to 3193627 tid 3 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3679: read oid 46 snap -1 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3679: expect deleted 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3680: delete oid 33 current snap is 407 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3676: finishing copy_from racing read to smithi04268048-25 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3676: finishing copy_from to smithi04268048-25 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 25 v 1268 (ObjNum 1251 snap 403 seq_num 1251) dirty exists 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3677: finishing write tid 1 to smithi04268048-4 2024-09-20T19:38:09.552 INFO:tasks.rados.rados.0.smithi042.stdout:3677: finishing write tid 2 to smithi04268048-4 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: finishing write tid 3 to smithi04268048-4 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: finishing write tid 4 to smithi04268048-4 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: finishing write tid 5 to smithi04268048-4 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: oid 4 updating version 0 to 1334 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: oid 4 updating version 1334 to 1336 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: oid 4 updating version 1336 to 1337 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: oid 4 version 1337 is already newer than 1335 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1337 (ObjNum 1260 snap 407 seq_num 1260) dirty exists 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3677: left oid 4 (ObjNum 1260 snap 407 seq_num 1260) 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3678: finishing write tid 1 to smithi04268048-49 2024-09-20T19:38:09.553 INFO:tasks.rados.rados.0.smithi042.stdout:3678: finishing write tid 2 to smithi04268048-49 2024-09-20T19:38:09.555 INFO:tasks.rados.rados.0.smithi042.stdout:3676: done (4 left) 2024-09-20T19:38:09.555 INFO:tasks.rados.rados.0.smithi042.stdout:3677: done (3 left) 2024-09-20T19:38:09.556 INFO:tasks.rados.rados.0.smithi042.stdout:3679: done (2 left) 2024-09-20T19:38:09.556 INFO:tasks.rados.rados.0.smithi042.stdout:3680: done (1 left) 2024-09-20T19:38:09.556 INFO:tasks.rados.rados.0.smithi042.stdout:3681: write (excl) oid 48 current snap is 407 2024-09-20T19:38:09.556 INFO:tasks.rados.rados.0.smithi042.stdout:3681: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-20T19:38:09.564 INFO:tasks.rados.rados.0.smithi042.stdout:3681: writing smithi04268048-48 from 427241 to 1181326 tid 1 2024-09-20T19:38:09.571 INFO:tasks.rados.rados.0.smithi042.stdout:3681: writing smithi04268048-48 from 1624287 to 2032728 tid 2 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3681: writing smithi04268048-48 from 2600534 to 3150498 tid 3 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3682: delete oid 35 current snap is 407 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: finishing write tid 3 to smithi04268048-49 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: finishing write tid 4 to smithi04268048-49 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: finishing write tid 5 to smithi04268048-49 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: oid 49 updating version 0 to 1610 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: oid 49 version 1610 is already newer than 1609 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: oid 49 version 1610 is already newer than 1608 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: oid 49 updating version 1610 to 1611 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1611 (ObjNum 1261 snap 407 seq_num 1261) dirty exists 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3678: left oid 49 (ObjNum 1261 snap 407 seq_num 1261) 2024-09-20T19:38:09.580 INFO:tasks.rados.rados.0.smithi042.stdout:3681: finishing write tid 1 to smithi04268048-48 2024-09-20T19:38:09.581 INFO:tasks.rados.rados.0.smithi042.stdout:3681: finishing write tid 2 to smithi04268048-48 2024-09-20T19:38:09.584 INFO:tasks.rados.rados.0.smithi042.stdout:3678: done (2 left) 2024-09-20T19:38:09.584 INFO:tasks.rados.rados.0.smithi042.stdout:3682: done (1 left) 2024-09-20T19:38:09.584 INFO:tasks.rados.rados.0.smithi042.stdout:3683: snap_remove snap 404 2024-09-20T19:38:09.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:09.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:09 smithi162 bash[24557]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:09 smithi162 bash[24557]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:09 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:09 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:09.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:09.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:09.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.899 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[32075]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.899 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[32075]: cluster 2024-09-20T19:38:08.469453+0000 mgr.y (mgr.24491) 755 : cluster [DBG] pgmap v1468: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-20T19:38:09.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:09 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.463193+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2024-09-20T19:38:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:09.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:10.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:10.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:10.176 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:10.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:10.311 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:10.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:10.457 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:38:10.457 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:38:10.457 INFO:teuthology.orchestra.run.smithi042.stdout: "complaint_time": 30, 2024-09-20T19:38:10.457 INFO:teuthology.orchestra.run.smithi042.stdout: "num_blocked_ops": 0 2024-09-20T19:38:10.458 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:38:10.501 INFO:tasks.rados.rados.0.smithi042.stdout:3683: done (1 left) 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3684: read oid 16 snap -1 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3684: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3685: delete oid 32 current snap is 407 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: finishing write tid 3 to smithi04268048-48 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: finishing write tid 4 to smithi04268048-48 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: finishing write tid 5 to smithi04268048-48 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: oid 48 updating version 0 to 1330 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: oid 48 updating version 1330 to 1332 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: oid 48 version 1332 is already newer than 1331 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: oid 48 updating version 1332 to 1333 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1333 (ObjNum 1263 snap 407 seq_num 1263) dirty exists 2024-09-20T19:38:10.502 INFO:tasks.rados.rados.0.smithi042.stdout:3681: left oid 48 (ObjNum 1263 snap 407 seq_num 1263) 2024-09-20T19:38:10.523 INFO:tasks.rados.rados.0.smithi042.stdout:3681: done (2 left) 2024-09-20T19:38:10.523 INFO:tasks.rados.rados.0.smithi042.stdout:3685: done (1 left) 2024-09-20T19:38:10.523 INFO:tasks.rados.rados.0.smithi042.stdout:3686: read oid 49 snap -1 2024-09-20T19:38:10.523 INFO:tasks.rados.rados.0.smithi042.stdout:3686: expect (ObjNum 1261 snap 407 seq_num 1261) 2024-09-20T19:38:10.523 INFO:tasks.rados.rados.0.smithi042.stdout:3687: delete oid 31 current snap is 407 2024-09-20T19:38:10.525 INFO:tasks.rados.rados.0.smithi042.stdout:3684: expect (ObjNum 1166 snap 381 seq_num 1166) 2024-09-20T19:38:10.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:10.570 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:10.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T19:38:10.618 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1530452 2024-09-20T19:38:10.641 INFO:tasks.rados.rados.0.smithi042.stdout:3686: expect (ObjNum 1261 snap 407 seq_num 1261) 2024-09-20T19:38:10.663 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2250807 2024-09-20T19:38:10.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:10.687 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:10.732 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1656952 2024-09-20T19:38:10.756 INFO:tasks.rados.rados.0.smithi042.stdout:3684: done (2 left) 2024-09-20T19:38:10.756 INFO:tasks.rados.rados.0.smithi042.stdout:3686: done (1 left) 2024-09-20T19:38:10.756 INFO:tasks.rados.rados.0.smithi042.stdout:3687: done (0 left) 2024-09-20T19:38:10.756 INFO:tasks.rados.rados.0.smithi042.stdout:3688: snap_create 2024-09-20T19:38:10.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:10.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:10.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:10.945 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:10.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:11.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.077 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:11.180 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:11.208 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:11.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.340 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:11.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:11.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:11.510 INFO:tasks.rados.rados.0.smithi042.stdout:3688: done (0 left) 2024-09-20T19:38:11.511 INFO:tasks.rados.rados.0.smithi042.stdout:3689: write oid 29 current snap is 408 2024-09-20T19:38:11.511 INFO:tasks.rados.rados.0.smithi042.stdout:3689: seq_num 1267 ranges {494469=655081,1741862=415779,2710226=584243} 2024-09-20T19:38:11.529 INFO:tasks.rados.rados.0.smithi042.stdout:3689: writing smithi04268048-29 from 494469 to 1149550 tid 1 2024-09-20T19:38:11.547 INFO:tasks.rados.rados.0.smithi042.stdout:3689: writing smithi04268048-29 from 1741862 to 2157641 tid 2 2024-09-20T19:38:11.565 INFO:tasks.rados.rados.0.smithi042.stdout:3689: writing smithi04268048-29 from 2710226 to 3294469 tid 3 2024-09-20T19:38:11.565 INFO:tasks.rados.rados.0.smithi042.stdout:3690: snap_remove snap 407 2024-09-20T19:38:11.573 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.586 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[32075]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.587 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.588 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:11 smithi042 bash[22706]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:11.703 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.710357+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:09.712709+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: audit 2024-09-20T19:38:09.902923+0000 mgr.y (mgr.24491) 756 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:10.469859+0000 mgr.y (mgr.24491) 757 : cluster [DBG] pgmap v1470: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 113 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 25 op/s 2024-09-20T19:38:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:11 smithi162 bash[24557]: cluster 2024-09-20T19:38:10.499901+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2024-09-20T19:38:11.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:11.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:11.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:11.996 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:11.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:12.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.122 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:12.216 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:38:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:38:12.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:12.234 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:38:12.234 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:38:12.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T19:38:12.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:12.254 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:38:12.254 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:38:12.271 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:38:12.271 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:38:12.289 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:38:12.289 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:38:12.308 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:38:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:38:12.340 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:38:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:38:12.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.361 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:38:12.361 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:38:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:12.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:12.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:12.524 INFO:tasks.rados.rados.0.smithi042.stdout:3690: done (1 left) 2024-09-20T19:38:12.524 INFO:tasks.rados.rados.0.smithi042.stdout:3691: copy_from oid 18 from oid 9 current snap is 408 2024-09-20T19:38:12.524 INFO:tasks.rados.rados.0.smithi042.stdout:3692: read oid 35 snap 405 2024-09-20T19:38:12.524 INFO:tasks.rados.rados.0.smithi042.stdout:3692: expect (ObjNum 1237 snap 401 seq_num 1237) 2024-09-20T19:38:12.524 INFO:tasks.rados.rados.0.smithi042.stdout:3693: snap_remove snap 406 2024-09-20T19:38:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:38:12.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[32075]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:12 smithi042 bash[22706]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:12.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: cluster 2024-09-20T19:38:11.507351+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.194540+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? v1:172.21.15.42:0/3096416112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.370256+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.42:0/3734418957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:12 smithi162 bash[24557]: audit 2024-09-20T19:38:12.370613+0000 mon.a (mon.0) 2608 : audit [INF] from='client.26063 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:38:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:12.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:12.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:13.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:13.175 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.204 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:13.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:13.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:13.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.466 INFO:tasks.rados.rados.0.smithi042.stdout:3693: done (3 left) 2024-09-20T19:38:13.466 INFO:tasks.rados.rados.0.smithi042.stdout:3694: copy_from oid 28 from oid 16 current snap is 408 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:3695: rollback oid 2 current snap is 408 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 2 to 405 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:3696: read oid 12 snap -1 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:3696: expect deleted 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:3697: write (excl) oid 39 current snap is 408 2024-09-20T19:38:13.467 INFO:tasks.rados.rados.0.smithi042.stdout:3697: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-20T19:38:13.474 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.475 INFO:tasks.rados.rados.0.smithi042.stdout:3697: writing smithi04268048-39 from 511276 to 1241606 tid 1 2024-09-20T19:38:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:13.482 INFO:tasks.rados.rados.0.smithi042.stdout:3697: writing smithi04268048-39 from 1800344 to 2359781 tid 2 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3697: writing smithi04268048-39 from 3021296 to 3311276 tid 3 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3698: copy_from oid 7 from oid 41 current snap is 408 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: finishing write tid 1 to smithi04268048-29 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: finishing write tid 2 to smithi04268048-29 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: finishing write tid 3 to smithi04268048-29 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: finishing write tid 4 to smithi04268048-29 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: finishing write tid 5 to smithi04268048-29 2024-09-20T19:38:13.488 INFO:tasks.rados.rados.0.smithi042.stdout:3689: oid 29 updating version 0 to 979 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:3689: oid 29 version 979 is already newer than 977 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:3689: oid 29 version 979 is already newer than 976 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:3689: oid 29 version 979 is already newer than 978 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 979 (ObjNum 1267 snap 408 seq_num 1267) dirty exists 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:3689: left oid 29 (ObjNum 1267 snap 408 seq_num 1267) 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:3692: expect (ObjNum 1237 snap 401 seq_num 1237) 2024-09-20T19:38:13.489 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:13.569 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:13.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.590 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:13 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:13.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.719 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.470730+0000 mgr.y (mgr.24491) 758 : cluster [DBG] pgmap v1473: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 114 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.510745+0000 mon.a (mon.0) 2609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: audit 2024-09-20T19:38:12.517271+0000 mon.a (mon.0) 2610 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.521910+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.762680+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.764379+0000 osd.6 (osd.6) 219 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.721 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:13 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.459466+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2024-09-20T19:38:13.725 INFO:tasks.rados.rados.0.smithi042.stdout:3691: finishing copy_from racing read to smithi04268048-18 2024-09-20T19:38:13.725 INFO:tasks.rados.rados.0.smithi042.stdout:3691: finishing copy_from to smithi04268048-18 2024-09-20T19:38:13.725 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1175 (ObjNum 1246 snap 401 seq_num 1246) dirty exists 2024-09-20T19:38:13.725 INFO:tasks.rados.rados.0.smithi042.stdout:3689: done (7 left) 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3691: done (6 left) 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3692: done (5 left) 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3696: done (4 left) 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3699: rollback oid 42 current snap is 408 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 42 to 405 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3695: finishing rollback tid 1 to smithi04268048-2 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 2 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: finishing write tid 1 to smithi04268048-39 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: finishing write tid 2 to smithi04268048-39 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: finishing write tid 3 to smithi04268048-39 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: finishing write tid 4 to smithi04268048-39 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: finishing write tid 5 to smithi04268048-39 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: oid 39 updating version 0 to 1261 2024-09-20T19:38:13.726 INFO:tasks.rados.rados.0.smithi042.stdout:3697: oid 39 updating version 1261 to 1263 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3697: oid 39 version 1263 is already newer than 1262 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3697: oid 39 version 1263 is already newer than 1260 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1263 (ObjNum 1268 snap 408 seq_num 1268) dirty exists 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3697: left oid 39 (ObjNum 1268 snap 408 seq_num 1268) 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3694: finishing copy_from racing read to smithi04268048-28 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3694: finishing copy_from to smithi04268048-28 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1444 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3698: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3698: finishing copy_from to smithi04268048-7 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3694: done (4 left) 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3695: done (3 left) 2024-09-20T19:38:13.727 INFO:tasks.rados.rados.0.smithi042.stdout:3697: done (2 left) 2024-09-20T19:38:13.728 INFO:tasks.rados.rados.0.smithi042.stdout:3698: done (1 left) 2024-09-20T19:38:13.728 INFO:tasks.rados.rados.0.smithi042.stdout:3700: read oid 23 snap -1 2024-09-20T19:38:13.728 INFO:tasks.rados.rados.0.smithi042.stdout:3700: expect (ObjNum 1231 snap 398 seq_num 1231) 2024-09-20T19:38:13.728 INFO:tasks.rados.rados.0.smithi042.stdout:3701: write oid 18 current snap is 408 2024-09-20T19:38:13.728 INFO:tasks.rados.rados.0.smithi042.stdout:3701: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-20T19:38:13.730 INFO:tasks.rados.rados.0.smithi042.stdout:3701: writing smithi04268048-18 from 528083 to 933662 tid 1 2024-09-20T19:38:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:13.735 INFO:tasks.rados.rados.0.smithi042.stdout:3701: writing smithi04268048-18 from 1342473 to 1761921 tid 2 2024-09-20T19:38:13.742 INFO:tasks.rados.rados.0.smithi042.stdout:3701: writing smithi04268048-18 from 2248719 to 2982668 tid 3 2024-09-20T19:38:13.742 INFO:tasks.rados.rados.0.smithi042.stdout:3702: read oid 14 snap -1 2024-09-20T19:38:13.742 INFO:tasks.rados.rados.0.smithi042.stdout:3702: expect (ObjNum 1141 snap 375 seq_num 1141) 2024-09-20T19:38:13.742 INFO:tasks.rados.rados.0.smithi042.stdout:3703: rollback oid 44 current snap is 408 2024-09-20T19:38:13.743 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 44 to 405 2024-09-20T19:38:13.743 INFO:tasks.rados.rados.0.smithi042.stdout:3700: expect (ObjNum 1231 snap 398 seq_num 1231) 2024-09-20T19:38:13.743 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:13.818 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 324067 2024-09-20T19:38:13.834 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3699: finishing rollback tid 0 to smithi04268048-42 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3699: finishing rollback tid 1 to smithi04268048-42 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3699: finishing rollback tid 2 to smithi04268048-42 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1341 (ObjNum 949 snap 315 seq_num 949) dirty exists 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3701: finishing write tid 1 to smithi04268048-18 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3699: done (4 left) 2024-09-20T19:38:13.857 INFO:tasks.rados.rados.0.smithi042.stdout:3700: done (3 left) 2024-09-20T19:38:13.858 INFO:tasks.rados.rados.0.smithi042.stdout:3704: write (excl) oid 22 current snap is 408 2024-09-20T19:38:13.858 INFO:tasks.rados.rados.0.smithi042.stdout:3701: finishing write tid 2 to smithi04268048-18 2024-09-20T19:38:13.858 INFO:tasks.rados.rados.0.smithi042.stdout:3704: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-20T19:38:13.858 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:13.863 INFO:tasks.rados.rados.0.smithi042.stdout:3704: writing smithi04268048-22 from 544890 to 1142071 tid 1 2024-09-20T19:38:13.871 INFO:tasks.rados.rados.0.smithi042.stdout:3704: writing smithi04268048-22 from 1917308 to 2480414 tid 2 2024-09-20T19:38:13.876 INFO:tasks.rados.rados.0.smithi042.stdout:3704: writing smithi04268048-22 from 3076142 to 3344890 tid 3 2024-09-20T19:38:13.876 INFO:tasks.rados.rados.0.smithi042.stdout:3705: rollback oid 4 current snap is 408 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 4 to 405 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3703: finishing rollback tid 1 to smithi04268048-44 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3703: finishing rollback tid 2 to smithi04268048-44 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1278 (ObjNum 1193 snap 388 seq_num 1193) dirty exists 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3701: finishing write tid 3 to smithi04268048-18 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3701: finishing write tid 4 to smithi04268048-18 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3703: done (4 left) 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3706: read oid 1 snap -1 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3706: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3707: rollback oid 24 current snap is 408 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 24 to 405 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3701: finishing write tid 5 to smithi04268048-18 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3701: oid 18 updating version 0 to 1606 2024-09-20T19:38:13.877 INFO:tasks.rados.rados.0.smithi042.stdout:3701: oid 18 updating version 1606 to 1607 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:3701: oid 18 version 1607 is already newer than 1605 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:3701: oid 18 version 1607 is already newer than 1604 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1607 (ObjNum 1269 snap 408 seq_num 1269) dirty exists 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:3701: left oid 18 (ObjNum 1269 snap 408 seq_num 1269) 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:3702: expect (ObjNum 1141 snap 375 seq_num 1141) 2024-09-20T19:38:13.878 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:13.928 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 330097 2024-09-20T19:38:13.960 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:13.977 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:13.984 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:13.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:14.028 INFO:tasks.rados.rados.0.smithi042.stdout:3704: finishing write tid 1 to smithi04268048-22 2024-09-20T19:38:14.028 INFO:tasks.rados.rados.0.smithi042.stdout:3704: finishing write tid 2 to smithi04268048-22 2024-09-20T19:38:14.028 INFO:tasks.rados.rados.0.smithi042.stdout:3705: finishing rollback tid 0 to smithi04268048-4 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3705: finishing rollback tid 1 to smithi04268048-4 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1343 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: finishing write tid 3 to smithi04268048-22 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: finishing write tid 4 to smithi04268048-22 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: finishing write tid 5 to smithi04268048-22 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: oid 22 updating version 0 to 1165 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: oid 22 version 1165 is already newer than 1164 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: oid 22 updating version 1165 to 1167 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: oid 22 version 1167 is already newer than 1166 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1167 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3704: left oid 22 (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3707: finishing rollback tid 0 to smithi04268048-24 2024-09-20T19:38:14.029 INFO:tasks.rados.rados.0.smithi042.stdout:3707: finishing rollback tid 1 to smithi04268048-24 2024-09-20T19:38:14.030 INFO:tasks.rados.rados.0.smithi042.stdout:3707: finishing rollback tid 2 to smithi04268048-24 2024-09-20T19:38:14.030 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1451 (ObjNum 1256 snap 404 seq_num 1256) dirty exists 2024-09-20T19:38:14.030 INFO:tasks.rados.rados.0.smithi042.stdout:3706: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:38:14.030 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:14.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.089 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1942114 2024-09-20T19:38:14.107 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:14.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:14.151 INFO:tasks.rados.rados.0.smithi042.stdout:3701: done (5 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3702: done (4 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3704: done (3 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3705: done (2 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3706: done (1 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3707: done (0 left) 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3708: copy_from oid 11 from oid 33 current snap is 408 2024-09-20T19:38:14.152 INFO:tasks.rados.rados.0.smithi042.stdout:3709: snap_remove snap 405 2024-09-20T19:38:14.217 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:14.243 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:14.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:14.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:14.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:14.547 INFO:tasks.rados.rados.0.smithi042.stdout:3709: done (1 left) 2024-09-20T19:38:14.547 INFO:tasks.rados.rados.0.smithi042.stdout:3710: delete oid 13 current snap is 408 2024-09-20T19:38:14.548 INFO:tasks.rados.rados.0.smithi042.stdout:3708: finishing copy_from to smithi04268048-11 2024-09-20T19:38:14.548 INFO:tasks.rados.rados.0.smithi042.stdout:3708: got expected ENOENT (src dne) 2024-09-20T19:38:14.548 INFO:tasks.rados.rados.0.smithi042.stdout:3708: finishing copy_from racing read to smithi04268048-11 2024-09-20T19:38:14.562 INFO:tasks.rados.rados.0.smithi042.stdout:3708: done (1 left) 2024-09-20T19:38:14.562 INFO:tasks.rados.rados.0.smithi042.stdout:3710: done (0 left) 2024-09-20T19:38:14.562 INFO:tasks.rados.rados.0.smithi042.stdout:3711: write (excl) oid 48 current snap is 408 2024-09-20T19:38:14.562 INFO:tasks.rados.rados.0.smithi042.stdout:3711: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-09-20T19:38:14.582 INFO:tasks.rados.rados.0.smithi042.stdout:3711: writing smithi04268048-48 from 578504 to 1326183 tid 1 2024-09-20T19:38:14.600 INFO:tasks.rados.rados.0.smithi042.stdout:3711: writing smithi04268048-48 from 2034272 to 2601047 tid 2 2024-09-20T19:38:14.608 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:14.609 INFO:tasks.rados.rados.0.smithi042.stdout:3711: writing smithi04268048-48 from 3130988 to 3378504 tid 3 2024-09-20T19:38:14.610 INFO:tasks.rados.rados.0.smithi042.stdout:3712: snap_create 2024-09-20T19:38:14.610 INFO:tasks.rados.rados.0.smithi042.stdout:3711: finishing write tid 1 to smithi04268048-48 2024-09-20T19:38:14.610 INFO:tasks.rados.rados.0.smithi042.stdout:3711: finishing write tid 2 to smithi04268048-48 2024-09-20T19:38:14.613 INFO:tasks.rados.rados.0.smithi042.stdout:3711: finishing write tid 3 to smithi04268048-48 2024-09-20T19:38:14.615 INFO:tasks.rados.rados.0.smithi042.stdout:3711: finishing write tid 4 to smithi04268048-48 2024-09-20T19:38:14.615 INFO:tasks.rados.rados.0.smithi042.stdout:3711: finishing write tid 5 to smithi04268048-48 2024-09-20T19:38:14.615 INFO:tasks.rados.rados.0.smithi042.stdout:3711: oid 48 updating version 0 to 1335 2024-09-20T19:38:14.616 INFO:tasks.rados.rados.0.smithi042.stdout:3711: oid 48 updating version 1335 to 1337 2024-09-20T19:38:14.616 INFO:tasks.rados.rados.0.smithi042.stdout:3711: oid 48 version 1337 is already newer than 1334 2024-09-20T19:38:14.616 INFO:tasks.rados.rados.0.smithi042.stdout:3711: oid 48 version 1337 is already newer than 1336 2024-09-20T19:38:14.616 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1337 (ObjNum 1272 snap 408 seq_num 1272) dirty exists 2024-09-20T19:38:14.616 INFO:tasks.rados.rados.0.smithi042.stdout:3711: left oid 48 (ObjNum 1272 snap 408 seq_num 1272) 2024-09-20T19:38:14.631 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:14.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.748 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:14 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:14.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:14.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.875 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.647995+0000 osd.2 (osd.2) 309 : cluster [DBG] 7.10 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.650085+0000 osd.2 (osd.2) 310 : cluster [DBG] 7.10 scrub ok 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.967318+0000 osd.5 (osd.5) 295 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:12.969328+0000 osd.5 (osd.5) 296 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: audit 2024-09-20T19:38:13.580609+0000 mon.a (mon.0) 2613 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.006243+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:14 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.008303+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:38:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:14.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:14.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:15.024 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:15.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:15.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:15.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:15.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:15.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:15.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:15.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:15.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:15.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:15.542 INFO:tasks.rados.rados.0.smithi042.stdout:3711: done (1 left) 2024-09-20T19:38:15.542 INFO:tasks.rados.rados.0.smithi042.stdout:3712: done (0 left) 2024-09-20T19:38:15.542 INFO:tasks.rados.rados.0.smithi042.stdout:3713: snap_remove snap 408 2024-09-20T19:38:15.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:15.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:15.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:15 smithi162 bash[24557]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:15.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[22706]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.697072+0000 osd.2 (osd.2) 311 : cluster [DBG] 7.20 scrub starts 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:13.700006+0000 osd.2 (osd.2) 312 : cluster [DBG] 7.20 scrub ok 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.313891+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.1b scrub starts 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.900 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.315636+0000 osd.0 (osd.0) 108 : cluster [DBG] 7.1b scrub ok 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.471627+0000 mgr.y (mgr.24491) 759 : cluster [DBG] pgmap v1476: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.531760+0000 osd.1 (osd.1) 333 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.535696+0000 osd.1 (osd.1) 334 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.544487+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.825744+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: cluster 2024-09-20T19:38:14.828842+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:15 smithi042 bash[32075]: audit 2024-09-20T19:38:15.119496+0000 mon.a (mon.0) 2615 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:38:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:15.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:16.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:16.033 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:16.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:16.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:16.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:16.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:16.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:38:16.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:16.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:16.540 INFO:tasks.rados.rados.0.smithi042.stdout:3713: done (0 left) 2024-09-20T19:38:16.540 INFO:tasks.rados.rados.0.smithi042.stdout:3714: read oid 20 snap -1 2024-09-20T19:38:16.540 INFO:tasks.rados.rados.0.smithi042.stdout:3714: expect deleted 2024-09-20T19:38:16.540 INFO:tasks.rados.rados.0.smithi042.stdout:3715: delete oid 40 current snap is 409 2024-09-20T19:38:16.542 INFO:tasks.rados.rados.0.smithi042.stdout:3715: done (1 left) 2024-09-20T19:38:16.542 INFO:tasks.rados.rados.0.smithi042.stdout:3716: delete oid 31 current snap is 409 2024-09-20T19:38:16.546 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.550 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:16.555 INFO:tasks.rados.rados.0.smithi042.stdout:3714: done (1 left) 2024-09-20T19:38:16.555 INFO:tasks.rados.rados.0.smithi042.stdout:3716: done (0 left) 2024-09-20T19:38:16.555 INFO:tasks.rados.rados.0.smithi042.stdout:3717: delete oid 2 current snap is 409 2024-09-20T19:38:16.556 INFO:tasks.rados.rados.0.smithi042.stdout:3717: done (0 left) 2024-09-20T19:38:16.556 INFO:tasks.rados.rados.0.smithi042.stdout:3718: delete oid 23 current snap is 409 2024-09-20T19:38:16.558 INFO:tasks.rados.rados.0.smithi042.stdout:3718: done (0 left) 2024-09-20T19:38:16.558 INFO:tasks.rados.rados.0.smithi042.stdout:3719: read oid 26 snap -1 2024-09-20T19:38:16.558 INFO:tasks.rados.rados.0.smithi042.stdout:3719: expect deleted 2024-09-20T19:38:16.558 INFO:tasks.rados.rados.0.smithi042.stdout:3720: snap_create 2024-09-20T19:38:16.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:16 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.681 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:16.734 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:38:16.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.044514+0000 osd.5 (osd.5) 297 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.047473+0000 osd.5 (osd.5) 298 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.540476+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.766005+0000 osd.6 (osd.6) 220 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:15.768270+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.893633+0000 mon.a (mon.0) 2617 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.895001+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: audit 2024-09-20T19:38:15.900930+0000 mon.a (mon.0) 2619 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:16 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.538573+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2024-09-20T19:38:16.808 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:16.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:16.931 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:16.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:17.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:17.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:17.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:17.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:17.224 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:17.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:17.322 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:17.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:17.453 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:17.555 INFO:tasks.rados.rados.0.smithi042.stdout:3719: done (1 left) 2024-09-20T19:38:17.555 INFO:tasks.rados.rados.0.smithi042.stdout:3720: done (0 left) 2024-09-20T19:38:17.555 INFO:tasks.rados.rados.0.smithi042.stdout:3721: read oid 7 snap -1 2024-09-20T19:38:17.555 INFO:tasks.rados.rados.0.smithi042.stdout:3721: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:38:17.555 INFO:tasks.rados.rados.0.smithi042.stdout:3722: snap_remove snap 409 2024-09-20T19:38:17.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:17.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.562 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.563 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.580 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:17 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:17.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:17.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:17.841 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:17.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.472161+0000 mgr.y (mgr.24491) 760 : cluster [DBG] pgmap v1479: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 127 MiB data, 677 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 3.1 MiB/s wr, 5 op/s 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.785227+0000 osd.6 (osd.6) 222 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:16.787679+0000 osd.6 (osd.6) 223 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.124337+0000 osd.7 (osd.7) 221 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:38:17.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.957 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:17 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.128046+0000 osd.7 (osd.7) 222 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:38:17.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:17.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:18.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:18.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:18.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:18.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3722: done (1 left) 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3723: read oid 41 snap -1 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3723: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3724: rollback oid 27 current snap is 410 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3724: done (2 left) 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3725: copy_from oid 15 from oid 10 current snap is 410 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:3721: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:38:18.562 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:18.594 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:18.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:18.746 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2988684 2024-09-20T19:38:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:18.757 INFO:tasks.rados.rados.0.smithi042.stdout:3721: done (2 left) 2024-09-20T19:38:18.757 INFO:tasks.rados.rados.0.smithi042.stdout:3726: write (excl) oid 13 current snap is 410 2024-09-20T19:38:18.757 INFO:tasks.rados.rados.0.smithi042.stdout:3723: expect (ObjNum 899 snap 296 seq_num 899) 2024-09-20T19:38:18.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:18 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.847 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1811290 2024-09-20T19:38:18.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:18.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:18.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.059994+0000 osd.5 (osd.5) 299 : cluster [DBG] 1.a scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.061611+0000 osd.5 (osd.5) 300 : cluster [DBG] 1.a scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.551853+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.641691+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.643848+0000 osd.1 (osd.1) 336 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.752577+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.754711+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.788772+0000 osd.6 (osd.6) 224 : cluster [DBG] 7.23 scrub starts 2024-09-20T19:38:18.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:17.790493+0000 osd.6 (osd.6) 225 : cluster [DBG] 7.23 scrub ok 2024-09-20T19:38:18.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:18 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.554872+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2024-09-20T19:38:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3725: finishing copy_from racing read to smithi04268048-15 2024-09-20T19:38:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3726: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-20T19:38:18.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:18.980 INFO:tasks.rados.rados.0.smithi042.stdout:3726: writing smithi04268048-13 from 662539 to 1386463 tid 1 2024-09-20T19:38:18.999 INFO:tasks.rados.rados.0.smithi042.stdout:3726: writing smithi04268048-13 from 2093976 to 2811747 tid 2 2024-09-20T19:38:19.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:19.007 INFO:tasks.rados.rados.0.smithi042.stdout:3726: writing smithi04268048-13 from 3435397 to 3462539 tid 3 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3723: done (2 left) 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3727: copy_from oid 44 from oid 22 current snap is 410 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3725: finishing copy_from to smithi04268048-15 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 972 (ObjNum 1209 snap 391 seq_num 1209) dirty exists 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3726: finishing write tid 1 to smithi04268048-13 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3726: finishing write tid 2 to smithi04268048-13 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3725: done (2 left) 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3728: copy_from oid 17 from oid 41 current snap is 410 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3729: copy_from oid 7 from oid 9 current snap is 410 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3730: write (excl) oid 3 current snap is 410 2024-09-20T19:38:19.008 INFO:tasks.rados.rados.0.smithi042.stdout:3730: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-20T19:38:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:19.015 INFO:tasks.rados.rados.0.smithi042.stdout:3730: writing smithi04268048-3 from 679346 to 1194872 tid 1 2024-09-20T19:38:19.022 INFO:tasks.rados.rados.0.smithi042.stdout:3730: writing smithi04268048-3 from 1868811 to 2446593 tid 2 2024-09-20T19:38:19.032 INFO:tasks.rados.rados.0.smithi042.stdout:3730: writing smithi04268048-3 from 3179173 to 3479346 tid 3 2024-09-20T19:38:19.032 INFO:tasks.rados.rados.0.smithi042.stdout:3731: read oid 40 snap -1 2024-09-20T19:38:19.032 INFO:tasks.rados.rados.0.smithi042.stdout:3731: expect deleted 2024-09-20T19:38:19.032 INFO:tasks.rados.rados.0.smithi042.stdout:3732: delete oid 5 current snap is 410 2024-09-20T19:38:19.032 INFO:tasks.rados.rados.0.smithi042.stdout:3726: finishing write tid 3 to smithi04268048-13 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: finishing write tid 4 to smithi04268048-13 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: finishing write tid 5 to smithi04268048-13 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: oid 13 updating version 0 to 1179 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: oid 13 updating version 1179 to 1181 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: oid 13 version 1181 is already newer than 1180 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: oid 13 version 1181 is already newer than 1178 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 1181 (ObjNum 1277 snap 410 seq_num 1277) dirty exists 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3726: left oid 13 (ObjNum 1277 snap 410 seq_num 1277) 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3729: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3729: finishing copy_from to smithi04268048-7 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1175 (ObjNum 1246 snap 401 seq_num 1246) dirty exists 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3730: finishing write tid 1 to smithi04268048-3 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3728: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:38:19.033 INFO:tasks.rados.rados.0.smithi042.stdout:3728: finishing copy_from to smithi04268048-17 2024-09-20T19:38:19.034 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1160 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:38:19.034 INFO:tasks.rados.rados.0.smithi042.stdout:3727: finishing copy_from racing read to smithi04268048-44 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3726: done (6 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3728: done (5 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3729: done (4 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3731: done (3 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3732: done (2 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3733: rollback oid 50 current snap is 410 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3733: done (2 left) 2024-09-20T19:38:19.035 INFO:tasks.rados.rados.0.smithi042.stdout:3734: delete oid 49 current snap is 410 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: finishing write tid 2 to smithi04268048-3 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: finishing write tid 3 to smithi04268048-3 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: finishing write tid 4 to smithi04268048-3 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: finishing write tid 5 to smithi04268048-3 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: oid 3 updating version 0 to 1606 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: oid 3 updating version 1606 to 1609 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: oid 3 version 1609 is already newer than 1608 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:3730: oid 3 version 1609 is already newer than 1607 2024-09-20T19:38:19.036 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1609 (ObjNum 1278 snap 410 seq_num 1278) dirty exists 2024-09-20T19:38:19.037 INFO:tasks.rados.rados.0.smithi042.stdout:3730: left oid 3 (ObjNum 1278 snap 410 seq_num 1278) 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:3727: finishing copy_from to smithi04268048-44 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 44 v 1167 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:3727: done (2 left) 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:3730: done (1 left) 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:3734: done (0 left) 2024-09-20T19:38:19.041 INFO:tasks.rados.rados.0.smithi042.stdout:3735: delete oid 23 current snap is 410 2024-09-20T19:38:19.043 INFO:tasks.rados.rados.0.smithi042.stdout:3735: done (0 left) 2024-09-20T19:38:19.043 INFO:tasks.rados.rados.0.smithi042.stdout:3736: delete oid 15 current snap is 410 2024-09-20T19:38:19.044 INFO:tasks.rados.rados.0.smithi042.stdout:3736: done (0 left) 2024-09-20T19:38:19.044 INFO:tasks.rados.rados.0.smithi042.stdout:3737: delete oid 50 current snap is 410 2024-09-20T19:38:19.046 INFO:tasks.rados.rados.0.smithi042.stdout:3737: done (0 left) 2024-09-20T19:38:19.046 INFO:tasks.rados.rados.0.smithi042.stdout:3738: write (excl) oid 4 current snap is 410 2024-09-20T19:38:19.046 INFO:tasks.rados.rados.0.smithi042.stdout:3738: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-20T19:38:19.060 INFO:tasks.rados.rados.0.smithi042.stdout:3738: writing smithi04268048-4 from 780188 to 1347208 tid 1 2024-09-20T19:38:19.073 INFO:tasks.rados.rados.0.smithi042.stdout:3738: writing smithi04268048-4 from 2103350 to 2575786 tid 2 2024-09-20T19:38:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:3738: writing smithi04268048-4 from 3227358 to 3580188 tid 3 2024-09-20T19:38:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:3739: snap_create 2024-09-20T19:38:19.081 INFO:tasks.rados.rados.0.smithi042.stdout:3738: finishing write tid 1 to smithi04268048-4 2024-09-20T19:38:19.082 INFO:tasks.rados.rados.0.smithi042.stdout:3738: finishing write tid 2 to smithi04268048-4 2024-09-20T19:38:19.085 INFO:tasks.rados.rados.0.smithi042.stdout:3738: finishing write tid 3 to smithi04268048-4 2024-09-20T19:38:19.085 INFO:tasks.rados.rados.0.smithi042.stdout:3738: finishing write tid 4 to smithi04268048-4 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: finishing write tid 5 to smithi04268048-4 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: oid 4 updating version 0 to 1348 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: oid 4 version 1348 is already newer than 1346 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: oid 4 updating version 1348 to 1349 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: oid 4 version 1349 is already newer than 1347 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1349 (ObjNum 1284 snap 410 seq_num 1284) dirty exists 2024-09-20T19:38:19.086 INFO:tasks.rados.rados.0.smithi042.stdout:3738: left oid 4 (ObjNum 1284 snap 410 seq_num 1284) 2024-09-20T19:38:19.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:19.136 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:19.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:19.264 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:19.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:19.396 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:19.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:19.527 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:19.582 INFO:tasks.rados.rados.0.smithi042.stdout:3738: done (1 left) 2024-09-20T19:38:19.582 INFO:tasks.rados.rados.0.smithi042.stdout:3739: done (0 left) 2024-09-20T19:38:19.582 INFO:tasks.rados.rados.0.smithi042.stdout:3740: rollback oid 7 current snap is 411 2024-09-20T19:38:19.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 7 to 410 2024-09-20T19:38:19.583 INFO:tasks.rados.rados.0.smithi042.stdout:3741: read oid 34 snap -1 2024-09-20T19:38:19.583 INFO:tasks.rados.rados.0.smithi042.stdout:3741: expect (ObjNum 1254 snap 404 seq_num 1254) 2024-09-20T19:38:19.583 INFO:tasks.rados.rados.0.smithi042.stdout:3742: rollback oid 28 current snap is 411 2024-09-20T19:38:19.583 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 410 2024-09-20T19:38:19.583 INFO:tasks.rados.rados.0.smithi042.stdout:3743: snap_create 2024-09-20T19:38:19.591 INFO:tasks.rados.rados.0.smithi042.stdout:3740: finishing rollback tid 0 to smithi04268048-7 2024-09-20T19:38:19.591 INFO:tasks.rados.rados.0.smithi042.stdout:3740: finishing rollback tid 1 to smithi04268048-7 2024-09-20T19:38:19.591 INFO:tasks.rados.rados.0.smithi042.stdout:3740: finishing rollback tid 2 to smithi04268048-7 2024-09-20T19:38:19.591 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1271 (ObjNum 1246 snap 401 seq_num 1246) dirty exists 2024-09-20T19:38:19.592 INFO:tasks.rados.rados.0.smithi042.stdout:3741: expect (ObjNum 1254 snap 404 seq_num 1254) 2024-09-20T19:38:19.606 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1309872 2024-09-20T19:38:19.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[24557]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:19.657 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 716732 2024-09-20T19:38:19.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[32075]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.473136+0000 mgr.y (mgr.24491) 761 : cluster [DBG] pgmap v1482: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 95 MiB data, 628 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.675330+0000 osd.1 (osd.1) 337 : cluster [DBG] 2.a scrub starts 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:19 smithi042 bash[22706]: cluster 2024-09-20T19:38:18.676837+0000 osd.1 (osd.1) 338 : cluster [DBG] 2.a scrub ok 2024-09-20T19:38:19.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:19.765 INFO:tasks.rados.rados.0.smithi042.stdout:3742: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:38:19.766 INFO:tasks.rados.rados.0.smithi042.stdout:3742: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:38:19.766 INFO:tasks.rados.rados.0.smithi042.stdout:3742: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:38:19.766 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1447 (ObjNum 1166 snap 381 seq_num 1166) dirty exists 2024-09-20T19:38:19.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:19.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:20.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:20.146 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:20.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:20.281 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:20.301 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:20.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:20.405 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:20.426 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:20.529 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.548 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:20.579 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3740: done (3 left) 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3741: done (2 left) 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3742: done (1 left) 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3743: done (0 left) 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3744: write (excl) oid 50 current snap is 412 2024-09-20T19:38:20.580 INFO:tasks.rados.rados.0.smithi042.stdout:3744: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-20T19:38:20.589 INFO:tasks.rados.rados.0.smithi042.stdout:3744: writing smithi04268048-50 from 796995 to 1555617 tid 1 2024-09-20T19:38:20.597 INFO:tasks.rados.rados.0.smithi042.stdout:3744: writing smithi04268048-50 from 2161832 to 2894279 tid 2 2024-09-20T19:38:20.598 INFO:tasks.rados.rados.0.smithi042.stdout:3745: read oid 34 snap 410 2024-09-20T19:38:20.598 INFO:tasks.rados.rados.0.smithi042.stdout:3745: expect (ObjNum 1254 snap 404 seq_num 1254) 2024-09-20T19:38:20.598 INFO:tasks.rados.rados.0.smithi042.stdout:3746: write oid 24 current snap is 412 2024-09-20T19:38:20.598 INFO:tasks.rados.rados.0.smithi042.stdout:3746: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-20T19:38:20.602 INFO:tasks.rados.rados.0.smithi042.stdout:3746: writing smithi04268048-24 from 413802 to 847673 tid 1 2024-09-20T19:38:20.608 INFO:tasks.rados.rados.0.smithi042.stdout:3746: writing smithi04268048-24 from 1420314 to 1896419 tid 2 2024-09-20T19:38:20.617 INFO:tasks.rados.rados.0.smithi042.stdout:3746: writing smithi04268048-24 from 2482204 to 3273189 tid 3 2024-09-20T19:38:20.617 INFO:tasks.rados.rados.0.smithi042.stdout:3747: copy_from oid 16 from oid 21 current snap is 412 2024-09-20T19:38:20.617 INFO:tasks.rados.rados.0.smithi042.stdout:3745: expect (ObjNum 1254 snap 404 seq_num 1254) 2024-09-20T19:38:20.634 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1742385 2024-09-20T19:38:20.651 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:20.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.659 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.660 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[22706]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.661 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:20 smithi042 bash[32075]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.672 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:20.673 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:20.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3744: finishing write tid 1 to smithi04268048-50 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3745: done (3 left) 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3748: rollback oid 22 current snap is 412 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 22 to 411 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3744: finishing write tid 2 to smithi04268048-50 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3744: finishing write tid 3 to smithi04268048-50 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3744: finishing write tid 4 to smithi04268048-50 2024-09-20T19:38:20.797 INFO:tasks.rados.rados.0.smithi042.stdout:3744: oid 50 updating version 0 to 1195 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3744: oid 50 updating version 1195 to 1196 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3744: oid 50 updating version 1196 to 1197 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1197 (ObjNum 1285 snap 412 seq_num 1285) dirty exists 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3744: left oid 50 (ObjNum 1285 snap 412 seq_num 1285) 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: finishing write tid 1 to smithi04268048-24 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: finishing write tid 2 to smithi04268048-24 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: finishing write tid 3 to smithi04268048-24 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: finishing write tid 4 to smithi04268048-24 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: finishing write tid 5 to smithi04268048-24 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: oid 24 updating version 0 to 1452 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: oid 24 updating version 1452 to 1453 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: oid 24 updating version 1453 to 1455 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:3746: oid 24 version 1455 is already newer than 1454 2024-09-20T19:38:20.798 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1455 (ObjNum 1286 snap 412 seq_num 1286) dirty exists 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3746: left oid 24 (ObjNum 1286 snap 412 seq_num 1286) 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3747: finishing copy_from racing read to smithi04268048-16 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3747: finishing copy_from to smithi04268048-16 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1595 (ObjNum 1236 snap 401 seq_num 1236) dirty exists 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3744: done (3 left) 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3746: done (2 left) 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3747: done (1 left) 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3749: write oid 17 current snap is 412 2024-09-20T19:38:20.799 INFO:tasks.rados.rados.0.smithi042.stdout:3749: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-20T19:38:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:20.803 INFO:tasks.rados.rados.0.smithi042.stdout:3749: writing smithi04268048-17 from 430609 to 939729 tid 1 2024-09-20T19:38:20.809 INFO:tasks.rados.rados.0.smithi042.stdout:3749: writing smithi04268048-17 from 1478796 to 2214912 tid 2 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3749: writing smithi04268048-17 from 2909627 to 3511884 tid 3 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3750: copy_from oid 31 from oid 45 current snap is 412 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3748: finishing rollback tid 0 to smithi04268048-22 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3748: finishing rollback tid 1 to smithi04268048-22 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3748: finishing rollback tid 2 to smithi04268048-22 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 22 v 1170 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3749: finishing write tid 1 to smithi04268048-17 2024-09-20T19:38:20.817 INFO:tasks.rados.rados.0.smithi042.stdout:3749: finishing write tid 2 to smithi04268048-17 2024-09-20T19:38:20.818 INFO:tasks.rados.rados.0.smithi042.stdout:3748: done (2 left) 2024-09-20T19:38:20.818 INFO:tasks.rados.rados.0.smithi042.stdout:3751: delete oid 10 current snap is 412 2024-09-20T19:38:20.821 INFO:tasks.rados.rados.0.smithi042.stdout:3751: done (2 left) 2024-09-20T19:38:20.821 INFO:tasks.rados.rados.0.smithi042.stdout:3752: rollback oid 48 current snap is 412 2024-09-20T19:38:20.821 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 411 2024-09-20T19:38:20.821 INFO:tasks.rados.rados.0.smithi042.stdout:3753: snap_create 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: finishing write tid 3 to smithi04268048-17 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: finishing write tid 4 to smithi04268048-17 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: finishing write tid 5 to smithi04268048-17 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: oid 17 updating version 0 to 1182 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: oid 17 version 1182 is already newer than 1179 2024-09-20T19:38:20.822 INFO:tasks.rados.rados.0.smithi042.stdout:3749: oid 17 version 1182 is already newer than 1180 2024-09-20T19:38:20.823 INFO:tasks.rados.rados.0.smithi042.stdout:3749: oid 17 version 1182 is already newer than 1181 2024-09-20T19:38:20.823 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1182 (ObjNum 1287 snap 412 seq_num 1287) dirty exists 2024-09-20T19:38:20.823 INFO:tasks.rados.rados.0.smithi042.stdout:3749: left oid 17 (ObjNum 1287 snap 412 seq_num 1287) 2024-09-20T19:38:20.824 INFO:tasks.rados.rados.0.smithi042.stdout:3752: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:38:20.825 INFO:tasks.rados.rados.0.smithi042.stdout:3752: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:38:20.825 INFO:tasks.rados.rados.0.smithi042.stdout:3752: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:38:20.825 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1340 (ObjNum 1272 snap 408 seq_num 1272) dirty exists 2024-09-20T19:38:20.830 INFO:tasks.rados.rados.0.smithi042.stdout:3750: finishing copy_from racing read to smithi04268048-31 2024-09-20T19:38:20.833 INFO:tasks.rados.rados.0.smithi042.stdout:3750: finishing copy_from to smithi04268048-31 2024-09-20T19:38:20.833 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 1602 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:38:20.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.047502+0000 osd.5 (osd.5) 301 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.049086+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.340609+0000 osd.0 (osd.0) 109 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.343816+0000 osd.0 (osd.0) 110 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: cluster 2024-09-20T19:38:19.580612+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2024-09-20T19:38:20.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: audit 2024-09-20T19:38:20.553650+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.42:0/3224985674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:20 smithi162 bash[24557]: audit 2024-09-20T19:38:20.553927+0000 mon.a (mon.0) 2624 : audit [INF] from='client.26075 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:38:20.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:20.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:21.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:21.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:21.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:21.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:21.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.320 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:21.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:21.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:21.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:21.603 INFO:tasks.rados.rados.0.smithi042.stdout:3749: done (3 left) 2024-09-20T19:38:21.603 INFO:tasks.rados.rados.0.smithi042.stdout:3750: done (2 left) 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3752: done (1 left) 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3753: done (0 left) 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3754: read oid 39 snap -1 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3754: expect (ObjNum 1268 snap 408 seq_num 1268) 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3755: rollback oid 12 current snap is 413 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 12 to 410 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3756: rollback oid 33 current snap is 413 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 411 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3757: rollback oid 41 current snap is 413 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 41 to 411 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3758: read oid 45 snap -1 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3758: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:38:21.604 INFO:tasks.rados.rados.0.smithi042.stdout:3759: delete oid 38 current snap is 413 2024-09-20T19:38:21.610 INFO:tasks.rados.rados.0.smithi042.stdout:3758: expect (ObjNum 1192 snap 387 seq_num 1192) 2024-09-20T19:38:21.635 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 101484 2024-09-20T19:38:21.657 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3756: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 0 (ObjNum 817 snap 0 seq_num 173721922) dirty dne 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3755: finishing rollback tid 1 to smithi04268048-12 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3755: done (5 left) 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3756: done (4 left) 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3758: done (3 left) 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3759: done (2 left) 2024-09-20T19:38:21.681 INFO:tasks.rados.rados.0.smithi042.stdout:3760: rollback oid 50 current snap is 413 2024-09-20T19:38:21.682 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 410 2024-09-20T19:38:21.682 INFO:tasks.rados.rados.0.smithi042.stdout:3754: expect (ObjNum 1268 snap 408 seq_num 1268) 2024-09-20T19:38:21.682 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:21.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:21 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:21.719 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1386866 2024-09-20T19:38:21.745 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:21.780 INFO:tasks.rados.rados.0.smithi042.stdout:3757: finishing rollback tid 0 to smithi04268048-41 2024-09-20T19:38:21.780 INFO:tasks.rados.rados.0.smithi042.stdout:3757: finishing rollback tid 1 to smithi04268048-41 2024-09-20T19:38:21.780 INFO:tasks.rados.rados.0.smithi042.stdout:3757: finishing rollback tid 2 to smithi04268048-41 2024-09-20T19:38:21.780 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1266 (ObjNum 899 snap 296 seq_num 899) dirty exists 2024-09-20T19:38:21.780 INFO:tasks.rados.rados.0.smithi042.stdout:3760: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:3760: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1200 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:3754: done (2 left) 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:3757: done (1 left) 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:3760: done (0 left) 2024-09-20T19:38:21.781 INFO:tasks.rados.rados.0.smithi042.stdout:3761: snap_remove snap 410 2024-09-20T19:38:21.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.841 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.878 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.879 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: audit 2024-09-20T19:38:19.914242+0000 mgr.y (mgr.24491) 762 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.050853+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:38:21.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.052801+0000 osd.5 (osd.5) 304 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.473745+0000 mgr.y (mgr.24491) 763 : cluster [DBG] pgmap v1485: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 89 MiB data, 633 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: audit 2024-09-20T19:38:20.575215+0000 mon.a (mon.0) 2625 : audit [INF] from='client.26075 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.578676+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.713968+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:20.716029+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:38:21.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.882 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.098871+0000 osd.7 (osd.7) 223 : cluster [DBG] 7.17 scrub starts 2024-09-20T19:38:21.882 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.882 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:21 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.101241+0000 osd.7 (osd.7) 224 : cluster [DBG] 7.17 scrub ok 2024-09-20T19:38:21.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:21.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:21.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:22.073 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:22.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:22.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.363 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:22.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:22.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.619 INFO:tasks.rados.rados.0.smithi042.stdout:3761: done (0 left) 2024-09-20T19:38:22.620 INFO:tasks.rados.rados.0.smithi042.stdout:3762: write oid 28 current snap is 413 2024-09-20T19:38:22.620 INFO:tasks.rados.rados.0.smithi042.stdout:3762: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-20T19:38:22.622 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:22.641 INFO:tasks.rados.rados.0.smithi042.stdout:3762: writing smithi04268048-28 from 481030 to 1215897 tid 1 2024-09-20T19:38:22.660 INFO:tasks.rados.rados.0.smithi042.stdout:3762: writing smithi04268048-28 from 1937889 to 2421332 tid 2 2024-09-20T19:38:22.673 INFO:tasks.rados.rados.0.smithi042.stdout:3762: writing smithi04268048-28 from 2875543 to 3427969 tid 3 2024-09-20T19:38:22.673 INFO:tasks.rados.rados.0.smithi042.stdout:3763: copy_from oid 2 from oid 15 current snap is 413 2024-09-20T19:38:22.674 INFO:tasks.rados.rados.0.smithi042.stdout:3762: finishing write tid 1 to smithi04268048-28 2024-09-20T19:38:22.674 INFO:tasks.rados.rados.0.smithi042.stdout:3762: finishing write tid 2 to smithi04268048-28 2024-09-20T19:38:22.674 INFO:tasks.rados.rados.0.smithi042.stdout:3764: snap_remove snap 411 2024-09-20T19:38:22.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:22.751 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:22.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.866 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.866 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.866 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:22 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.881 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:22.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:22.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.060884+0000 osd.5 (osd.5) 305 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.062673+0000 osd.5 (osd.5) 306 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:21.601492+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2024-09-20T19:38:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.526656+0000 osd.2 (osd.2) 313 : cluster [DBG] 7.14 scrub starts 2024-09-20T19:38:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:22 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.531855+0000 osd.2 (osd.2) 314 : cluster [DBG] 7.14 scrub ok 2024-09-20T19:38:22.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:23.007 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:23.079 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T19:38:23.109 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:38:23.109 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:38:23.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:23.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:23.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:23.268 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:23.371 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:23.393 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:23.466 INFO:tasks.rados.rados.0.smithi042.stdout:3764: done (2 left) 2024-09-20T19:38:23.466 INFO:tasks.rados.rados.0.smithi042.stdout:3765: read oid 9 snap -1 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3765: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3766: rollback oid 46 current snap is 413 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 46 to 412 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3767: copy_from oid 40 from oid 47 current snap is 413 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3768: delete oid 8 current snap is 413 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3763: finishing copy_from to smithi04268048-2 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3763: got expected ENOENT (src dne) 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3763: finishing copy_from racing read to smithi04268048-2 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: finishing write tid 3 to smithi04268048-28 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: finishing write tid 4 to smithi04268048-28 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: finishing write tid 5 to smithi04268048-28 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: oid 28 updating version 0 to 1451 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: oid 28 version 1451 is already newer than 1448 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: oid 28 version 1451 is already newer than 1449 2024-09-20T19:38:23.467 INFO:tasks.rados.rados.0.smithi042.stdout:3762: oid 28 version 1451 is already newer than 1450 2024-09-20T19:38:23.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1451 (ObjNum 1290 snap 413 seq_num 1290) dirty exists 2024-09-20T19:38:23.468 INFO:tasks.rados.rados.0.smithi042.stdout:3762: left oid 28 (ObjNum 1290 snap 413 seq_num 1290) 2024-09-20T19:38:23.468 INFO:tasks.rados.rados.0.smithi042.stdout:3766: finishing rollback tid 1 to smithi04268048-46 2024-09-20T19:38:23.468 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:23.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:23.497 INFO:tasks.rados.rados.0.smithi042.stdout:3767: finishing copy_from to smithi04268048-40 2024-09-20T19:38:23.497 INFO:tasks.rados.rados.0.smithi042.stdout:3767: got expected ENOENT (src dne) 2024-09-20T19:38:23.497 INFO:tasks.rados.rados.0.smithi042.stdout:3767: finishing copy_from racing read to smithi04268048-40 2024-09-20T19:38:23.499 INFO:tasks.rados.rados.0.smithi042.stdout:3762: done (5 left) 2024-09-20T19:38:23.499 INFO:tasks.rados.rados.0.smithi042.stdout:3763: done (4 left) 2024-09-20T19:38:23.499 INFO:tasks.rados.rados.0.smithi042.stdout:3766: done (3 left) 2024-09-20T19:38:23.499 INFO:tasks.rados.rados.0.smithi042.stdout:3767: done (2 left) 2024-09-20T19:38:23.500 INFO:tasks.rados.rados.0.smithi042.stdout:3768: done (1 left) 2024-09-20T19:38:23.500 INFO:tasks.rados.rados.0.smithi042.stdout:3769: delete oid 18 current snap is 413 2024-09-20T19:38:23.506 INFO:tasks.rados.rados.0.smithi042.stdout:3765: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:23.507 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:23.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:23.550 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:23.562 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:23.621 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1463662 2024-09-20T19:38:23.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:23.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.635 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.636 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.637 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.638 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:23 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.648 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:23.666 INFO:tasks.rados.rados.0.smithi042.stdout:3765: done (1 left) 2024-09-20T19:38:23.666 INFO:tasks.rados.rados.0.smithi042.stdout:3769: done (0 left) 2024-09-20T19:38:23.666 INFO:tasks.rados.rados.0.smithi042.stdout:3770: write oid 11 current snap is 413 2024-09-20T19:38:23.667 INFO:tasks.rados.rados.0.smithi042.stdout:3770: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-09-20T19:38:23.674 INFO:tasks.rados.rados.0.smithi042.stdout:3770: writing smithi04268048-11 from 531451 to 1208418 tid 1 2024-09-20T19:38:23.683 INFO:tasks.rados.rados.0.smithi042.stdout:3770: writing smithi04268048-11 from 1713335 to 2460458 tid 2 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3770: writing smithi04268048-11 from 2957812 to 3460407 tid 3 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3771: copy_from oid 21 from oid 43 current snap is 413 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3770: finishing write tid 1 to smithi04268048-11 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3770: finishing write tid 2 to smithi04268048-11 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3772: copy_from oid 48 from oid 15 current snap is 413 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3773: write oid 6 current snap is 413 2024-09-20T19:38:23.690 INFO:tasks.rados.rados.0.smithi042.stdout:3773: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-09-20T19:38:23.698 INFO:tasks.rados.rados.0.smithi042.stdout:3773: writing smithi04268048-6 from 548258 to 1300474 tid 1 2024-09-20T19:38:23.704 INFO:tasks.rados.rados.0.smithi042.stdout:3773: writing smithi04268048-6 from 1771817 to 2262598 tid 2 2024-09-20T19:38:23.716 INFO:tasks.rados.rados.0.smithi042.stdout:3773: writing smithi04268048-6 from 2985235 to 3699102 tid 3 2024-09-20T19:38:23.716 INFO:tasks.rados.rados.0.smithi042.stdout:3774: delete oid 19 current snap is 413 2024-09-20T19:38:23.716 INFO:tasks.rados.rados.0.smithi042.stdout:3771: finishing copy_from to smithi04268048-21 2024-09-20T19:38:23.716 INFO:tasks.rados.rados.0.smithi042.stdout:3771: got expected ENOENT (src dne) 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3771: finishing copy_from racing read to smithi04268048-21 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3772: finishing copy_from to smithi04268048-48 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3772: got expected ENOENT (src dne) 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3772: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: finishing write tid 3 to smithi04268048-11 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: finishing write tid 4 to smithi04268048-11 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: finishing write tid 5 to smithi04268048-11 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: oid 11 updating version 0 to 1203 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: oid 11 version 1203 is already newer than 1202 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: oid 11 updating version 1203 to 1204 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: oid 11 updating version 1204 to 1205 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1205 (ObjNum 1293 snap 413 seq_num 1293) dirty exists 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3770: left oid 11 (ObjNum 1293 snap 413 seq_num 1293) 2024-09-20T19:38:23.717 INFO:tasks.rados.rados.0.smithi042.stdout:3773: finishing write tid 1 to smithi04268048-6 2024-09-20T19:38:23.718 INFO:tasks.rados.rados.0.smithi042.stdout:3773: finishing write tid 2 to smithi04268048-6 2024-09-20T19:38:23.720 INFO:tasks.rados.rados.0.smithi042.stdout:3770: done (4 left) 2024-09-20T19:38:23.720 INFO:tasks.rados.rados.0.smithi042.stdout:3771: done (3 left) 2024-09-20T19:38:23.720 INFO:tasks.rados.rados.0.smithi042.stdout:3772: done (2 left) 2024-09-20T19:38:23.720 INFO:tasks.rados.rados.0.smithi042.stdout:3774: done (1 left) 2024-09-20T19:38:23.720 INFO:tasks.rados.rados.0.smithi042.stdout:3775: snap_create 2024-09-20T19:38:23.722 INFO:tasks.rados.rados.0.smithi042.stdout:3773: finishing write tid 3 to smithi04268048-6 2024-09-20T19:38:23.722 INFO:tasks.rados.rados.0.smithi042.stdout:3773: finishing write tid 4 to smithi04268048-6 2024-09-20T19:38:23.723 INFO:tasks.rados.rados.0.smithi042.stdout:3773: finishing write tid 5 to smithi04268048-6 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:3773: oid 6 updating version 0 to 1605 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:3773: oid 6 updating version 1605 to 1606 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:3773: oid 6 updating version 1606 to 1607 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:3773: oid 6 version 1607 is already newer than 1604 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1607 (ObjNum 1294 snap 413 seq_num 1294) dirty exists 2024-09-20T19:38:23.724 INFO:tasks.rados.rados.0.smithi042.stdout:3773: left oid 6 (ObjNum 1294 snap 413 seq_num 1294) 2024-09-20T19:38:23.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.004349+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.006447+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.126107+0000 osd.5 (osd.5) 307 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.129032+0000 osd.5 (osd.5) 308 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.475371+0000 mgr.y (mgr.24491) 764 : cluster [DBG] pgmap v1488: 178 pgs: 178 active+clean; 99 MiB data, 653 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 74 op/s 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.616691+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.731338+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:22.733736+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:23 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.463164+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2024-09-20T19:38:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:23.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:23.906 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:23.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:24.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:24.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:24.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.293 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:24.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:24.423 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:24.526 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:24.626 INFO:tasks.rados.rados.0.smithi042.stdout:3773: done (1 left) 2024-09-20T19:38:24.626 INFO:tasks.rados.rados.0.smithi042.stdout:3775: done (0 left) 2024-09-20T19:38:24.626 INFO:tasks.rados.rados.0.smithi042.stdout:3776: rollback oid 34 current snap is 414 2024-09-20T19:38:24.626 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 34 to 412 2024-09-20T19:38:24.626 INFO:tasks.rados.rados.0.smithi042.stdout:3777: read oid 18 snap -1 2024-09-20T19:38:24.627 INFO:tasks.rados.rados.0.smithi042.stdout:3777: expect deleted 2024-09-20T19:38:24.627 INFO:tasks.rados.rados.0.smithi042.stdout:3778: rollback oid 48 current snap is 414 2024-09-20T19:38:24.627 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 413 2024-09-20T19:38:24.627 INFO:tasks.rados.rados.0.smithi042.stdout:3779: copy_from oid 5 from oid 45 current snap is 414 2024-09-20T19:38:24.627 INFO:tasks.rados.rados.0.smithi042.stdout:3780: snap_create 2024-09-20T19:38:24.634 INFO:tasks.rados.rados.0.smithi042.stdout:3778: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:38:24.634 INFO:tasks.rados.rados.0.smithi042.stdout:3778: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:38:24.635 INFO:tasks.rados.rados.0.smithi042.stdout:3776: finishing rollback tid 0 to smithi04268048-34 2024-09-20T19:38:24.636 INFO:tasks.rados.rados.0.smithi042.stdout:3776: finishing rollback tid 1 to smithi04268048-34 2024-09-20T19:38:24.636 INFO:tasks.rados.rados.0.smithi042.stdout:3778: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:38:24.636 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1343 (ObjNum 1272 snap 408 seq_num 1272) dirty exists 2024-09-20T19:38:24.636 INFO:tasks.rados.rados.0.smithi042.stdout:3776: finishing rollback tid 2 to smithi04268048-34 2024-09-20T19:38:24.636 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1253 (ObjNum 1254 snap 404 seq_num 1254) dirty exists 2024-09-20T19:38:24.652 INFO:tasks.rados.rados.0.smithi042.stdout:3779: finishing copy_from racing read to smithi04268048-5 2024-09-20T19:38:24.655 INFO:tasks.rados.rados.0.smithi042.stdout:3779: finishing copy_from to smithi04268048-5 2024-09-20T19:38:24.656 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1602 (ObjNum 1192 snap 387 seq_num 1192) dirty exists 2024-09-20T19:38:24.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:24.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:24.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:24.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:24.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:24.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:24.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:24 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:24.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:25.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.237877+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.239992+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.700823+0000 osd.6 (osd.6) 228 : cluster [DBG] 7.d scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:23.703689+0000 osd.6 (osd.6) 229 : cluster [DBG] 7.d scrub ok 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.551436+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.b scrub starts 2024-09-20T19:38:25.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:25.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:24 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.552994+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.b scrub ok 2024-09-20T19:38:25.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:25.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:25.203 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:25.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:25.333 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:25.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:25.464 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:25.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:25.596 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3776: done (4 left) 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3777: done (3 left) 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3778: done (2 left) 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3779: done (1 left) 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3780: done (0 left) 2024-09-20T19:38:25.639 INFO:tasks.rados.rados.0.smithi042.stdout:3781: delete oid 33 current snap is 415 2024-09-20T19:38:25.643 INFO:tasks.rados.rados.0.smithi042.stdout:3781: done (0 left) 2024-09-20T19:38:25.643 INFO:tasks.rados.rados.0.smithi042.stdout:3782: snap_create 2024-09-20T19:38:25.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:25 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:25.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:25.854 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.218027+0000 osd.0 (osd.0) 115 : cluster [DBG] 7.f scrub starts 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.221324+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.476348+0000 mgr.y (mgr.24491) 765 : cluster [DBG] pgmap v1491: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 31 op/s 2024-09-20T19:38:25.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:24.623461+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.141089+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.143078+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.533355+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:25 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.535698+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-20T19:38:25.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:25.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:26.088 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:26.109 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:26.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:26.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:26.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:26.352 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:26.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T19:38:26.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:26.479 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:38:26.479 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:38:26.488 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:26.498 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:38:26.498 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:38:26.515 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:38:26.516 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:38:26.533 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:38:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:38:26.554 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:38:26.554 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:38:26.582 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:38:26.582 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:38:26.591 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:26.601 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:38:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:38:26.612 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:26.620 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:38:26.620 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:38:26.671 INFO:tasks.rados.rados.0.smithi042.stdout:3782: done (0 left) 2024-09-20T19:38:26.671 INFO:tasks.rados.rados.0.smithi042.stdout:3783: snap_create 2024-09-20T19:38:26.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:26.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:26.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:26.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:26.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:26.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[22706]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:26.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:26.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:26.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:26 smithi042 bash[32075]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:26.974 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:27.001 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.202152+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:38:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.204536+0000 osd.0 (osd.0) 118 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:25.635785+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.109318+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.111290+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.310678+0000 osd.1 (osd.1) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.311545+0000 osd.1 (osd.1) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:26 smithi162 bash[24557]: audit 2024-09-20T19:38:26.460116+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.42:0/4237305361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:27.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:27.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:27.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:27.396 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:27.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:27.524 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:27.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:27.649 INFO:tasks.rados.rados.0.smithi042.stdout:3783: done (0 left) 2024-09-20T19:38:27.649 INFO:tasks.rados.rados.0.smithi042.stdout:3784: rollback oid 40 current snap is 417 2024-09-20T19:38:27.650 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 40 to 414 2024-09-20T19:38:27.650 INFO:tasks.rados.rados.0.smithi042.stdout:3785: write (excl) oid 41 current snap is 417 2024-09-20T19:38:27.650 INFO:tasks.rados.rados.0.smithi042.stdout:3785: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-20T19:38:27.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:27.666 INFO:tasks.rados.rados.0.smithi042.stdout:3785: writing smithi04268048-41 from 598679 to 1176642 tid 1 2024-09-20T19:38:27.685 INFO:tasks.rados.rados.0.smithi042.stdout:3785: writing smithi04268048-41 from 1947263 to 2585371 tid 2 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3785: writing smithi04268048-41 from 3351151 to 3798679 tid 3 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3786: read oid 1 snap -1 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3786: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3787: write oid 28 current snap is 417 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3784: finishing rollback tid 1 to smithi04268048-40 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 0 (ObjNum 49 snap 0 seq_num 1390888752) dirty dne 2024-09-20T19:38:27.701 INFO:tasks.rados.rados.0.smithi042.stdout:3785: finishing write tid 1 to smithi04268048-41 2024-09-20T19:38:27.702 INFO:tasks.rados.rados.0.smithi042.stdout:3785: finishing write tid 2 to smithi04268048-41 2024-09-20T19:38:27.702 INFO:tasks.rados.rados.0.smithi042.stdout:3787: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-09-20T19:38:27.718 INFO:tasks.rados.rados.0.smithi042.stdout:3787: writing smithi04268048-28 from 615486 to 1268698 tid 1 2024-09-20T19:38:27.730 INFO:tasks.rados.rados.0.smithi042.stdout:3787: writing smithi04268048-28 from 1889392 to 2387511 tid 2 2024-09-20T19:38:27.741 INFO:tasks.rados.rados.0.smithi042.stdout:3787: writing smithi04268048-28 from 2978574 to 3453882 tid 3 2024-09-20T19:38:27.741 INFO:tasks.rados.rados.0.smithi042.stdout:3784: done (3 left) 2024-09-20T19:38:27.741 INFO:tasks.rados.rados.0.smithi042.stdout:3788: delete oid 47 current snap is 417 2024-09-20T19:38:27.741 INFO:tasks.rados.rados.0.smithi042.stdout:3785: finishing write tid 3 to smithi04268048-41 2024-09-20T19:38:27.741 INFO:tasks.rados.rados.0.smithi042.stdout:3785: finishing write tid 4 to smithi04268048-41 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: finishing write tid 5 to smithi04268048-41 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: oid 41 updating version 0 to 1268 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: oid 41 version 1268 is already newer than 1267 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: oid 41 updating version 1268 to 1269 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: oid 41 updating version 1269 to 1270 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 41 v 1270 (ObjNum 1297 snap 417 seq_num 1297) dirty exists 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3785: left oid 41 (ObjNum 1297 snap 417 seq_num 1297) 2024-09-20T19:38:27.742 INFO:tasks.rados.rados.0.smithi042.stdout:3786: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:38:27.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:27 smithi162 bash[24557]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.829 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:27.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:27.895 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2134344 2024-09-20T19:38:27.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[32075]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.476767+0000 mgr.y (mgr.24491) 766 : cluster [DBG] pgmap v1494: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 105 MiB data, 666 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 5 op/s 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.663476+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.f scrub starts 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.665522+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.f scrub ok 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.670280+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.683245+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:27 smithi042 bash[22706]: cluster 2024-09-20T19:38:26.684253+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:38:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3787: finishing write tid 1 to smithi04268048-28 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3787: finishing write tid 2 to smithi04268048-28 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3785: done (3 left) 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3786: done (2 left) 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3788: done (1 left) 2024-09-20T19:38:27.914 INFO:tasks.rados.rados.0.smithi042.stdout:3789: delete oid 8 current snap is 417 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: finishing write tid 3 to smithi04268048-28 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: finishing write tid 4 to smithi04268048-28 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: finishing write tid 5 to smithi04268048-28 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: oid 28 updating version 0 to 1452 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: oid 28 updating version 1452 to 1453 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: oid 28 updating version 1453 to 1454 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: oid 28 updating version 1454 to 1455 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1455 (ObjNum 1298 snap 417 seq_num 1298) dirty exists 2024-09-20T19:38:27.915 INFO:tasks.rados.rados.0.smithi042.stdout:3787: left oid 28 (ObjNum 1298 snap 417 seq_num 1298) 2024-09-20T19:38:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:27.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:27.917 INFO:tasks.rados.rados.0.smithi042.stdout:3787: done (1 left) 2024-09-20T19:38:27.917 INFO:tasks.rados.rados.0.smithi042.stdout:3789: done (0 left) 2024-09-20T19:38:27.917 INFO:tasks.rados.rados.0.smithi042.stdout:3790: read oid 16 snap -1 2024-09-20T19:38:27.918 INFO:tasks.rados.rados.0.smithi042.stdout:3790: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:38:27.918 INFO:tasks.rados.rados.0.smithi042.stdout:3791: delete oid 23 current snap is 417 2024-09-20T19:38:27.920 INFO:tasks.rados.rados.0.smithi042.stdout:3791: done (1 left) 2024-09-20T19:38:27.920 INFO:tasks.rados.rados.0.smithi042.stdout:3792: snap_remove snap 414 2024-09-20T19:38:28.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:28.046 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:28.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:28.176 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:28.279 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:28.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:28.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:28.482 INFO:tasks.rados.rados.0.smithi042.stdout:3792: done (1 left) 2024-09-20T19:38:28.482 INFO:tasks.rados.rados.0.smithi042.stdout:3793: write oid 48 current snap is 417 2024-09-20T19:38:28.482 INFO:tasks.rados.rados.0.smithi042.stdout:3793: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-20T19:38:28.485 INFO:tasks.rados.rados.0.smithi042.stdout:3793: writing smithi04268048-48 from 682714 to 1353275 tid 1 2024-09-20T19:38:28.494 INFO:tasks.rados.rados.0.smithi042.stdout:3793: writing smithi04268048-48 from 2123320 to 2628777 tid 2 2024-09-20T19:38:28.499 INFO:tasks.rados.rados.0.smithi042.stdout:3793: writing smithi04268048-48 from 3088266 to 3725015 tid 3 2024-09-20T19:38:28.499 INFO:tasks.rados.rados.0.smithi042.stdout:3794: snap_remove snap 413 2024-09-20T19:38:28.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:28.563 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:28.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:28 smithi162 bash[24557]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:28.794 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:28.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:28.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[22706]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.642854+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.668959+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.671193+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.707374+0000 osd.4 (osd.4) 173 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:27.709242+0000 osd.4 (osd.4) 174 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.469430+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:28 smithi042 bash[32075]: audit 2024-09-20T19:38:28.609215+0000 mon.a (mon.0) 2635 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:28.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:29.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:29.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:29.303 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:29.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:29.471 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:38:29.481 INFO:tasks.rados.rados.0.smithi042.stdout:3794: done (2 left) 2024-09-20T19:38:29.481 INFO:tasks.rados.rados.0.smithi042.stdout:3795: delete oid 19 current snap is 417 2024-09-20T19:38:29.482 INFO:tasks.rados.rados.0.smithi042.stdout:3790: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:38:29.482 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:29.568 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:29.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:29.593 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:29.637 INFO:tasks.rados.rados.0.smithi042.stdout:3793: finishing write tid 1 to smithi04268048-48 2024-09-20T19:38:29.637 INFO:tasks.rados.rados.0.smithi042.stdout:3793: finishing write tid 2 to smithi04268048-48 2024-09-20T19:38:29.637 INFO:tasks.rados.rados.0.smithi042.stdout:3793: finishing write tid 3 to smithi04268048-48 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: finishing write tid 4 to smithi04268048-48 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: finishing write tid 5 to smithi04268048-48 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: oid 48 updating version 0 to 1346 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: oid 48 version 1346 is already newer than 1345 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: oid 48 version 1346 is already newer than 1344 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: oid 48 updating version 1346 to 1347 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1347 (ObjNum 1302 snap 417 seq_num 1302) dirty exists 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: left oid 48 (ObjNum 1302 snap 417 seq_num 1302) 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3790: done (2 left) 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3793: done (1 left) 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3795: done (0 left) 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3796: rollback oid 15 current snap is 417 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 412 2024-09-20T19:38:29.638 INFO:tasks.rados.rados.0.smithi042.stdout:3797: snap_remove snap 416 2024-09-20T19:38:29.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:29.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:29.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:29.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.835 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:29 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.477799+0000 mgr.y (mgr.24491) 767 : cluster [DBG] pgmap v1498: 178 pgs: 178 active+clean; 103 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-20T19:38:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.707877+0000 osd.1 (osd.1) 343 : cluster [DBG] 7.1e scrub starts 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.709177+0000 osd.1 (osd.1) 344 : cluster [DBG] 7.1e scrub ok 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.744245+0000 osd.4 (osd.4) 175 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:28.745663+0000 osd.4 (osd.4) 176 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: audit 2024-09-20T19:38:29.027778+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.462451+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.462488+0000 mon.a (mon.0) 2638 : cluster [INF] Cluster is now healthy 2024-09-20T19:38:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: audit 2024-09-20T19:38:29.469954+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? v1:172.21.15.42:0/3399804008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.474678+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T19:38:29.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:29.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:30.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:30.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:30.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:30.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:30.235 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:30.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:30.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:30.468 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:30.485 INFO:tasks.rados.rados.0.smithi042.stdout:3797: done (1 left) 2024-09-20T19:38:30.485 INFO:tasks.rados.rados.0.smithi042.stdout:3798: snap_remove snap 415 2024-09-20T19:38:30.502 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:30.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:30.634 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:30.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:38:30.736 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:30.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:30.749 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:30.750 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:30.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:30.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:30.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:30.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:30.751 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:30.752 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:30.753 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:30 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:30.763 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:30.866 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:30.891 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:30.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:30.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.014 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.723427+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.c scrub starts 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:29.725628+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.c scrub ok 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.009142+0000 osd.7 (osd.7) 229 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:38:31.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:31.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.011268+0000 osd.7 (osd.7) 230 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:38:31.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:31.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:30 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.480704+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T19:38:31.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:31.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:31.213 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:38:31.249 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:31.379 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:31.481 INFO:tasks.rados.rados.0.smithi042.stdout:3798: done (1 left) 2024-09-20T19:38:31.482 INFO:tasks.rados.rados.0.smithi042.stdout:3799: snap_create 2024-09-20T19:38:31.482 INFO:tasks.rados.rados.0.smithi042.stdout:3796: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:38:31.482 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 0 (ObjNum 817 snap 0 seq_num 173721922) dirty dne 2024-09-20T19:38:31.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:31.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:31.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.782 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.783 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.784 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.785 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:31 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:31.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:31.909 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.909 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: audit 2024-09-20T19:38:29.925621+0000 mgr.y (mgr.24491) 768 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:31.909 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.125389+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.13 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.128560+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.13 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.187161+0000 osd.5 (osd.5) 309 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.190014+0000 osd.5 (osd.5) 310 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.478578+0000 mgr.y (mgr.24491) 769 : cluster [DBG] pgmap v1501: 178 pgs: 178 active+clean; 104 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.628269+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.630307+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:38:31.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.684981+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.686610+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.740874+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:30.744132+0000 osd.4 (osd.4) 180 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.911 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:31 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.478860+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T19:38:31.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:31.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:32.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:32.056 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:32.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:32.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:32.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:32.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:32.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3796: done (1 left) 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3799: done (0 left) 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3800: read oid 46 snap 417 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3800: expect deleted 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3801: write (excl) oid 45 current snap is 418 2024-09-20T19:38:32.491 INFO:tasks.rados.rados.0.smithi042.stdout:3801: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-20T19:38:32.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:38:32.503 INFO:tasks.rados.rados.0.smithi042.stdout:3801: writing smithi04268048-45 from 716328 to 1137387 tid 1 2024-09-20T19:38:32.511 INFO:tasks.rados.rados.0.smithi042.stdout:3801: writing smithi04268048-45 from 1840284 to 2349410 tid 2 2024-09-20T19:38:32.522 INFO:tasks.rados.rados.0.smithi042.stdout:3801: writing smithi04268048-45 from 3143112 to 3802405 tid 3 2024-09-20T19:38:32.523 INFO:tasks.rados.rados.0.smithi042.stdout:3802: read oid 26 snap -1 2024-09-20T19:38:32.523 INFO:tasks.rados.rados.0.smithi042.stdout:3802: expect deleted 2024-09-20T19:38:32.523 INFO:tasks.rados.rados.0.smithi042.stdout:3803: snap_remove snap 412 2024-09-20T19:38:32.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:32.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:32.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:32.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.817 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:32 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:32.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.225540+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:38:32.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.227387+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.648595+0000 osd.6 (osd.6) 234 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.650761+0000 osd.6 (osd.6) 235 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.661944+0000 osd.1 (osd.1) 345 : cluster [DBG] 7.b scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.665793+0000 osd.1 (osd.1) 346 : cluster [DBG] 7.b scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.726042+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.728009+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.780196+0000 osd.4 (osd.4) 181 : cluster [DBG] 2.f scrub starts 2024-09-20T19:38:32.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:31.783208+0000 osd.4 (osd.4) 182 : cluster [DBG] 2.f scrub ok 2024-09-20T19:38:32.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:32 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.485677+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T19:38:32.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:32.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:32.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:33.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:33.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.190 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:38:33.191 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:38:33.191 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:38:33.191 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:38:33.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:33.308 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:33.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:33.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3803: done (3 left) 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3804: delete oid 41 current snap is 418 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: finishing write tid 1 to smithi04268048-45 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: finishing write tid 2 to smithi04268048-45 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: finishing write tid 3 to smithi04268048-45 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: finishing write tid 4 to smithi04268048-45 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: finishing write tid 5 to smithi04268048-45 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: oid 45 updating version 0 to 1627 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: oid 45 updating version 1627 to 1628 2024-09-20T19:38:33.481 INFO:tasks.rados.rados.0.smithi042.stdout:3801: oid 45 version 1628 is already newer than 1626 2024-09-20T19:38:33.482 INFO:tasks.rados.rados.0.smithi042.stdout:3801: oid 45 updating version 1628 to 1629 2024-09-20T19:38:33.482 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1629 (ObjNum 1304 snap 418 seq_num 1304) dirty exists 2024-09-20T19:38:33.482 INFO:tasks.rados.rados.0.smithi042.stdout:3801: left oid 45 (ObjNum 1304 snap 418 seq_num 1304) 2024-09-20T19:38:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T19:38:33.506 INFO:tasks.rados.rados.0.smithi042.stdout:3800: done (3 left) 2024-09-20T19:38:33.506 INFO:tasks.rados.rados.0.smithi042.stdout:3801: done (2 left) 2024-09-20T19:38:33.506 INFO:tasks.rados.rados.0.smithi042.stdout:3802: done (1 left) 2024-09-20T19:38:33.507 INFO:tasks.rados.rados.0.smithi042.stdout:3804: done (0 left) 2024-09-20T19:38:33.507 INFO:tasks.rados.rados.0.smithi042.stdout:3805: read oid 18 snap -1 2024-09-20T19:38:33.507 INFO:tasks.rados.rados.0.smithi042.stdout:3805: expect deleted 2024-09-20T19:38:33.510 INFO:tasks.rados.rados.0.smithi042.stdout:3806: write oid 31 current snap is 418 2024-09-20T19:38:33.511 INFO:tasks.rados.rados.0.smithi042.stdout:3806: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-09-20T19:38:33.522 INFO:tasks.rados.rados.0.smithi042.stdout:3806: writing smithi04268048-31 from 749942 to 1321499 tid 1 2024-09-20T19:38:33.536 INFO:tasks.rados.rados.0.smithi042.stdout:3806: writing smithi04268048-31 from 1840895 to 2637337 tid 2 2024-09-20T19:38:33.549 INFO:tasks.rados.rados.0.smithi042.stdout:3806: writing smithi04268048-31 from 3365252 to 3949942 tid 3 2024-09-20T19:38:33.549 INFO:tasks.rados.rados.0.smithi042.stdout:3807: snap_create 2024-09-20T19:38:33.549 INFO:tasks.rados.rados.0.smithi042.stdout:3806: finishing write tid 1 to smithi04268048-31 2024-09-20T19:38:33.549 INFO:tasks.rados.rados.0.smithi042.stdout:3806: finishing write tid 2 to smithi04268048-31 2024-09-20T19:38:33.553 INFO:tasks.rados.rados.0.smithi042.stdout:3806: finishing write tid 3 to smithi04268048-31 2024-09-20T19:38:33.553 INFO:tasks.rados.rados.0.smithi042.stdout:3806: finishing write tid 4 to smithi04268048-31 2024-09-20T19:38:33.553 INFO:tasks.rados.rados.0.smithi042.stdout:3806: finishing write tid 5 to smithi04268048-31 2024-09-20T19:38:33.553 INFO:tasks.rados.rados.0.smithi042.stdout:3806: oid 31 updating version 0 to 1608 2024-09-20T19:38:33.553 INFO:tasks.rados.rados.0.smithi042.stdout:3806: oid 31 updating version 1608 to 1609 2024-09-20T19:38:33.554 INFO:tasks.rados.rados.0.smithi042.stdout:3806: oid 31 updating version 1609 to 1611 2024-09-20T19:38:33.554 INFO:tasks.rados.rados.0.smithi042.stdout:3806: oid 31 version 1611 is already newer than 1610 2024-09-20T19:38:33.554 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 1611 (ObjNum 1306 snap 418 seq_num 1306) dirty exists 2024-09-20T19:38:33.554 INFO:tasks.rados.rados.0.smithi042.stdout:3806: left oid 31 (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:38:33.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:33.596 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:33.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.712 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:33 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:33.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.268041+0000 osd.5 (osd.5) 313 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.270364+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:32.480606+0000 mgr.y (mgr.24491) 770 : cluster [DBG] pgmap v1504: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 106 MiB data, 679 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 804 KiB/s wr, 8 op/s 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:32.917095+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.42:0/3312192960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:32.917697+0000 mon.a (mon.0) 2644 : audit [INF] from='client.26093 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: audit 2024-09-20T19:38:33.465102+0000 mon.a (mon.0) 2645 : audit [INF] from='client.26093 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:33 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.473254+0000 mon.a (mon.0) 2646 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T19:38:33.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:33.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:33.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:33.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T19:38:34.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.112 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:34.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:34.243 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:34.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.376 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:34.479 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:34.511 INFO:tasks.rados.rados.0.smithi042.stdout:3805: done (2 left) 2024-09-20T19:38:34.511 INFO:tasks.rados.rados.0.smithi042.stdout:3806: done (1 left) 2024-09-20T19:38:34.511 INFO:tasks.rados.rados.0.smithi042.stdout:3807: done (0 left) 2024-09-20T19:38:34.511 INFO:tasks.rados.rados.0.smithi042.stdout:3808: write (excl) oid 40 current snap is 419 2024-09-20T19:38:34.511 INFO:tasks.rados.rados.0.smithi042.stdout:3808: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-20T19:38:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:34.520 INFO:tasks.rados.rados.0.smithi042.stdout:3808: writing smithi04268048-40 from 766749 to 1413555 tid 1 2024-09-20T19:38:34.529 INFO:tasks.rados.rados.0.smithi042.stdout:3808: writing smithi04268048-40 from 1899377 to 2555830 tid 2 2024-09-20T19:38:34.536 INFO:tasks.rados.rados.0.smithi042.stdout:3808: writing smithi04268048-40 from 3109028 to 3718490 tid 3 2024-09-20T19:38:34.536 INFO:tasks.rados.rados.0.smithi042.stdout:3809: write (excl) oid 33 current snap is 419 2024-09-20T19:38:34.537 INFO:tasks.rados.rados.0.smithi042.stdout:3808: finishing write tid 1 to smithi04268048-40 2024-09-20T19:38:34.537 INFO:tasks.rados.rados.0.smithi042.stdout:3808: finishing write tid 2 to smithi04268048-40 2024-09-20T19:38:34.537 INFO:tasks.rados.rados.0.smithi042.stdout:3809: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-20T19:38:34.542 INFO:tasks.rados.rados.0.smithi042.stdout:3809: writing smithi04268048-33 from 783556 to 1221964 tid 1 2024-09-20T19:38:34.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T19:38:34.548 INFO:tasks.rados.rados.0.smithi042.stdout:3809: writing smithi04268048-33 from 1674212 to 2074323 tid 2 2024-09-20T19:38:34.554 INFO:tasks.rados.rados.0.smithi042.stdout:3809: writing smithi04268048-33 from 2736451 to 3157185 tid 3 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3809: writing smithi04268048-33 from 3882270 to 3983556 tid 4 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3810: snap_create 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: finishing write tid 3 to smithi04268048-40 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: finishing write tid 4 to smithi04268048-40 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: finishing write tid 5 to smithi04268048-40 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: oid 40 updating version 0 to 1069 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: oid 40 version 1069 is already newer than 1068 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: oid 40 updating version 1069 to 1070 2024-09-20T19:38:34.559 INFO:tasks.rados.rados.0.smithi042.stdout:3808: oid 40 updating version 1070 to 1071 2024-09-20T19:38:34.560 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 40 v 1071 (ObjNum 1307 snap 419 seq_num 1307) dirty exists 2024-09-20T19:38:34.560 INFO:tasks.rados.rados.0.smithi042.stdout:3808: left oid 40 (ObjNum 1307 snap 419 seq_num 1307) 2024-09-20T19:38:34.560 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 1 to smithi04268048-33 2024-09-20T19:38:34.560 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 2 to smithi04268048-33 2024-09-20T19:38:34.560 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 3 to smithi04268048-33 2024-09-20T19:38:34.563 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 4 to smithi04268048-33 2024-09-20T19:38:34.563 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 5 to smithi04268048-33 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: finishing write tid 6 to smithi04268048-33 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: oid 33 updating version 0 to 1331 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: oid 33 version 1331 is already newer than 1330 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: oid 33 version 1331 is already newer than 1329 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: oid 33 updating version 1331 to 1333 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: oid 33 version 1333 is already newer than 1332 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1333 (ObjNum 1308 snap 419 seq_num 1308) dirty exists 2024-09-20T19:38:34.564 INFO:tasks.rados.rados.0.smithi042.stdout:3809: left oid 33 (ObjNum 1308 snap 419 seq_num 1308) 2024-09-20T19:38:34.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:34.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:34.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:34.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:34.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:34.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.903 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:34.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:34.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:34 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:35.007 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:35.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:35.039 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.281014+0000 osd.5 (osd.5) 315 : cluster [DBG] 7.0 scrub starts 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.283799+0000 osd.5 (osd.5) 316 : cluster [DBG] 7.0 scrub ok 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:35.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.620316+0000 osd.1 (osd.1) 347 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.621776+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.635848+0000 osd.6 (osd.6) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:33.637351+0000 osd.6 (osd.6) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:35.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:34 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.508734+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T19:38:35.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:35.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:35.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:35.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:35.399 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:35.507 INFO:tasks.rados.rados.0.smithi042.stdout:3808: done (2 left) 2024-09-20T19:38:35.507 INFO:tasks.rados.rados.0.smithi042.stdout:3809: done (1 left) 2024-09-20T19:38:35.507 INFO:tasks.rados.rados.0.smithi042.stdout:3810: done (0 left) 2024-09-20T19:38:35.507 INFO:tasks.rados.rados.0.smithi042.stdout:3811: read oid 20 snap -1 2024-09-20T19:38:35.508 INFO:tasks.rados.rados.0.smithi042.stdout:3811: expect deleted 2024-09-20T19:38:35.508 INFO:tasks.rados.rados.0.smithi042.stdout:3812: snap_create 2024-09-20T19:38:35.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.559 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:35.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:35.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:35.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:35 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:35.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.246207+0000 osd.5 (osd.5) 317 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.247288+0000 osd.5 (osd.5) 318 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:34.481590+0000 mgr.y (mgr.24491) 771 : cluster [DBG] pgmap v1506: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 5.4 MiB/s rd, 848 KiB/s wr, 8 op/s 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.025694+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.028092+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:35 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.504505+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T19:38:35.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:35.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:36.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:36.088 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:36.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:36.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:36.334 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:36.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:36.482 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:36.518 INFO:tasks.rados.rados.0.smithi042.stdout:3811: done (1 left) 2024-09-20T19:38:36.518 INFO:tasks.rados.rados.0.smithi042.stdout:3812: done (0 left) 2024-09-20T19:38:36.518 INFO:tasks.rados.rados.0.smithi042.stdout:3813: snap_remove snap 420 2024-09-20T19:38:36.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:36.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:36.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:36.723 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:38:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:36 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:36.741 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:38:36.741 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:38:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:36.758 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:38:36.758 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:38:36.776 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:38:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:38:36.797 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:38:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:38:36.814 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:38:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:38:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:38:36.833 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:38:36.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:36.850 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:38:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:38:36.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.233497+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.9 scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.237156+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.250371+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.d scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.252415+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.d scrub ok 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.795031+0000 osd.4 (osd.4) 183 : cluster [DBG] 5.e scrub starts 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.863 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:35.798237+0000 osd.4 (osd.4) 184 : cluster [DBG] 5.e scrub ok 2024-09-20T19:38:36.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.864 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:36 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.509985+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2024-09-20T19:38:36.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:36.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:36.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:36.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:37.079 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:37.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:37.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:37.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:37.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:37.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:37.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:37.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:37.520 INFO:tasks.rados.rados.0.smithi042.stdout:3813: done (0 left) 2024-09-20T19:38:37.520 INFO:tasks.rados.rados.0.smithi042.stdout:3814: snap_remove snap 417 2024-09-20T19:38:37.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:37.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:37 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:37.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:37.906 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:37.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.482401+0000 mgr.y (mgr.24491) 772 : cluster [DBG] pgmap v1509: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 105 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 442 KiB/s wr, 1 op/s 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: audit 2024-09-20T19:38:36.704655+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.42:0/194124785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.776114+0000 osd.4 (osd.4) 185 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.778413+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:38:37.929 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.785646+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:38:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:36.786798+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:38:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:37.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:37 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.514323+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2024-09-20T19:38:38.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:38.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:38.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:38.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:38.399 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:38.428 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:38.482 INFO:tasks.rados.rados.0.smithi042.stdout:3814: done (0 left) 2024-09-20T19:38:38.483 INFO:tasks.rados.rados.0.smithi042.stdout:3815: write (excl) oid 3 current snap is 421 2024-09-20T19:38:38.483 INFO:tasks.rados.rados.0.smithi042.stdout:3815: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-20T19:38:38.483 INFO:tasks.rados.rados.0.smithi042.stdout:3815: writing smithi04268048-3 from 400363 to 914020 tid 1 2024-09-20T19:38:38.489 INFO:tasks.rados.rados.0.smithi042.stdout:3815: writing smithi04268048-3 from 1332694 to 1992816 tid 2 2024-09-20T19:38:38.489 INFO:tasks.rados.rados.0.smithi042.stdout:3816: snap_remove snap 418 2024-09-20T19:38:38.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:38.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:38.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:38.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:38.813 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:38.814 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:38:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:38.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:38.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:39.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:39.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:39.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:39.200 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:39.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:39.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:39.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:39.461 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:39.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.497 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.497 INFO:tasks.rados.rados.0.smithi042.stdout:3816: done (1 left) 2024-09-20T19:38:39.497 INFO:tasks.rados.rados.0.smithi042.stdout:3817: snap_remove snap 419 2024-09-20T19:38:39.564 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.692 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.762 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:39 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.604594+0000 osd.1 (osd.1) 349 : cluster [DBG] 7.1d scrub starts 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:37.609485+0000 osd.1 (osd.1) 350 : cluster [DBG] 7.1d scrub ok 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.471287+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2024-09-20T19:38:39.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.483002+0000 mgr.y (mgr.24491) 773 : cluster [DBG] pgmap v1513: 178 pgs: 178 active+clean; 101 MiB data, 661 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:39.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:39.852 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:39.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:39.983 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:39.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:40.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:40.113 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:40.216 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:40.245 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:40.291 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:38:40.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:40.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:40.376 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:40.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.501 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3817: done (1 left) 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3818: snap_create 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: finishing write tid 1 to smithi04268048-3 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: finishing write tid 2 to smithi04268048-3 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: finishing write tid 3 to smithi04268048-3 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: finishing write tid 4 to smithi04268048-3 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: oid 3 updating version 0 to 1612 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: oid 3 version 1612 is already newer than 1611 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: oid 3 version 1612 is already newer than 1610 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1612 (ObjNum 1309 snap 421 seq_num 1309) dirty exists 2024-09-20T19:38:40.507 INFO:tasks.rados.rados.0.smithi042.stdout:3815: left oid 3 (ObjNum 1309 snap 421 seq_num 1309) 2024-09-20T19:38:40.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.614 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:40 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:40.640 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:38:40.640 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:38:40.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.588804+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.c scrub starts 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:38.590493+0000 osd.1 (osd.1) 352 : cluster [DBG] 5.c scrub ok 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:39.494713+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.078637+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:40 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.080852+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:38:40.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:40.752 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:40.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:40.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:40.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:41.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:41.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:41.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:41.269 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:41.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:41.400 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:41.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:41.510 INFO:tasks.rados.rados.0.smithi042.stdout:3815: done (1 left) 2024-09-20T19:38:41.510 INFO:tasks.rados.rados.0.smithi042.stdout:3818: done (0 left) 2024-09-20T19:38:41.510 INFO:tasks.rados.rados.0.smithi042.stdout:3819: rollback oid 1 current snap is 422 2024-09-20T19:38:41.510 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 1 to 421 2024-09-20T19:38:41.510 INFO:tasks.rados.rados.0.smithi042.stdout:3820: rollback oid 13 current snap is 422 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 13 to 421 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:3821: read oid 26 snap -1 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:3821: expect deleted 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:3822: rollback oid 15 current snap is 422 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 421 2024-09-20T19:38:41.511 INFO:tasks.rados.rados.0.smithi042.stdout:3823: delete oid 5 current snap is 422 2024-09-20T19:38:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:38:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3820: finishing rollback tid 0 to smithi04268048-13 2024-09-20T19:38:41.527 INFO:tasks.rados.rados.0.smithi042.stdout:3820: finishing rollback tid 1 to smithi04268048-13 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3821: done (4 left) 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3823: done (3 left) 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3824: rollback oid 31 current snap is 422 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 31 to 421 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3825: read oid 37 snap -1 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3825: expect (ObjNum 1250 snap 402 seq_num 1250) 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3826: read oid 39 snap 421 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3826: expect (ObjNum 1268 snap 408 seq_num 1268) 2024-09-20T19:38:41.528 INFO:tasks.rados.rados.0.smithi042.stdout:3827: copy_from oid 46 from oid 12 current snap is 422 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:3820: finishing rollback tid 2 to smithi04268048-13 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 1184 (ObjNum 1277 snap 410 seq_num 1277) dirty exists 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:3820: done (6 left) 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:3828: rollback oid 33 current snap is 422 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 33 to 421 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:3829: write (excl) oid 49 current snap is 422 2024-09-20T19:38:41.529 INFO:tasks.rados.rados.0.smithi042.stdout:3829: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-20T19:38:41.535 INFO:tasks.rados.rados.0.smithi042.stdout:3829: writing smithi04268048-49 from 433977 to 1098132 tid 1 2024-09-20T19:38:41.535 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:41.542 INFO:tasks.rados.rados.0.smithi042.stdout:3829: writing smithi04268048-49 from 1733305 to 2033977 tid 2 2024-09-20T19:38:41.542 INFO:tasks.rados.rados.0.smithi042.stdout:3830: rollback oid 9 current snap is 422 2024-09-20T19:38:41.543 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 9 to 421 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3827: finishing copy_from to smithi04268048-46 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3827: got expected ENOENT (src dne) 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3827: finishing copy_from racing read to smithi04268048-46 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3822: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3819: finishing rollback tid 0 to smithi04268048-1 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3819: finishing rollback tid 1 to smithi04268048-1 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:3825: expect (ObjNum 1250 snap 402 seq_num 1250) 2024-09-20T19:38:41.544 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:41.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T19:38:41.630 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:41.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:41.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:41.715 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:41.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.778 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.779 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:41 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.789 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3819: finishing rollback tid 2 to smithi04268048-1 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1148 (ObjNum 1244 snap 401 seq_num 1244) dirty exists 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3828: finishing rollback tid 0 to smithi04268048-33 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3828: finishing rollback tid 1 to smithi04268048-33 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3828: finishing rollback tid 2 to smithi04268048-33 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1336 (ObjNum 1308 snap 419 seq_num 1308) dirty exists 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3824: finishing rollback tid 0 to smithi04268048-31 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:3826: expect (ObjNum 1268 snap 408 seq_num 1268) 2024-09-20T19:38:41.803 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:41.892 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3824: finishing rollback tid 1 to smithi04268048-31 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3829: finishing write tid 1 to smithi04268048-49 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3824: finishing rollback tid 2 to smithi04268048-31 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 31 v 1614 (ObjNum 1306 snap 418 seq_num 1306) dirty exists 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3829: finishing write tid 2 to smithi04268048-49 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3829: finishing write tid 3 to smithi04268048-49 2024-09-20T19:38:41.895 INFO:tasks.rados.rados.0.smithi042.stdout:3830: finishing rollback tid 0 to smithi04268048-9 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3830: finishing rollback tid 1 to smithi04268048-9 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3829: finishing write tid 4 to smithi04268048-49 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3829: oid 49 updating version 0 to 1636 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3829: oid 49 version 1636 is already newer than 1635 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3829: oid 49 version 1636 is already newer than 1634 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 49 v 1636 (ObjNum 1311 snap 422 seq_num 1311) dirty exists 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3829: left oid 49 (ObjNum 1311 snap 422 seq_num 1311) 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3830: finishing rollback tid 2 to smithi04268048-9 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1213 (ObjNum 1246 snap 401 seq_num 1246) dirty exists 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3819: done (8 left) 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3822: done (7 left) 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3824: done (6 left) 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3825: done (5 left) 2024-09-20T19:38:41.896 INFO:tasks.rados.rados.0.smithi042.stdout:3826: done (4 left) 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3827: done (3 left) 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3828: done (2 left) 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3829: done (1 left) 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3830: done (0 left) 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3831: read oid 41 snap -1 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3831: expect deleted 2024-09-20T19:38:41.897 INFO:tasks.rados.rados.0.smithi042.stdout:3832: delete oid 32 current snap is 422 2024-09-20T19:38:41.898 INFO:tasks.rados.rados.0.smithi042.stdout:3831: done (1 left) 2024-09-20T19:38:41.898 INFO:tasks.rados.rados.0.smithi042.stdout:3832: done (0 left) 2024-09-20T19:38:41.898 INFO:tasks.rados.rados.0.smithi042.stdout:3833: snap_remove snap 421 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.901 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.902 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:39.937121+0000 mgr.y (mgr.24491) 774 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.084567+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.086835+0000 osd.0 (osd.0) 124 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.483521+0000 mgr.y (mgr.24491) 775 : cluster [DBG] pgmap v1515: 178 pgs: 178 active+clean; 101 MiB data, 663 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.503981+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:40.711834+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.42:0/3050411348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.903 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: audit 2024-09-20T19:38:40.712422+0000 mon.a (mon.0) 2654 : audit [INF] from='client.26105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.802248+0000 osd.4 (osd.4) 187 : cluster [DBG] 7.16 scrub starts 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:40.806411+0000 osd.4 (osd.4) 188 : cluster [DBG] 7.16 scrub ok 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.034901+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:41 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.037026+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:38:41.914 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:41.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:42.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:42.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:42.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:42.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:42.277 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:42.407 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T19:38:42.527 INFO:tasks.rados.rados.0.smithi042.stdout:3833: done (0 left) 2024-09-20T19:38:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3834: write oid 32 current snap is 422 2024-09-20T19:38:42.528 INFO:tasks.rados.rados.0.smithi042.stdout:3834: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-20T19:38:42.532 INFO:tasks.rados.rados.0.smithi042.stdout:3834: writing smithi04268048-32 from 467591 to 882244 tid 1 2024-09-20T19:38:42.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3834: writing smithi04268048-32 from 1450269 to 2067591 tid 2 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3835: copy_from oid 38 from oid 28 current snap is 422 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3836: copy_from oid 48 from oid 3 current snap is 422 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3837: rollback oid 26 current snap is 422 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3837: done (3 left) 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3838: write oid 7 current snap is 422 2024-09-20T19:38:42.540 INFO:tasks.rados.rados.0.smithi042.stdout:3838: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-20T19:38:42.545 INFO:tasks.rados.rados.0.smithi042.stdout:3838: writing smithi04268048-7 from 484398 to 974300 tid 1 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.548 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[32075]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.550 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:42 smithi042 bash[22706]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.551 INFO:tasks.rados.rados.0.smithi042.stdout:3838: writing smithi04268048-7 from 1392398 to 1803516 tid 2 2024-09-20T19:38:42.551 INFO:tasks.rados.rados.0.smithi042.stdout:3839: write oid 30 current snap is 422 2024-09-20T19:38:42.552 INFO:tasks.rados.rados.0.smithi042.stdout:3834: finishing write tid 1 to smithi04268048-32 2024-09-20T19:38:42.552 INFO:tasks.rados.rados.0.smithi042.stdout:3839: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-20T19:38:42.558 INFO:tasks.rados.rados.0.smithi042.stdout:3839: writing smithi04268048-30 from 501205 to 1066356 tid 1 2024-09-20T19:38:42.564 INFO:tasks.rados.rados.0.smithi042.stdout:3839: writing smithi04268048-30 from 1850880 to 2101205 tid 2 2024-09-20T19:38:42.564 INFO:tasks.rados.rados.0.smithi042.stdout:3840: read oid 9 snap -1 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3840: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: finishing write tid 2 to smithi04268048-32 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: finishing write tid 3 to smithi04268048-32 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: finishing write tid 4 to smithi04268048-32 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: oid 32 updating version 0 to 1638 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: oid 32 updating version 1638 to 1640 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: oid 32 version 1640 is already newer than 1639 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1640 (ObjNum 1313 snap 422 seq_num 1313) dirty exists 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: left oid 32 (ObjNum 1313 snap 422 seq_num 1313) 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3834: done (5 left) 2024-09-20T19:38:42.565 INFO:tasks.rados.rados.0.smithi042.stdout:3841: snap_create 2024-09-20T19:38:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:42.575 INFO:tasks.rados.rados.0.smithi042.stdout:3836: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:38:42.579 INFO:tasks.rados.rados.0.smithi042.stdout:3835: finishing copy_from racing read to smithi04268048-38 2024-09-20T19:38:42.579 INFO:tasks.rados.rados.0.smithi042.stdout:3840: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:42.626 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 605398 2024-09-20T19:38:42.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3836: finishing copy_from to smithi04268048-48 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1612 (ObjNum 1309 snap 421 seq_num 1309) dirty exists 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: finishing write tid 1 to smithi04268048-30 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: finishing write tid 2 to smithi04268048-30 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: finishing write tid 3 to smithi04268048-30 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3838: finishing write tid 1 to smithi04268048-7 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: finishing write tid 4 to smithi04268048-30 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: oid 30 updating version 0 to 1614 2024-09-20T19:38:42.721 INFO:tasks.rados.rados.0.smithi042.stdout:3839: oid 30 updating version 1614 to 1616 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3839: oid 30 version 1616 is already newer than 1615 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1616 (ObjNum 1315 snap 422 seq_num 1315) dirty exists 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3839: left oid 30 (ObjNum 1315 snap 422 seq_num 1315) 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3835: finishing copy_from to smithi04268048-38 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 38 v 1455 (ObjNum 1298 snap 417 seq_num 1298) dirty exists 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: finishing write tid 2 to smithi04268048-7 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: finishing write tid 3 to smithi04268048-7 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: finishing write tid 4 to smithi04268048-7 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: oid 7 updating version 0 to 1274 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: oid 7 version 1274 is already newer than 1273 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: oid 7 version 1274 is already newer than 1272 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1274 (ObjNum 1314 snap 422 seq_num 1314) dirty exists 2024-09-20T19:38:42.722 INFO:tasks.rados.rados.0.smithi042.stdout:3838: left oid 7 (ObjNum 1314 snap 422 seq_num 1314) 2024-09-20T19:38:42.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:42.804 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.050631+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.052941+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.499420+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: audit 2024-09-20T19:38:41.501432+0000 mon.a (mon.0) 2656 : audit [INF] from='client.26105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: cluster 2024-09-20T19:38:41.507690+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.805 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:42 smithi162 bash[24557]: audit 2024-09-20T19:38:41.596374+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:38:42.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:42.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:42.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:42.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:43.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:43.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:43.209 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:43.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:43.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T19:38:43.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:43.476 INFO:tasks.rados.rados.0.smithi042.stdout:3835: done (5 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3836: done (4 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3838: done (3 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3839: done (2 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3840: done (1 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3841: done (0 left) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3842: read oid 13 snap -1 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3842: expect (ObjNum 1277 snap 410 seq_num 1277) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3843: read oid 28 snap -1 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3843: expect (ObjNum 1298 snap 417 seq_num 1298) 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3844: read oid 20 snap -1 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3844: expect deleted 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3845: read oid 35 snap 422 2024-09-20T19:38:43.477 INFO:tasks.rados.rados.0.smithi042.stdout:3845: expect deleted 2024-09-20T19:38:43.478 INFO:tasks.rados.rados.0.smithi042.stdout:3846: snap_create 2024-09-20T19:38:43.495 INFO:tasks.rados.rados.0.smithi042.stdout:3842: expect (ObjNum 1277 snap 410 seq_num 1277) 2024-09-20T19:38:43.495 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 567827 2024-09-20T19:38:43.542 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:43.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:43.624 INFO:tasks.rados.rados.0.smithi042.stdout:3843: expect (ObjNum 1298 snap 417 seq_num 1298) 2024-09-20T19:38:43.624 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:43.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:43.707 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:43.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.709 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:43 smithi162 bash[24557]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:43.787 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:43.811 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:43.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.821 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[22706]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:42.484391+0000 mgr.y (mgr.24491) 776 : cluster [DBG] pgmap v1518: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 96 MiB data, 658 MiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 520 KiB/s wr, 8 op/s 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: audit 2024-09-20T19:38:42.520255+0000 mon.a (mon.0) 2659 : audit [INF] from='client.? v1:172.21.15.42:0/1121547802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.822 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:42.525546+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2024-09-20T19:38:43.823 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.823 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:43 smithi042 bash[32075]: cluster 2024-09-20T19:38:43.474485+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2024-09-20T19:38:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:43.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:43.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:43.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:44.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:44.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:44.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:44.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:44.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:44.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:44.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3842: done (4 left) 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3843: done (3 left) 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3844: done (2 left) 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3845: done (1 left) 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3846: done (0 left) 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3847: rollback oid 15 current snap is 424 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 15 to 423 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3848: write (excl) oid 33 current snap is 424 2024-09-20T19:38:44.485 INFO:tasks.rados.rados.0.smithi042.stdout:3848: seq_num 1316 ranges {518012=756753,2025715=92297} 2024-09-20T19:38:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:44.496 INFO:tasks.rados.rados.0.smithi042.stdout:3848: writing smithi04268048-33 from 518012 to 1274765 tid 1 2024-09-20T19:38:44.498 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:38:44.503 INFO:tasks.rados.rados.0.smithi042.stdout:3848: writing smithi04268048-33 from 2025715 to 2118012 tid 2 2024-09-20T19:38:44.503 INFO:tasks.rados.rados.0.smithi042.stdout:3849: delete oid 2 current snap is 424 2024-09-20T19:38:44.504 INFO:tasks.rados.rados.0.smithi042.stdout:3847: finishing rollback tid 1 to smithi04268048-15 2024-09-20T19:38:44.504 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 0 (ObjNum 241 snap 0 seq_num 1415635712) dirty dne 2024-09-20T19:38:44.504 INFO:tasks.rados.rados.0.smithi042.stdout:3848: finishing write tid 1 to smithi04268048-33 2024-09-20T19:38:44.506 INFO:tasks.rados.rados.0.smithi042.stdout:3848: finishing write tid 2 to smithi04268048-33 2024-09-20T19:38:44.506 INFO:tasks.rados.rados.0.smithi042.stdout:3848: finishing write tid 3 to smithi04268048-33 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: finishing write tid 4 to smithi04268048-33 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: oid 33 updating version 0 to 1339 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: oid 33 version 1339 is already newer than 1337 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: oid 33 version 1339 is already newer than 1338 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 33 v 1339 (ObjNum 1316 snap 424 seq_num 1316) dirty exists 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: left oid 33 (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3847: done (2 left) 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3848: done (1 left) 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3849: done (0 left) 2024-09-20T19:38:44.507 INFO:tasks.rados.rados.0.smithi042.stdout:3850: write oid 15 current snap is 424 2024-09-20T19:38:44.508 INFO:tasks.rados.rados.0.smithi042.stdout:3850: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-20T19:38:44.515 INFO:tasks.rados.rados.0.smithi042.stdout:3850: writing smithi04268048-15 from 551626 to 1058877 tid 1 2024-09-20T19:38:44.523 INFO:tasks.rados.rados.0.smithi042.stdout:3850: writing smithi04268048-15 from 1626326 to 2044782 tid 2 2024-09-20T19:38:44.524 INFO:tasks.rados.rados.0.smithi042.stdout:3851: read oid 44 snap -1 2024-09-20T19:38:44.524 INFO:tasks.rados.rados.0.smithi042.stdout:3851: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:44.524 INFO:tasks.rados.rados.0.smithi042.stdout:3850: finishing write tid 1 to smithi04268048-15 2024-09-20T19:38:44.524 INFO:tasks.rados.rados.0.smithi042.stdout:3852: write (excl) oid 48 current snap is 424 2024-09-20T19:38:44.524 INFO:tasks.rados.rados.0.smithi042.stdout:3852: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-09-20T19:38:44.537 INFO:tasks.rados.rados.0.smithi042.stdout:3852: writing smithi04268048-48 from 568433 to 1150933 tid 1 2024-09-20T19:38:44.547 INFO:tasks.rados.rados.0.smithi042.stdout:3852: writing smithi04268048-48 from 1684808 to 2168433 tid 2 2024-09-20T19:38:44.547 INFO:tasks.rados.rados.0.smithi042.stdout:3853: rollback oid 43 current snap is 424 2024-09-20T19:38:44.547 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 423 2024-09-20T19:38:44.547 INFO:tasks.rados.rados.0.smithi042.stdout:3850: finishing write tid 2 to smithi04268048-15 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: finishing write tid 3 to smithi04268048-15 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: finishing write tid 4 to smithi04268048-15 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: oid 15 updating version 0 to 1155 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: oid 15 updating version 1155 to 1156 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: oid 15 version 1156 is already newer than 1154 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1156 (ObjNum 1318 snap 424 seq_num 1318) dirty exists 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3850: left oid 15 (ObjNum 1318 snap 424 seq_num 1318) 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:3851: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:44.548 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:44.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:44.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:44.625 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:44.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[22706]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[22706]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[22706]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[22706]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[32075]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[32075]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[32075]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:44 smithi042 bash[32075]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:44.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:44 smithi162 bash[24557]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:44 smithi162 bash[24557]: audit 2024-09-20T19:38:43.638388+0000 mon.a (mon.0) 2662 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:44.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:44 smithi162 bash[24557]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:44 smithi162 bash[24557]: cluster 2024-09-20T19:38:44.481239+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2024-09-20T19:38:44.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:44.761 INFO:tasks.rados.rados.0.smithi042.stdout:3852: finishing write tid 1 to smithi04268048-48 2024-09-20T19:38:44.761 INFO:tasks.rados.rados.0.smithi042.stdout:3853: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 0 (ObjNum 289 snap 0 seq_num 173955665) dirty dne 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: finishing write tid 2 to smithi04268048-48 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: finishing write tid 3 to smithi04268048-48 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: finishing write tid 4 to smithi04268048-48 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: oid 48 updating version 0 to 1615 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: oid 48 version 1615 is already newer than 1613 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: oid 48 version 1615 is already newer than 1614 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1615 (ObjNum 1319 snap 424 seq_num 1319) dirty exists 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: left oid 48 (ObjNum 1319 snap 424 seq_num 1319) 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3850: done (3 left) 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3851: done (2 left) 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3852: done (1 left) 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3853: done (0 left) 2024-09-20T19:38:44.762 INFO:tasks.rados.rados.0.smithi042.stdout:3854: snap_create 2024-09-20T19:38:44.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:44.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:44.968 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:44.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:45.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:45.230 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:45.361 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:45.390 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:45.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:45.518 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:45.532 INFO:tasks.rados.rados.0.smithi042.stdout:3854: done (0 left) 2024-09-20T19:38:45.532 INFO:tasks.rados.rados.0.smithi042.stdout:3855: write (excl) oid 17 current snap is 425 2024-09-20T19:38:45.532 INFO:tasks.rados.rados.0.smithi042.stdout:3855: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-20T19:38:45.539 INFO:tasks.rados.rados.0.smithi042.stdout:3855: writing smithi04268048-17 from 585240 to 1242989 tid 1 2024-09-20T19:38:45.545 INFO:tasks.rados.rados.0.smithi042.stdout:3855: writing smithi04268048-17 from 1743290 to 2165415 tid 2 2024-09-20T19:38:45.545 INFO:tasks.rados.rados.0.smithi042.stdout:3856: copy_from oid 3 from oid 32 current snap is 425 2024-09-20T19:38:45.545 INFO:tasks.rados.rados.0.smithi042.stdout:3857: snap_remove snap 423 2024-09-20T19:38:45.622 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:45.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:45.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:45 smithi162 bash[24557]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:45.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:45 smithi162 bash[24557]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:45.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:45.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:45.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:45 smithi042 bash[32075]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:45.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:45 smithi042 bash[32075]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:45 smithi042 bash[22706]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:45.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:45 smithi042 bash[22706]: cluster 2024-09-20T19:38:44.485275+0000 mgr.y (mgr.24491) 777 : cluster [DBG] pgmap v1522: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:38:46.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:46.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:46.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:46.168 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:46.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:46.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:46.373 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:38:46.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:46.412 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:46.429 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:46.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3857: done (2 left) 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3858: snap_create 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3855: finishing write tid 1 to smithi04268048-17 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3855: finishing write tid 2 to smithi04268048-17 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3855: finishing write tid 3 to smithi04268048-17 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3855: finishing write tid 4 to smithi04268048-17 2024-09-20T19:38:46.551 INFO:tasks.rados.rados.0.smithi042.stdout:3855: oid 17 updating version 0 to 1183 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:3855: oid 17 updating version 1183 to 1184 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:3855: oid 17 updating version 1184 to 1185 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1185 (ObjNum 1320 snap 425 seq_num 1320) dirty exists 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:3855: left oid 17 (ObjNum 1320 snap 425 seq_num 1320) 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:3856: finishing copy_from racing read to smithi04268048-3 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:3856: finishing copy_from to smithi04268048-3 2024-09-20T19:38:46.552 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 3 v 1640 (ObjNum 1313 snap 422 seq_num 1313) dirty exists 2024-09-20T19:38:46.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:46.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:46.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:46 smithi042 bash[32075]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:46 smithi042 bash[32075]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:46 smithi042 bash[22706]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:46 smithi042 bash[22706]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:46.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:46.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:46 smithi162 bash[24557]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:46 smithi162 bash[24557]: cluster 2024-09-20T19:38:45.529966+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2024-09-20T19:38:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:46.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:46.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:47.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:47.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:47.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:47.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:47.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:47.342 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:47.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:47.475 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:47.558 INFO:tasks.rados.rados.0.smithi042.stdout:3855: done (2 left) 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:3856: done (1 left) 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:3858: done (0 left) 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:3859: rollback oid 30 current snap is 426 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 30 to 425 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:3860: write (excl) oid 23 current snap is 426 2024-09-20T19:38:47.559 INFO:tasks.rados.rados.0.smithi042.stdout:3860: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-20T19:38:47.574 INFO:tasks.rados.rados.0.smithi042.stdout:3860: writing smithi04268048-23 from 602047 to 1335045 tid 1 2024-09-20T19:38:47.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:47.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[22706]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[22706]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[22706]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.579 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[22706]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[32075]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[32075]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[32075]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:47 smithi042 bash[32075]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.581 INFO:tasks.rados.rados.0.smithi042.stdout:3860: writing smithi04268048-23 from 1801772 to 2202047 tid 2 2024-09-20T19:38:47.581 INFO:tasks.rados.rados.0.smithi042.stdout:3861: snap_remove snap 422 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:47.708 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:47.730 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:38:47.730 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:38:47.748 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:38:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:38:47.765 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:38:47.765 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:38:47.782 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:38:47.782 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:38:47.802 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:38:47.803 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:38:47.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:47 smithi162 bash[24557]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:47 smithi162 bash[24557]: cluster 2024-09-20T19:38:46.485874+0000 mgr.y (mgr.24491) 778 : cluster [DBG] pgmap v1524: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 94 MiB data, 651 MiB used, 625 GiB / 626 GiB avail; 9.8 MiB/s rd, 708 KiB/s wr, 7 op/s 2024-09-20T19:38:47.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:47 smithi162 bash[24557]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:47 smithi162 bash[24557]: cluster 2024-09-20T19:38:46.549125+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2024-09-20T19:38:47.824 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:38:47.824 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:38:47.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:47.846 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:38:47.847 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:38:47.857 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:47.865 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:38:47.865 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:38:47.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:47.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:47.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:48.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:48.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:48.231 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2310 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.500235374, 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078455948, 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2310 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730441+0000", 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730444+0000", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6960000000000001e-06 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730468+0000", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4110000000000001e-05 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730484+0000", 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6342e-05 2024-09-20T19:38:48.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730499+0000", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5379999999999998e-05 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.734975+0000", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044754510000000001 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.734992+0000", 2024-09-20T19:38:48.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7053000000000001e-05 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782595+0000", 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047602575 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782717+0000", 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012212200000000001 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808530+0000", 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025813250999999999 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808636+0000", 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010632300000000001 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808897+0000", 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000260645 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.237 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2321 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.50015742699998, 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078792609, 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.238 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2321 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730519+0000", 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730521+0000", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.623e-06 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730528+0000", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.08e-06 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730545+0000", 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7880000000000002e-05 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730561+0000", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5353e-05 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735052+0000", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044915340000000002 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735069+0000", 2024-09-20T19:38:48.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6889e-05 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782803+0000", 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0477337369999999 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782886+0000", 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2651999999999994e-05 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.808959+0000", 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026073354999999999 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809062+0000", 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010259000000000001 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809312+0000", 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024991599999999998 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2331 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.50009878200001, 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0791174560000001, 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.244 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2331 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730578+0000", 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730579+0000", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.367e-06 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730586+0000", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6660000000000002e-06 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730600+0000", 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4643e-05 2024-09-20T19:38:48.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730624+0000", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3506000000000002e-05 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735238+0000", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046146870000000001 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735257+0000", 2024-09-20T19:38:48.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9015999999999999e-05 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.782943+0000", 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047685296 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783055+0000", 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112483 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809397+0000", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026341956 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809500+0000", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102439 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809695+0000", 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019539700000000001 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2341 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.50003695400005, 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0794075430000001, 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2341 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730639+0000", 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730641+0000", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3e-06 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730647+0000", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3559999999999998e-06 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730661+0000", 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4433999999999999e-05 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730675+0000", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3866000000000001e-05 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735322+0000", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046470920000000002 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735338+0000", 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5906999999999999e-05 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783131+0000", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0477926209999999 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783244+0000", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112748 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809759+0000", 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026515001 2024-09-20T19:38:48.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.809860+0000", 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100936 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810047+0000", 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018728199999999999 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2351 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.255 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49998234500003, 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0797808179999999, 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2351 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730694+0000", 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730696+0000", 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5659999999999999e-06 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730701+0000", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9370000000000001e-06 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730720+0000", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8488999999999999e-05 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730733+0000", 2024-09-20T19:38:48.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3403e-05 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735398+0000", 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046646819999999999 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735415+0000", 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6852000000000001e-05 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783331+0000", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0479157459999999 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783426+0000", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5086000000000004e-05 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810119+0000", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026693063 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810237+0000", 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011857399999999999 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810475+0000", 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023741999999999999 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2361 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49992784400001, 2024-09-20T19:38:48.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0800758349999999, 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2361 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730748+0000", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730750+0000", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4470000000000001e-06 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730761+0000", 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.151e-05 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730776+0000", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4629e-05 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730789+0000", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.342e-05 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735477+0000", 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046872600000000004 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735492+0000", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5483999999999999e-05 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783503+0000", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048011147 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783561+0000", 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7921000000000003e-05 2024-09-20T19:38:48.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810544+0000", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026982448999999999 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810641+0000", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7238999999999995e-05 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810824+0000", 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332900000000001 2024-09-20T19:38:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2371 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49987159900002, 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080392268, 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2371 2024-09-20T19:38:48.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730805+0000", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730806+0000", 2024-09-20T19:38:48.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4130000000000001e-06 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730812+0000", 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7130000000000002e-06 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730827+0000", 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.482e-05 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.730840+0000", 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3239e-05 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735549+0000", 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047094379999999998 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735565+0000", 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5903e-05 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783599+0000", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0480339780000001 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783748+0000", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014867099999999999 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810884+0000", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027136239999999999 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.810960+0000", 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5442e-05 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811197+0000", 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023741100000000001 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2386 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:38:48.272 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49856310899997, 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.07942407, 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2386 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732113+0000", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732116+0000", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8930000000000001e-06 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.732128+0000", 2024-09-20T19:38:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1545999999999999e-05 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733339+0000", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012117230000000001 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733364+0000", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4851999999999999e-05 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735635+0000", 2024-09-20T19:38:48.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022705329999999999 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735651+0000", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6252999999999998e-05 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783859+0000", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0482082370000001 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.783959+0000", 2024-09-20T19:38:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9746000000000003e-05 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811276+0000", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027317174999999999 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811366+0000", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9417000000000001e-05 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811537+0000", 2024-09-20T19:38:48.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171695 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2396 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49728823600003, 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078525953, 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2396 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733388+0000", 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733390+0000", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3329999999999999e-06 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733400+0000", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5570000000000001e-06 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733417+0000", 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6739999999999999e-05 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733431+0000", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4112e-05 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735711+0000", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022797160000000002 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735727+0000", 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-05 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784043+0000", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0483154619999999 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784182+0000", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013889600000000001 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811601+0000", 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027418898000000001 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811709+0000", 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010836300000000001 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811914+0000", 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020494600000000001 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2406 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.283 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49722947700002, 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.078899925, 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2406 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:38:48.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733447+0000", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733448+0000", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.423e-06 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733459+0000", 2024-09-20T19:38:48.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0285999999999999e-05 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733475+0000", 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6563e-05 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733488+0000", 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3139000000000001e-05 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735846+0000", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023578169999999999 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735863+0000", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6549e-05 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784317+0000", 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0484547749999999 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784360+0000", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2880000000000003e-05 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.811994+0000", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027633993999999999 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812107+0000", 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113169 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812347+0000", 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023933 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2416 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49717022899995, 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0792861229999999, 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2416 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733506+0000", 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733508+0000", 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.457e-06 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733514+0000", 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6220000000000003e-06 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733528+0000", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3983e-05 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733540+0000", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.206e-05 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735933+0000", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023926220000000001 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.735949+0000", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6552999999999999e-05 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784410+0000", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0484610160000001 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784474+0000", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3885000000000002e-05 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812437+0000", 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027962655999999999 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812571+0000", 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134232 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812792+0000", 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022103699999999999 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.294 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2426 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49712106799996, 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0795227549999999, 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2426 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733555+0000", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733557+0000", 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.384e-06 2024-09-20T19:38:48.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733563+0000", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2859999999999996e-06 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733578+0000", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4693e-05 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733591+0000", 2024-09-20T19:38:48.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.346e-05 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736007+0000", 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024159749999999999 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736024+0000", 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6952e-05 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784553+0000", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048529026 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784649+0000", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6232999999999998e-05 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812877+0000", 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028227498 2024-09-20T19:38:48.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.812961+0000", 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4085999999999994e-05 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813078+0000", 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117162 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2436 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49707005200003, 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.079874939, 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2436 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733606+0000", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733607+0000", 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.223e-06 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733614+0000", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.336e-06 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733628+0000", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4157e-05 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733641+0000", 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3363000000000001e-05 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736084+0000", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024431499999999998 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736101+0000", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6039999999999999e-05 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784727+0000", 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048626016 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784794+0000", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7896000000000002e-05 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813130+0000", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028335289 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813213+0000", 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3163000000000003e-05 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813481+0000", 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000268306 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2446 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.497018981, 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080036373, 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2446 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733657+0000", 2024-09-20T19:38:48.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733659+0000", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8500000000000001e-06 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733666+0000", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5599999999999999e-06 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733681+0000", 2024-09-20T19:38:48.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5288999999999999e-05 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733696+0000", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4707e-05 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736162+0000", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024662260000000002 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736178+0000", 2024-09-20T19:38:48.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6422999999999998e-05 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.784966+0000", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.048787887 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785132+0000", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016522100000000001 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813533+0000", 2024-09-20T19:38:48.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02840123 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813583+0000", 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0760999999999997e-05 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813694+0000", 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110219 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2456 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49696562300005, 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080351998, 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2456 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733711+0000", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733712+0000", 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.204e-06 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733735+0000", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759e-05 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733752+0000", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6963e-05 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733765+0000", 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3716999999999999e-05 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736238+0000", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024724230000000001 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736254+0000", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6245000000000001e-05 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785216+0000", 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0489618709999999 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785294+0000", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.852e-05 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813766+0000", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028471195000000001 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.813907+0000", 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014175 2024-09-20T19:38:48.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814063+0000", 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155351 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2466 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49689384800001, 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0804979480000001, 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.317 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2466 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733782+0000", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733784+0000", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733791+0000", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733807+0000", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6022999999999999e-05 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733821+0000", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4059999999999999e-05 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736314+0000", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002493655 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736330+0000", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5650000000000001e-05 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785356+0000", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049026096 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785471+0000", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114505 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814112+0000", 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02864152 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814164+0000", 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2030999999999997e-05 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814280+0000", 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011619499999999999 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2476 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49684035600001, 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0808984619999999, 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2476 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733836+0000", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733837+0000", 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3060000000000001e-06 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733843+0000", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8869999999999997e-06 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733857+0000", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4146e-05 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733871+0000", 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3366e-05 2024-09-20T19:38:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736389+0000", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025186549999999999 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736406+0000", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6500000000000001e-05 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785587+0000", 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049180741 2024-09-20T19:38:48.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785739+0000", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152609 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814343+0000", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028604258 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814462+0000", 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118318 2024-09-20T19:38:48.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814734+0000", 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027267600000000002 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2486 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49679151400005, 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081119497, 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.328 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2486 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733885+0000", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733886+0000", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2729999999999999e-06 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733892+0000", 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7429999999999999e-06 2024-09-20T19:38:48.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733905+0000", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3653e-05 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733919+0000", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3626e-05 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736467+0000", 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025480009999999998 2024-09-20T19:38:48.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.736483+0000", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6273000000000002e-05 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785878+0000", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.049395021 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.785995+0000", 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011697300000000001 2024-09-20T19:38:48.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814815+0000", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028819918999999999 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.814875+0000", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0208999999999999e-05 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815004+0000", 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128806 2024-09-20T19:38:48.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2496 5.e 5:7759931f:::notify.3:head [watch ping cookie 94817798633088 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49674299900005, 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0813325680000001, 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2496 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733933+0000", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733935+0000", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2559999999999999e-06 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733941+0000", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9429999999999997e-06 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733954+0000", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3890000000000001e-05 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.733968+0000", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3569e-05 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743265+0000", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092973120000000003 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743287+0000", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1846000000000001e-05 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.786078+0000", 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0427906039999999 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.786224+0000", 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014672700000000001 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815055+0000", 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028830130999999998 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815112+0000", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7178e-05 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.815266+0000", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154112 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14478.0:2543 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94817798633088 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:38:48.339 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.49090046699996, 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.080228567, 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14478", 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.42:0/1056266775", 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2543 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739776+0000", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739777+0000", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4729999999999999e-06 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739783+0000", 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9599999999999997e-06 2024-09-20T19:38:48.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739798+0000", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4493e-05 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.739811+0000", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3468999999999999e-05 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743681+0000", 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038696569999999999 2024-09-20T19:38:48.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:08.743698+0000", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7110000000000001e-05 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.787209+0000", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0435105220000001 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.787321+0000", 2024-09-20T19:38:48.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112125 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.817383+0000", 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030062286000000001 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.817453+0000", 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0094999999999999e-05 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-20T19:38:48.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-20T19:30:09.820004+0000", 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025513770000000001 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-20T19:38:48.345 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:38:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:48.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:48.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:3861: done (2 left) 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:3862: copy_from oid 24 from oid 28 current snap is 426 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:3859: finishing rollback tid 0 to smithi04268048-30 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:3859: finishing rollback tid 1 to smithi04268048-30 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:3859: finishing rollback tid 2 to smithi04268048-30 2024-09-20T19:38:48.483 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1620 (ObjNum 1315 snap 422 seq_num 1315) dirty exists 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: finishing write tid 1 to smithi04268048-23 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: finishing write tid 2 to smithi04268048-23 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: finishing write tid 3 to smithi04268048-23 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: finishing write tid 4 to smithi04268048-23 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: oid 23 updating version 0 to 1187 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: oid 23 updating version 1187 to 1188 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: oid 23 version 1188 is already newer than 1186 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1188 (ObjNum 1321 snap 426 seq_num 1321) dirty exists 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: left oid 23 (ObjNum 1321 snap 426 seq_num 1321) 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3859: done (2 left) 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3860: done (1 left) 2024-09-20T19:38:48.484 INFO:tasks.rados.rados.0.smithi042.stdout:3863: snap_remove snap 424 2024-09-20T19:38:48.577 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:48.602 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:48.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:48.731 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:48 smithi162 bash[24557]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[22706]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: cluster 2024-09-20T19:38:47.553277+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: audit 2024-09-20T19:38:47.708777+0000 mon.a (mon.0) 2667 : audit [DBG] from='client.? v1:172.21.15.42:0/3716270362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:48 smithi042 bash[32075]: cluster 2024-09-20T19:38:48.481173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2024-09-20T19:38:48.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:48.989 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:48.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:49.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:49.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:49.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T19:38:49.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:49.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:49.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:49.498 INFO:tasks.rados.rados.0.smithi042.stdout:3863: done (1 left) 2024-09-20T19:38:49.499 INFO:tasks.rados.rados.0.smithi042.stdout:3864: read oid 20 snap 425 2024-09-20T19:38:49.499 INFO:tasks.rados.rados.0.smithi042.stdout:3864: expect deleted 2024-09-20T19:38:49.499 INFO:tasks.rados.rados.0.smithi042.stdout:3865: write (excl) oid 42 current snap is 426 2024-09-20T19:38:49.499 INFO:tasks.rados.rados.0.smithi042.stdout:3865: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-20T19:38:49.503 INFO:tasks.rados.rados.0.smithi042.stdout:3865: writing smithi04268048-42 from 618854 to 1027101 tid 1 2024-09-20T19:38:49.513 INFO:tasks.rados.rados.0.smithi042.stdout:3865: writing smithi04268048-42 from 1743901 to 2169695 tid 2 2024-09-20T19:38:49.514 INFO:tasks.rados.rados.0.smithi042.stdout:3862: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:38:49.514 INFO:tasks.rados.rados.0.smithi042.stdout:3862: finishing copy_from to smithi04268048-24 2024-09-20T19:38:49.514 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1455 (ObjNum 1298 snap 417 seq_num 1298) dirty exists 2024-09-20T19:38:49.514 INFO:tasks.rados.rados.0.smithi042.stdout:3862: done (2 left) 2024-09-20T19:38:49.515 INFO:tasks.rados.rados.0.smithi042.stdout:3864: done (1 left) 2024-09-20T19:38:49.515 INFO:tasks.rados.rados.0.smithi042.stdout:3866: read oid 10 snap -1 2024-09-20T19:38:49.515 INFO:tasks.rados.rados.0.smithi042.stdout:3866: expect deleted 2024-09-20T19:38:49.515 INFO:tasks.rados.rados.0.smithi042.stdout:3867: write oid 20 current snap is 426 2024-09-20T19:38:49.515 INFO:tasks.rados.rados.0.smithi042.stdout:3867: seq_num 1323 ranges {635661=599849,1918736=316925} 2024-09-20T19:38:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:49.522 INFO:tasks.rados.rados.0.smithi042.stdout:3867: writing smithi04268048-20 from 635661 to 1235510 tid 1 2024-09-20T19:38:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:49.528 INFO:tasks.rados.rados.0.smithi042.stdout:3867: writing smithi04268048-20 from 1918736 to 2235661 tid 2 2024-09-20T19:38:49.528 INFO:tasks.rados.rados.0.smithi042.stdout:3868: write oid 34 current snap is 426 2024-09-20T19:38:49.528 INFO:tasks.rados.rados.0.smithi042.stdout:3868: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-20T19:38:49.535 INFO:tasks.rados.rados.0.smithi042.stdout:3868: writing smithi04268048-34 from 652468 to 1327566 tid 1 2024-09-20T19:38:49.541 INFO:tasks.rados.rados.0.smithi042.stdout:3868: writing smithi04268048-34 from 1977218 to 2252468 tid 2 2024-09-20T19:38:49.541 INFO:tasks.rados.rados.0.smithi042.stdout:3869: write oid 4 current snap is 426 2024-09-20T19:38:49.541 INFO:tasks.rados.rados.0.smithi042.stdout:3869: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-20T19:38:49.549 INFO:tasks.rados.rados.0.smithi042.stdout:3869: writing smithi04268048-4 from 669275 to 1419622 tid 1 2024-09-20T19:38:49.554 INFO:tasks.rados.rados.0.smithi042.stdout:3869: writing smithi04268048-4 from 2035700 to 2269275 tid 2 2024-09-20T19:38:49.554 INFO:tasks.rados.rados.0.smithi042.stdout:3870: write (excl) oid 27 current snap is 426 2024-09-20T19:38:49.554 INFO:tasks.rados.rados.0.smithi042.stdout:3870: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-20T19:38:49.560 INFO:tasks.rados.rados.0.smithi042.stdout:3870: writing smithi04268048-27 from 686082 to 1111678 tid 1 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3870: writing smithi04268048-27 from 1577829 to 2010961 tid 2 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3871: read oid 7 snap -1 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3871: expect (ObjNum 1314 snap 422 seq_num 1314) 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3872: delete oid 49 current snap is 426 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: finishing write tid 1 to smithi04268048-42 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: finishing write tid 2 to smithi04268048-42 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: finishing write tid 3 to smithi04268048-42 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: finishing write tid 4 to smithi04268048-42 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: oid 42 updating version 0 to 1353 2024-09-20T19:38:49.567 INFO:tasks.rados.rados.0.smithi042.stdout:3865: oid 42 updating version 1353 to 1354 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3865: oid 42 version 1354 is already newer than 1352 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1354 (ObjNum 1322 snap 426 seq_num 1322) dirty exists 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3865: left oid 42 (ObjNum 1322 snap 426 seq_num 1322) 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: finishing write tid 1 to smithi04268048-20 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: finishing write tid 2 to smithi04268048-20 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: finishing write tid 3 to smithi04268048-20 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: finishing write tid 4 to smithi04268048-20 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: oid 20 updating version 0 to 1241 2024-09-20T19:38:49.568 INFO:tasks.rados.rados.0.smithi042.stdout:3867: oid 20 version 1241 is already newer than 1239 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3867: oid 20 version 1241 is already newer than 1240 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 20 v 1241 (ObjNum 1323 snap 426 seq_num 1323) dirty exists 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3867: left oid 20 (ObjNum 1323 snap 426 seq_num 1323) 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3868: finishing write tid 1 to smithi04268048-34 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3868: finishing write tid 2 to smithi04268048-34 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3868: finishing write tid 3 to smithi04268048-34 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3868: finishing write tid 4 to smithi04268048-34 2024-09-20T19:38:49.569 INFO:tasks.rados.rados.0.smithi042.stdout:3868: oid 34 updating version 0 to 1256 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3868: oid 34 version 1256 is already newer than 1254 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3868: oid 34 version 1256 is already newer than 1255 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1256 (ObjNum 1324 snap 426 seq_num 1324) dirty exists 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3868: left oid 34 (ObjNum 1324 snap 426 seq_num 1324) 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3869: finishing write tid 1 to smithi04268048-4 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3869: finishing write tid 2 to smithi04268048-4 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3869: finishing write tid 3 to smithi04268048-4 2024-09-20T19:38:49.570 INFO:tasks.rados.rados.0.smithi042.stdout:3869: finishing write tid 4 to smithi04268048-4 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:3869: oid 4 updating version 0 to 1357 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:3869: oid 4 version 1357 is already newer than 1356 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:3869: oid 4 updating version 1357 to 1358 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 4 v 1358 (ObjNum 1325 snap 426 seq_num 1325) dirty exists 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:3869: left oid 4 (ObjNum 1325 snap 426 seq_num 1325) 2024-09-20T19:38:49.571 INFO:tasks.rados.rados.0.smithi042.stdout:3870: finishing write tid 1 to smithi04268048-27 2024-09-20T19:38:49.572 INFO:tasks.rados.rados.0.smithi042.stdout:3865: done (7 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3866: done (6 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3867: done (5 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3868: done (4 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3869: done (3 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3872: done (2 left) 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3873: rollback oid 32 current snap is 426 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 32 to 425 2024-09-20T19:38:49.573 INFO:tasks.rados.rados.0.smithi042.stdout:3874: copy_from oid 10 from oid 1 current snap is 426 2024-09-20T19:38:49.574 INFO:tasks.rados.rados.0.smithi042.stdout:3875: copy_from oid 48 from oid 40 current snap is 426 2024-09-20T19:38:49.574 INFO:tasks.rados.rados.0.smithi042.stdout:3876: read oid 9 snap 425 2024-09-20T19:38:49.574 INFO:tasks.rados.rados.0.smithi042.stdout:3876: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:49.574 INFO:tasks.rados.rados.0.smithi042.stdout:3877: snap_create 2024-09-20T19:38:49.575 INFO:tasks.rados.rados.0.smithi042.stdout:3871: expect (ObjNum 1314 snap 422 seq_num 1314) 2024-09-20T19:38:49.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:49.691 INFO:tasks.rados.rados.0.smithi042.stdout:3873: finishing rollback tid 0 to smithi04268048-32 2024-09-20T19:38:49.691 INFO:tasks.rados.rados.0.smithi042.stdout:3873: finishing rollback tid 1 to smithi04268048-32 2024-09-20T19:38:49.691 INFO:tasks.rados.rados.0.smithi042.stdout:3873: finishing rollback tid 2 to smithi04268048-32 2024-09-20T19:38:49.692 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 32 v 1646 (ObjNum 1313 snap 422 seq_num 1313) dirty exists 2024-09-20T19:38:49.692 INFO:tasks.rados.rados.0.smithi042.stdout:3870: finishing write tid 2 to smithi04268048-27 2024-09-20T19:38:49.692 INFO:tasks.rados.rados.0.smithi042.stdout:3870: finishing write tid 3 to smithi04268048-27 2024-09-20T19:38:49.692 INFO:tasks.rados.rados.0.smithi042.stdout:3876: expect (ObjNum 1246 snap 401 seq_num 1246) 2024-09-20T19:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:49.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:49 smithi162 bash[24557]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:49 smithi162 bash[24557]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:49 smithi162 bash[24557]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:49.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:49 smithi162 bash[24557]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:49.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:49.827 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2868116 2024-09-20T19:38:49.833 INFO:tasks.rados.rados.0.smithi042.stdout:3870: finishing write tid 4 to smithi04268048-27 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3870: oid 27 updating version 0 to 1360 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3870: oid 27 updating version 1360 to 1362 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3870: oid 27 version 1362 is already newer than 1361 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 27 v 1362 (ObjNum 1326 snap 426 seq_num 1326) dirty exists 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3870: left oid 27 (ObjNum 1326 snap 426 seq_num 1326) 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3874: finishing copy_from racing read to smithi04268048-10 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3875: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3875: finishing copy_from to smithi04268048-48 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1071 (ObjNum 1307 snap 419 seq_num 1307) dirty exists 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:3874: finishing copy_from to smithi04268048-10 2024-09-20T19:38:49.834 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 10 v 1148 (ObjNum 1244 snap 401 seq_num 1244) dirty exists 2024-09-20T19:38:49.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:49.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:49.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[22706]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[22706]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[22706]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:49.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[22706]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:49.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[32075]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[32075]: cluster 2024-09-20T19:38:48.487482+0000 mgr.y (mgr.24491) 779 : cluster [DBG] pgmap v1528: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:49.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[32075]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:49.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:49 smithi042 bash[32075]: cluster 2024-09-20T19:38:49.489339+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2024-09-20T19:38:50.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:50.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:49 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:38:50.141 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:50.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:50.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:50.299 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:50.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:50.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:50.581 INFO:tasks.rados.rados.0.smithi042.stdout:3870: done (6 left) 2024-09-20T19:38:50.581 INFO:tasks.rados.rados.0.smithi042.stdout:3871: done (5 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3873: done (4 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3874: done (3 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3875: done (2 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3876: done (1 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3877: done (0 left) 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3878: rollback oid 47 current snap is 427 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 47 to 426 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3879: read oid 36 snap -1 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3879: expect deleted 2024-09-20T19:38:50.582 INFO:tasks.rados.rados.0.smithi042.stdout:3880: snap_remove snap 425 2024-09-20T19:38:50.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:50.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:50.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:50.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:50.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:50.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:51.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:51.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:51.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:51.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:51.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:38:51.302 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:51.446 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:51.471 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:51.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:51.587 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T19:38:51.590 INFO:tasks.rados.rados.0.smithi042.stdout:3880: done (2 left) 2024-09-20T19:38:51.591 INFO:tasks.rados.rados.0.smithi042.stdout:3881: write (excl) oid 14 current snap is 427 2024-09-20T19:38:51.591 INFO:tasks.rados.rados.0.smithi042.stdout:3881: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-20T19:38:51.599 INFO:tasks.rados.rados.0.smithi042.stdout:3881: writing smithi04268048-14 from 719696 to 1295790 tid 1 2024-09-20T19:38:51.606 INFO:tasks.rados.rados.0.smithi042.stdout:3881: writing smithi04268048-14 from 2094793 to 2319696 tid 2 2024-09-20T19:38:51.606 INFO:tasks.rados.rados.0.smithi042.stdout:3882: read oid 17 snap -1 2024-09-20T19:38:51.606 INFO:tasks.rados.rados.0.smithi042.stdout:3882: expect (ObjNum 1320 snap 425 seq_num 1320) 2024-09-20T19:38:51.606 INFO:tasks.rados.rados.0.smithi042.stdout:3883: write oid 19 current snap is 427 2024-09-20T19:38:51.606 INFO:tasks.rados.rados.0.smithi042.stdout:3883: seq_num 1329 ranges {736503=651343,2153275=183228} 2024-09-20T19:38:51.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:51.614 INFO:tasks.rados.rados.0.smithi042.stdout:3883: writing smithi04268048-19 from 736503 to 1387846 tid 1 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3883: writing smithi04268048-19 from 2153275 to 2336503 tid 2 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3884: copy_from oid 48 from oid 21 current snap is 427 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3878: finishing rollback tid 1 to smithi04268048-47 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 47 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3878: done (5 left) 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3879: done (4 left) 2024-09-20T19:38:51.620 INFO:tasks.rados.rados.0.smithi042.stdout:3885: snap_create 2024-09-20T19:38:51.621 INFO:tasks.rados.rados.0.smithi042.stdout:3882: expect (ObjNum 1320 snap 425 seq_num 1320) 2024-09-20T19:38:51.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:51.733 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:51.790 INFO:tasks.rados.rados.0.smithi042.stdout:3881: finishing write tid 1 to smithi04268048-14 2024-09-20T19:38:51.790 INFO:tasks.rados.rados.0.smithi042.stdout:3881: finishing write tid 2 to smithi04268048-14 2024-09-20T19:38:51.790 INFO:tasks.rados.rados.0.smithi042.stdout:3881: finishing write tid 3 to smithi04268048-14 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3881: finishing write tid 4 to smithi04268048-14 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3881: oid 14 updating version 0 to 1605 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3881: oid 14 version 1605 is already newer than 1604 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3881: oid 14 updating version 1605 to 1606 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1606 (ObjNum 1328 snap 427 seq_num 1328) dirty exists 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3881: left oid 14 (ObjNum 1328 snap 427 seq_num 1328) 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: finishing write tid 1 to smithi04268048-19 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: finishing write tid 2 to smithi04268048-19 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: finishing write tid 3 to smithi04268048-19 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: finishing write tid 4 to smithi04268048-19 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: oid 19 updating version 0 to 1457 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: oid 19 updating version 1457 to 1458 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:3883: oid 19 version 1458 is already newer than 1456 2024-09-20T19:38:51.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 19 v 1458 (ObjNum 1329 snap 427 seq_num 1329) dirty exists 2024-09-20T19:38:51.792 INFO:tasks.rados.rados.0.smithi042.stdout:3883: left oid 19 (ObjNum 1329 snap 427 seq_num 1329) 2024-09-20T19:38:51.792 INFO:tasks.rados.rados.0.smithi042.stdout:3884: finishing copy_from racing read to smithi04268048-48 2024-09-20T19:38:51.792 INFO:tasks.rados.rados.0.smithi042.stdout:3884: finishing copy_from to smithi04268048-48 2024-09-20T19:38:51.792 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1595 (ObjNum 1236 snap 401 seq_num 1236) dirty exists 2024-09-20T19:38:51.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:51.854 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:51 smithi162 bash[24557]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:51.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[22706]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:49.947429+0000 mgr.y (mgr.24491) 780 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: cluster 2024-09-20T19:38:50.488249+0000 mgr.y (mgr.24491) 781 : cluster [DBG] pgmap v1530: 178 pgs: 4 active+clean+snaptrim, 174 active+clean; 104 MiB data, 669 MiB used, 625 GiB / 626 GiB avail; 8.8 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-20T19:38:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: cluster 2024-09-20T19:38:50.578307+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2024-09-20T19:38:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:51.318879+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.42:0/121682921' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:51 smithi042 bash[32075]: audit 2024-09-20T19:38:51.319191+0000 mon.a (mon.0) 2671 : audit [INF] from='client.26123 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:38:51.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:51.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:51.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:52.094 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:52.118 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:52.221 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:52.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:52.374 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:52.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:52.599 INFO:tasks.rados.rados.0.smithi042.stdout:3881: done (4 left) 2024-09-20T19:38:52.599 INFO:tasks.rados.rados.0.smithi042.stdout:3882: done (3 left) 2024-09-20T19:38:52.599 INFO:tasks.rados.rados.0.smithi042.stdout:3883: done (2 left) 2024-09-20T19:38:52.600 INFO:tasks.rados.rados.0.smithi042.stdout:3884: done (1 left) 2024-09-20T19:38:52.600 INFO:tasks.rados.rados.0.smithi042.stdout:3885: done (0 left) 2024-09-20T19:38:52.600 INFO:tasks.rados.rados.0.smithi042.stdout:3886: read oid 45 snap -1 2024-09-20T19:38:52.600 INFO:tasks.rados.rados.0.smithi042.stdout:3886: expect (ObjNum 1304 snap 418 seq_num 1304) 2024-09-20T19:38:52.600 INFO:tasks.rados.rados.0.smithi042.stdout:3887: snap_create 2024-09-20T19:38:52.600 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:38:52.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:52.615 INFO:tasks.rados.rados.0.smithi042.stdout:3886: expect (ObjNum 1304 snap 418 seq_num 1304) 2024-09-20T19:38:52.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.618 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[32075]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:52 smithi042 bash[22706]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.633 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:52.662 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:52.737 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:52.756 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:52.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:51.581049+0000 mon.a (mon.0) 2672 : audit [INF] from='client.26123 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: cluster 2024-09-20T19:38:51.589098+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:52.120704+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.42:0/2816257295' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:52 smithi162 bash[24557]: audit 2024-09-20T19:38:52.121019+0000 mon.a (mon.0) 2674 : audit [INF] from='client.26129 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:38:52.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:52.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:52.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:53.018 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:53.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T19:38:53.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:53.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:53.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:53.274 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:53.379 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:53.484 INFO:tasks.rados.rados.0.smithi042.stdout:3886: done (1 left) 2024-09-20T19:38:53.484 INFO:tasks.rados.rados.0.smithi042.stdout:3887: done (0 left) 2024-09-20T19:38:53.485 INFO:tasks.rados.rados.0.smithi042.stdout:3888: write oid 50 current snap is 429 2024-09-20T19:38:53.485 INFO:tasks.rados.rados.0.smithi042.stdout:3888: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-20T19:38:53.498 INFO:tasks.rados.rados.0.smithi042.stdout:3888: writing smithi04268048-50 from 753310 to 1479902 tid 1 2024-09-20T19:38:53.505 INFO:tasks.rados.rados.0.smithi042.stdout:3888: writing smithi04268048-50 from 2095404 to 2353310 tid 2 2024-09-20T19:38:53.505 INFO:tasks.rados.rados.0.smithi042.stdout:3889: copy_from oid 1 from oid 20 current snap is 429 2024-09-20T19:38:53.505 INFO:tasks.rados.rados.0.smithi042.stdout:3890: read oid 44 snap -1 2024-09-20T19:38:53.505 INFO:tasks.rados.rados.0.smithi042.stdout:3890: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:53.505 INFO:tasks.rados.rados.0.smithi042.stdout:3891: snap_create 2024-09-20T19:38:53.507 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:53.520 INFO:tasks.rados.rados.0.smithi042.stdout:3890: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:53.533 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:53.586 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:53.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:53.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[22706]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:53 smithi042 bash[32075]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.713 INFO:tasks.rados.rados.0.smithi042.stdout:3888: finishing write tid 1 to smithi04268048-50 2024-09-20T19:38:53.713 INFO:tasks.rados.rados.0.smithi042.stdout:3888: finishing write tid 2 to smithi04268048-50 2024-09-20T19:38:53.713 INFO:tasks.rados.rados.0.smithi042.stdout:3888: finishing write tid 3 to smithi04268048-50 2024-09-20T19:38:53.713 INFO:tasks.rados.rados.0.smithi042.stdout:3888: finishing write tid 4 to smithi04268048-50 2024-09-20T19:38:53.713 INFO:tasks.rados.rados.0.smithi042.stdout:3888: oid 50 updating version 0 to 1216 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:3888: oid 50 updating version 1216 to 1218 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:3888: oid 50 version 1218 is already newer than 1217 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1218 (ObjNum 1330 snap 429 seq_num 1330) dirty exists 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:3888: left oid 50 (ObjNum 1330 snap 429 seq_num 1330) 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:3889: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:3889: finishing copy_from to smithi04268048-1 2024-09-20T19:38:53.714 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1241 (ObjNum 1323 snap 426 seq_num 1323) dirty exists 2024-09-20T19:38:53.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:53.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:53.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.490045+0000 mgr.y (mgr.24491) 782 : cluster [DBG] pgmap v1533: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 98 MiB data, 681 MiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.581379+0000 mon.a (mon.0) 2675 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: audit 2024-09-20T19:38:52.583714+0000 mon.a (mon.0) 2676 : audit [INF] from='client.26129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:38:53.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:52.591831+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2024-09-20T19:38:53.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:53 smithi162 bash[24557]: cluster 2024-09-20T19:38:53.483082+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2024-09-20T19:38:53.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:53.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:38:54.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:54.056 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:54.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:54.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:54.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:54.448 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3888: done (3 left) 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3889: done (2 left) 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3890: done (1 left) 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3891: done (0 left) 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3892: read oid 28 snap 429 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3892: expect (ObjNum 1298 snap 417 seq_num 1298) 2024-09-20T19:38:54.484 INFO:tasks.rados.rados.0.smithi042.stdout:3893: read oid 44 snap 427 2024-09-20T19:38:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:3893: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:3894: read oid 42 snap 426 2024-09-20T19:38:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:3894: expect (ObjNum 1322 snap 426 seq_num 1322) 2024-09-20T19:38:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:3895: write (excl) oid 39 current snap is 430 2024-09-20T19:38:54.485 INFO:tasks.rados.rados.0.smithi042.stdout:3895: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-20T19:38:54.493 INFO:tasks.rados.rados.0.smithi042.stdout:3895: writing smithi04268048-39 from 770117 to 1288311 tid 1 2024-09-20T19:38:54.501 INFO:tasks.rados.rados.0.smithi042.stdout:3895: writing smithi04268048-39 from 1870239 to 2370117 tid 2 2024-09-20T19:38:54.501 INFO:tasks.rados.rados.0.smithi042.stdout:3896: read oid 11 snap -1 2024-09-20T19:38:54.502 INFO:tasks.rados.rados.0.smithi042.stdout:3896: expect (ObjNum 1293 snap 413 seq_num 1293) 2024-09-20T19:38:54.502 INFO:tasks.rados.rados.0.smithi042.stdout:3897: delete oid 35 current snap is 430 2024-09-20T19:38:54.502 INFO:tasks.rados.rados.0.smithi042.stdout:3893: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:38:54.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:54.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:54.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:54.703 INFO:tasks.rados.rados.0.smithi042.stdout:3892: expect (ObjNum 1298 snap 417 seq_num 1298) 2024-09-20T19:38:54.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.827 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:54.909 INFO:tasks.rados.rados.0.smithi042.stdout:3894: expect (ObjNum 1322 snap 426 seq_num 1322) 2024-09-20T19:38:54.909 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:54.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.955 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:54.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:54.969 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: finishing write tid 1 to smithi04268048-39 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: finishing write tid 2 to smithi04268048-39 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: finishing write tid 3 to smithi04268048-39 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3892: done (5 left) 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3893: done (4 left) 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3894: done (3 left) 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3897: done (2 left) 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3898: copy_from oid 17 from oid 44 current snap is 430 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: finishing write tid 4 to smithi04268048-39 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: oid 39 updating version 0 to 1272 2024-09-20T19:38:55.004 INFO:tasks.rados.rados.0.smithi042.stdout:3895: oid 39 updating version 1272 to 1273 2024-09-20T19:38:55.005 INFO:tasks.rados.rados.0.smithi042.stdout:3895: oid 39 updating version 1273 to 1274 2024-09-20T19:38:55.005 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1274 (ObjNum 1331 snap 430 seq_num 1331) dirty exists 2024-09-20T19:38:55.005 INFO:tasks.rados.rados.0.smithi042.stdout:3895: left oid 39 (ObjNum 1331 snap 430 seq_num 1331) 2024-09-20T19:38:55.005 INFO:tasks.rados.rados.0.smithi042.stdout:3896: expect (ObjNum 1293 snap 413 seq_num 1293) 2024-09-20T19:38:55.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:55.076 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:55.150 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 732559 2024-09-20T19:38:55.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3898: finishing copy_from racing read to smithi04268048-17 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3898: finishing copy_from to smithi04268048-17 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 17 v 1167 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3895: done (2 left) 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3896: done (1 left) 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3898: done (0 left) 2024-09-20T19:38:55.212 INFO:tasks.rados.rados.0.smithi042.stdout:3899: snap_remove snap 429 2024-09-20T19:38:55.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:55.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:55.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:55.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:55.506 INFO:tasks.rados.rados.0.smithi042.stdout:3899: done (0 left) 2024-09-20T19:38:55.507 INFO:tasks.rados.rados.0.smithi042.stdout:3900: write oid 8 current snap is 430 2024-09-20T19:38:55.507 INFO:tasks.rados.rados.0.smithi042.stdout:3900: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-20T19:38:55.523 INFO:tasks.rados.rados.0.smithi042.stdout:3900: writing smithi04268048-8 from 403731 to 1072423 tid 1 2024-09-20T19:38:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:3900: writing smithi04268048-8 from 1587203 to 2175000 tid 2 2024-09-20T19:38:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:3901: snap_create 2024-09-20T19:38:55.534 INFO:tasks.rados.rados.0.smithi042.stdout:3900: finishing write tid 1 to smithi04268048-8 2024-09-20T19:38:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:3900: finishing write tid 2 to smithi04268048-8 2024-09-20T19:38:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:3900: finishing write tid 3 to smithi04268048-8 2024-09-20T19:38:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:3900: finishing write tid 4 to smithi04268048-8 2024-09-20T19:38:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:3900: oid 8 updating version 0 to 1615 2024-09-20T19:38:55.538 INFO:tasks.rados.rados.0.smithi042.stdout:3900: oid 8 updating version 1615 to 1617 2024-09-20T19:38:55.539 INFO:tasks.rados.rados.0.smithi042.stdout:3900: oid 8 version 1617 is already newer than 1616 2024-09-20T19:38:55.539 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1617 (ObjNum 1333 snap 430 seq_num 1333) dirty exists 2024-09-20T19:38:55.539 INFO:tasks.rados.rados.0.smithi042.stdout:3900: left oid 8 (ObjNum 1333 snap 430 seq_num 1333) 2024-09-20T19:38:55.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:55.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[32075]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[32075]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.580 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[32075]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.581 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[32075]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[22706]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[22706]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[22706]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.581 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:55 smithi042 bash[22706]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:55.677 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:38:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:38:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:38:55.678 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:38:55.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:55.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:55.715 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:55 smithi162 bash[24557]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.715 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:55 smithi162 bash[24557]: cluster 2024-09-20T19:38:54.481866+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2024-09-20T19:38:55.715 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:55 smithi162 bash[24557]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.715 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:55 smithi162 bash[24557]: cluster 2024-09-20T19:38:54.490857+0000 mgr.y (mgr.24491) 783 : cluster [DBG] pgmap v1537: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2024-09-20T19:38:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:55.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:38:55.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:55.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:55.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:55.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:56.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:56.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:56.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:56.256 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:56.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:56.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:56.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:56.505 INFO:tasks.rados.rados.0.smithi042.stdout:3900: done (1 left) 2024-09-20T19:38:56.505 INFO:tasks.rados.rados.0.smithi042.stdout:3901: done (0 left) 2024-09-20T19:38:56.505 INFO:tasks.rados.rados.0.smithi042.stdout:3902: snap_create 2024-09-20T19:38:56.506 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:38:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:38:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:38:56.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:56.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T19:38:56.534 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:56.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:56.649 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:56.752 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:56.763 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:56 smithi042 bash[32075]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.763 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:56 smithi042 bash[32075]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:56 smithi042 bash[22706]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:56 smithi042 bash[22706]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.780 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:56 smithi162 bash[24557]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:56 smithi162 bash[24557]: cluster 2024-09-20T19:38:55.501768+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2024-09-20T19:38:56.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:56.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:57.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:57.039 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:57.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:57.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:57.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:57.408 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:57.526 INFO:tasks.rados.rados.0.smithi042.stdout:3902: done (0 left) 2024-09-20T19:38:57.527 INFO:tasks.rados.rados.0.smithi042.stdout:3903: copy_from oid 6 from oid 47 current snap is 432 2024-09-20T19:38:57.527 INFO:tasks.rados.rados.0.smithi042.stdout:3904: write (excl) oid 50 current snap is 432 2024-09-20T19:38:57.527 INFO:tasks.rados.rados.0.smithi042.stdout:3904: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-20T19:38:57.538 INFO:tasks.rados.rados.0.smithi042.stdout:3904: writing smithi04268048-50 from 420538 to 1164479 tid 1 2024-09-20T19:38:57.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3904: writing smithi04268048-50 from 1929332 to 2420538 tid 2 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3905: read oid 5 snap -1 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3905: expect deleted 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3906: read oid 29 snap -1 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3906: expect (ObjNum 1267 snap 408 seq_num 1267) 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3903: finishing copy_from to smithi04268048-6 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3903: got expected ENOENT (src dne) 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3903: finishing copy_from racing read to smithi04268048-6 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3904: finishing write tid 1 to smithi04268048-50 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3903: done (3 left) 2024-09-20T19:38:57.557 INFO:tasks.rados.rados.0.smithi042.stdout:3907: copy_from oid 9 from oid 35 current snap is 432 2024-09-20T19:38:57.558 INFO:tasks.rados.rados.0.smithi042.stdout:3908: write (excl) oid 15 current snap is 432 2024-09-20T19:38:57.558 INFO:tasks.rados.rados.0.smithi042.stdout:3908: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-20T19:38:57.560 INFO:tasks.rados.rados.0.smithi042.stdout:3908: writing smithi04268048-15 from 437345 to 856535 tid 1 2024-09-20T19:38:57.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3908: writing smithi04268048-15 from 1587814 to 2179280 tid 2 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3909: read oid 43 snap -1 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3909: expect deleted 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3910: write oid 37 current snap is 432 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3904: finishing write tid 2 to smithi04268048-50 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3904: finishing write tid 3 to smithi04268048-50 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3904: finishing write tid 4 to smithi04268048-50 2024-09-20T19:38:57.571 INFO:tasks.rados.rados.0.smithi042.stdout:3904: oid 50 updating version 0 to 1222 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3904: oid 50 version 1222 is already newer than 1220 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3904: oid 50 version 1222 is already newer than 1221 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1222 (ObjNum 1334 snap 432 seq_num 1334) dirty exists 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3904: left oid 50 (ObjNum 1334 snap 432 seq_num 1334) 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3907: finishing copy_from to smithi04268048-9 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3907: got expected ENOENT (src dne) 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3907: finishing copy_from racing read to smithi04268048-9 2024-09-20T19:38:57.572 INFO:tasks.rados.rados.0.smithi042.stdout:3906: expect (ObjNum 1267 snap 408 seq_num 1267) 2024-09-20T19:38:57.627 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1682471 2024-09-20T19:38:57.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:57.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[22706]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[22706]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[22706]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[22706]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[32075]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[32075]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[32075]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:57 smithi042 bash[32075]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:57.708 INFO:tasks.rados.rados.0.smithi042.stdout:3908: finishing write tid 1 to smithi04268048-15 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: finishing write tid 2 to smithi04268048-15 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: finishing write tid 3 to smithi04268048-15 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: finishing write tid 4 to smithi04268048-15 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: oid 15 updating version 0 to 1159 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: oid 15 version 1159 is already newer than 1157 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: oid 15 version 1159 is already newer than 1158 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 15 v 1159 (ObjNum 1335 snap 432 seq_num 1335) dirty exists 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3908: left oid 15 (ObjNum 1335 snap 432 seq_num 1335) 2024-09-20T19:38:57.709 INFO:tasks.rados.rados.0.smithi042.stdout:3910: seq_num 1336 ranges {454152=494439,1646296=735124} 2024-09-20T19:38:57.718 INFO:tasks.rados.rados.0.smithi042.stdout:3910: writing smithi04268048-37 from 454152 to 948591 tid 1 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3910: writing smithi04268048-37 from 1646296 to 2381420 tid 2 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3904: done (6 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3905: done (5 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3906: done (4 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3907: done (3 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3908: done (2 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3909: done (1 left) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3911: read oid 40 snap -1 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3911: expect (ObjNum 1307 snap 419 seq_num 1307) 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3912: copy_from oid 8 from oid 10 current snap is 432 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3910: finishing write tid 1 to smithi04268048-37 2024-09-20T19:38:57.735 INFO:tasks.rados.rados.0.smithi042.stdout:3913: read oid 2 snap -1 2024-09-20T19:38:57.736 INFO:tasks.rados.rados.0.smithi042.stdout:3913: expect deleted 2024-09-20T19:38:57.736 INFO:tasks.rados.rados.0.smithi042.stdout:3914: write oid 30 current snap is 432 2024-09-20T19:38:57.736 INFO:tasks.rados.rados.0.smithi042.stdout:3914: seq_num 1337 ranges {470959=569688,1704778=595135} 2024-09-20T19:38:57.748 INFO:tasks.rados.rados.0.smithi042.stdout:3914: writing smithi04268048-30 from 470959 to 1040647 tid 1 2024-09-20T19:38:57.762 INFO:tasks.rados.rados.0.smithi042.stdout:3914: writing smithi04268048-30 from 1704778 to 2299913 tid 2 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3915: rollback oid 29 current snap is 432 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 29 to 431 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3916: copy_from oid 1 from oid 25 current snap is 432 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: finishing write tid 2 to smithi04268048-37 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: finishing write tid 3 to smithi04268048-37 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: finishing write tid 4 to smithi04268048-37 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: oid 37 updating version 0 to 1180 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: oid 37 version 1180 is already newer than 1178 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: oid 37 version 1180 is already newer than 1179 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1180 (ObjNum 1336 snap 432 seq_num 1336) dirty exists 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3910: left oid 37 (ObjNum 1336 snap 432 seq_num 1336) 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:3911: expect (ObjNum 1307 snap 419 seq_num 1307) 2024-09-20T19:38:57.763 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:57.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:57 smithi162 bash[24557]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:57 smithi162 bash[24557]: cluster 2024-09-20T19:38:56.491715+0000 mgr.y (mgr.24491) 784 : cluster [DBG] pgmap v1540: 178 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 171 active+clean; 106 MiB data, 689 MiB used, 625 GiB / 626 GiB avail; 853 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2024-09-20T19:38:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:57 smithi162 bash[24557]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:57 smithi162 bash[24557]: cluster 2024-09-20T19:38:56.495486+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2024-09-20T19:38:57.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:57.847 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1346159 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:3914: finishing write tid 1 to smithi04268048-30 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:3912: finishing copy_from racing read to smithi04268048-8 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:3912: finishing copy_from to smithi04268048-8 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 8 v 1148 (ObjNum 1244 snap 401 seq_num 1244) dirty exists 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:3915: finishing rollback tid 0 to smithi04268048-29 2024-09-20T19:38:57.876 INFO:tasks.rados.rados.0.smithi042.stdout:3915: finishing rollback tid 1 to smithi04268048-29 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: finishing write tid 2 to smithi04268048-30 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: finishing write tid 3 to smithi04268048-30 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3915: finishing rollback tid 2 to smithi04268048-29 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 29 v 1151 (ObjNum 1267 snap 408 seq_num 1267) dirty exists 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: finishing write tid 4 to smithi04268048-30 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: oid 30 updating version 0 to 1630 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: oid 30 version 1630 is already newer than 1629 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: oid 30 version 1630 is already newer than 1628 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1630 (ObjNum 1337 snap 432 seq_num 1337) dirty exists 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: left oid 30 (ObjNum 1337 snap 432 seq_num 1337) 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3910: done (6 left) 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3911: done (5 left) 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3912: done (4 left) 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3913: done (3 left) 2024-09-20T19:38:57.877 INFO:tasks.rados.rados.0.smithi042.stdout:3914: done (2 left) 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:3915: done (1 left) 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:3917: write oid 9 current snap is 432 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:3916: finishing copy_from racing read to smithi04268048-1 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:3916: finishing copy_from to smithi04268048-1 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 1 v 1268 (ObjNum 1251 snap 403 seq_num 1251) dirty exists 2024-09-20T19:38:57.878 INFO:tasks.rados.rados.0.smithi042.stdout:3917: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-20T19:38:57.888 INFO:tasks.rados.rados.0.smithi042.stdout:3917: writing smithi04268048-9 from 487766 to 1132703 tid 1 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3917: writing smithi04268048-9 from 1646907 to 2385700 tid 2 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3916: done (1 left) 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3918: read oid 31 snap -1 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3918: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3919: write oid 18 current snap is 432 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3917: finishing write tid 1 to smithi04268048-9 2024-09-20T19:38:57.903 INFO:tasks.rados.rados.0.smithi042.stdout:3919: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-20T19:38:57.914 INFO:tasks.rados.rados.0.smithi042.stdout:3919: writing smithi04268048-18 from 504573 to 941112 tid 1 2024-09-20T19:38:57.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:57.926 INFO:tasks.rados.rados.0.smithi042.stdout:3919: writing smithi04268048-18 from 1421742 to 2020546 tid 2 2024-09-20T19:38:57.931 INFO:tasks.rados.rados.0.smithi042.stdout:3919: writing smithi04268048-18 from 2437505 to 2504573 tid 3 2024-09-20T19:38:57.931 INFO:tasks.rados.rados.0.smithi042.stdout:3920: snap_remove snap 427 2024-09-20T19:38:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:57.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:58.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:58.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:38:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:38:58.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:58.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.336 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:58.439 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3920: done (3 left) 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3921: read oid 39 snap -1 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3921: expect (ObjNum 1331 snap 430 seq_num 1331) 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3922: copy_from oid 14 from oid 22 current snap is 432 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3917: finishing write tid 2 to smithi04268048-9 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3917: finishing write tid 3 to smithi04268048-9 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3917: finishing write tid 4 to smithi04268048-9 2024-09-20T19:38:58.487 INFO:tasks.rados.rados.0.smithi042.stdout:3917: oid 9 updating version 0 to 1225 2024-09-20T19:38:58.488 INFO:tasks.rados.rados.0.smithi042.stdout:3917: oid 9 updating version 1225 to 1226 2024-09-20T19:38:58.488 INFO:tasks.rados.rados.0.smithi042.stdout:3917: oid 9 version 1226 is already newer than 1224 2024-09-20T19:38:58.488 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1226 (ObjNum 1338 snap 432 seq_num 1338) dirty exists 2024-09-20T19:38:58.488 INFO:tasks.rados.rados.0.smithi042.stdout:3917: left oid 9 (ObjNum 1338 snap 432 seq_num 1338) 2024-09-20T19:38:58.488 INFO:tasks.rados.rados.0.smithi042.stdout:3918: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:38:58.513 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2457359 2024-09-20T19:38:58.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:58.588 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:38:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: finishing write tid 1 to smithi04268048-18 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: finishing write tid 2 to smithi04268048-18 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: finishing write tid 3 to smithi04268048-18 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: finishing write tid 4 to smithi04268048-18 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: finishing write tid 5 to smithi04268048-18 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: oid 18 updating version 0 to 1631 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: oid 18 updating version 1631 to 1634 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: oid 18 version 1634 is already newer than 1633 2024-09-20T19:38:58.637 INFO:tasks.rados.rados.0.smithi042.stdout:3919: oid 18 version 1634 is already newer than 1632 2024-09-20T19:38:58.638 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1634 (ObjNum 1339 snap 432 seq_num 1339) dirty exists 2024-09-20T19:38:58.638 INFO:tasks.rados.rados.0.smithi042.stdout:3919: left oid 18 (ObjNum 1339 snap 432 seq_num 1339) 2024-09-20T19:38:58.638 INFO:tasks.rados.rados.0.smithi042.stdout:3921: expect (ObjNum 1331 snap 430 seq_num 1331) 2024-09-20T19:38:58.638 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:38:58.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:58.717 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:3922: finishing copy_from racing read to smithi04268048-14 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:3922: finishing copy_from to smithi04268048-14 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 14 v 1170 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:3917: done (4 left) 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:3918: done (3 left) 2024-09-20T19:38:58.736 INFO:tasks.rados.rados.0.smithi042.stdout:3919: done (2 left) 2024-09-20T19:38:58.737 INFO:tasks.rados.rados.0.smithi042.stdout:3921: done (1 left) 2024-09-20T19:38:58.737 INFO:tasks.rados.rados.0.smithi042.stdout:3922: done (0 left) 2024-09-20T19:38:58.737 INFO:tasks.rados.rados.0.smithi042.stdout:3923: write oid 46 current snap is 432 2024-09-20T19:38:58.737 INFO:tasks.rados.rados.0.smithi042.stdout:3923: seq_num 1340 ranges {521380=511788,1480224=742462} 2024-09-20T19:38:58.742 INFO:tasks.rados.rados.0.smithi042.stdout:3923: writing smithi04268048-46 from 521380 to 1033168 tid 1 2024-09-20T19:38:58.749 INFO:tasks.rados.rados.0.smithi042.stdout:3923: writing smithi04268048-46 from 1480224 to 2222686 tid 2 2024-09-20T19:38:58.749 INFO:tasks.rados.rados.0.smithi042.stdout:3924: rollback oid 45 current snap is 432 2024-09-20T19:38:58.749 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 431 2024-09-20T19:38:58.749 INFO:tasks.rados.rados.0.smithi042.stdout:3923: finishing write tid 1 to smithi04268048-46 2024-09-20T19:38:58.749 INFO:tasks.rados.rados.0.smithi042.stdout:3925: snap_remove snap 426 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[22706]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[22706]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[32075]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[32075]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.770 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:58 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:58 smithi162 bash[24557]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:58 smithi162 bash[24557]: cluster 2024-09-20T19:38:57.514395+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2024-09-20T19:38:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:58 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:58 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.481569+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2024-09-20T19:38:58.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:58.848 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:38:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:38:58.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:38:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:58.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:38:59.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:59.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:38:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:38:59.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:59.340 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:59.471 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:59.497 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:38:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:38:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:3925: done (2 left) 2024-09-20T19:38:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:3926: snap_create 2024-09-20T19:38:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:3924: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:38:59.537 INFO:tasks.rados.rados.0.smithi042.stdout:3923: finishing write tid 2 to smithi04268048-46 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3924: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: finishing write tid 3 to smithi04268048-46 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3924: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1654 (ObjNum 1304 snap 418 seq_num 1304) dirty exists 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: finishing write tid 4 to smithi04268048-46 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: oid 46 updating version 0 to 1284 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: oid 46 version 1284 is already newer than 1283 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: oid 46 version 1284 is already newer than 1282 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 46 v 1284 (ObjNum 1340 snap 432 seq_num 1340) dirty exists 2024-09-20T19:38:59.538 INFO:tasks.rados.rados.0.smithi042.stdout:3923: left oid 46 (ObjNum 1340 snap 432 seq_num 1340) 2024-09-20T19:38:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:38:59.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[32075]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:38:59 smithi042 bash[22706]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:59.757 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:38:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:38:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.492454+0000 mgr.y (mgr.24491) 785 : cluster [DBG] pgmap v1543: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 701 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2024-09-20T19:38:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: audit 2024-09-20T19:38:58.664261+0000 mon.a (mon.0) 2684 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:38:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.730963+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:38:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[24557]: cluster 2024-09-20T19:38:58.733003+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:38:59.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:38:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:38:59.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:38:59.987 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:00.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:00.238 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:39:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:39:00.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:00.256 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:39:00.256 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:39:00.258 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:00.284 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:39:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:39:00.301 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:39:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:39:00.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:38:59 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:00.323 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:39:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:39:00.344 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:39:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:39:00.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:00.366 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:39:00.366 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:39:00.387 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:39:00.387 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:39:00.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:00.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:00.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3923: done (2 left) 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3924: done (1 left) 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3926: done (0 left) 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3927: copy_from oid 12 from oid 17 current snap is 433 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3928: read oid 27 snap 432 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3928: expect (ObjNum 1326 snap 426 seq_num 1326) 2024-09-20T19:39:00.535 INFO:tasks.rados.rados.0.smithi042.stdout:3929: snap_remove snap 431 2024-09-20T19:39:00.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:00.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:00 smithi162 bash[24557]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:00 smithi162 bash[24557]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.633 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:00 smithi162 bash[24557]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:00.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:00 smithi162 bash[24557]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:00.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:00.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:00.777 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:00.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:00.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:00.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:00.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[32075]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[32075]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[32075]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:00.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[32075]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[22706]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[22706]: cluster 2024-09-20T19:38:59.531140+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2024-09-20T19:39:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[22706]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:00 smithi042 bash[22706]: audit 2024-09-20T19:39:00.217760+0000 mon.a (mon.0) 2686 : audit [DBG] from='client.? v1:172.21.15.42:0/2717523286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:01.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.035 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:01.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.165 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:01.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:01.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:01.373 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:39:01.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:01.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:01.523 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:01.551 INFO:tasks.rados.rados.0.smithi042.stdout:3929: done (2 left) 2024-09-20T19:39:01.551 INFO:tasks.rados.rados.0.smithi042.stdout:3930: delete oid 10 current snap is 433 2024-09-20T19:39:01.551 INFO:tasks.rados.rados.0.smithi042.stdout:3928: expect (ObjNum 1326 snap 426 seq_num 1326) 2024-09-20T19:39:01.551 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:01.557 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:01.593 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:01.633 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:01.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3927: finishing copy_from racing read to smithi04268048-12 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3927: finishing copy_from to smithi04268048-12 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 12 v 1167 (ObjNum 1270 snap 408 seq_num 1270) dirty exists 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3927: done (2 left) 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3928: done (1 left) 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3930: done (0 left) 2024-09-20T19:39:01.672 INFO:tasks.rados.rados.0.smithi042.stdout:3931: snap_create 2024-09-20T19:39:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:01.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.789 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:01.819 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: audit 2024-09-20T19:38:59.958891+0000 mgr.y (mgr.24491) 786 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:00.493167+0000 mgr.y (mgr.24491) 787 : cluster [DBG] pgmap v1545: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 117 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:00.532611+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2024-09-20T19:39:01.922 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:01.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:02.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:02.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:02.183 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:02.211 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:02.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.341 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:02.444 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.472 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:02.563 INFO:tasks.rados.rados.0.smithi042.stdout:3931: done (0 left) 2024-09-20T19:39:02.564 INFO:tasks.rados.rados.0.smithi042.stdout:3932: read oid 33 snap -1 2024-09-20T19:39:02.564 INFO:tasks.rados.rados.0.smithi042.stdout:3932: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:02.564 INFO:tasks.rados.rados.0.smithi042.stdout:3933: delete oid 49 current snap is 434 2024-09-20T19:39:02.570 INFO:tasks.rados.rados.0.smithi042.stdout:3933: done (1 left) 2024-09-20T19:39:02.570 INFO:tasks.rados.rados.0.smithi042.stdout:3934: read oid 23 snap -1 2024-09-20T19:39:02.570 INFO:tasks.rados.rados.0.smithi042.stdout:3934: expect (ObjNum 1321 snap 426 seq_num 1321) 2024-09-20T19:39:02.570 INFO:tasks.rados.rados.0.smithi042.stdout:3935: write (excl) oid 28 current snap is 434 2024-09-20T19:39:02.570 INFO:tasks.rados.rados.0.smithi042.stdout:3935: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-20T19:39:02.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:02.578 INFO:tasks.rados.rados.0.smithi042.stdout:3935: writing smithi04268048-28 from 571801 to 1309336 tid 1 2024-09-20T19:39:02.585 INFO:tasks.rados.rados.0.smithi042.stdout:3935: writing smithi04268048-28 from 1939317 to 2429106 tid 2 2024-09-20T19:39:02.585 INFO:tasks.rados.rados.0.smithi042.stdout:3936: write (excl) oid 13 current snap is 434 2024-09-20T19:39:02.585 INFO:tasks.rados.rados.0.smithi042.stdout:3936: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-20T19:39:02.591 INFO:tasks.rados.rados.0.smithi042.stdout:3936: writing smithi04268048-13 from 588608 to 1001392 tid 1 2024-09-20T19:39:02.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:02.605 INFO:tasks.rados.rados.0.smithi042.stdout:3936: writing smithi04268048-13 from 1597799 to 2347599 tid 2 2024-09-20T19:39:02.605 INFO:tasks.rados.rados.0.smithi042.stdout:3937: snap_create 2024-09-20T19:39:02.605 INFO:tasks.rados.rados.0.smithi042.stdout:3934: expect (ObjNum 1321 snap 426 seq_num 1321) 2024-09-20T19:39:02.691 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1019878 2024-09-20T19:39:02.703 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.704 INFO:tasks.rados.rados.0.smithi042.stdout:3932: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:02.727 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 447631 2024-09-20T19:39:02.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: finishing write tid 1 to smithi04268048-28 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: finishing write tid 2 to smithi04268048-28 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: finishing write tid 3 to smithi04268048-28 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: finishing write tid 4 to smithi04268048-28 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: oid 28 updating version 0 to 1460 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: oid 28 updating version 1460 to 1461 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: oid 28 version 1461 is already newer than 1459 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1461 (ObjNum 1343 snap 434 seq_num 1343) dirty exists 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3935: left oid 28 (ObjNum 1343 snap 434 seq_num 1343) 2024-09-20T19:39:02.791 INFO:tasks.rados.rados.0.smithi042.stdout:3936: finishing write tid 1 to smithi04268048-13 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: finishing write tid 2 to smithi04268048-13 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: finishing write tid 3 to smithi04268048-13 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: finishing write tid 4 to smithi04268048-13 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: oid 13 updating version 0 to 1185 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: oid 13 updating version 1185 to 1187 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: oid 13 version 1187 is already newer than 1186 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 1187 (ObjNum 1344 snap 434 seq_num 1344) dirty exists 2024-09-20T19:39:02.792 INFO:tasks.rados.rados.0.smithi042.stdout:3936: left oid 13 (ObjNum 1344 snap 434 seq_num 1344) 2024-09-20T19:39:02.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:02 smithi162 bash[24557]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:02 smithi162 bash[24557]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:02 smithi042 bash[32075]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.831 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:02 smithi042 bash[32075]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:02 smithi042 bash[22706]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:02 smithi042 bash[22706]: cluster 2024-09-20T19:39:01.539987+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2024-09-20T19:39:02.833 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:02.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:02.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:02.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:03.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:39:03.089 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:39:03.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:03.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:03.239 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:03.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:03.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:03.487 INFO:tasks.rados.rados.0.smithi042.stdout:3932: done (4 left) 2024-09-20T19:39:03.487 INFO:tasks.rados.rados.0.smithi042.stdout:3934: done (3 left) 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3935: done (2 left) 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3936: done (1 left) 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3937: done (0 left) 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3938: read oid 15 snap 432 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3938: expect (ObjNum 1335 snap 432 seq_num 1335) 2024-09-20T19:39:03.488 INFO:tasks.rados.rados.0.smithi042.stdout:3939: snap_remove snap 428 2024-09-20T19:39:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:03.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:03 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:03.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:02.494251+0000 mgr.y (mgr.24491) 788 : cluster [DBG] pgmap v1548: 178 pgs: 7 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 164 active+clean; 131 MiB data, 763 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 24 op/s 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:02.561803+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:03 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.485288+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2024-09-20T19:39:03.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:03.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:03.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:03.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:03.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T19:39:03.999 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:04.024 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:04.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:04.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:04.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:04.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:04.406 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:04.488 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3939: done (1 left) 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3940: rollback oid 45 current snap is 435 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 45 to 430 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3941: read oid 50 snap -1 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3941: expect (ObjNum 1334 snap 432 seq_num 1334) 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3942: read oid 2 snap -1 2024-09-20T19:39:04.501 INFO:tasks.rados.rados.0.smithi042.stdout:3942: expect deleted 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3943: read oid 33 snap 433 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3943: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3944: read oid 22 snap -1 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3944: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3945: write (excl) oid 5 current snap is 435 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3945: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-20T19:39:04.502 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:39:04.502 INFO:tasks.rados.rados.0.smithi042.stdout:3945: writing smithi04268048-5 from 605415 to 1093448 tid 1 2024-09-20T19:39:04.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:04.511 INFO:tasks.rados.rados.0.smithi042.stdout:3945: writing smithi04268048-5 from 1656281 to 2149739 tid 2 2024-09-20T19:39:04.512 INFO:tasks.rados.rados.0.smithi042.stdout:3946: write oid 37 current snap is 435 2024-09-20T19:39:04.512 INFO:tasks.rados.rados.0.smithi042.stdout:3946: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-20T19:39:04.520 INFO:tasks.rados.rados.0.smithi042.stdout:3946: writing smithi04268048-37 from 622222 to 1301857 tid 1 2024-09-20T19:39:04.530 INFO:tasks.rados.rados.0.smithi042.stdout:3946: writing smithi04268048-37 from 1831116 to 2584585 tid 2 2024-09-20T19:39:04.530 INFO:tasks.rados.rados.0.smithi042.stdout:3947: read oid 8 snap 433 2024-09-20T19:39:04.530 INFO:tasks.rados.rados.0.smithi042.stdout:3947: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:39:04.530 INFO:tasks.rados.rados.0.smithi042.stdout:3948: write (excl) oid 23 current snap is 435 2024-09-20T19:39:04.530 INFO:tasks.rados.rados.0.smithi042.stdout:3948: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-20T19:39:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:04.539 INFO:tasks.rados.rados.0.smithi042.stdout:3948: writing smithi04268048-23 from 639029 to 1393913 tid 1 2024-09-20T19:39:04.547 INFO:tasks.rados.rados.0.smithi042.stdout:3948: writing smithi04268048-23 from 2173245 to 2639029 tid 2 2024-09-20T19:39:04.547 INFO:tasks.rados.rados.0.smithi042.stdout:3949: delete oid 3 current snap is 435 2024-09-20T19:39:04.547 INFO:tasks.rados.rados.0.smithi042.stdout:3938: expect (ObjNum 1335 snap 432 seq_num 1335) 2024-09-20T19:39:04.547 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:04.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:04.646 INFO:tasks.rados.rados.0.smithi042.stdout:3940: finishing rollback tid 0 to smithi04268048-45 2024-09-20T19:39:04.646 INFO:tasks.rados.rados.0.smithi042.stdout:3940: finishing rollback tid 1 to smithi04268048-45 2024-09-20T19:39:04.646 INFO:tasks.rados.rados.0.smithi042.stdout:3940: finishing rollback tid 2 to smithi04268048-45 2024-09-20T19:39:04.647 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 45 v 1665 (ObjNum 1304 snap 418 seq_num 1304) dirty exists 2024-09-20T19:39:04.647 INFO:tasks.rados.rados.0.smithi042.stdout:3944: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:04.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:04.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:04 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:04.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:04.788 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1199750 2024-09-20T19:39:04.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:04.839 INFO:tasks.rados.rados.0.smithi042.stdout:3945: finishing write tid 1 to smithi04268048-5 2024-09-20T19:39:04.840 INFO:tasks.rados.rados.0.smithi042.stdout:3943: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:04.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:04.904 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:04.928 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:04.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:04.937 INFO:tasks.rados.rados.0.smithi042.stdout:3941: expect (ObjNum 1334 snap 432 seq_num 1334) 2024-09-20T19:39:05.016 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2191075 2024-09-20T19:39:05.018 INFO:tasks.rados.rados.0.smithi042.stdout:3945: finishing write tid 2 to smithi04268048-5 2024-09-20T19:39:05.018 INFO:tasks.rados.rados.0.smithi042.stdout:3945: finishing write tid 3 to smithi04268048-5 2024-09-20T19:39:05.018 INFO:tasks.rados.rados.0.smithi042.stdout:3945: finishing write tid 4 to smithi04268048-5 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3945: oid 5 updating version 0 to 1608 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3945: oid 5 updating version 1608 to 1609 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3945: oid 5 version 1609 is already newer than 1607 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 5 v 1609 (ObjNum 1345 snap 435 seq_num 1345) dirty exists 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3945: left oid 5 (ObjNum 1345 snap 435 seq_num 1345) 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3946: finishing write tid 1 to smithi04268048-37 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3946: finishing write tid 2 to smithi04268048-37 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3946: finishing write tid 3 to smithi04268048-37 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3938: done (10 left) 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3940: done (9 left) 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3941: done (8 left) 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3942: done (7 left) 2024-09-20T19:39:05.019 INFO:tasks.rados.rados.0.smithi042.stdout:3943: done (6 left) 2024-09-20T19:39:05.020 INFO:tasks.rados.rados.0.smithi042.stdout:3944: done (5 left) 2024-09-20T19:39:05.020 INFO:tasks.rados.rados.0.smithi042.stdout:3945: done (4 left) 2024-09-20T19:39:05.020 INFO:tasks.rados.rados.0.smithi042.stdout:3949: done (3 left) 2024-09-20T19:39:05.020 INFO:tasks.rados.rados.0.smithi042.stdout:3950: snap_remove snap 432 2024-09-20T19:39:05.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.049 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:05.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:03.733791+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.42:0/1262081944' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:03.734352+0000 mon.a (mon.0) 2691 : audit [INF] from='client.26141 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.090927+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.42:0/885330996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.091531+0000 mon.a (mon.0) 2692 : audit [INF] from='client.26147 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.480849+0000 mon.a (mon.0) 2693 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.482820+0000 mon.a (mon.0) 2694 : audit [INF] from='client.26141 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: audit 2024-09-20T19:39:04.482986+0000 mon.a (mon.0) 2695 : audit [INF] from='client.26147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:04 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.491597+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e1036: 8 total, 8 up, 7 in 2024-09-20T19:39:05.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:05.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.308 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:05.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:05.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:05.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:39:05.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:05.574 INFO:tasks.rados.rados.0.smithi042.stdout:3950: done (3 left) 2024-09-20T19:39:05.575 INFO:tasks.rados.rados.0.smithi042.stdout:3951: snap_remove snap 434 2024-09-20T19:39:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:05.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[32075]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:05 smithi042 bash[22706]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:05.693 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.655763+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:03.657468+0000 osd.2 (osd.2) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.494725+0000 mgr.y (mgr.24491) 789 : cluster [DBG] pgmap v1552: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 27 op/s 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.838587+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:05 smithi162 bash[24557]: cluster 2024-09-20T19:39:04.841299+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:39:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:05.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:05.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:05.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.967 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:05.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:06.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:06.138 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T19:39:06.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:39:06.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:06.224 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:06.327 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:06.351 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:06.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:06.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:06.569 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:06.583 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:06.598 INFO:tasks.rados.rados.0.smithi042.stdout:3951: done (3 left) 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3952: read oid 44 snap 433 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3952: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3953: read oid 25 snap -1 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3953: expect (ObjNum 1251 snap 403 seq_num 1251) 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3954: snap_create 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3946: finishing write tid 4 to smithi04268048-37 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3946: oid 37 updating version 0 to 1182 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3946: oid 37 updating version 1182 to 1183 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3946: oid 37 version 1183 is already newer than 1181 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 37 v 1183 (ObjNum 1346 snap 435 seq_num 1346) dirty exists 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3946: left oid 37 (ObjNum 1346 snap 435 seq_num 1346) 2024-09-20T19:39:06.599 INFO:tasks.rados.rados.0.smithi042.stdout:3947: expect (ObjNum 1244 snap 401 seq_num 1244) 2024-09-20T19:39:06.611 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:06.621 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2767233 2024-09-20T19:39:06.628 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:06.714 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:06.715 INFO:tasks.rados.rados.0.smithi042.stdout:3948: finishing write tid 1 to smithi04268048-23 2024-09-20T19:39:06.715 INFO:tasks.rados.rados.0.smithi042.stdout:3948: finishing write tid 2 to smithi04268048-23 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: finishing write tid 3 to smithi04268048-23 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: finishing write tid 4 to smithi04268048-23 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: oid 23 updating version 0 to 1190 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: oid 23 version 1190 is already newer than 1189 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: oid 23 updating version 1190 to 1191 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 23 v 1191 (ObjNum 1347 snap 435 seq_num 1347) dirty exists 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3948: left oid 23 (ObjNum 1347 snap 435 seq_num 1347) 2024-09-20T19:39:06.716 INFO:tasks.rados.rados.0.smithi042.stdout:3952: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:06.756 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2805224 2024-09-20T19:39:06.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:06 smithi162 bash[24557]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.843 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:06.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[22706]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.854 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.441185+0000 osd.5 (osd.5) 321 : cluster [DBG] 6.a scrub starts 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.442712+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.a scrub ok 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.569305+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.794839+0000 osd.4 (osd.4) 189 : cluster [DBG] 6.e scrub starts 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.855 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:06 smithi042 bash[32075]: cluster 2024-09-20T19:39:05.796897+0000 osd.4 (osd.4) 190 : cluster [DBG] 6.e scrub ok 2024-09-20T19:39:06.867 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:06.902 INFO:tasks.rados.rados.0.smithi042.stdout:3953: expect (ObjNum 1251 snap 403 seq_num 1251) 2024-09-20T19:39:06.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:06.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:06.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:07.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.124 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:07.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:07.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:07.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:07.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:07.598 INFO:tasks.rados.rados.0.smithi042.stdout:3946: done (5 left) 2024-09-20T19:39:07.598 INFO:tasks.rados.rados.0.smithi042.stdout:3947: done (4 left) 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3948: done (3 left) 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3952: done (2 left) 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3953: done (1 left) 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3954: done (0 left) 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3955: write oid 35 current snap is 436 2024-09-20T19:39:07.599 INFO:tasks.rados.rados.0.smithi042.stdout:3955: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-20T19:39:07.617 INFO:tasks.rados.rados.0.smithi042.stdout:3955: writing smithi04268048-35 from 672643 to 1178025 tid 1 2024-09-20T19:39:07.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:3955: writing smithi04268048-35 from 1890209 to 2391005 tid 2 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:3956: read oid 33 snap -1 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:3956: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:3957: rollback oid 48 current snap is 436 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 48 to 435 2024-09-20T19:39:07.638 INFO:tasks.rados.rados.0.smithi042.stdout:3955: finishing write tid 1 to smithi04268048-35 2024-09-20T19:39:07.639 INFO:tasks.rados.rados.0.smithi042.stdout:3958: delete oid 12 current snap is 436 2024-09-20T19:39:07.643 INFO:tasks.rados.rados.0.smithi042.stdout:3955: finishing write tid 2 to smithi04268048-35 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: finishing write tid 3 to smithi04268048-35 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: finishing write tid 4 to smithi04268048-35 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: oid 35 updating version 0 to 1259 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: oid 35 version 1259 is already newer than 1258 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: oid 35 version 1259 is already newer than 1257 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 35 v 1259 (ObjNum 1349 snap 436 seq_num 1349) dirty exists 2024-09-20T19:39:07.646 INFO:tasks.rados.rados.0.smithi042.stdout:3955: left oid 35 (ObjNum 1349 snap 436 seq_num 1349) 2024-09-20T19:39:07.649 INFO:tasks.rados.rados.0.smithi042.stdout:3955: done (3 left) 2024-09-20T19:39:07.649 INFO:tasks.rados.rados.0.smithi042.stdout:3958: done (2 left) 2024-09-20T19:39:07.650 INFO:tasks.rados.rados.0.smithi042.stdout:3959: snap_remove snap 430 2024-09-20T19:39:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:07.753 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:07.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:07 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:07 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:07 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:07 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.882 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:07.893 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.495530+0000 mgr.y (mgr.24491) 790 : cluster [DBG] pgmap v1554: 178 pgs: 7 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 163 active+clean; 135 MiB data, 767 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.894 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:07 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.585501+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2024-09-20T19:39:07.909 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:07.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:08.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:08.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:08.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:39:08.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:08.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:08.296 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:08.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:08.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:08.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:08.606 INFO:tasks.rados.rados.0.smithi042.stdout:3959: done (2 left) 2024-09-20T19:39:08.606 INFO:tasks.rados.rados.0.smithi042.stdout:3960: rollback oid 50 current snap is 436 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 50 to 435 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3961: read oid 17 snap -1 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3961: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3962: read oid 21 snap 435 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3962: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3963: snap_create 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3957: finishing rollback tid 0 to smithi04268048-48 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3957: finishing rollback tid 1 to smithi04268048-48 2024-09-20T19:39:08.607 INFO:tasks.rados.rados.0.smithi042.stdout:3956: expect (ObjNum 1316 snap 424 seq_num 1316) 2024-09-20T19:39:08.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:08 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:08.703 INFO:tasks.rados.rados.0.smithi042.stdout:3957: finishing rollback tid 2 to smithi04268048-48 2024-09-20T19:39:08.703 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 48 v 1618 (ObjNum 1236 snap 401 seq_num 1236) dirty exists 2024-09-20T19:39:08.703 INFO:tasks.rados.rados.0.smithi042.stdout:3962: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:39:08.710 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 1604680 2024-09-20T19:39:08.738 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:39:08.782 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:08.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:08.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:08.825 INFO:tasks.rados.rados.0.smithi042.stdout:3960: finishing rollback tid 0 to smithi04268048-50 2024-09-20T19:39:08.825 INFO:tasks.rados.rados.0.smithi042.stdout:3960: finishing rollback tid 1 to smithi04268048-50 2024-09-20T19:39:08.825 INFO:tasks.rados.rados.0.smithi042.stdout:3960: finishing rollback tid 2 to smithi04268048-50 2024-09-20T19:39:08.825 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 50 v 1236 (ObjNum 1334 snap 432 seq_num 1334) dirty exists 2024-09-20T19:39:08.825 INFO:tasks.rados.rados.0.smithi042.stdout:3961: expect (ObjNum 1270 snap 408 seq_num 1270) 2024-09-20T19:39:08.826 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:08.891 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:08.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.908367+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:06.910809+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.493659+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.498273+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.924 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:08 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.595186+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2024-09-20T19:39:08.934 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:08.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:08.957 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:09.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:09.064 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:09.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:09.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:09.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:09.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:09.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:09.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:09.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:09.580 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:09.601 INFO:tasks.rados.rados.0.smithi042.stdout:3956: done (5 left) 2024-09-20T19:39:09.601 INFO:tasks.rados.rados.0.smithi042.stdout:3957: done (4 left) 2024-09-20T19:39:09.601 INFO:tasks.rados.rados.0.smithi042.stdout:3960: done (3 left) 2024-09-20T19:39:09.602 INFO:tasks.rados.rados.0.smithi042.stdout:3961: done (2 left) 2024-09-20T19:39:09.602 INFO:tasks.rados.rados.0.smithi042.stdout:3962: done (1 left) 2024-09-20T19:39:09.602 INFO:tasks.rados.rados.0.smithi042.stdout:3963: done (0 left) 2024-09-20T19:39:09.602 INFO:tasks.rados.rados.0.smithi042.stdout:3964: write (excl) oid 24 current snap is 437 2024-09-20T19:39:09.602 INFO:tasks.rados.rados.0.smithi042.stdout:3964: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-20T19:39:09.624 INFO:tasks.rados.rados.0.smithi042.stdout:3964: writing smithi04268048-24 from 706257 to 1362137 tid 1 2024-09-20T19:39:09.642 INFO:tasks.rados.rados.0.smithi042.stdout:3964: writing smithi04268048-24 from 1890820 to 2395285 tid 2 2024-09-20T19:39:09.642 INFO:tasks.rados.rados.0.smithi042.stdout:3965: read oid 21 snap -1 2024-09-20T19:39:09.643 INFO:tasks.rados.rados.0.smithi042.stdout:3965: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:39:09.643 INFO:tasks.rados.rados.0.smithi042.stdout:3966: snap_remove snap 435 2024-09-20T19:39:09.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:09.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.694 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.695 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:09.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:09.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.877549+0000 osd.2 (osd.2) 329 : cluster [DBG] 7.8 scrub starts 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:07.884687+0000 osd.2 (osd.2) 330 : cluster [DBG] 7.8 scrub ok 2024-09-20T19:39:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.496709+0000 mgr.y (mgr.24491) 791 : cluster [DBG] pgmap v1557: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 885 KiB/s wr, 22 op/s 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.598318+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.847848+0000 osd.1 (osd.1) 355 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:08.849731+0000 osd.1 (osd.1) 356 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:09 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.599282+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2024-09-20T19:39:09.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:09.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:09.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:10.070 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:10.100 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:10.203 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:10.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:39:09 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:10.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:10.314 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:10.315 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:10.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:10.348 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:39:10.349 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:39:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:10.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:10.484 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:10.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:10.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:10.634 INFO:tasks.rados.rados.0.smithi042.stdout:3966: done (2 left) 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3967: write (excl) oid 39 current snap is 437 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: finishing write tid 1 to smithi04268048-24 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: finishing write tid 2 to smithi04268048-24 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: finishing write tid 3 to smithi04268048-24 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: finishing write tid 4 to smithi04268048-24 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: oid 24 updating version 0 to 1596 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: oid 24 updating version 1596 to 1598 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: oid 24 version 1598 is already newer than 1597 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1598 (ObjNum 1351 snap 437 seq_num 1351) dirty exists 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3964: left oid 24 (ObjNum 1351 snap 437 seq_num 1351) 2024-09-20T19:39:10.635 INFO:tasks.rados.rados.0.smithi042.stdout:3965: expect (ObjNum 1236 snap 401 seq_num 1236) 2024-09-20T19:39:10.717 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2267591 2024-09-20T19:39:10.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:10.729 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[22706]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.731 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.731 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.731 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:10 smithi042 bash[32075]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.741 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:10.752 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 2387988 2024-09-20T19:39:10.761 INFO:tasks.rados.rados.0.smithi042.stdout:3967: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-20T19:39:10.770 INFO:tasks.rados.rados.0.smithi042.stdout:3967: writing smithi04268048-39 from 723064 to 1454193 tid 1 2024-09-20T19:39:10.779 INFO:tasks.rados.rados.0.smithi042.stdout:3967: writing smithi04268048-39 from 1949302 to 2713778 tid 2 2024-09-20T19:39:10.779 INFO:tasks.rados.rados.0.smithi042.stdout:3964: done (2 left) 2024-09-20T19:39:10.779 INFO:tasks.rados.rados.0.smithi042.stdout:3965: done (1 left) 2024-09-20T19:39:10.779 INFO:tasks.rados.rados.0.smithi042.stdout:3968: copy_from oid 7 from oid 43 current snap is 437 2024-09-20T19:39:10.779 INFO:tasks.rados.rados.0.smithi042.stdout:3969: snap_create 2024-09-20T19:39:10.781 INFO:tasks.rados.rados.0.smithi042.stdout:3968: finishing copy_from to smithi04268048-7 2024-09-20T19:39:10.781 INFO:tasks.rados.rados.0.smithi042.stdout:3968: got expected ENOENT (src dne) 2024-09-20T19:39:10.781 INFO:tasks.rados.rados.0.smithi042.stdout:3968: finishing copy_from racing read to smithi04268048-7 2024-09-20T19:39:10.790 INFO:tasks.rados.rados.0.smithi042.stdout:3967: finishing write tid 1 to smithi04268048-39 2024-09-20T19:39:10.790 INFO:tasks.rados.rados.0.smithi042.stdout:3967: finishing write tid 2 to smithi04268048-39 2024-09-20T19:39:10.790 INFO:tasks.rados.rados.0.smithi042.stdout:3967: finishing write tid 3 to smithi04268048-39 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:3967: finishing write tid 4 to smithi04268048-39 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:3967: oid 39 updating version 0 to 1276 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:3967: oid 39 updating version 1276 to 1277 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:3967: oid 39 version 1277 is already newer than 1275 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 39 v 1277 (ObjNum 1352 snap 437 seq_num 1352) dirty exists 2024-09-20T19:39:10.791 INFO:tasks.rados.rados.0.smithi042.stdout:3967: left oid 39 (ObjNum 1352 snap 437 seq_num 1352) 2024-09-20T19:39:10.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.809923+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:39:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.812310+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:39:10.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.968093+0000 osd.2 (osd.2) 331 : cluster [DBG] 7.a scrub starts 2024-09-20T19:39:10.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:10 smithi162 bash[24557]: cluster 2024-09-20T19:39:09.971004+0000 osd.2 (osd.2) 332 : cluster [DBG] 7.a scrub ok 2024-09-20T19:39:10.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:10.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:10.975 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:11.004 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:11.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:11.134 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:11.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T19:39:11.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:11.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:11.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:11.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:11.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:3967: done (2 left) 2024-09-20T19:39:11.630 INFO:tasks.rados.rados.0.smithi042.stdout:3968: done (1 left) 2024-09-20T19:39:11.631 INFO:tasks.rados.rados.0.smithi042.stdout:3969: done (0 left) 2024-09-20T19:39:11.631 INFO:tasks.rados.rados.0.smithi042.stdout:3970: snap_remove snap 436 2024-09-20T19:39:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:11.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:11.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:11.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:11 smithi162 bash[24557]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:11.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:11.898 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:11.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[22706]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: audit 2024-09-20T19:39:09.967557+0000 mgr.y (mgr.24491) 792 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: cluster 2024-09-20T19:39:10.497356+0000 mgr.y (mgr.24491) 793 : cluster [DBG] pgmap v1560: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 134 MiB data, 770 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 888 KiB/s wr, 24 op/s 2024-09-20T19:39:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:11 smithi042 bash[32075]: cluster 2024-09-20T19:39:10.628944+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2024-09-20T19:39:12.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:12.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:12.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:12.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:12.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:12.289 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:12.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:12.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:12.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:12.549 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:12.631 INFO:tasks.rados.rados.0.smithi042.stdout:3970: done (0 left) 2024-09-20T19:39:12.632 INFO:tasks.rados.rados.0.smithi042.stdout:3971: copy_from oid 42 from oid 34 current snap is 438 2024-09-20T19:39:12.632 INFO:tasks.rados.rados.0.smithi042.stdout:3972: snap_remove snap 433 2024-09-20T19:39:12.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:12.677 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:12.780 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:12 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:12.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:11.626962+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.347345+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.350090+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:12 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.629026+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2024-09-20T19:39:12.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:12.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:13.040 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:13.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:13.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:13.197 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:13.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:13.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:13.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:13.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:13.506 INFO:tasks.rados.rados.0.smithi042.stdout:3972: done (1 left) 2024-09-20T19:39:13.507 INFO:tasks.rados.rados.0.smithi042.stdout:3973: write oid 9 current snap is 438 2024-09-20T19:39:13.507 INFO:tasks.rados.rados.0.smithi042.stdout:3973: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-20T19:39:13.511 INFO:tasks.rados.rados.0.smithi042.stdout:3973: writing smithi04268048-9 from 739871 to 1146249 tid 1 2024-09-20T19:39:13.523 INFO:tasks.rados.rados.0.smithi042.stdout:3973: writing smithi04268048-9 from 1607784 to 2115918 tid 2 2024-09-20T19:39:13.524 INFO:tasks.rados.rados.0.smithi042.stdout:3974: write (excl) oid 7 current snap is 438 2024-09-20T19:39:13.524 INFO:tasks.rados.rados.0.smithi042.stdout:3974: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-20T19:39:13.542 INFO:tasks.rados.rados.0.smithi042.stdout:3974: writing smithi04268048-7 from 756678 to 1354658 tid 1 2024-09-20T19:39:13.552 INFO:tasks.rados.rados.0.smithi042.stdout:3974: writing smithi04268048-7 from 1782619 to 2550764 tid 2 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3975: delete oid 15 current snap is 438 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3971: finishing copy_from racing read to smithi04268048-42 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3971: finishing copy_from to smithi04268048-42 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 42 v 1256 (ObjNum 1324 snap 426 seq_num 1324) dirty exists 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: finishing write tid 1 to smithi04268048-9 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: finishing write tid 2 to smithi04268048-9 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: finishing write tid 3 to smithi04268048-9 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: finishing write tid 4 to smithi04268048-9 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: oid 9 updating version 0 to 1246 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: oid 9 version 1246 is already newer than 1244 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: oid 9 version 1246 is already newer than 1245 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 9 v 1246 (ObjNum 1353 snap 438 seq_num 1353) dirty exists 2024-09-20T19:39:13.553 INFO:tasks.rados.rados.0.smithi042.stdout:3973: left oid 9 (ObjNum 1353 snap 438 seq_num 1353) 2024-09-20T19:39:13.554 INFO:tasks.rados.rados.0.smithi042.stdout:3974: finishing write tid 1 to smithi04268048-7 2024-09-20T19:39:13.556 INFO:tasks.rados.rados.0.smithi042.stdout:3971: done (3 left) 2024-09-20T19:39:13.556 INFO:tasks.rados.rados.0.smithi042.stdout:3973: done (2 left) 2024-09-20T19:39:13.556 INFO:tasks.rados.rados.0.smithi042.stdout:3975: done (1 left) 2024-09-20T19:39:13.556 INFO:tasks.rados.rados.0.smithi042.stdout:3976: write (excl) oid 13 current snap is 438 2024-09-20T19:39:13.556 INFO:tasks.rados.rados.0.smithi042.stdout:3976: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-09-20T19:39:13.558 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:13.570 INFO:tasks.rados.rados.0.smithi042.stdout:3976: writing smithi04268048-13 from 790292 to 1538770 tid 1 2024-09-20T19:39:13.578 INFO:tasks.rados.rados.0.smithi042.stdout:3976: writing smithi04268048-13 from 2183230 to 2790292 tid 2 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3977: snap_create 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: finishing write tid 2 to smithi04268048-7 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: finishing write tid 3 to smithi04268048-7 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: finishing write tid 4 to smithi04268048-7 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: oid 7 updating version 0 to 1276 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: oid 7 version 1276 is already newer than 1275 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: oid 7 updating version 1276 to 1277 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 7 v 1277 (ObjNum 1354 snap 438 seq_num 1354) dirty exists 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3974: left oid 7 (ObjNum 1354 snap 438 seq_num 1354) 2024-09-20T19:39:13.579 INFO:tasks.rados.rados.0.smithi042.stdout:3976: finishing write tid 1 to smithi04268048-13 2024-09-20T19:39:13.582 INFO:tasks.rados.rados.0.smithi042.stdout:3976: finishing write tid 2 to smithi04268048-13 2024-09-20T19:39:13.582 INFO:tasks.rados.rados.0.smithi042.stdout:3976: finishing write tid 3 to smithi04268048-13 2024-09-20T19:39:13.583 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:3976: finishing write tid 4 to smithi04268048-13 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:3976: oid 13 updating version 0 to 1188 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:3976: oid 13 updating version 1188 to 1190 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:3976: oid 13 version 1190 is already newer than 1189 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 13 v 1190 (ObjNum 1356 snap 438 seq_num 1356) dirty exists 2024-09-20T19:39:13.583 INFO:tasks.rados.rados.0.smithi042.stdout:3976: left oid 13 (ObjNum 1356 snap 438 seq_num 1356) 2024-09-20T19:39:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:13.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.699 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:13 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:13.817 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:13.830 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.250405+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.256591+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.832 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.498801+0000 mgr.y (mgr.24491) 794 : cluster [DBG] pgmap v1563: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 125 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 894 KiB/s wr, 16 op/s 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.812648+0000 osd.4 (osd.4) 191 : cluster [DBG] 7.1 scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:12.816941+0000 osd.4 (osd.4) 192 : cluster [DBG] 7.1 scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.075222+0000 osd.7 (osd.7) 239 : cluster [DBG] 7.c scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.078643+0000 osd.7 (osd.7) 240 : cluster [DBG] 7.c scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.316507+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.318182+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.833 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:13 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.495732+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2024-09-20T19:39:13.845 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:13.949 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:13.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:14.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:14.224 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:39:14.224 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:39:14.243 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:39:14.246 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:39:14.248 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:39:14.248 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:39:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:14.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:14.262 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:39:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:39:14.280 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:39:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:39:14.299 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:39:14.299 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:39:14.321 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:39:14.322 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:39:14.345 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:39:14.345 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:39:14.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:14.368 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:39:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:39:14.369 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:14.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:14.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:14.515 INFO:tasks.rados.rados.0.smithi042.stdout:3974: done (2 left) 2024-09-20T19:39:14.515 INFO:tasks.rados.rados.0.smithi042.stdout:3976: done (1 left) 2024-09-20T19:39:14.515 INFO:tasks.rados.rados.0.smithi042.stdout:3977: done (0 left) 2024-09-20T19:39:14.515 INFO:tasks.rados.rados.0.smithi042.stdout:3978: write oid 6 current snap is 439 2024-09-20T19:39:14.515 INFO:tasks.rados.rados.0.smithi042.stdout:3978: seq_num 1357 ranges {407099=423727,1441712=515472,2582060=225039} 2024-09-20T19:39:14.522 INFO:tasks.rados.rados.0.smithi042.stdout:3978: writing smithi04268048-6 from 407099 to 830826 tid 1 2024-09-20T19:39:14.531 INFO:tasks.rados.rados.0.smithi042.stdout:3978: writing smithi04268048-6 from 1441712 to 1957184 tid 2 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3978: writing smithi04268048-6 from 2582060 to 2807099 tid 3 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3979: read oid 30 snap -1 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3979: expect (ObjNum 1337 snap 432 seq_num 1337) 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3980: copy_from oid 24 from oid 4 current snap is 439 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3978: finishing write tid 1 to smithi04268048-6 2024-09-20T19:39:14.538 INFO:tasks.rados.rados.0.smithi042.stdout:3978: finishing write tid 2 to smithi04268048-6 2024-09-20T19:39:14.539 INFO:tasks.rados.rados.0.smithi042.stdout:3981: read oid 31 snap -1 2024-09-20T19:39:14.539 INFO:tasks.rados.rados.0.smithi042.stdout:3981: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:39:14.539 INFO:tasks.rados.rados.0.smithi042.stdout:3982: snap_remove snap 438 2024-09-20T19:39:14.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:14.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:14.727 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:14.753 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:14 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:14.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.916 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: audit 2024-09-20T19:39:13.692507+0000 mon.a (mon.0) 2706 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.875649+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:13.877269+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: audit 2024-09-20T19:39:14.204781+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.42:0/3784265828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:14 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.512412+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2024-09-20T19:39:14.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:15.013 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:15.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:15.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:15.275 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:15.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:15.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:39:15.522 INFO:tasks.rados.rados.0.smithi042.stdout:3982: done (4 left) 2024-09-20T19:39:15.523 INFO:tasks.rados.rados.0.smithi042.stdout:3983: write oid 18 current snap is 439 2024-09-20T19:39:15.523 INFO:tasks.rados.rados.0.smithi042.stdout:3983: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-20T19:39:15.532 INFO:tasks.rados.rados.0.smithi042.stdout:3983: writing smithi04268048-18 from 423906 to 922882 tid 1 2024-09-20T19:39:15.541 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:15.545 INFO:tasks.rados.rados.0.smithi042.stdout:3983: writing smithi04268048-18 from 1500194 to 2159324 tid 2 2024-09-20T19:39:15.550 INFO:tasks.rados.rados.0.smithi042.stdout:3983: writing smithi04268048-18 from 2609483 to 2823906 tid 3 2024-09-20T19:39:15.551 INFO:tasks.rados.rados.0.smithi042.stdout:3984: snap_create 2024-09-20T19:39:15.551 INFO:tasks.rados.rados.0.smithi042.stdout:3979: expect (ObjNum 1337 snap 432 seq_num 1337) 2024-09-20T19:39:15.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: finishing write tid 3 to smithi04268048-6 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: finishing write tid 4 to smithi04268048-6 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3980: finishing copy_from racing read to smithi04268048-24 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: finishing write tid 5 to smithi04268048-6 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: oid 6 updating version 0 to 1621 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: oid 6 version 1621 is already newer than 1619 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: oid 6 version 1621 is already newer than 1620 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:3978: oid 6 version 1621 is already newer than 1618 2024-09-20T19:39:15.683 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 6 v 1621 (ObjNum 1357 snap 439 seq_num 1357) dirty exists 2024-09-20T19:39:15.684 INFO:tasks.rados.rados.0.smithi042.stdout:3978: left oid 6 (ObjNum 1357 snap 439 seq_num 1357) 2024-09-20T19:39:15.684 INFO:tasks.rados.rados.0.smithi042.stdout:3981: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:39:15.740 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:15.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:15.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3980: finishing copy_from to smithi04268048-24 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 24 v 1358 (ObjNum 1325 snap 426 seq_num 1325) dirty exists 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: finishing write tid 1 to smithi04268048-18 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: finishing write tid 2 to smithi04268048-18 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: finishing write tid 3 to smithi04268048-18 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: finishing write tid 4 to smithi04268048-18 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: finishing write tid 5 to smithi04268048-18 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: oid 18 updating version 0 to 1648 2024-09-20T19:39:15.843 INFO:tasks.rados.rados.0.smithi042.stdout:3983: oid 18 version 1648 is already newer than 1646 2024-09-20T19:39:15.844 INFO:tasks.rados.rados.0.smithi042.stdout:3983: oid 18 updating version 1648 to 1649 2024-09-20T19:39:15.844 INFO:tasks.rados.rados.0.smithi042.stdout:3983: oid 18 version 1649 is already newer than 1647 2024-09-20T19:39:15.844 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 18 v 1649 (ObjNum 1358 snap 439 seq_num 1358) dirty exists 2024-09-20T19:39:15.844 INFO:tasks.rados.rados.0.smithi042.stdout:3983: left oid 18 (ObjNum 1358 snap 439 seq_num 1358) 2024-09-20T19:39:15.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.916 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:15.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:15 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:15.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:15.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:14.230041+0000 mgr.y (mgr.24491) 795 : audit [DBG] from='client.16311 v1:172.21.15.42:0/3254483659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.499462+0000 mgr.y (mgr.24491) 796 : cluster [DBG] pgmap v1566: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.524479+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.f scrub starts 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.526469+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.f scrub ok 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:14.685021+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.895224+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:14.900176+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.005944+0000 osd.2 (osd.2) 333 : cluster [DBG] 7.3 scrub starts 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.008548+0000 osd.2 (osd.2) 334 : cluster [DBG] 7.3 scrub ok 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.505673+0000 mon.a (mon.0) 2709 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.505734+0000 mon.a (mon.0) 2710 : cluster [INF] Cluster is now healthy 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:15.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:15.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: audit 2024-09-20T19:39:15.507979+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? v1:172.21.15.42:0/178630907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:15.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:15.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:15 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.520743+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2024-09-20T19:39:16.031 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:16.035 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:39:16.060 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:16.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:16.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:16.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:16.408 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:16.409 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2024-09-20T19:39:13.488402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":128946452,"num_objects":252,"num_object_clones":11,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":2,"num_read":6290,"num_read_kb":2732549,"num_write":3441,"num_write_kb":919115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29011,"ondisk_log_size":29011,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":760052,"kb_used_data":304804,"kb_used_omap":92,"kb_used_meta":452643,"kb_avail":655456012,"statfs":{"total":671965249536,"available":671186956288,"internally_reserved":0,"allocated":312119296,"data_stored":343952288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":463507129},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11267800,"num_objects":-8,"num_object_clones":-8,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":-1,"num_read":46,"num_read_kb":38455,"num_write":14,"num_write_kb":3579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.895007"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":4767,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.781980+0000","last_change":"2024-09-20T19:39:10.783330+0000","last_active":"2024-09-20T19:39:10.781980+0000","last_peered":"2024-09-20T19:39:10.781980+0000","last_clean":"2024-09-20T19:39:10.781980+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:39:10.781980+0000","last_undegraded":"2024-09-20T19:39:10.781980+0000","last_fullsized":"2024-09-20T19:39:10.781980+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:38:17.790503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.790503+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.090810+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0040795340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"7.22","version":"995'368","reported_seq":4553,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594975+0000","last_change":"2024-09-20T19:39:08.596242+0000","last_active":"2024-09-20T19:39:08.594975+0000","last_peered":"2024-09-20T19:39:08.594975+0000","last_clean":"2024-09-20T19:39:08.594975+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:39:08.594975+0000","last_undegraded":"2024-09-20T19:39:08.594975+0000","last_fullsized":"2024-09-20T19:39:08.594975+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"938'336","last_scrub_stamp":"2024-09-20T19:37:36.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:36.773213+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017920060000000001,"stat_sum":{"num_bytes":3344890,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":40573,"num_write":37,"num_write_kb":14578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"7.21","version":"1042'837","reported_seq":5515,"reported_epoch":1043,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:11.907900+0000","last_change":"2024-09-20T19:39:10.813900+0000","last_active":"2024-09-20T19:39:11.907900+0000","last_peered":"2024-09-20T19:39:11.907900+0000","last_clean":"2024-09-20T19:39:11.907900+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:39:11.907900+0000","last_undegraded":"2024-09-20T19:39:11.907900+0000","last_fullsized":"2024-09-20T19:39:11.907900+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"953'788","last_scrub_stamp":"2024-09-20T19:37:42.892118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.892118+0000","objects_scrubbed":3,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002227064,"stat_sum":{"num_bytes":7749283,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":143,"num_read_kb":155074,"num_write":59,"num_write_kb":32034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"7.20","version":"1014'991","reported_seq":5578,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.628034+0000","last_change":"2024-09-20T19:39:10.727424+0000","last_active":"2024-09-20T19:39:11.628034+0000","last_peered":"2024-09-20T19:39:11.628034+0000","last_clean":"2024-09-20T19:39:11.628034+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:39:11.628034+0000","last_undegraded":"2024-09-20T19:39:11.628034+0000","last_fullsized":"2024-09-20T19:39:11.628034+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'982","last_scrub_stamp":"2024-09-20T19:38:13.700013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:13.700013+0000","objects_scrubbed":1,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:18.899481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041796869999999996,"stat_sum":{"num_bytes":3815486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":67428,"num_write":32,"num_write_kb":14158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"7.1a","version":"1040'792","reported_seq":5122,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.598375+0000","last_change":"2024-09-20T19:39:08.599651+0000","last_active":"2024-09-20T19:39:08.598375+0000","last_peered":"2024-09-20T19:39:08.598375+0000","last_clean":"2024-09-20T19:39:08.598375+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:39:08.598375+0000","last_undegraded":"2024-09-20T19:39:08.598375+0000","last_fullsized":"2024-09-20T19:39:08.598375+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'725","last_scrub_stamp":"2024-09-20T19:37:24.658564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:24.658564+0000","objects_scrubbed":6,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.004820062,"stat_sum":{"num_bytes":7379313,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":116514,"num_write":94,"num_write_kb":37050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":1489,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642569+0000","last_change":"2024-09-20T19:36:31.860620+0000","last_active":"2024-09-20T19:39:10.642569+0000","last_peered":"2024-09-20T19:39:10.642569+0000","last_clean":"2024-09-20T19:39:10.642569+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:39:10.642569+0000","last_undegraded":"2024-09-20T19:39:10.642569+0000","last_fullsized":"2024-09-20T19:39:10.642569+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.860530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.860530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1813,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.650338+0000","last_change":"2024-09-20T19:37:25.135773+0000","last_active":"2024-09-20T19:39:12.650338+0000","last_peered":"2024-09-20T19:39:12.650338+0000","last_clean":"2024-09-20T19:39:12.650338+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:39:12.650338+0000","last_undegraded":"2024-09-20T19:39:12.650338+0000","last_fullsized":"2024-09-20T19:39:12.650338+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.135646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.135646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"987'39","reported_seq":1751,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.627255+0000","last_change":"2024-09-20T19:37:41.511382+0000","last_active":"2024-09-20T19:39:11.627255+0000","last_peered":"2024-09-20T19:39:11.627255+0000","last_clean":"2024-09-20T19:39:11.627255+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:39:11.627255+0000","last_undegraded":"2024-09-20T19:39:11.627255+0000","last_fullsized":"2024-09-20T19:39:11.627255+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:37:41.511307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:41.511307+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1783,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594076+0000","last_change":"2024-09-20T19:38:28.745755+0000","last_active":"2024-09-20T19:39:08.594076+0000","last_peered":"2024-09-20T19:39:08.594076+0000","last_clean":"2024-09-20T19:39:08.594076+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:39:08.594076+0000","last_undegraded":"2024-09-20T19:39:08.594076+0000","last_fullsized":"2024-09-20T19:39:08.594076+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:28.745668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.745668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.949968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":5051,"reported_epoch":1044,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:12.646036+0000","last_change":"2024-09-20T19:39:12.645961+0000","last_active":"2024-09-20T19:39:12.646036+0000","last_peered":"2024-09-20T19:39:12.646036+0000","last_clean":"2024-09-20T19:39:12.646036+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:39:12.646036+0000","last_undegraded":"2024-09-20T19:39:12.646036+0000","last_fullsized":"2024-09-20T19:39:12.646036+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:38:14.315644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.315644+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:32.616009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.38744605599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"2.1e","version":"0'0","reported_seq":1548,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.651138+0000","last_change":"2024-09-20T19:37:21.156138+0000","last_active":"2024-09-20T19:39:12.651138+0000","last_peered":"2024-09-20T19:39:12.651138+0000","last_clean":"2024-09-20T19:39:12.651138+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:39:12.651138+0000","last_undegraded":"2024-09-20T19:39:12.651138+0000","last_fullsized":"2024-09-20T19:39:12.651138+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.156013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.156013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"987'21","reported_seq":1896,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595646+0000","last_change":"2024-09-20T19:36:44.033709+0000","last_active":"2024-09-20T19:39:08.595646+0000","last_peered":"2024-09-20T19:39:08.595646+0000","last_clean":"2024-09-20T19:39:08.595646+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:39:08.595646+0000","last_undegraded":"2024-09-20T19:39:08.595646+0000","last_fullsized":"2024-09-20T19:39:08.595646+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:36:44.033568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:44.033568+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.776685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":2021,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625495+0000","last_change":"2024-09-20T19:37:20.943858+0000","last_active":"2024-09-20T19:39:11.625495+0000","last_peered":"2024-09-20T19:39:11.625495+0000","last_clean":"2024-09-20T19:39:11.625495+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:39:11.625495+0000","last_undegraded":"2024-09-20T19:39:11.625495+0000","last_fullsized":"2024-09-20T19:39:11.625495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:20.943776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:20.943776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1845,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595051+0000","last_change":"2024-09-20T19:38:27.709303+0000","last_active":"2024-09-20T19:39:08.595051+0000","last_peered":"2024-09-20T19:39:08.595051+0000","last_clean":"2024-09-20T19:39:08.595051+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:39:08.595051+0000","last_undegraded":"2024-09-20T19:39:08.595051+0000","last_fullsized":"2024-09-20T19:39:08.595051+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:27.709247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.709247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.024307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"1030'1284","reported_seq":5831,"reported_epoch":1044,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:12.643681+0000","last_change":"2024-09-20T19:39:12.643600+0000","last_active":"2024-09-20T19:39:12.643681+0000","last_peered":"2024-09-20T19:39:12.643681+0000","last_clean":"2024-09-20T19:39:12.643681+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:39:12.643681+0000","last_undegraded":"2024-09-20T19:39:12.643681+0000","last_fullsized":"2024-09-20T19:39:12.643681+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"895'1218","last_scrub_stamp":"2024-09-20T19:36:46.063473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:46.063473+0000","objects_scrubbed":6,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.33645950000000002,"stat_sum":{"num_bytes":5866270,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":128956,"num_write":108,"num_write_kb":39081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"2.1d","version":"0'0","reported_seq":2017,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643968+0000","last_change":"2024-09-20T19:38:40.080992+0000","last_active":"2024-09-20T19:39:10.643968+0000","last_peered":"2024-09-20T19:39:10.643968+0000","last_clean":"2024-09-20T19:39:10.643968+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:39:10.643968+0000","last_undegraded":"2024-09-20T19:39:10.643968+0000","last_fullsized":"2024-09-20T19:39:10.643968+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.080865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.080865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.792937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"987'9","reported_seq":1893,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594657+0000","last_change":"2024-09-20T19:37:44.725836+0000","last_active":"2024-09-20T19:39:08.594657+0000","last_peered":"2024-09-20T19:39:08.594657+0000","last_clean":"2024-09-20T19:39:08.594657+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:39:08.594657+0000","last_undegraded":"2024-09-20T19:39:08.594657+0000","last_fullsized":"2024-09-20T19:39:08.594657+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:37:44.725701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:44.725701+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":2001,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642632+0000","last_change":"2024-09-20T19:34:50.140557+0000","last_active":"2024-09-20T19:39:10.642632+0000","last_peered":"2024-09-20T19:39:10.642632+0000","last_clean":"2024-09-20T19:39:10.642632+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:39:10.642632+0000","last_undegraded":"2024-09-20T19:39:10.642632+0000","last_fullsized":"2024-09-20T19:39:10.642632+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:50.140498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:50.140498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2022,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908454+0000","last_change":"2024-09-20T19:36:22.545289+0000","last_active":"2024-09-20T19:39:11.908454+0000","last_peered":"2024-09-20T19:39:11.908454+0000","last_clean":"2024-09-20T19:39:11.908454+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:39:11.908454+0000","last_undegraded":"2024-09-20T19:39:11.908454+0000","last_fullsized":"2024-09-20T19:39:11.908454+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:22.545231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:22.545231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.331104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"950'482","reported_seq":5066,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778058+0000","last_change":"2024-09-20T19:39:10.779772+0000","last_active":"2024-09-20T19:39:10.778058+0000","last_peered":"2024-09-20T19:39:10.778058+0000","last_clean":"2024-09-20T19:39:10.778058+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:39:10.778058+0000","last_undegraded":"2024-09-20T19:39:10.778058+0000","last_fullsized":"2024-09-20T19:39:10.778058+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"950'482","last_scrub_stamp":"2024-09-20T19:37:42.727140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.727140+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022960229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":55935,"num_write":41,"num_write_kb":15366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"2.1c","version":"0'0","reported_seq":2006,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643682+0000","last_change":"2024-09-20T19:38:26.111416+0000","last_active":"2024-09-20T19:39:10.643682+0000","last_peered":"2024-09-20T19:39:10.643682+0000","last_clean":"2024-09-20T19:39:10.643682+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:39:10.643682+0000","last_undegraded":"2024-09-20T19:39:10.643682+0000","last_fullsized":"2024-09-20T19:39:10.643682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:26.111300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.111300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:33.731393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"987'21","reported_seq":1932,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594338+0000","last_change":"2024-09-20T19:38:30.744236+0000","last_active":"2024-09-20T19:39:08.594338+0000","last_peered":"2024-09-20T19:39:08.594338+0000","last_clean":"2024-09-20T19:39:08.594338+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:39:08.594338+0000","last_undegraded":"2024-09-20T19:39:08.594338+0000","last_fullsized":"2024-09-20T19:39:08.594338+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.744141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.744141+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.937516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2003,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908406+0000","last_change":"2024-09-20T19:36:23.582321+0000","last_active":"2024-09-20T19:39:11.908406+0000","last_peered":"2024-09-20T19:39:11.908406+0000","last_clean":"2024-09-20T19:39:11.908406+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:39:11.908406+0000","last_undegraded":"2024-09-20T19:39:11.908406+0000","last_fullsized":"2024-09-20T19:39:11.908406+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.582265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.377154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":982,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.642624+0000","last_change":"2024-09-20T19:37:24.118390+0000","last_active":"2024-09-20T19:39:12.642624+0000","last_peered":"2024-09-20T19:39:12.642624+0000","last_clean":"2024-09-20T19:39:12.642624+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:39:12.642624+0000","last_undegraded":"2024-09-20T19:39:12.642624+0000","last_fullsized":"2024-09-20T19:39:12.642624+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:24.118312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:24.118312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1639,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.907773+0000","last_change":"2024-09-20T19:38:15.047645+0000","last_active":"2024-09-20T19:39:11.907773+0000","last_peered":"2024-09-20T19:39:11.907773+0000","last_clean":"2024-09-20T19:39:11.907773+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:39:11.907773+0000","last_undegraded":"2024-09-20T19:39:11.907773+0000","last_fullsized":"2024-09-20T19:39:11.907773+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:15.047486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:15.047486+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:17.135448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1021'427","reported_seq":5012,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.596306+0000","last_change":"2024-09-20T19:39:08.598286+0000","last_active":"2024-09-20T19:39:08.596306+0000","last_peered":"2024-09-20T19:39:08.596306+0000","last_clean":"2024-09-20T19:39:08.596306+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:39:08.596306+0000","last_undegraded":"2024-09-20T19:39:08.596306+0000","last_fullsized":"2024-09-20T19:39:08.596306+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'423","last_scrub_stamp":"2024-09-20T19:37:33.788168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:33.788168+0000","objects_scrubbed":1,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031930000000000001,"stat_sum":{"num_bytes":2235661,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":27202,"num_write":27,"num_write_kb":10018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.d","version":"987'45","reported_seq":1886,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595114+0000","last_change":"2024-09-20T19:38:14.828944+0000","last_active":"2024-09-20T19:39:08.595114+0000","last_peered":"2024-09-20T19:39:08.595114+0000","last_clean":"2024-09-20T19:39:08.595114+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:39:08.595114+0000","last_undegraded":"2024-09-20T19:39:08.595114+0000","last_fullsized":"2024-09-20T19:39:08.595114+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:38:14.828850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.828850+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:15.054682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2002,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643350+0000","last_change":"2024-09-20T19:34:49.185014+0000","last_active":"2024-09-20T19:39:10.643350+0000","last_peered":"2024-09-20T19:39:10.643350+0000","last_clean":"2024-09-20T19:39:10.643350+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:39:10.643350+0000","last_undegraded":"2024-09-20T19:39:10.643350+0000","last_fullsized":"2024-09-20T19:39:10.643350+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:49.184958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:49.184958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":2032,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625725+0000","last_change":"2024-09-20T19:38:38.590561+0000","last_active":"2024-09-20T19:39:11.625725+0000","last_peered":"2024-09-20T19:39:11.625725+0000","last_clean":"2024-09-20T19:39:11.625725+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:39:11.625725+0000","last_undegraded":"2024-09-20T19:39:11.625725+0000","last_fullsized":"2024-09-20T19:39:11.625725+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:38.590498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:38.590498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.175561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":592,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632960+0000","last_change":"2024-09-20T19:37:21.611924+0000","last_active":"2024-09-20T19:39:11.632960+0000","last_peered":"2024-09-20T19:39:11.632960+0000","last_clean":"2024-09-20T19:39:11.632960+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:39:11.632960+0000","last_undegraded":"2024-09-20T19:39:11.632960+0000","last_fullsized":"2024-09-20T19:39:11.632960+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.611767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.611767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":2095,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642826+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:39:10.642826+0000","last_peered":"2024-09-20T19:39:10.642826+0000","last_clean":"2024-09-20T19:39:10.642826+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:39:10.642826+0000","last_undegraded":"2024-09-20T19:39:10.642826+0000","last_fullsized":"2024-09-20T19:39:10.642826+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:45.070277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1042'709","reported_seq":5079,"reported_epoch":1044,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:12.650558+0000","last_change":"2024-09-20T19:39:12.641745+0000","last_active":"2024-09-20T19:39:12.650558+0000","last_peered":"2024-09-20T19:39:12.650558+0000","last_clean":"2024-09-20T19:39:12.650558+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:39:12.650558+0000","last_undegraded":"2024-09-20T19:39:12.650558+0000","last_fullsized":"2024-09-20T19:39:12.650558+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'687","last_scrub_stamp":"2024-09-20T19:38:24.221337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.221337+0000","objects_scrubbed":1,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.143255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.387010034,"stat_sum":{"num_bytes":3924893,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":35389,"num_write":68,"num_write_kb":28356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.c","version":"987'30","reported_seq":1931,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594685+0000","last_change":"2024-09-20T19:38:29.725721+0000","last_active":"2024-09-20T19:39:08.594685+0000","last_peered":"2024-09-20T19:39:08.594685+0000","last_clean":"2024-09-20T19:39:08.594685+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:39:08.594685+0000","last_undegraded":"2024-09-20T19:39:08.594685+0000","last_fullsized":"2024-09-20T19:39:08.594685+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:29.725634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:29.725634+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.585770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":2048,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.627624+0000","last_change":"2024-09-20T19:38:18.676904+0000","last_active":"2024-09-20T19:39:11.627624+0000","last_peered":"2024-09-20T19:39:11.627624+0000","last_clean":"2024-09-20T19:39:11.627624+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:39:11.627624+0000","last_undegraded":"2024-09-20T19:39:11.627624+0000","last_fullsized":"2024-09-20T19:39:11.627624+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:18.676841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:18.676841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:22.424549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":2294,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632742+0000","last_change":"2024-09-20T19:39:06.910897+0000","last_active":"2024-09-20T19:39:11.632742+0000","last_peered":"2024-09-20T19:39:11.632742+0000","last_clean":"2024-09-20T19:39:11.632742+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:39:11.632742+0000","last_undegraded":"2024-09-20T19:39:11.632742+0000","last_fullsized":"2024-09-20T19:39:11.632742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:39:06.910817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:06.910817+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.382750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1855,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595429+0000","last_change":"2024-09-20T19:39:05.797046+0000","last_active":"2024-09-20T19:39:08.595429+0000","last_peered":"2024-09-20T19:39:08.595429+0000","last_clean":"2024-09-20T19:39:08.595429+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:39:08.595429+0000","last_undegraded":"2024-09-20T19:39:08.595429+0000","last_fullsized":"2024-09-20T19:39:08.595429+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.796915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.796915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:13.456865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1610,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.907885+0000","last_change":"2024-09-20T19:38:17.061687+0000","last_active":"2024-09-20T19:39:11.907885+0000","last_peered":"2024-09-20T19:39:11.907885+0000","last_clean":"2024-09-20T19:39:11.907885+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:39:11.907885+0000","last_undegraded":"2024-09-20T19:39:11.907885+0000","last_fullsized":"2024-09-20T19:39:11.907885+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:17.061616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.061616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.609939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1041'961","reported_seq":5622,"reported_epoch":1042,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:10.643326+0000","last_change":"2024-09-20T19:39:10.638839+0000","last_active":"2024-09-20T19:39:10.643326+0000","last_peered":"2024-09-20T19:39:10.643326+0000","last_clean":"2024-09-20T19:39:10.643326+0000","last_became_active":"2024-09-20T19:35:34.395504+0000","last_became_peered":"2024-09-20T19:35:34.395504+0000","last_unstale":"2024-09-20T19:39:10.643326+0000","last_undegraded":"2024-09-20T19:39:10.643326+0000","last_fullsized":"2024-09-20T19:39:10.643326+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":826,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'835","last_scrub_stamp":"2024-09-20T19:33:55.628320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:33:55.628320+0000","objects_scrubbed":2,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.91905855299999994,"stat_sum":{"num_bytes":3025557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":55890,"num_write":39,"num_write_kb":17667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b1","length":"1"}]},{"pgid":"4.f","version":"987'39","reported_seq":2133,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629645+0000","last_change":"2024-09-20T19:38:26.665615+0000","last_active":"2024-09-20T19:39:11.629645+0000","last_peered":"2024-09-20T19:39:11.629645+0000","last_clean":"2024-09-20T19:39:11.629645+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:39:11.629645+0000","last_undegraded":"2024-09-20T19:39:11.629645+0000","last_fullsized":"2024-09-20T19:39:11.629645+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:38:26.665526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.665526+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:32.401246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":2048,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.626216+0000","last_change":"2024-09-20T19:38:17.643987+0000","last_active":"2024-09-20T19:39:11.626216+0000","last_peered":"2024-09-20T19:39:11.626216+0000","last_clean":"2024-09-20T19:39:11.626216+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:39:11.626216+0000","last_undegraded":"2024-09-20T19:39:11.626216+0000","last_fullsized":"2024-09-20T19:39:11.626216+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:17.643862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.643862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:31.220331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":2073,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595764+0000","last_change":"2024-09-20T19:38:35.798379+0000","last_active":"2024-09-20T19:39:08.595764+0000","last_peered":"2024-09-20T19:39:08.595764+0000","last_clean":"2024-09-20T19:39:08.595764+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:39:08.595764+0000","last_undegraded":"2024-09-20T19:39:08.595764+0000","last_fullsized":"2024-09-20T19:39:08.595764+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:38:35.798250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.798250+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.803640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2024,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908609+0000","last_change":"2024-09-20T19:38:35.252514+0000","last_active":"2024-09-20T19:39:11.908609+0000","last_peered":"2024-09-20T19:39:11.908609+0000","last_clean":"2024-09-20T19:39:11.908609+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:39:11.908609+0000","last_undegraded":"2024-09-20T19:39:11.908609+0000","last_fullsized":"2024-09-20T19:39:11.908609+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.252424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.252424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.236879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1041'654","reported_seq":5281,"reported_epoch":1042,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:10.780224+0000","last_change":"2024-09-20T19:39:10.561151+0000","last_active":"2024-09-20T19:39:10.780224+0000","last_peered":"2024-09-20T19:39:10.780224+0000","last_clean":"2024-09-20T19:39:10.780224+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:39:10.780224+0000","last_undegraded":"2024-09-20T19:39:10.780224+0000","last_fullsized":"2024-09-20T19:39:10.780224+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'630","last_scrub_stamp":"2024-09-20T19:38:23.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.703700+0000","objects_scrubbed":1,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:48.848902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.024858551,"stat_sum":{"num_bytes":2067591,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":34,"num_read_kb":20016,"num_write":45,"num_write_kb":14853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b1","length":"2"},{"start":"1b4","length":"1"}]},{"pgid":"4.e","version":"987'27","reported_seq":1909,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.596051+0000","last_change":"2024-09-20T19:37:37.778246+0000","last_active":"2024-09-20T19:39:08.596051+0000","last_peered":"2024-09-20T19:39:08.596051+0000","last_clean":"2024-09-20T19:39:08.596051+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:39:08.596051+0000","last_undegraded":"2024-09-20T19:39:08.596051+0000","last_fullsized":"2024-09-20T19:39:08.596051+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:37:37.778165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:37.778165+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":675,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642231+0000","last_change":"2024-09-20T19:36:37.827773+0000","last_active":"2024-09-20T19:39:10.642231+0000","last_peered":"2024-09-20T19:39:10.642231+0000","last_clean":"2024-09-20T19:39:10.642231+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:39:10.642231+0000","last_undegraded":"2024-09-20T19:39:10.642231+0000","last_fullsized":"2024-09-20T19:39:10.642231+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:37.827682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:37.827682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2012,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909267+0000","last_change":"2024-09-20T19:34:48.070464+0000","last_active":"2024-09-20T19:39:11.909267+0000","last_peered":"2024-09-20T19:39:11.909267+0000","last_clean":"2024-09-20T19:39:11.909267+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:39:11.909267+0000","last_undegraded":"2024-09-20T19:39:11.909267+0000","last_fullsized":"2024-09-20T19:39:11.909267+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.070340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.070340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1485,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.779251+0000","last_change":"2024-09-20T19:37:21.733870+0000","last_active":"2024-09-20T19:39:10.779251+0000","last_peered":"2024-09-20T19:39:10.779251+0000","last_clean":"2024-09-20T19:39:10.779251+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:39:10.779251+0000","last_undegraded":"2024-09-20T19:39:10.779251+0000","last_fullsized":"2024-09-20T19:39:10.779251+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.733745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.733745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":840,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908569+0000","last_change":"2024-09-20T19:35:03.930343+0000","last_active":"2024-09-20T19:39:11.908569+0000","last_peered":"2024-09-20T19:39:11.908569+0000","last_clean":"2024-09-20T19:39:11.908569+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:39:11.908569+0000","last_undegraded":"2024-09-20T19:39:11.908569+0000","last_fullsized":"2024-09-20T19:39:11.908569+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:35:03.930289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.930289+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":4330,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.624912+0000","last_change":"2024-09-20T19:39:10.635918+0000","last_active":"2024-09-20T19:39:11.624912+0000","last_peered":"2024-09-20T19:39:11.624912+0000","last_clean":"2024-09-20T19:39:11.624912+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:39:11.624912+0000","last_undegraded":"2024-09-20T19:39:11.624912+0000","last_fullsized":"2024-09-20T19:39:11.624912+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:37:42.138190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.138190+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0054333999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.1","version":"987'42","reported_seq":1888,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594524+0000","last_change":"2024-09-20T19:38:36.778506+0000","last_active":"2024-09-20T19:39:08.594524+0000","last_peered":"2024-09-20T19:39:08.594524+0000","last_clean":"2024-09-20T19:39:08.594524+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:39:08.594524+0000","last_undegraded":"2024-09-20T19:39:08.594524+0000","last_fullsized":"2024-09-20T19:39:08.594524+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:38:36.778420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.778420+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.256363+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2008,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.782664+0000","last_change":"2024-09-20T19:37:25.730287+0000","last_active":"2024-09-20T19:39:10.782664+0000","last_peered":"2024-09-20T19:39:10.782664+0000","last_clean":"2024-09-20T19:39:10.782664+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:39:10.782664+0000","last_undegraded":"2024-09-20T19:39:10.782664+0000","last_fullsized":"2024-09-20T19:39:10.782664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.730226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.730226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":956,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.623882+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:39:11.623882+0000","last_peered":"2024-09-20T19:39:11.623882+0000","last_clean":"2024-09-20T19:39:11.623882+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:39:11.623882+0000","last_undegraded":"2024-09-20T19:39:11.623882+0000","last_fullsized":"2024-09-20T19:39:11.623882+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:03.844414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1988,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642835+0000","last_change":"2024-09-20T19:34:16.144529+0000","last_active":"2024-09-20T19:39:10.642835+0000","last_peered":"2024-09-20T19:39:10.642835+0000","last_clean":"2024-09-20T19:39:10.642835+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:39:10.642835+0000","last_undegraded":"2024-09-20T19:39:10.642835+0000","last_fullsized":"2024-09-20T19:39:10.642835+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:16.144473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:16.144473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1422,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595357+0000","last_change":"2024-09-20T19:38:26.684327+0000","last_active":"2024-09-20T19:39:08.595357+0000","last_peered":"2024-09-20T19:39:08.595357+0000","last_clean":"2024-09-20T19:39:08.595357+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:39:08.595357+0000","last_undegraded":"2024-09-20T19:39:08.595357+0000","last_fullsized":"2024-09-20T19:39:08.595357+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:26.684257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.684257+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.590882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1016'507","reported_seq":4933,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.630757+0000","last_change":"2024-09-20T19:39:10.728179+0000","last_active":"2024-09-20T19:39:11.630757+0000","last_peered":"2024-09-20T19:39:11.630757+0000","last_clean":"2024-09-20T19:39:11.630757+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:39:11.630757+0000","last_undegraded":"2024-09-20T19:39:11.630757+0000","last_fullsized":"2024-09-20T19:39:11.630757+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"937'499","last_scrub_stamp":"2024-09-20T19:37:29.555257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:29.555257+0000","objects_scrubbed":1,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0054262349999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":7626,"num_write":23,"num_write_kb":5767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.0","version":"987'42","reported_seq":1752,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642453+0000","last_change":"2024-09-20T19:37:30.646115+0000","last_active":"2024-09-20T19:39:10.642453+0000","last_peered":"2024-09-20T19:39:10.642453+0000","last_clean":"2024-09-20T19:39:10.642453+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:39:10.642453+0000","last_undegraded":"2024-09-20T19:39:10.642453+0000","last_fullsized":"2024-09-20T19:39:10.642453+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:37:30.645963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:30.645963+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":2026,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.628944+0000","last_change":"2024-09-20T19:37:26.984507+0000","last_active":"2024-09-20T19:39:11.628944+0000","last_peered":"2024-09-20T19:39:11.628944+0000","last_clean":"2024-09-20T19:39:11.628944+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:39:11.628944+0000","last_undegraded":"2024-09-20T19:39:11.628944+0000","last_fullsized":"2024-09-20T19:39:11.628944+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:37:26.984377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:26.984377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1826,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594069+0000","last_change":"2024-09-20T19:37:27.633138+0000","last_active":"2024-09-20T19:39:08.594069+0000","last_peered":"2024-09-20T19:39:08.594069+0000","last_clean":"2024-09-20T19:39:08.594069+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:39:08.594069+0000","last_undegraded":"2024-09-20T19:39:08.594069+0000","last_fullsized":"2024-09-20T19:39:08.594069+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:27.633075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:27.633075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1780,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594464+0000","last_change":"2024-09-20T19:36:48.027651+0000","last_active":"2024-09-20T19:39:08.594464+0000","last_peered":"2024-09-20T19:39:08.594464+0000","last_clean":"2024-09-20T19:39:08.594464+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:39:08.594464+0000","last_undegraded":"2024-09-20T19:39:08.594464+0000","last_fullsized":"2024-09-20T19:39:08.594464+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:48.027589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.027589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.737059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1737,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.780994+0000","last_change":"2024-09-20T19:38:31.650903+0000","last_active":"2024-09-20T19:39:10.780994+0000","last_peered":"2024-09-20T19:39:10.780994+0000","last_clean":"2024-09-20T19:39:10.780994+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:39:10.780994+0000","last_undegraded":"2024-09-20T19:39:10.780994+0000","last_fullsized":"2024-09-20T19:39:10.780994+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:31.650776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.650776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.460344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1040'1014","reported_seq":5819,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.907795+0000","last_change":"2024-09-20T19:39:10.816395+0000","last_active":"2024-09-20T19:39:11.907795+0000","last_peered":"2024-09-20T19:39:11.907795+0000","last_clean":"2024-09-20T19:39:11.907795+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:39:11.907795+0000","last_undegraded":"2024-09-20T19:39:11.907795+0000","last_fullsized":"2024-09-20T19:39:11.907795+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'991","last_scrub_stamp":"2024-09-20T19:38:33.283803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.283803+0000","objects_scrubbed":1,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:37.210988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0034197189999999999,"stat_sum":{"num_bytes":3294469,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":88272,"num_write":58,"num_write_kb":15972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.3","version":"987'39","reported_seq":1846,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.643578+0000","last_change":"2024-09-20T19:36:24.022617+0000","last_active":"2024-09-20T19:39:12.643578+0000","last_peered":"2024-09-20T19:39:12.643578+0000","last_clean":"2024-09-20T19:39:12.643578+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:39:12.643578+0000","last_undegraded":"2024-09-20T19:39:12.643578+0000","last_fullsized":"2024-09-20T19:39:12.643578+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:36:24.022527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:24.022527+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1976,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632960+0000","last_change":"2024-09-20T19:36:31.559020+0000","last_active":"2024-09-20T19:39:11.632960+0000","last_peered":"2024-09-20T19:39:11.632960+0000","last_clean":"2024-09-20T19:39:11.632960+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:39:11.632960+0000","last_undegraded":"2024-09-20T19:39:11.632960+0000","last_fullsized":"2024-09-20T19:39:11.632960+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.558923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.558923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.839139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2020,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778277+0000","last_change":"2024-09-20T19:38:30.630442+0000","last_active":"2024-09-20T19:39:10.778277+0000","last_peered":"2024-09-20T19:39:10.778277+0000","last_clean":"2024-09-20T19:39:10.778277+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:39:10.778277+0000","last_undegraded":"2024-09-20T19:39:10.778277+0000","last_fullsized":"2024-09-20T19:39:10.778277+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.630318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.630318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.617338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":2017,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.624809+0000","last_change":"2024-09-20T19:39:08.849859+0000","last_active":"2024-09-20T19:39:11.624809+0000","last_peered":"2024-09-20T19:39:11.624809+0000","last_clean":"2024-09-20T19:39:11.624809+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:39:11.624809+0000","last_undegraded":"2024-09-20T19:39:11.624809+0000","last_fullsized":"2024-09-20T19:39:11.624809+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:08.849741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:08.849741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:27.397133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1722,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.781708+0000","last_change":"2024-09-20T19:38:22.733902+0000","last_active":"2024-09-20T19:39:10.781708+0000","last_peered":"2024-09-20T19:39:10.781708+0000","last_clean":"2024-09-20T19:39:10.781708+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:39:10.781708+0000","last_undegraded":"2024-09-20T19:39:10.781708+0000","last_fullsized":"2024-09-20T19:39:10.781708+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:22.733746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.733746+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:23.387040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":5101,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594645+0000","last_change":"2024-09-20T19:39:08.595888+0000","last_active":"2024-09-20T19:39:08.594645+0000","last_peered":"2024-09-20T19:39:08.594645+0000","last_clean":"2024-09-20T19:39:08.594645+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:39:08.594645+0000","last_undegraded":"2024-09-20T19:39:08.594645+0000","last_fullsized":"2024-09-20T19:39:08.594645+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:37:23.690413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.690413+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015501880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.2","version":"987'30","reported_seq":2078,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625915+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:39:11.625915+0000","last_peered":"2024-09-20T19:39:11.625915+0000","last_clean":"2024-09-20T19:39:11.625915+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:39:11.625915+0000","last_undegraded":"2024-09-20T19:39:11.625915+0000","last_fullsized":"2024-09-20T19:39:11.625915+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.824143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2018,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.628117+0000","last_change":"2024-09-20T19:34:48.237100+0000","last_active":"2024-09-20T19:39:11.628117+0000","last_peered":"2024-09-20T19:39:11.628117+0000","last_clean":"2024-09-20T19:39:11.628117+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:39:11.628117+0000","last_undegraded":"2024-09-20T19:39:11.628117+0000","last_fullsized":"2024-09-20T19:39:11.628117+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:48.236983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.236983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":2001,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.647125+0000","last_change":"2024-09-20T19:38:19.343965+0000","last_active":"2024-09-20T19:39:12.647125+0000","last_peered":"2024-09-20T19:39:12.647125+0000","last_clean":"2024-09-20T19:39:12.647125+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:39:12.647125+0000","last_undegraded":"2024-09-20T19:39:12.647125+0000","last_fullsized":"2024-09-20T19:39:12.647125+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.343828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.343828+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.328086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1816,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.645014+0000","last_change":"2024-09-20T19:35:47.661521+0000","last_active":"2024-09-20T19:39:12.645014+0000","last_peered":"2024-09-20T19:39:12.645014+0000","last_clean":"2024-09-20T19:39:12.645014+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:39:12.645014+0000","last_undegraded":"2024-09-20T19:39:12.645014+0000","last_fullsized":"2024-09-20T19:39:12.645014+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.661410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.661410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1614,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908024+0000","last_change":"2024-09-20T19:38:12.969468+0000","last_active":"2024-09-20T19:39:11.908024+0000","last_peered":"2024-09-20T19:39:11.908024+0000","last_clean":"2024-09-20T19:39:11.908024+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:39:11.908024+0000","last_undegraded":"2024-09-20T19:39:11.908024+0000","last_fullsized":"2024-09-20T19:39:11.908024+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:12.969339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.969339+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:22.236227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1042'722","reported_seq":5443,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.795107+0000","last_change":"2024-09-20T19:39:10.797783+0000","last_active":"2024-09-20T19:39:10.795107+0000","last_peered":"2024-09-20T19:39:10.795107+0000","last_clean":"2024-09-20T19:39:10.795107+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:39:10.795107+0000","last_undegraded":"2024-09-20T19:39:10.795107+0000","last_fullsized":"2024-09-20T19:39:10.795107+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'654","last_scrub_stamp":"2024-09-20T19:37:44.764602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:44.764602+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015569819,"stat_sum":{"num_bytes":3875008,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":136098,"num_write":135,"num_write_kb":47643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.4","version":"987'57","reported_seq":2418,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.627835+0000","last_change":"2024-09-20T19:36:44.910823+0000","last_active":"2024-09-20T19:39:11.627835+0000","last_peered":"2024-09-20T19:39:11.627835+0000","last_clean":"2024-09-20T19:39:11.627835+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:39:11.627835+0000","last_undegraded":"2024-09-20T19:39:11.627835+0000","last_fullsized":"2024-09-20T19:39:11.627835+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'45","last_scrub_stamp":"2024-09-20T19:36:44.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:44.910638+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2034,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908712+0000","last_change":"2024-09-20T19:38:32.270500+0000","last_active":"2024-09-20T19:39:11.908712+0000","last_peered":"2024-09-20T19:39:11.908712+0000","last_clean":"2024-09-20T19:39:11.908712+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:39:11.908712+0000","last_undegraded":"2024-09-20T19:39:11.908712+0000","last_fullsized":"2024-09-20T19:39:11.908712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:32.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:32.270373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.192387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1806,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.649984+0000","last_change":"2024-09-20T19:35:49.735749+0000","last_active":"2024-09-20T19:39:12.649984+0000","last_peered":"2024-09-20T19:39:12.649984+0000","last_clean":"2024-09-20T19:39:12.649984+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:39:12.649984+0000","last_undegraded":"2024-09-20T19:39:12.649984+0000","last_fullsized":"2024-09-20T19:39:12.649984+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:49.735628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.735628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1461,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595544+0000","last_change":"2024-09-20T19:36:23.041088+0000","last_active":"2024-09-20T19:39:08.595544+0000","last_peered":"2024-09-20T19:39:08.595544+0000","last_clean":"2024-09-20T19:39:08.595544+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:39:08.595544+0000","last_undegraded":"2024-09-20T19:39:08.595544+0000","last_fullsized":"2024-09-20T19:39:08.595544+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.040967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.040967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.759148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":2082,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642851+0000","last_change":"2024-09-20T19:38:41.037160+0000","last_active":"2024-09-20T19:39:10.642851+0000","last_peered":"2024-09-20T19:39:10.642851+0000","last_clean":"2024-09-20T19:39:10.642851+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:39:10.642851+0000","last_undegraded":"2024-09-20T19:39:10.642851+0000","last_fullsized":"2024-09-20T19:39:10.642851+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:41.037037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.037037+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.295442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1044'1198","reported_seq":5473,"reported_epoch":1044,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:12.651661+0000","last_change":"2024-09-20T19:39:12.646478+0000","last_active":"2024-09-20T19:39:12.651661+0000","last_peered":"2024-09-20T19:39:12.651661+0000","last_clean":"2024-09-20T19:39:12.651661+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:39:12.651661+0000","last_undegraded":"2024-09-20T19:39:12.651661+0000","last_fullsized":"2024-09-20T19:39:12.651661+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'984","last_scrub_stamp":"2024-09-20T19:35:46.717005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:46.717005+0000","objects_scrubbed":10,"log_size":1198,"log_dups_size":0,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.33299669900000001,"stat_sum":{"num_bytes":5799009,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":159971,"num_write":135,"num_write_kb":55283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.7","version":"987'21","reported_seq":1992,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.633579+0000","last_change":"2024-09-20T19:36:23.692393+0000","last_active":"2024-09-20T19:39:11.633579+0000","last_peered":"2024-09-20T19:39:11.633579+0000","last_clean":"2024-09-20T19:39:11.633579+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:39:11.633579+0000","last_undegraded":"2024-09-20T19:39:11.633579+0000","last_fullsized":"2024-09-20T19:39:11.633579+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:36:23.692257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.692257+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.833506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":3277,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.624873+0000","last_change":"2024-09-20T19:37:22.980274+0000","last_active":"2024-09-20T19:39:11.624873+0000","last_peered":"2024-09-20T19:39:11.624873+0000","last_clean":"2024-09-20T19:39:11.624873+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:39:11.624873+0000","last_undegraded":"2024-09-20T19:39:11.624873+0000","last_fullsized":"2024-09-20T19:39:11.624873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:37:22.980139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:37:22.980139+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1246,"num_read_kb":1241,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2046,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.627421+0000","last_change":"2024-09-20T19:37:39.514447+0000","last_active":"2024-09-20T19:39:11.627421+0000","last_peered":"2024-09-20T19:39:11.627421+0000","last_clean":"2024-09-20T19:39:11.627421+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:39:11.627421+0000","last_undegraded":"2024-09-20T19:39:11.627421+0000","last_fullsized":"2024-09-20T19:39:11.627421+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:39.514392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:39.514392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":600,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632609+0000","last_change":"2024-09-20T19:37:34.496871+0000","last_active":"2024-09-20T19:39:11.632609+0000","last_peered":"2024-09-20T19:39:11.632609+0000","last_clean":"2024-09-20T19:39:11.632609+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:39:11.632609+0000","last_undegraded":"2024-09-20T19:39:11.632609+0000","last_fullsized":"2024-09-20T19:39:11.632609+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:34.496752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:34.496752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2006,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643510+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:39:10.643510+0000","last_peered":"2024-09-20T19:39:10.643510+0000","last_clean":"2024-09-20T19:39:10.643510+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:39:10.643510+0000","last_undegraded":"2024-09-20T19:39:10.643510+0000","last_fullsized":"2024-09-20T19:39:10.643510+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.132300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1617,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629294+0000","last_change":"2024-09-20T19:38:14.535923+0000","last_active":"2024-09-20T19:39:11.629294+0000","last_peered":"2024-09-20T19:39:11.629294+0000","last_clean":"2024-09-20T19:39:11.629294+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:39:11.629294+0000","last_undegraded":"2024-09-20T19:39:11.629294+0000","last_fullsized":"2024-09-20T19:39:11.629294+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:14.535715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.535715+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:27.281420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1041'1240","reported_seq":5912,"reported_epoch":1042,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:10.643314+0000","last_change":"2024-09-20T19:39:10.552791+0000","last_active":"2024-09-20T19:39:10.643314+0000","last_peered":"2024-09-20T19:39:10.643314+0000","last_clean":"2024-09-20T19:39:10.643314+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:39:10.643314+0000","last_undegraded":"2024-09-20T19:39:10.643314+0000","last_fullsized":"2024-09-20T19:39:10.643314+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"892'1099","last_scrub_stamp":"2024-09-20T19:36:40.791824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.791824+0000","objects_scrubbed":12,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.90603955999999997,"stat_sum":{"num_bytes":8639755,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":194445,"num_write":169,"num_write_kb":58522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b1","length":"2"},{"start":"1b4","length":"1"}]},{"pgid":"4.6","version":"987'36","reported_seq":1864,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.650940+0000","last_change":"2024-09-20T19:36:33.028171+0000","last_active":"2024-09-20T19:39:12.650940+0000","last_peered":"2024-09-20T19:39:12.650940+0000","last_clean":"2024-09-20T19:39:12.650940+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:39:12.650940+0000","last_undegraded":"2024-09-20T19:39:12.650940+0000","last_fullsized":"2024-09-20T19:39:12.650940+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:36:33.028057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:33.028057+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1760,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.643942+0000","last_change":"2024-09-20T19:38:25.204605+0000","last_active":"2024-09-20T19:39:12.643942+0000","last_peered":"2024-09-20T19:39:12.643942+0000","last_clean":"2024-09-20T19:39:12.643942+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:39:12.643942+0000","last_undegraded":"2024-09-20T19:39:12.643942+0000","last_fullsized":"2024-09-20T19:39:12.643942+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:38:25.204541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.204541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.668929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2019,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642701+0000","last_change":"2024-09-20T19:38:25.143232+0000","last_active":"2024-09-20T19:39:10.642701+0000","last_peered":"2024-09-20T19:39:10.642701+0000","last_clean":"2024-09-20T19:39:10.642701+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:39:10.642701+0000","last_undegraded":"2024-09-20T19:39:10.642701+0000","last_fullsized":"2024-09-20T19:39:10.642701+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.143087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.143087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:32.092810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2020,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909321+0000","last_change":"2024-09-20T19:38:20.052968+0000","last_active":"2024-09-20T19:39:11.909321+0000","last_peered":"2024-09-20T19:39:11.909321+0000","last_clean":"2024-09-20T19:39:11.909321+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:39:11.909321+0000","last_undegraded":"2024-09-20T19:39:11.909321+0000","last_fullsized":"2024-09-20T19:39:11.909321+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.052811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.052811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.621676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2045,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625466+0000","last_change":"2024-09-20T19:37:25.979434+0000","last_active":"2024-09-20T19:39:11.625466+0000","last_peered":"2024-09-20T19:39:11.625466+0000","last_clean":"2024-09-20T19:39:11.625466+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:39:11.625466+0000","last_undegraded":"2024-09-20T19:39:11.625466+0000","last_fullsized":"2024-09-20T19:39:11.625466+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.979338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.979338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":4531,"reported_epoch":1044,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:12.651334+0000","last_change":"2024-09-20T19:39:12.651318+0000","last_active":"2024-09-20T19:39:12.651334+0000","last_peered":"2024-09-20T19:39:12.651334+0000","last_clean":"2024-09-20T19:39:12.651334+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:39:12.651334+0000","last_undegraded":"2024-09-20T19:39:12.651334+0000","last_fullsized":"2024-09-20T19:39:12.651334+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:35:06.711015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:35:06.711015+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.38988613599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.5","version":"987'42","reported_seq":2328,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.780621+0000","last_change":"2024-09-20T19:36:48.949647+0000","last_active":"2024-09-20T19:39:10.780621+0000","last_peered":"2024-09-20T19:39:10.780621+0000","last_clean":"2024-09-20T19:39:10.780621+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:39:10.780621+0000","last_undegraded":"2024-09-20T19:39:10.780621+0000","last_fullsized":"2024-09-20T19:39:10.780621+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:36:48.949462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.949462+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.572488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":872,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908329+0000","last_change":"2024-09-20T19:38:22.129174+0000","last_active":"2024-09-20T19:39:11.908329+0000","last_peered":"2024-09-20T19:39:11.908329+0000","last_clean":"2024-09-20T19:39:11.908329+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:39:11.908329+0000","last_undegraded":"2024-09-20T19:39:11.908329+0000","last_fullsized":"2024-09-20T19:39:11.908329+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:38:22.129045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.129045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.992187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2002,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908969+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:39:11.908969+0000","last_peered":"2024-09-20T19:39:11.908969+0000","last_clean":"2024-09-20T19:39:11.908969+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:39:11.908969+0000","last_undegraded":"2024-09-20T19:39:11.908969+0000","last_fullsized":"2024-09-20T19:39:11.908969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.413613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":2061,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642958+0000","last_change":"2024-09-20T19:34:48.152641+0000","last_active":"2024-09-20T19:39:10.642958+0000","last_peered":"2024-09-20T19:39:10.642958+0000","last_clean":"2024-09-20T19:39:10.642958+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:39:10.642958+0000","last_undegraded":"2024-09-20T19:39:10.642958+0000","last_fullsized":"2024-09-20T19:39:10.642958+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:34:48.152576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:34:48.152576+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1992,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643527+0000","last_change":"2024-09-20T19:38:30.011404+0000","last_active":"2024-09-20T19:39:10.643527+0000","last_peered":"2024-09-20T19:39:10.643527+0000","last_clean":"2024-09-20T19:39:10.643527+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:39:10.643527+0000","last_undegraded":"2024-09-20T19:39:10.643527+0000","last_fullsized":"2024-09-20T19:39:10.643527+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.011277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.011277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.873402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2030,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909465+0000","last_change":"2024-09-20T19:38:34.247348+0000","last_active":"2024-09-20T19:39:11.909465+0000","last_peered":"2024-09-20T19:39:11.909465+0000","last_clean":"2024-09-20T19:39:11.909465+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:39:11.909465+0000","last_undegraded":"2024-09-20T19:39:11.909465+0000","last_fullsized":"2024-09-20T19:39:11.909465+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:34.247292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:34.247292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.593318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"1040'466","reported_seq":4785,"reported_epoch":1044,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:12.646298+0000","last_change":"2024-09-20T19:39:12.646420+0000","last_active":"2024-09-20T19:39:12.646298+0000","last_peered":"2024-09-20T19:39:12.646298+0000","last_clean":"2024-09-20T19:39:12.646298+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:39:12.646298+0000","last_undegraded":"2024-09-20T19:39:12.646298+0000","last_fullsized":"2024-09-20T19:39:12.646298+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'450","last_scrub_stamp":"2024-09-20T19:38:35.237169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.237169+0000","objects_scrubbed":1,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:00.212236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023323179999999999,"stat_sum":{"num_bytes":4625123,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":32970,"num_write":68,"num_write_kb":17761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.a","version":"987'51","reported_seq":2075,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778881+0000","last_change":"2024-09-20T19:37:20.784206+0000","last_active":"2024-09-20T19:39:10.778881+0000","last_peered":"2024-09-20T19:39:10.778881+0000","last_clean":"2024-09-20T19:39:10.778881+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:39:10.778881+0000","last_undegraded":"2024-09-20T19:39:10.778881+0000","last_fullsized":"2024-09-20T19:39:10.778881+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'35","last_scrub_stamp":"2024-09-20T19:37:20.784063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:20.784063+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2047,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.634456+0000","last_change":"2024-09-20T19:38:25.535832+0000","last_active":"2024-09-20T19:39:11.634456+0000","last_peered":"2024-09-20T19:39:11.634456+0000","last_clean":"2024-09-20T19:39:11.634456+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:39:11.634456+0000","last_undegraded":"2024-09-20T19:39:11.634456+0000","last_fullsized":"2024-09-20T19:39:11.634456+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.535708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.535708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.031136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":600,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.631277+0000","last_change":"2024-09-20T19:38:24.553105+0000","last_active":"2024-09-20T19:39:11.631277+0000","last_peered":"2024-09-20T19:39:11.631277+0000","last_clean":"2024-09-20T19:39:11.631277+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:39:11.631277+0000","last_undegraded":"2024-09-20T19:39:11.631277+0000","last_fullsized":"2024-09-20T19:39:11.631277+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:24.553002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.553002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:27.621618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1997,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642871+0000","last_change":"2024-09-20T19:38:14.008440+0000","last_active":"2024-09-20T19:39:10.642871+0000","last_peered":"2024-09-20T19:39:10.642871+0000","last_clean":"2024-09-20T19:39:10.642871+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:39:10.642871+0000","last_undegraded":"2024-09-20T19:39:10.642871+0000","last_fullsized":"2024-09-20T19:39:10.642871+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:14.008316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.008316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.394501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1040'1671","reported_seq":6246,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.626731+0000","last_change":"2024-09-20T19:39:10.724545+0000","last_active":"2024-09-20T19:39:11.626731+0000","last_peered":"2024-09-20T19:39:11.626731+0000","last_clean":"2024-09-20T19:39:11.626731+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:39:11.626731+0000","last_undegraded":"2024-09-20T19:39:11.626731+0000","last_fullsized":"2024-09-20T19:39:11.626731+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'1669","last_scrub_stamp":"2024-09-20T19:39:07.884701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.884701+0000","objects_scrubbed":4,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:33.988373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014277000999999999,"stat_sum":{"num_bytes":5983919,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":272,"num_read_kb":211320,"num_write":187,"num_write_kb":73661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.b","version":"987'21","reported_seq":1830,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.646815+0000","last_change":"2024-09-20T19:35:50.754361+0000","last_active":"2024-09-20T19:39:12.646815+0000","last_peered":"2024-09-20T19:39:12.646815+0000","last_clean":"2024-09-20T19:39:12.646815+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:39:12.646815+0000","last_undegraded":"2024-09-20T19:39:12.646815+0000","last_fullsized":"2024-09-20T19:39:12.646815+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:50.754272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:50.754272+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":2046,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.631121+0000","last_change":"2024-09-20T19:36:34.027338+0000","last_active":"2024-09-20T19:39:11.631121+0000","last_peered":"2024-09-20T19:39:11.631121+0000","last_clean":"2024-09-20T19:39:11.631121+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:39:11.631121+0000","last_undegraded":"2024-09-20T19:39:11.631121+0000","last_fullsized":"2024-09-20T19:39:11.631121+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:34.027257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:34.027257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.639569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2053,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.631791+0000","last_change":"2024-09-20T19:38:31.728072+0000","last_active":"2024-09-20T19:39:11.631791+0000","last_peered":"2024-09-20T19:39:11.631791+0000","last_clean":"2024-09-20T19:39:11.631791+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:39:11.631791+0000","last_undegraded":"2024-09-20T19:39:11.631791+0000","last_fullsized":"2024-09-20T19:39:11.631791+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.728014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.728014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.454777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1822,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.644051+0000","last_change":"2024-09-20T19:35:51.726463+0000","last_active":"2024-09-20T19:39:12.644051+0000","last_peered":"2024-09-20T19:39:12.644051+0000","last_clean":"2024-09-20T19:39:12.644051+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:39:12.644051+0000","last_undegraded":"2024-09-20T19:39:12.644051+0000","last_fullsized":"2024-09-20T19:39:12.644051+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:35:51.726405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:51.726405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"1039'678","reported_seq":5285,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.628020+0000","last_change":"2024-09-20T19:39:10.637285+0000","last_active":"2024-09-20T19:39:11.628020+0000","last_peered":"2024-09-20T19:39:11.628020+0000","last_clean":"2024-09-20T19:39:11.628020+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:39:11.628020+0000","last_undegraded":"2024-09-20T19:39:11.628020+0000","last_fullsized":"2024-09-20T19:39:11.628020+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'666","last_scrub_stamp":"2024-09-20T19:38:31.665799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.665799+0000","objects_scrubbed":2,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.736247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003388168,"stat_sum":{"num_bytes":4925111,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":148484,"num_write":57,"num_write_kb":16671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.8","version":"987'39","reported_seq":2082,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908107+0000","last_change":"2024-09-20T19:39:07.498428+0000","last_active":"2024-09-20T19:39:11.908107+0000","last_peered":"2024-09-20T19:39:11.908107+0000","last_clean":"2024-09-20T19:39:11.908107+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:39:11.908107+0000","last_undegraded":"2024-09-20T19:39:11.908107+0000","last_fullsized":"2024-09-20T19:39:11.908107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:07.498283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.498283+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:11.846306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":604,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632818+0000","last_change":"2024-09-20T19:37:32.482435+0000","last_active":"2024-09-20T19:39:11.632818+0000","last_peered":"2024-09-20T19:39:11.632818+0000","last_clean":"2024-09-20T19:39:11.632818+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:39:11.632818+0000","last_undegraded":"2024-09-20T19:39:11.632818+0000","last_fullsized":"2024-09-20T19:39:11.632818+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:37:32.482370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.482370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.892648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":2268,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642700+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:39:10.642700+0000","last_peered":"2024-09-20T19:39:10.642700+0000","last_clean":"2024-09-20T19:39:10.642700+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:39:10.642700+0000","last_undegraded":"2024-09-20T19:39:10.642700+0000","last_fullsized":"2024-09-20T19:39:10.642700+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.039100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2016,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908065+0000","last_change":"2024-09-20T19:39:05.442773+0000","last_active":"2024-09-20T19:39:11.908065+0000","last_peered":"2024-09-20T19:39:11.908065+0000","last_clean":"2024-09-20T19:39:11.908065+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:39:11.908065+0000","last_undegraded":"2024-09-20T19:39:11.908065+0000","last_fullsized":"2024-09-20T19:39:11.908065+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.442716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.442716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:22.768089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":4686,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.630501+0000","last_change":"2024-09-20T19:39:10.725070+0000","last_active":"2024-09-20T19:39:11.630501+0000","last_peered":"2024-09-20T19:39:11.630501+0000","last_clean":"2024-09-20T19:39:11.630501+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:39:11.630501+0000","last_undegraded":"2024-09-20T19:39:11.630501+0000","last_fullsized":"2024-09-20T19:39:11.630501+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:39:09.971030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.971030+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.319285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025114199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.9","version":"987'24","reported_seq":1895,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594459+0000","last_change":"2024-09-20T19:36:40.974050+0000","last_active":"2024-09-20T19:39:08.594459+0000","last_peered":"2024-09-20T19:39:08.594459+0000","last_clean":"2024-09-20T19:39:08.594459+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:39:08.594459+0000","last_undegraded":"2024-09-20T19:39:08.594459+0000","last_fullsized":"2024-09-20T19:39:08.594459+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:36:40.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.973907+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.768011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1881,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595385+0000","last_change":"2024-09-20T19:38:31.783351+0000","last_active":"2024-09-20T19:39:08.595385+0000","last_peered":"2024-09-20T19:39:08.595385+0000","last_clean":"2024-09-20T19:39:08.595385+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:39:08.595385+0000","last_undegraded":"2024-09-20T19:39:08.595385+0000","last_fullsized":"2024-09-20T19:39:08.595385+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:38:31.783222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.783222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:48.587593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2031,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629065+0000","last_change":"2024-09-20T19:38:36.786857+0000","last_active":"2024-09-20T19:39:11.629065+0000","last_peered":"2024-09-20T19:39:11.629065+0000","last_clean":"2024-09-20T19:39:11.629065+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:39:11.629065+0000","last_undegraded":"2024-09-20T19:39:11.629065+0000","last_fullsized":"2024-09-20T19:39:11.629065+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:36.786803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.786803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:39.996355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1502,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642638+0000","last_change":"2024-09-20T19:37:32.585551+0000","last_active":"2024-09-20T19:39:10.642638+0000","last_peered":"2024-09-20T19:39:10.642638+0000","last_clean":"2024-09-20T19:39:10.642638+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:39:10.642638+0000","last_undegraded":"2024-09-20T19:39:10.642638+0000","last_fullsized":"2024-09-20T19:39:10.642638+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:32.585494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.585494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"1025'1368","reported_seq":6344,"reported_epoch":1042,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:10.643973+0000","last_change":"2024-09-20T19:39:10.643962+0000","last_active":"2024-09-20T19:39:10.643973+0000","last_peered":"2024-09-20T19:39:10.643973+0000","last_clean":"2024-09-20T19:39:10.643973+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:39:10.643973+0000","last_undegraded":"2024-09-20T19:39:10.643973+0000","last_fullsized":"2024-09-20T19:39:10.643973+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'1072","last_scrub_stamp":"2024-09-20T19:34:09.034773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:34:09.034773+0000","objects_scrubbed":12,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.004220858,"stat_sum":{"num_bytes":6774211,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":292,"num_read_kb":216218,"num_write":227,"num_write_kb":88919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.16","version":"987'21","reported_seq":1831,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.650479+0000","last_change":"2024-09-20T19:37:23.121422+0000","last_active":"2024-09-20T19:39:12.650479+0000","last_peered":"2024-09-20T19:39:12.650479+0000","last_clean":"2024-09-20T19:39:12.650479+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:39:12.650479+0000","last_undegraded":"2024-09-20T19:39:12.650479+0000","last_fullsized":"2024-09-20T19:39:12.650479+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:37:23.121282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.121282+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2063,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629640+0000","last_change":"2024-09-20T19:38:30.686760+0000","last_active":"2024-09-20T19:39:11.629640+0000","last_peered":"2024-09-20T19:39:11.629640+0000","last_clean":"2024-09-20T19:39:11.629640+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:39:11.629640+0000","last_undegraded":"2024-09-20T19:39:11.629640+0000","last_fullsized":"2024-09-20T19:39:11.629640+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.686621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.686621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.603556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1499,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629385+0000","last_change":"2024-09-20T19:36:43.911478+0000","last_active":"2024-09-20T19:39:11.629385+0000","last_peered":"2024-09-20T19:39:11.629385+0000","last_clean":"2024-09-20T19:39:11.629385+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:39:11.629385+0000","last_undegraded":"2024-09-20T19:39:11.629385+0000","last_fullsized":"2024-09-20T19:39:11.629385+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:43.911422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:36:43.911422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2018,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.632464+0000","last_change":"2024-09-20T19:37:37.478973+0000","last_active":"2024-09-20T19:39:11.632464+0000","last_peered":"2024-09-20T19:39:11.632464+0000","last_clean":"2024-09-20T19:39:11.632464+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:39:11.632464+0000","last_undegraded":"2024-09-20T19:39:11.632464+0000","last_fullsized":"2024-09-20T19:39:11.632464+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:37.478823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:37.478823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.857932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1040'1644","reported_seq":6276,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.631484+0000","last_change":"2024-09-20T19:39:10.729329+0000","last_active":"2024-09-20T19:39:11.631484+0000","last_peered":"2024-09-20T19:39:11.631484+0000","last_clean":"2024-09-20T19:39:11.631484+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:39:11.631484+0000","last_undegraded":"2024-09-20T19:39:11.631484+0000","last_fullsized":"2024-09-20T19:39:11.631484+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"986'1609","last_scrub_stamp":"2024-09-20T19:38:22.531869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.531869+0000","objects_scrubbed":3,"log_size":1644,"log_dups_size":0,"ondisk_log_size":1644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:36.106701+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0054884560000000001,"stat_sum":{"num_bytes":7748984,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":304,"num_read_kb":200572,"num_write":228,"num_write_kb":84794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"2.11","version":"0'0","reported_seq":2011,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778471+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:39:10.778471+0000","last_peered":"2024-09-20T19:39:10.778471+0000","last_clean":"2024-09-20T19:39:10.778471+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:39:10.778471+0000","last_undegraded":"2024-09-20T19:39:10.778471+0000","last_fullsized":"2024-09-20T19:39:10.778471+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.535576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"987'18","reported_seq":1679,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643839+0000","last_change":"2024-09-20T19:36:39.739712+0000","last_active":"2024-09-20T19:39:10.643839+0000","last_peered":"2024-09-20T19:39:10.643839+0000","last_clean":"2024-09-20T19:39:10.643839+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:39:10.643839+0000","last_undegraded":"2024-09-20T19:39:10.643839+0000","last_fullsized":"2024-09-20T19:39:10.643839+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:36:39.739615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:39.739615+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2034,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909097+0000","last_change":"2024-09-20T19:37:43.848462+0000","last_active":"2024-09-20T19:39:11.909097+0000","last_peered":"2024-09-20T19:39:11.909097+0000","last_clean":"2024-09-20T19:39:11.909097+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:39:11.909097+0000","last_undegraded":"2024-09-20T19:39:11.909097+0000","last_fullsized":"2024-09-20T19:39:11.909097+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:43.848400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:43.848400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1989,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643226+0000","last_change":"2024-09-20T19:34:44.066655+0000","last_active":"2024-09-20T19:39:10.643226+0000","last_peered":"2024-09-20T19:39:10.643226+0000","last_clean":"2024-09-20T19:39:10.643226+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:39:10.643226+0000","last_undegraded":"2024-09-20T19:39:10.643226+0000","last_fullsized":"2024-09-20T19:39:10.643226+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1020'831","reported_seq":5336,"reported_epoch":1042,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:10.644107+0000","last_change":"2024-09-20T19:39:10.644095+0000","last_active":"2024-09-20T19:39:10.644107+0000","last_peered":"2024-09-20T19:39:10.644107+0000","last_clean":"2024-09-20T19:39:10.644107+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:39:10.644107+0000","last_undegraded":"2024-09-20T19:39:10.644107+0000","last_fullsized":"2024-09-20T19:39:10.644107+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"988'810","last_scrub_stamp":"2024-09-20T19:38:21.101246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:21.101246+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:24.230150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0033877229999999,"stat_sum":{"num_bytes":2118012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":145,"num_read_kb":108369,"num_write":90,"num_write_kb":32667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.14","version":"987'30","reported_seq":1704,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.626271+0000","last_change":"2024-09-20T19:36:36.023059+0000","last_active":"2024-09-20T19:39:11.626271+0000","last_peered":"2024-09-20T19:39:11.626271+0000","last_clean":"2024-09-20T19:39:11.626271+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:39:11.626271+0000","last_undegraded":"2024-09-20T19:39:11.626271+0000","last_fullsized":"2024-09-20T19:39:11.626271+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:36:36.022993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:36.022993+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:51.646412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":2043,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908729+0000","last_change":"2024-09-20T19:38:31.227463+0000","last_active":"2024-09-20T19:39:11.908729+0000","last_peered":"2024-09-20T19:39:11.908729+0000","last_clean":"2024-09-20T19:39:11.908729+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:39:11.908729+0000","last_undegraded":"2024-09-20T19:39:11.908729+0000","last_fullsized":"2024-09-20T19:39:11.908729+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.227391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.227391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.357562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":2285,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908796+0000","last_change":"2024-09-20T19:38:19.049146+0000","last_active":"2024-09-20T19:39:11.908796+0000","last_peered":"2024-09-20T19:39:11.908796+0000","last_clean":"2024-09-20T19:39:11.908796+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:39:11.908796+0000","last_undegraded":"2024-09-20T19:39:11.908796+0000","last_fullsized":"2024-09-20T19:39:11.908796+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.049091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.049091+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:31.883013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1794,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.651546+0000","last_change":"2024-09-20T19:35:56.751414+0000","last_active":"2024-09-20T19:39:12.651546+0000","last_peered":"2024-09-20T19:39:12.651546+0000","last_clean":"2024-09-20T19:39:12.651546+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:39:12.651546+0000","last_undegraded":"2024-09-20T19:39:12.651546+0000","last_fullsized":"2024-09-20T19:39:12.651546+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:56.751367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:56.751367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1014'757","reported_seq":5070,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.598380+0000","last_change":"2024-09-20T19:39:08.600476+0000","last_active":"2024-09-20T19:39:08.598380+0000","last_peered":"2024-09-20T19:39:08.598380+0000","last_clean":"2024-09-20T19:39:08.598380+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:39:08.598380+0000","last_undegraded":"2024-09-20T19:39:08.598380+0000","last_fullsized":"2024-09-20T19:39:08.598380+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"994'754","last_scrub_stamp":"2024-09-20T19:38:40.806420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.806420+0000","objects_scrubbed":1,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:51.484650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0045057810000000004,"stat_sum":{"num_bytes":2084398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":184,"num_read_kb":132045,"num_write":133,"num_write_kb":50709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.15","version":"987'21","reported_seq":2089,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908848+0000","last_change":"2024-09-20T19:38:30.190106+0000","last_active":"2024-09-20T19:39:11.908848+0000","last_peered":"2024-09-20T19:39:11.908848+0000","last_clean":"2024-09-20T19:39:11.908848+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:39:11.908848+0000","last_undegraded":"2024-09-20T19:39:11.908848+0000","last_fullsized":"2024-09-20T19:39:11.908848+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.190020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.190020+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.116547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1767,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.647471+0000","last_change":"2024-09-20T19:39:12.256710+0000","last_active":"2024-09-20T19:39:12.647471+0000","last_peered":"2024-09-20T19:39:12.647471+0000","last_clean":"2024-09-20T19:39:12.647471+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:39:12.647471+0000","last_undegraded":"2024-09-20T19:39:12.647471+0000","last_fullsized":"2024-09-20T19:39:12.647471+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:12.256601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.256601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.448150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":734,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778644+0000","last_change":"2024-09-20T19:38:27.671255+0000","last_active":"2024-09-20T19:39:10.778644+0000","last_peered":"2024-09-20T19:39:10.778644+0000","last_clean":"2024-09-20T19:39:10.778644+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:39:10.778644+0000","last_undegraded":"2024-09-20T19:39:10.778644+0000","last_fullsized":"2024-09-20T19:39:10.778644+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'17","last_scrub_stamp":"2024-09-20T19:38:27.671197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.671197+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.930015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1843,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595251+0000","last_change":"2024-09-20T19:36:49.011748+0000","last_active":"2024-09-20T19:39:08.595251+0000","last_peered":"2024-09-20T19:39:08.595251+0000","last_clean":"2024-09-20T19:39:08.595251+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:39:08.595251+0000","last_undegraded":"2024-09-20T19:39:08.595251+0000","last_fullsized":"2024-09-20T19:39:08.595251+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:49.011625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:49.011625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.796400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1040'624","reported_seq":5232,"reported_epoch":1042,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T19:39:10.642468+0000","last_change":"2024-09-20T19:39:10.642445+0000","last_active":"2024-09-20T19:39:10.642468+0000","last_peered":"2024-09-20T19:39:10.642468+0000","last_clean":"2024-09-20T19:39:10.642468+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:39:10.642468+0000","last_undegraded":"2024-09-20T19:39:10.642468+0000","last_fullsized":"2024-09-20T19:39:10.642468+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"870'563","last_scrub_stamp":"2024-09-20T19:36:28.907373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:28.907373+0000","objects_scrubbed":6,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.00487017,"stat_sum":{"num_bytes":2437345,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":54472,"num_write":51,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.12","version":"987'21","reported_seq":2052,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625766+0000","last_change":"2024-09-20T19:35:49.126133+0000","last_active":"2024-09-20T19:39:11.625766+0000","last_peered":"2024-09-20T19:39:11.625766+0000","last_clean":"2024-09-20T19:39:11.625766+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:39:11.625766+0000","last_undegraded":"2024-09-20T19:39:11.625766+0000","last_fullsized":"2024-09-20T19:39:11.625766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:49.126032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.126032+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.626720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2010,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.779620+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:39:10.779620+0000","last_peered":"2024-09-20T19:39:10.779620+0000","last_clean":"2024-09-20T19:39:10.779620+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:39:10.779620+0000","last_undegraded":"2024-09-20T19:39:10.779620+0000","last_fullsized":"2024-09-20T19:39:10.779620+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.462527+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1523,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.643072+0000","last_change":"2024-09-20T19:37:20.195581+0000","last_active":"2024-09-20T19:39:12.643072+0000","last_peered":"2024-09-20T19:39:12.643072+0000","last_clean":"2024-09-20T19:39:12.643072+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:39:12.643072+0000","last_undegraded":"2024-09-20T19:39:12.643072+0000","last_fullsized":"2024-09-20T19:39:12.643072+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:20.195458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:20.195458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1769,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.645472+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:39:12.645472+0000","last_peered":"2024-09-20T19:39:12.645472+0000","last_clean":"2024-09-20T19:39:12.645472+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:39:12.645472+0000","last_undegraded":"2024-09-20T19:39:12.645472+0000","last_fullsized":"2024-09-20T19:39:12.645472+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.129003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1006'762","reported_seq":5331,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.626037+0000","last_change":"2024-09-20T19:39:10.725387+0000","last_active":"2024-09-20T19:39:11.626037+0000","last_peered":"2024-09-20T19:39:11.626037+0000","last_clean":"2024-09-20T19:39:11.626037+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:39:11.626037+0000","last_undegraded":"2024-09-20T19:39:11.626037+0000","last_fullsized":"2024-09-20T19:39:11.626037+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"979'756","last_scrub_stamp":"2024-09-20T19:38:12.650090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.650090+0000","objects_scrubbed":1,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:24.285129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015233426,"stat_sum":{"num_bytes":3966749,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":86291,"num_write":62,"num_write_kb":17826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.13","version":"987'27","reported_seq":1862,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594448+0000","last_change":"2024-09-20T19:38:17.754780+0000","last_active":"2024-09-20T19:39:08.594448+0000","last_peered":"2024-09-20T19:39:08.594448+0000","last_clean":"2024-09-20T19:39:08.594448+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:39:08.594448+0000","last_undegraded":"2024-09-20T19:39:08.594448+0000","last_fullsized":"2024-09-20T19:39:08.594448+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.754717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.754717+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:21.650602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":2040,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.631084+0000","last_change":"2024-09-20T19:39:09.812428+0000","last_active":"2024-09-20T19:39:11.631084+0000","last_peered":"2024-09-20T19:39:11.631084+0000","last_clean":"2024-09-20T19:39:11.631084+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:39:11.631084+0000","last_undegraded":"2024-09-20T19:39:11.631084+0000","last_fullsized":"2024-09-20T19:39:11.631084+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:09.812317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.812317+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.550714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":2035,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625286+0000","last_change":"2024-09-20T19:37:23.932269+0000","last_active":"2024-09-20T19:39:11.625286+0000","last_peered":"2024-09-20T19:39:11.625286+0000","last_clean":"2024-09-20T19:39:11.625286+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:39:11.625286+0000","last_undegraded":"2024-09-20T19:39:11.625286+0000","last_fullsized":"2024-09-20T19:39:11.625286+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:23.932204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.932204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1527,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.643824+0000","last_change":"2024-09-20T19:38:40.086970+0000","last_active":"2024-09-20T19:39:12.643824+0000","last_peered":"2024-09-20T19:39:12.643824+0000","last_clean":"2024-09-20T19:39:12.643824+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:39:12.643824+0000","last_undegraded":"2024-09-20T19:39:12.643824+0000","last_fullsized":"2024-09-20T19:39:12.643824+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.086844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.086844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:00.305793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"1043'547","reported_seq":5131,"reported_epoch":1044,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:12.643356+0000","last_change":"2024-09-20T19:39:12.588779+0000","last_active":"2024-09-20T19:39:12.643356+0000","last_peered":"2024-09-20T19:39:12.643356+0000","last_clean":"2024-09-20T19:39:12.643356+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:39:12.643356+0000","last_undegraded":"2024-09-20T19:39:12.643356+0000","last_fullsized":"2024-09-20T19:39:12.643356+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'542","last_scrub_stamp":"2024-09-20T19:38:30.128571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.128571+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.498904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.94673084500000004,"stat_sum":{"num_bytes":3344890,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":17,"num_read_kb":8,"num_write":50,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"1"}]},{"pgid":"4.10","version":"987'12","reported_seq":1687,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625641+0000","last_change":"2024-09-20T19:39:04.841388+0000","last_active":"2024-09-20T19:39:11.625641+0000","last_peered":"2024-09-20T19:39:11.625641+0000","last_clean":"2024-09-20T19:39:11.625641+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:39:11.625641+0000","last_undegraded":"2024-09-20T19:39:11.625641+0000","last_fullsized":"2024-09-20T19:39:11.625641+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12","last_scrub_stamp":"2024-09-20T19:39:04.841305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:04.841305+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.241113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":2022,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909185+0000","last_change":"2024-09-20T19:37:36.911119+0000","last_active":"2024-09-20T19:39:11.909185+0000","last_peered":"2024-09-20T19:39:11.909185+0000","last_clean":"2024-09-20T19:39:11.909185+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:39:11.909185+0000","last_undegraded":"2024-09-20T19:39:11.909185+0000","last_fullsized":"2024-09-20T19:39:11.909185+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:36.910997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:36.910997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2003,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.777983+0000","last_change":"2024-09-20T19:35:03.419683+0000","last_active":"2024-09-20T19:39:10.777983+0000","last_peered":"2024-09-20T19:39:10.777983+0000","last_clean":"2024-09-20T19:39:10.777983+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:39:10.777983+0000","last_undegraded":"2024-09-20T19:39:10.777983+0000","last_fullsized":"2024-09-20T19:39:10.777983+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:03.419628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:03.419628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1981,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642470+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:39:10.642470+0000","last_peered":"2024-09-20T19:39:10.642470+0000","last_clean":"2024-09-20T19:39:10.642470+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:39:10.642470+0000","last_undegraded":"2024-09-20T19:39:10.642470+0000","last_fullsized":"2024-09-20T19:39:10.642470+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.047922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":4631,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.779951+0000","last_change":"2024-09-20T19:39:10.781638+0000","last_active":"2024-09-20T19:39:10.779951+0000","last_peered":"2024-09-20T19:39:10.779951+0000","last_clean":"2024-09-20T19:39:10.779951+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:39:10.779951+0000","last_undegraded":"2024-09-20T19:39:10.779951+0000","last_fullsized":"2024-09-20T19:39:10.779951+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:38:16.787693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:16.787693+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:19.615323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032148820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"4.11","version":"987'27","reported_seq":898,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642866+0000","last_change":"2024-09-20T19:38:17.128241+0000","last_active":"2024-09-20T19:39:10.642866+0000","last_peered":"2024-09-20T19:39:10.642866+0000","last_clean":"2024-09-20T19:39:10.642866+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:39:10.642866+0000","last_undegraded":"2024-09-20T19:39:10.642866+0000","last_fullsized":"2024-09-20T19:39:10.642866+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.128056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.128056+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:20.660206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2007,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.782585+0000","last_change":"2024-09-20T19:37:28.709229+0000","last_active":"2024-09-20T19:39:10.782585+0000","last_peered":"2024-09-20T19:39:10.782585+0000","last_clean":"2024-09-20T19:39:10.782585+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:39:10.782585+0000","last_undegraded":"2024-09-20T19:39:10.782585+0000","last_fullsized":"2024-09-20T19:39:10.782585+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:28.709108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:28.709108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":2010,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.642384+0000","last_change":"2024-09-20T19:38:35.028245+0000","last_active":"2024-09-20T19:39:10.642384+0000","last_peered":"2024-09-20T19:39:10.642384+0000","last_clean":"2024-09-20T19:39:10.642384+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:39:10.642384+0000","last_undegraded":"2024-09-20T19:39:10.642384+0000","last_fullsized":"2024-09-20T19:39:10.642384+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.028101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.028101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:56.459400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1535,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.645438+0000","last_change":"2024-09-20T19:36:47.076088+0000","last_active":"2024-09-20T19:39:12.645438+0000","last_peered":"2024-09-20T19:39:12.645438+0000","last_clean":"2024-09-20T19:39:12.645438+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:39:12.645438+0000","last_undegraded":"2024-09-20T19:39:12.645438+0000","last_fullsized":"2024-09-20T19:39:12.645438+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:47.075960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:47.075960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"1041'844","reported_seq":5442,"reported_epoch":1043,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T19:39:11.630004+0000","last_change":"2024-09-20T19:39:10.519499+0000","last_active":"2024-09-20T19:39:11.630004+0000","last_peered":"2024-09-20T19:39:11.630004+0000","last_clean":"2024-09-20T19:39:11.630004+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:39:11.630004+0000","last_undegraded":"2024-09-20T19:39:11.630004+0000","last_fullsized":"2024-09-20T19:39:11.630004+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"999'811","last_scrub_stamp":"2024-09-20T19:38:37.609506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:37.609506+0000","objects_scrubbed":1,"log_size":844,"log_dups_size":0,"ondisk_log_size":844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.568004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.91680477400000004,"stat_sum":{"num_bytes":6767605,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":99462,"num_write":87,"num_write_kb":36359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b1","length":"2"},{"start":"1b4","length":"1"}]},{"pgid":"6.1c","version":"77'1","reported_seq":1988,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643426+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:39:10.643426+0000","last_peered":"2024-09-20T19:39:10.643426+0000","last_clean":"2024-09-20T19:39:10.643426+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:39:10.643426+0000","last_undegraded":"2024-09-20T19:39:10.643426+0000","last_fullsized":"2024-09-20T19:39:10.643426+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.063854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":2027,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.908180+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:39:11.908180+0000","last_peered":"2024-09-20T19:39:11.908180+0000","last_clean":"2024-09-20T19:39:11.908180+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:39:11.908180+0000","last_undegraded":"2024-09-20T19:39:11.908180+0000","last_fullsized":"2024-09-20T19:39:11.908180+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.325845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"987'30","reported_seq":1837,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.650833+0000","last_change":"2024-09-20T19:38:41.053008+0000","last_active":"2024-09-20T19:39:12.650833+0000","last_peered":"2024-09-20T19:39:12.650833+0000","last_clean":"2024-09-20T19:39:12.650833+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:39:12.650833+0000","last_undegraded":"2024-09-20T19:39:12.650833+0000","last_fullsized":"2024-09-20T19:39:12.650833+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:41.052945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.052945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.592511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":2281,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.782093+0000","last_change":"2024-09-20T19:38:12.764542+0000","last_active":"2024-09-20T19:39:10.782093+0000","last_peered":"2024-09-20T19:39:10.782093+0000","last_clean":"2024-09-20T19:39:10.782093+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:39:10.782093+0000","last_undegraded":"2024-09-20T19:39:10.782093+0000","last_fullsized":"2024-09-20T19:39:10.782093+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:38:12.764391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.764391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.592596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"1040'1118","reported_seq":5901,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.909262+0000","last_change":"2024-09-20T19:39:10.819371+0000","last_active":"2024-09-20T19:39:11.909262+0000","last_peered":"2024-09-20T19:39:11.909262+0000","last_clean":"2024-09-20T19:39:11.909262+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:39:11.909262+0000","last_undegraded":"2024-09-20T19:39:11.909262+0000","last_fullsized":"2024-09-20T19:39:11.909262+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"946'1053","last_scrub_stamp":"2024-09-20T19:37:35.945540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:35.945540+0000","objects_scrubbed":3,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0042217460000000002,"stat_sum":{"num_bytes":10606108,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":90953,"num_write":82,"num_write_kb":31274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":2012,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.626611+0000","last_change":"2024-09-20T19:36:23.037110+0000","last_active":"2024-09-20T19:39:11.626611+0000","last_peered":"2024-09-20T19:39:11.626611+0000","last_clean":"2024-09-20T19:39:11.626611+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:39:11.626611+0000","last_undegraded":"2024-09-20T19:39:11.626611+0000","last_fullsized":"2024-09-20T19:39:11.626611+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.036982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.036982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.561071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":848,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.778196+0000","last_change":"2024-09-20T19:36:42.052037+0000","last_active":"2024-09-20T19:39:10.778196+0000","last_peered":"2024-09-20T19:39:10.778196+0000","last_clean":"2024-09-20T19:39:10.778196+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:39:10.778196+0000","last_undegraded":"2024-09-20T19:39:10.778196+0000","last_fullsized":"2024-09-20T19:39:10.778196+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:36:42.051911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:42.051911+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"987'27","reported_seq":2070,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.782381+0000","last_change":"2024-09-20T19:38:15.768347+0000","last_active":"2024-09-20T19:39:10.782381+0000","last_peered":"2024-09-20T19:39:10.782381+0000","last_clean":"2024-09-20T19:39:10.782381+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:39:10.782381+0000","last_undegraded":"2024-09-20T19:39:10.782381+0000","last_fullsized":"2024-09-20T19:39:10.782381+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:38:15.768276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:15.768276+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.350115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1807,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.651405+0000","last_change":"2024-09-20T19:35:55.754020+0000","last_active":"2024-09-20T19:39:12.651405+0000","last_peered":"2024-09-20T19:39:12.651405+0000","last_clean":"2024-09-20T19:39:12.651405+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:39:12.651405+0000","last_undegraded":"2024-09-20T19:39:12.651405+0000","last_fullsized":"2024-09-20T19:39:12.651405+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:55.753897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:55.753897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"1040'740","reported_seq":5134,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.783489+0000","last_change":"2024-09-20T19:39:10.794924+0000","last_active":"2024-09-20T19:39:10.783489+0000","last_peered":"2024-09-20T19:39:10.783489+0000","last_clean":"2024-09-20T19:39:10.783489+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:39:10.783489+0000","last_undegraded":"2024-09-20T19:39:10.783489+0000","last_fullsized":"2024-09-20T19:39:10.783489+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'671","last_scrub_stamp":"2024-09-20T19:36:40.118767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.118767+0000","objects_scrubbed":7,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014330816,"stat_sum":{"num_bytes":5950305,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":59453,"num_write":41,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":1825,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.595653+0000","last_change":"2024-09-20T19:36:27.046038+0000","last_active":"2024-09-20T19:39:08.595653+0000","last_peered":"2024-09-20T19:39:08.595653+0000","last_clean":"2024-09-20T19:39:08.595653+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:39:08.595653+0000","last_undegraded":"2024-09-20T19:39:08.595653+0000","last_fullsized":"2024-09-20T19:39:08.595653+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:27.045955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:27.045955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.803434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2018,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.779163+0000","last_change":"2024-09-20T19:35:47.234215+0000","last_active":"2024-09-20T19:39:10.779163+0000","last_peered":"2024-09-20T19:39:10.779163+0000","last_clean":"2024-09-20T19:39:10.779163+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:39:10.779163+0000","last_undegraded":"2024-09-20T19:39:10.779163+0000","last_fullsized":"2024-09-20T19:39:10.779163+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.234087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.234087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"987'39","reported_seq":633,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.629234+0000","last_change":"2024-09-20T19:37:44.510960+0000","last_active":"2024-09-20T19:39:11.629234+0000","last_peered":"2024-09-20T19:39:11.629234+0000","last_clean":"2024-09-20T19:39:11.629234+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:39:11.629234+0000","last_undegraded":"2024-09-20T19:39:11.629234+0000","last_fullsized":"2024-09-20T19:39:11.629234+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'27","last_scrub_stamp":"2024-09-20T19:37:44.510895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:44.510895+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":2031,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.630542+0000","last_change":"2024-09-20T19:38:33.621829+0000","last_active":"2024-09-20T19:39:11.630542+0000","last_peered":"2024-09-20T19:39:11.630542+0000","last_clean":"2024-09-20T19:39:11.630542+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:39:11.630542+0000","last_undegraded":"2024-09-20T19:39:11.630542+0000","last_fullsized":"2024-09-20T19:39:11.630542+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:33.621779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.621779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.572657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":4833,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.625258+0000","last_change":"2024-09-20T19:39:10.638904+0000","last_active":"2024-09-20T19:39:11.625258+0000","last_peered":"2024-09-20T19:39:11.625258+0000","last_clean":"2024-09-20T19:39:11.625258+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:39:11.625258+0000","last_undegraded":"2024-09-20T19:39:11.625258+0000","last_fullsized":"2024-09-20T19:39:11.625258+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:38:28.709181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.709181+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:45.339627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004263948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ac","length":"3"},{"start":"1b0","length":"3"},{"start":"1b4","length":"2"}]},{"pgid":"6.1f","version":"0'0","reported_seq":660,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-09-20T19:39:12.649983+0000","last_change":"2024-09-20T19:38:23.240112+0000","last_active":"2024-09-20T19:39:12.649983+0000","last_peered":"2024-09-20T19:39:12.649983+0000","last_clean":"2024-09-20T19:39:12.649983+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:39:12.649983+0000","last_undegraded":"2024-09-20T19:39:12.649983+0000","last_fullsized":"2024-09-20T19:39:12.649983+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:23.240001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.240001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.870584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1495,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-20T19:39:10.643209+0000","last_change":"2024-09-20T19:34:51.148618+0000","last_active":"2024-09-20T19:39:10.643209+0000","last_peered":"2024-09-20T19:39:10.643209+0000","last_clean":"2024-09-20T19:39:10.643209+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:39:10.643209+0000","last_undegraded":"2024-09-20T19:39:10.643209+0000","last_fullsized":"2024-09-20T19:39:10.643209+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.148494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.148494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"987'24","reported_seq":910,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-09-20T19:39:11.630797+0000","last_change":"2024-09-20T19:37:28.963853+0000","last_active":"2024-09-20T19:39:11.630797+0000","last_peered":"2024-09-20T19:39:11.630797+0000","last_clean":"2024-09-20T19:39:11.630797+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:39:11.630797+0000","last_undegraded":"2024-09-20T19:39:11.630797+0000","last_fullsized":"2024-09-20T19:39:11.630797+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:37:28.963655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:28.963655+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1838,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-20T19:39:08.594676+0000","last_change":"2024-09-20T19:38:20.716172+0000","last_active":"2024-09-20T19:39:08.594676+0000","last_peered":"2024-09-20T19:39:08.594676+0000","last_clean":"2024-09-20T19:39:08.594676+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:39:08.594676+0000","last_undegraded":"2024-09-20T19:39:08.594676+0000","last_fullsized":"2024-09-20T19:39:08.594676+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.716039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.716039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:23.156821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":128349945,"num_objects":53,"num_object_clones":11,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":2,"num_read":3499,"num_read_kb":2730006,"num_write":2433,"num_write_kb":919070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":363806720,"data_stored":429581241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":27043,"ondisk_log_size":27043,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1246,"num_read_kb":1241,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907206,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129708,"kb_used_data":57900,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93615444,"statfs":{"total":95995035648,"available":95862214656,"internally_reserved":0,"allocated":59289600,"data_stored":64952400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234257,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112980,"kb_used_data":36828,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93632172,"statfs":{"total":95995035648,"available":95879344128,"internally_reserved":0,"allocated":37711872,"data_stored":39104058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561307,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92572,"kb_used_data":38812,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93652580,"statfs":{"total":95995035648,"available":95900241920,"internally_reserved":0,"allocated":39743488,"data_stored":41854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928617,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93228,"kb_used_data":34652,"kb_used_omap":35,"kb_used_meta":58332,"kb_avail":93651924,"statfs":{"total":95995035648,"available":95899570176,"internally_reserved":0,"allocated":35483648,"data_stored":44010337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":59732913},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378943,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111044,"kb_used_data":35260,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93634108,"statfs":{"total":95995035648,"available":95881326592,"internally_reserved":0,"allocated":36106240,"data_stored":39190767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575179,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115916,"kb_used_data":48084,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93629236,"statfs":{"total":95995035648,"available":95876337664,"internally_reserved":0,"allocated":49238016,"data_stored":52668617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648110,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104604,"kb_used_data":53268,"kb_used_omap":49,"kb_used_meta":51086,"kb_avail":93640548,"statfs":{"total":95995035648,"available":95887921152,"internally_reserved":0,"allocated":54546432,"data_stored":62171197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":52312617},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53293056,"data_stored":61450228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48156672,"data_stored":52053120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34754560,"data_stored":38312833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33894400,"data_stored":43076771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37904384,"data_stored":40543304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35848192,"data_stored":37792527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57716736,"data_stored":63904127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:39:16.420 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:39:16.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.444 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:16.519 INFO:tasks.rados.rados.0.smithi042.stdout:3978: done (5 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3979: done (4 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3980: done (3 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3981: done (2 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3983: done (1 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3984: done (0 left) 2024-09-20T19:39:16.520 INFO:tasks.rados.rados.0.smithi042.stdout:3985: delete oid 11 current snap is 440 2024-09-20T19:39:16.538 INFO:tasks.rados.rados.0.smithi042.stdout:3985: done (0 left) 2024-09-20T19:39:16.538 INFO:tasks.rados.rados.0.smithi042.stdout:3986: write oid 30 current snap is 440 2024-09-20T19:39:16.538 INFO:tasks.rados.rados.0.smithi042.stdout:3986: seq_num 1360 ranges {457520=649474,1900805=662799} 2024-09-20T19:39:16.546 INFO:tasks.rados.rados.0.smithi042.stdout:3986: writing smithi04268048-30 from 457520 to 1106994 tid 1 2024-09-20T19:39:16.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.569 INFO:tasks.rados.rados.0.smithi042.stdout:3986: writing smithi04268048-30 from 1900805 to 2563604 tid 2 2024-09-20T19:39:16.569 INFO:tasks.rados.rados.0.smithi042.stdout:3987: delete oid 9 current snap is 440 2024-09-20T19:39:16.569 INFO:tasks.rados.rados.0.smithi042.stdout:3986: finishing write tid 1 to smithi04268048-30 2024-09-20T19:39:16.572 INFO:tasks.rados.rados.0.smithi042.stdout:3987: done (1 left) 2024-09-20T19:39:16.572 INFO:tasks.rados.rados.0.smithi042.stdout:3988: snap_create 2024-09-20T19:39:16.572 INFO:tasks.rados.rados.0.smithi042.stdout:3986: finishing write tid 2 to smithi04268048-30 2024-09-20T19:39:16.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:16.573 INFO:tasks.rados.rados.0.smithi042.stdout:3986: finishing write tid 3 to smithi04268048-30 2024-09-20T19:39:16.573 INFO:tasks.rados.rados.0.smithi042.stdout:3986: finishing write tid 4 to smithi04268048-30 2024-09-20T19:39:16.573 INFO:tasks.rados.rados.0.smithi042.stdout:3986: oid 30 updating version 0 to 1651 2024-09-20T19:39:16.574 INFO:tasks.rados.rados.0.smithi042.stdout:3986: oid 30 updating version 1651 to 1653 2024-09-20T19:39:16.574 INFO:tasks.rados.rados.0.smithi042.stdout:3986: oid 30 version 1653 is already newer than 1652 2024-09-20T19:39:16.574 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 30 v 1653 (ObjNum 1360 snap 440 seq_num 1360) dirty exists 2024-09-20T19:39:16.574 INFO:tasks.rados.rados.0.smithi042.stdout:3986: left oid 30 (ObjNum 1360 snap 440 seq_num 1360) 2024-09-20T19:39:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:16.665 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.665 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.665 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.666 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.667 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.676 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:16.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:16.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:16.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:16 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:16.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.490843+0000 osd.5 (osd.5) 327 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.494125+0000 osd.5 (osd.5) 328 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.908345+0000 osd.1 (osd.1) 363 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:39:16.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:15.910677+0000 osd.1 (osd.1) 364 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.000201+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.001688+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: audit 2024-09-20T19:39:16.342317+0000 mon.a (mon.0) 2713 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.381017+0000 osd.6 (osd.6) 242 : cluster [DBG] 7.1f scrub starts 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.382721+0000 osd.6 (osd.6) 243 : cluster [DBG] 7.1f scrub ok 2024-09-20T19:39:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:16 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.517440+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2024-09-20T19:39:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:16.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:17.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:17.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:17.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.199 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:17.303 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:17.324 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:17.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.447 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:17.517 INFO:tasks.rados.rados.0.smithi042.stdout:3986: done (1 left) 2024-09-20T19:39:17.517 INFO:tasks.rados.rados.0.smithi042.stdout:3988: done (0 left) 2024-09-20T19:39:17.517 INFO:tasks.rados.rados.0.smithi042.stdout:3989: snap_remove snap 440 2024-09-20T19:39:17.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:17.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:17.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.694 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:17.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:17.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[22706]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.813 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:17 smithi042 bash[32075]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.818 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:17.870 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:39:17.870 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:39:17.870 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-20T19:39:17.870 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.499970+0000 mgr.y (mgr.24491) 797 : cluster [DBG] pgmap v1569: 178 pgs: 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 163 active+clean; 122 MiB data, 740 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 334 KiB/s wr, 1 op/s 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.900891+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.906330+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.926276+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:16.927567+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.942766+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:16.949206+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.035176+0000 osd.7 (osd.7) 243 : cluster [DBG] 7.15 scrub starts 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.039945+0000 osd.7 (osd.7) 244 : cluster [DBG] 7.15 scrub ok 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.376596+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.377846+0000 osd.6 (osd.6) 245 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.516076+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.582385+0000 mon.a (mon.0) 2720 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:39:17.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.583404+0000 mon.a (mon.0) 2721 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:39:17.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:17 smithi162 bash[24557]: audit 2024-09-20T19:39:17.591242+0000 mon.a (mon.0) 2722 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:17.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:17.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:17.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:18.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:18.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:18.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:18.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:39:18.294 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:18.318 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:18.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:18.445 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:18.518 INFO:tasks.rados.rados.0.smithi042.stdout:3989: done (0 left) 2024-09-20T19:39:18.518 INFO:tasks.rados.rados.0.smithi042.stdout:3990: write (excl) oid 34 current snap is 441 2024-09-20T19:39:18.518 INFO:tasks.rados.rados.0.smithi042.stdout:3990: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-09-20T19:39:18.524 INFO:tasks.rados.rados.0.smithi042.stdout:3990: writing smithi04268048-34 from 491134 to 1007459 tid 1 2024-09-20T19:39:18.532 INFO:tasks.rados.rados.0.smithi042.stdout:3990: writing smithi04268048-34 from 1734122 to 2400590 tid 2 2024-09-20T19:39:18.532 INFO:tasks.rados.rados.0.smithi042.stdout:3991: write oid 11 current snap is 441 2024-09-20T19:39:18.532 INFO:tasks.rados.rados.0.smithi042.stdout:3991: seq_num 1363 ranges {507941=591574,1676251=526479,2746598=161343} 2024-09-20T19:39:18.543 INFO:tasks.rados.rados.0.smithi042.stdout:3991: writing smithi04268048-11 from 507941 to 1099515 tid 1 2024-09-20T19:39:18.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:18.554 INFO:tasks.rados.rados.0.smithi042.stdout:3991: writing smithi04268048-11 from 1676251 to 2202730 tid 2 2024-09-20T19:39:18.560 INFO:tasks.rados.rados.0.smithi042.stdout:3991: writing smithi04268048-11 from 2746598 to 2907941 tid 3 2024-09-20T19:39:18.560 INFO:tasks.rados.rados.0.smithi042.stdout:3992: read oid 35 snap -1 2024-09-20T19:39:18.560 INFO:tasks.rados.rados.0.smithi042.stdout:3992: expect (ObjNum 1349 snap 436 seq_num 1349) 2024-09-20T19:39:18.560 INFO:tasks.rados.rados.0.smithi042.stdout:3993: write oid 16 current snap is 441 2024-09-20T19:39:18.560 INFO:tasks.rados.rados.0.smithi042.stdout:3993: seq_num 1364 ranges {524748=666823,1734733=670137} 2024-09-20T19:39:18.567 INFO:tasks.rados.rados.0.smithi042.stdout:3993: writing smithi04268048-16 from 524748 to 1191571 tid 1 2024-09-20T19:39:18.569 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:18.575 INFO:tasks.rados.rados.0.smithi042.stdout:3993: writing smithi04268048-16 from 1734733 to 2404870 tid 2 2024-09-20T19:39:18.575 INFO:tasks.rados.rados.0.smithi042.stdout:3994: read oid 9 snap 439 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3994: expect (ObjNum 1353 snap 438 seq_num 1353) 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3995: read oid 31 snap -1 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3995: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3996: read oid 49 snap -1 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3996: expect deleted 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3997: rollback oid 28 current snap is 441 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 28 to 439 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: finishing write tid 1 to smithi04268048-34 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: finishing write tid 2 to smithi04268048-34 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: finishing write tid 3 to smithi04268048-34 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: finishing write tid 4 to smithi04268048-34 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: oid 34 updating version 0 to 1261 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: oid 34 version 1261 is already newer than 1260 2024-09-20T19:39:18.576 INFO:tasks.rados.rados.0.smithi042.stdout:3990: oid 34 updating version 1261 to 1262 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 34 v 1262 (ObjNum 1362 snap 441 seq_num 1362) dirty exists 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3990: left oid 34 (ObjNum 1362 snap 441 seq_num 1362) 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3990: done (7 left) 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3998: rollback oid 43 current snap is 441 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:rollback oid 43 to 437 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3991: finishing write tid 1 to smithi04268048-11 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3991: finishing write tid 2 to smithi04268048-11 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3991: finishing write tid 3 to smithi04268048-11 2024-09-20T19:39:18.577 INFO:tasks.rados.rados.0.smithi042.stdout:3992: expect (ObjNum 1349 snap 436 seq_num 1349) 2024-09-20T19:39:18.626 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:18.655 INFO:tasks.rados.rados.0.smithi042.stdout:3991: finishing write tid 4 to smithi04268048-11 2024-09-20T19:39:18.655 INFO:tasks.rados.rados.0.smithi042.stdout:3991: finishing write tid 5 to smithi04268048-11 2024-09-20T19:39:18.655 INFO:tasks.rados.rados.0.smithi042.stdout:3991: oid 11 updating version 0 to 1255 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3991: oid 11 updating version 1255 to 1256 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3991: oid 11 version 1256 is already newer than 1253 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3991: oid 11 version 1256 is already newer than 1254 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 11 v 1256 (ObjNum 1363 snap 441 seq_num 1363) dirty exists 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3991: left oid 11 (ObjNum 1363 snap 441 seq_num 1363) 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3993: finishing write tid 1 to smithi04268048-16 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3998: finishing rollback tid 1 to smithi04268048-43 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 43 v 0 (ObjNum 49 snap 0 seq_num 1390888752) dirty dne 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:3994: expect (ObjNum 1353 snap 438 seq_num 1353) 2024-09-20T19:39:18.656 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:18.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:18.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:18 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:18.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T19:39:18.767 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:18.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:18.821 INFO:tasks.rados.rados.0.smithi042.stdout:3995: expect (ObjNum 1306 snap 418 seq_num 1306) 2024-09-20T19:39:18.821 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 0 2024-09-20T19:39:18.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:18.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:18.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.989591+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-09-20T19:39:18.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:17.990940+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-09-20T19:39:18.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.374466+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.a scrub starts 2024-09-20T19:39:18.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.378901+0000 osd.6 (osd.6) 247 : cluster [DBG] 4.a scrub ok 2024-09-20T19:39:18.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:18 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.516504+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2024-09-20T19:39:18.945 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:18.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:18.951 INFO:tasks.rados.rados.0.smithi042.stdout:seeking to 3647008 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: finishing write tid 2 to smithi04268048-16 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: finishing write tid 3 to smithi04268048-16 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: finishing write tid 4 to smithi04268048-16 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: oid 16 updating version 0 to 1599 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: oid 16 updating version 1599 to 1600 2024-09-20T19:39:18.954 INFO:tasks.rados.rados.0.smithi042.stdout:3993: oid 16 updating version 1600 to 1601 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 16 v 1601 (ObjNum 1364 snap 441 seq_num 1364) dirty exists 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3993: left oid 16 (ObjNum 1364 snap 441 seq_num 1364) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3997: finishing rollback tid 0 to smithi04268048-28 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3997: finishing rollback tid 1 to smithi04268048-28 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3997: finishing rollback tid 2 to smithi04268048-28 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:update_object_version oid 28 v 1464 (ObjNum 1343 snap 434 seq_num 1343) dirty exists 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3991: done (7 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3992: done (6 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3993: done (5 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3994: done (4 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3995: done (3 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3996: done (2 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3997: done (1 left) 2024-09-20T19:39:18.955 INFO:tasks.rados.rados.0.smithi042.stdout:3998: done (0 left) 2024-09-20T19:39:18.956 INFO:tasks.rados.rados.0.smithi042.stdout:3999: snap_create 2024-09-20T19:39:19.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:19.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T19:39:19.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:19.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:19.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:19.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:19.429 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:19.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:19.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:19.585 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:19.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.668 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.683 INFO:tasks.rados.rados.0.smithi042.stdout:3999: done (0 left) 2024-09-20T19:39:19.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:19.692 INFO:tasks.rados.rados.0.smithi042.stderr:0 errors. 2024-09-20T19:39:19.692 INFO:tasks.rados.rados.0.smithi042.stderr: 2024-09-20T19:39:19.700 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:39:19.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:19.816 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:19.842 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:19.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[22706]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:19 smithi042 bash[32075]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.500618+0000 mgr.y (mgr.24491) 798 : cluster [DBG] pgmap v1572: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 737 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.930480+0000 osd.4 (osd.4) 193 : cluster [DBG] 7.e scrub starts 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[24557]: cluster 2024-09-20T19:39:18.932923+0000 osd.4 (osd.4) 194 : cluster [DBG] 7.e scrub ok 2024-09-20T19:39:19.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:19.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:20.075 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:20.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:20.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:39:19 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:20.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:20.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.360 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:20.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:20.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:20.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:20.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:20.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:20.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:20.866 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:20.866 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:20.867 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:20 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:20.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:20.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:20.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:21.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:21.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.367935+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.6 scrub starts 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.369825+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub ok 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:19.680559+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.030057+0000 osd.7 (osd.7) 247 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:21.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:20 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.031867+0000 osd.7 (osd.7) 248 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:39:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:21.245 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:21.265 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:21.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:21.393 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:21.496 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:21.524 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:21.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:21.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:21.755 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:21.766 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.766 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.766 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.767 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.768 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.769 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:21 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:21.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: audit 2024-09-20T19:39:19.975674+0000 mgr.y (mgr.24491) 799 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.372945+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.4 scrub starts 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.376807+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.501125+0000 mgr.y (mgr.24491) 800 : cluster [DBG] pgmap v1575: 178 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 169 active+clean; 121 MiB data, 735 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.931317+0000 osd.1 (osd.1) 367 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.933241+0000 osd.1 (osd.1) 368 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:20.999529+0000 osd.4 (osd.4) 195 : cluster [DBG] 7.1a scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.003031+0000 osd.4 (osd.4) 196 : cluster [DBG] 7.1a scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.050817+0000 osd.2 (osd.2) 335 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.055255+0000 osd.2 (osd.2) 336 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.055853+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.057243+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.323048+0000 osd.6 (osd.6) 248 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:21 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.325627+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:39:21.912 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:21.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:22.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:22.041 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:22.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:22.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:22.399 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.427 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:22.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:22.558 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:22.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.689 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:22.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.805 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:22 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.553442+0000 osd.5 (osd.5) 329 : cluster [DBG] 7.1c scrub starts 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:21.556190+0000 osd.5 (osd.5) 330 : cluster [DBG] 7.1c scrub ok 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.038473+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.039733+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.052769+0000 osd.2 (osd.2) 337 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.057427+0000 osd.2 (osd.2) 338 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.312717+0000 osd.6 (osd.6) 250 : cluster [DBG] 6.c scrub starts 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:22 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.314856+0000 osd.6 (osd.6) 251 : cluster [DBG] 6.c scrub ok 2024-09-20T19:39:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:22.921 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.950 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:22.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:23.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:23.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:23.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:23.264 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:23.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:23.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:23.550 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:39:23.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:23.596 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:23.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:23.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:23.843 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:23.843 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:23.843 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:23 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:23.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:23.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:23.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:24.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:24.101 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:24.102 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.502199+0000 mgr.y (mgr.24491) 801 : cluster [DBG] pgmap v1576: 178 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 171 active+clean; 126 MiB data, 745 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.536436+0000 osd.5 (osd.5) 331 : cluster [DBG] 7.21 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.541851+0000 osd.5 (osd.5) 332 : cluster [DBG] 7.21 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.922435+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:22.924783+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.312414+0000 osd.6 (osd.6) 252 : cluster [DBG] 7.19 scrub starts 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.315326+0000 osd.6 (osd.6) 253 : cluster [DBG] 7.19 scrub ok 2024-09-20T19:39:24.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:24.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:23 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.495166+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2024-09-20T19:39:24.116 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:24.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:24.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:24.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:24.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:24.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:24.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:24.605 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:24.635 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:24.738 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:24.765 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:24.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:24.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:24 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:24.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:24.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:24.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:24.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:25.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:25.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.154 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:25.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:25.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.358891+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:25.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.360492+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.543548+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.545679+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.949720+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:39:25.185 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:25.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:23.951761+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:39:25.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:25.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.142039+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.f scrub starts 2024-09-20T19:39:25.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:25.186 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:24 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.143696+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.f scrub ok 2024-09-20T19:39:25.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:25.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:25.324 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-20T19:39:25.324 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-20T19:39:25.324 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-20T19:39:25.324 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-20T19:39:25.324 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-20T19:39:25.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:25.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.533 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:25.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:25.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:25.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:25.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:25 smithi162 bash[24557]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:25.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:25.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:25.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:26.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:26.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[22706]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.503137+0000 mgr.y (mgr.24491) 802 : cluster [DBG] pgmap v1578: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 619 KiB/s wr, 8 op/s 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.551125+0000 osd.5 (osd.5) 335 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:26.042 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.552922+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.960030+0000 osd.7 (osd.7) 253 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:24.961214+0000 osd.7 (osd.7) 254 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.108950+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.111334+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.265635+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:26.043 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:25 smithi042 bash[32075]: cluster 2024-09-20T19:39:25.267316+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:39:26.053 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:26.156 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:26.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T19:39:26.184 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:26.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:26.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:26.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.424 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:26.541 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:26.594 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:39:26.596 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:39:26.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.693 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:26.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:26.920 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.945 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:26.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:27.032 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.032 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.033 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:26 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:26 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:26 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.296599+0000 osd.6 (osd.6) 256 : cluster [DBG] 7.7 scrub starts 2024-09-20T19:39:27.056 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:26 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:26 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.299873+0000 osd.6 (osd.6) 257 : cluster [DBG] 7.7 scrub ok 2024-09-20T19:39:27.074 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:27.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:27.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:27.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:27.334 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:27.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:27.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:27.597 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:27.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:27.726 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:27.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:27.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:27.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:27.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:27.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:27.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:27.972 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:27.973 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:27 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:27.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:27.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:28.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:28.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.371479+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.372824+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.503863+0000 mgr.y (mgr.24491) 803 : cluster [DBG] pgmap v1579: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 465 KiB/s wr, 6 op/s 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: audit 2024-09-20T19:39:26.582188+0000 mgr.y (mgr.24491) 804 : audit [DBG] from='client.26171 v1:172.21.15.42:0/2709889931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.951067+0000 osd.4 (osd.4) 199 : cluster [DBG] 4.e scrub starts 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:26.955039+0000 osd.4 (osd.4) 200 : cluster [DBG] 4.e scrub ok 2024-09-20T19:39:28.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:28.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.273453+0000 osd.6 (osd.6) 258 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:39:28.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:28.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:27 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.274750+0000 osd.6 (osd.6) 259 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:39:28.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:28.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:28.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:28.376 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:28.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:28.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.637 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:28.740 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:28.766 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:28 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:28.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.981 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.362866+0000 osd.0 (osd.0) 137 : cluster [DBG] 4.b scrub starts 2024-09-20T19:39:28.982 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.366725+0000 osd.0 (osd.0) 138 : cluster [DBG] 4.b scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.525036+0000 osd.5 (osd.5) 337 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.527743+0000 osd.5 (osd.5) 338 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.927111+0000 osd.1 (osd.1) 371 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.929206+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.941044+0000 osd.4 (osd.4) 201 : cluster [DBG] 7.22 scrub starts 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:27.944198+0000 osd.4 (osd.4) 202 : cluster [DBG] 7.22 scrub ok 2024-09-20T19:39:28.983 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.984 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.239843+0000 osd.6 (osd.6) 260 : cluster [DBG] 7.12 scrub starts 2024-09-20T19:39:28.984 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.984 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.241797+0000 osd.6 (osd.6) 261 : cluster [DBG] 7.12 scrub ok 2024-09-20T19:39:28.984 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.984 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:28 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.500102+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2024-09-20T19:39:28.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:29.106 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:39:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:39:29.124 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:39:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:39:29.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:29.142 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:39:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:39:29.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:29.160 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:39:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:39:29.179 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:39:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:39:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:39:29.197 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:39:29.218 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:39:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:39:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:39:29.240 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:39:29.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:29.274 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:29.322 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1579,"stamp":"2024-09-20T19:39:26.503406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132451911,"num_objects":253,"num_object_clones":14,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":2,"num_read":6356,"num_read_kb":2775133,"num_write":3487,"num_write_kb":934562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29098,"ondisk_log_size":29098,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":765064,"kb_used_data":308024,"kb_used_omap":92,"kb_used_meta":456803,"kb_avail":655451000,"statfs":{"total":671965249536,"available":671181824000,"internally_reserved":0,"allocated":315416576,"data_stored":340315249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":467766969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6069534,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":-1,"num_read":35,"num_read_kb":28555,"num_write":15,"num_write_kb":3714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.989541"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":4810,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152918+0000","last_change":"2024-09-20T19:39:18.538563+0000","last_active":"2024-09-20T19:39:20.152918+0000","last_peered":"2024-09-20T19:39:20.152918+0000","last_clean":"2024-09-20T19:39:20.152918+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:39:20.152918+0000","last_undegraded":"2024-09-20T19:39:20.152918+0000","last_fullsized":"2024-09-20T19:39:20.152918+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:38:17.790503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.790503+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:33.413374+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0057066840000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"7.22","version":"995'368","reported_seq":4608,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.496997+0000","last_change":"2024-09-20T19:39:18.594230+0000","last_active":"2024-09-20T19:39:23.496997+0000","last_peered":"2024-09-20T19:39:23.496997+0000","last_clean":"2024-09-20T19:39:23.496997+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:39:23.496997+0000","last_undegraded":"2024-09-20T19:39:23.496997+0000","last_fullsized":"2024-09-20T19:39:23.496997+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"938'336","last_scrub_stamp":"2024-09-20T19:37:36.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:36.773213+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090287879999999994,"stat_sum":{"num_bytes":3344890,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":40573,"num_write":37,"num_write_kb":14578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b3","length":"1"},{"start":"1b5","length":"2"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"7.21","version":"1050'849","reported_seq":5569,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.156659+0000","last_change":"2024-09-20T19:39:18.834737+0000","last_active":"2024-09-20T19:39:20.156659+0000","last_peered":"2024-09-20T19:39:20.156659+0000","last_clean":"2024-09-20T19:39:20.156659+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:39:20.156659+0000","last_undegraded":"2024-09-20T19:39:20.156659+0000","last_fullsized":"2024-09-20T19:39:20.156659+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"953'788","last_scrub_stamp":"2024-09-20T19:37:42.892118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.892118+0000","objects_scrubbed":3,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.29602658100000001,"stat_sum":{"num_bytes":9237240,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":145,"num_read_kb":155075,"num_write":64,"num_write_kb":35558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"7.20","version":"1014'991","reported_seq":5618,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147238+0000","last_change":"2024-09-20T19:39:18.554134+0000","last_active":"2024-09-20T19:39:20.147238+0000","last_peered":"2024-09-20T19:39:20.147238+0000","last_clean":"2024-09-20T19:39:20.147238+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:39:20.147238+0000","last_undegraded":"2024-09-20T19:39:20.147238+0000","last_fullsized":"2024-09-20T19:39:20.147238+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'982","last_scrub_stamp":"2024-09-20T19:38:13.700013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:13.700013+0000","objects_scrubbed":1,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:22.806732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077880600000000005,"stat_sum":{"num_bytes":3815486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":67428,"num_write":32,"num_write_kb":14158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"7.1a","version":"1050'798","reported_seq":5190,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499238+0000","last_change":"2024-09-20T19:39:21.003139+0000","last_active":"2024-09-20T19:39:23.499238+0000","last_peered":"2024-09-20T19:39:23.499238+0000","last_clean":"2024-09-20T19:39:23.499238+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:39:23.499238+0000","last_undegraded":"2024-09-20T19:39:23.499238+0000","last_fullsized":"2024-09-20T19:39:23.499238+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"1050'798","last_scrub_stamp":"2024-09-20T19:39:21.003036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.003036+0000","objects_scrubbed":3,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.152836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0091316239999999996,"stat_sum":{"num_bytes":4908304,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":116514,"num_write":96,"num_write_kb":37050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b3","length":"1"},{"start":"1b5","length":"2"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":1507,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152626+0000","last_change":"2024-09-20T19:36:31.860620+0000","last_active":"2024-09-20T19:39:20.152626+0000","last_peered":"2024-09-20T19:39:20.152626+0000","last_clean":"2024-09-20T19:39:20.152626+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:39:20.152626+0000","last_undegraded":"2024-09-20T19:39:20.152626+0000","last_fullsized":"2024-09-20T19:39:20.152626+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.860530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.860530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1827,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144055+0000","last_change":"2024-09-20T19:37:25.135773+0000","last_active":"2024-09-20T19:39:20.144055+0000","last_peered":"2024-09-20T19:39:20.144055+0000","last_clean":"2024-09-20T19:39:20.144055+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:39:20.144055+0000","last_undegraded":"2024-09-20T19:39:20.144055+0000","last_fullsized":"2024-09-20T19:39:20.144055+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.135646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.135646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"987'39","reported_seq":1775,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:22.057603+0000","last_change":"2024-09-20T19:39:22.057603+0000","last_active":"2024-09-20T19:39:22.057603+0000","last_peered":"2024-09-20T19:39:22.057603+0000","last_clean":"2024-09-20T19:39:22.057603+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:39:22.057603+0000","last_undegraded":"2024-09-20T19:39:22.057603+0000","last_fullsized":"2024-09-20T19:39:22.057603+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:22.057437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.057437+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.474404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1807,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.496149+0000","last_change":"2024-09-20T19:38:28.745755+0000","last_active":"2024-09-20T19:39:23.496149+0000","last_peered":"2024-09-20T19:39:23.496149+0000","last_clean":"2024-09-20T19:39:23.496149+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:39:23.496149+0000","last_undegraded":"2024-09-20T19:39:23.496149+0000","last_fullsized":"2024-09-20T19:39:23.496149+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:28.745668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.745668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.949968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":5087,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143681+0000","last_change":"2024-09-20T19:39:18.608387+0000","last_active":"2024-09-20T19:39:20.143681+0000","last_peered":"2024-09-20T19:39:20.143681+0000","last_clean":"2024-09-20T19:39:20.143681+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:39:20.143681+0000","last_undegraded":"2024-09-20T19:39:20.143681+0000","last_fullsized":"2024-09-20T19:39:20.143681+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:38:14.315644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.315644+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.091726462999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"2.1e","version":"0'0","reported_seq":1562,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144343+0000","last_change":"2024-09-20T19:37:21.156138+0000","last_active":"2024-09-20T19:39:20.144343+0000","last_peered":"2024-09-20T19:39:20.144343+0000","last_clean":"2024-09-20T19:39:20.144343+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:39:20.144343+0000","last_undegraded":"2024-09-20T19:39:20.144343+0000","last_fullsized":"2024-09-20T19:39:20.144343+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.156013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.156013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"987'21","reported_seq":1920,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.496515+0000","last_change":"2024-09-20T19:36:44.033709+0000","last_active":"2024-09-20T19:39:23.496515+0000","last_peered":"2024-09-20T19:39:23.496515+0000","last_clean":"2024-09-20T19:39:23.496515+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:39:23.496515+0000","last_undegraded":"2024-09-20T19:39:23.496515+0000","last_fullsized":"2024-09-20T19:39:23.496515+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:36:44.033568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:44.033568+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.776685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":2045,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.933373+0000","last_change":"2024-09-20T19:39:20.933373+0000","last_active":"2024-09-20T19:39:20.933373+0000","last_peered":"2024-09-20T19:39:20.933373+0000","last_clean":"2024-09-20T19:39:20.933373+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:39:20.933373+0000","last_undegraded":"2024-09-20T19:39:20.933373+0000","last_fullsized":"2024-09-20T19:39:20.933373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:20.933250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.933250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.308194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1869,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499274+0000","last_change":"2024-09-20T19:38:27.709303+0000","last_active":"2024-09-20T19:39:23.499274+0000","last_peered":"2024-09-20T19:39:23.499274+0000","last_clean":"2024-09-20T19:39:23.499274+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:39:23.499274+0000","last_undegraded":"2024-09-20T19:39:23.499274+0000","last_fullsized":"2024-09-20T19:39:23.499274+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:27.709247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.709247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.024307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"1030'1284","reported_seq":5867,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.142739+0000","last_change":"2024-09-20T19:39:18.608130+0000","last_active":"2024-09-20T19:39:20.142739+0000","last_peered":"2024-09-20T19:39:20.142739+0000","last_clean":"2024-09-20T19:39:20.142739+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:39:20.142739+0000","last_undegraded":"2024-09-20T19:39:20.142739+0000","last_fullsized":"2024-09-20T19:39:20.142739+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"895'1218","last_scrub_stamp":"2024-09-20T19:36:46.063473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:46.063473+0000","objects_scrubbed":6,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.092706255000000001,"stat_sum":{"num_bytes":5866270,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":128956,"num_write":108,"num_write_kb":39081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"2.1d","version":"0'0","reported_seq":2035,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.159919+0000","last_change":"2024-09-20T19:38:40.080992+0000","last_active":"2024-09-20T19:39:20.159919+0000","last_peered":"2024-09-20T19:39:20.159919+0000","last_clean":"2024-09-20T19:39:20.159919+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:39:20.159919+0000","last_undegraded":"2024-09-20T19:39:20.159919+0000","last_fullsized":"2024-09-20T19:39:20.159919+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.080865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.080865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.792937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"987'9","reported_seq":1917,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499239+0000","last_change":"2024-09-20T19:37:44.725836+0000","last_active":"2024-09-20T19:39:23.499239+0000","last_peered":"2024-09-20T19:39:23.499239+0000","last_clean":"2024-09-20T19:39:23.499239+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:39:23.499239+0000","last_undegraded":"2024-09-20T19:39:23.499239+0000","last_fullsized":"2024-09-20T19:39:23.499239+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'7","last_scrub_stamp":"2024-09-20T19:37:44.725701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:44.725701+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":2027,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:21.057305+0000","last_change":"2024-09-20T19:39:21.057305+0000","last_active":"2024-09-20T19:39:21.057305+0000","last_peered":"2024-09-20T19:39:21.057305+0000","last_clean":"2024-09-20T19:39:21.057305+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:39:21.057305+0000","last_undegraded":"2024-09-20T19:39:21.057305+0000","last_fullsized":"2024-09-20T19:39:21.057305+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:21.057247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.057247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.649619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2038,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158291+0000","last_change":"2024-09-20T19:36:22.545289+0000","last_active":"2024-09-20T19:39:20.158291+0000","last_peered":"2024-09-20T19:39:20.158291+0000","last_clean":"2024-09-20T19:39:20.158291+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:39:20.158291+0000","last_undegraded":"2024-09-20T19:39:20.158291+0000","last_fullsized":"2024-09-20T19:39:20.158291+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:22.545231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:22.545231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.331104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"950'482","reported_seq":5109,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150976+0000","last_change":"2024-09-20T19:39:18.537411+0000","last_active":"2024-09-20T19:39:20.150976+0000","last_peered":"2024-09-20T19:39:20.150976+0000","last_clean":"2024-09-20T19:39:20.150976+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:39:20.150976+0000","last_undegraded":"2024-09-20T19:39:20.150976+0000","last_fullsized":"2024-09-20T19:39:20.150976+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"950'482","last_scrub_stamp":"2024-09-20T19:37:42.727140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.727140+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049883319999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":55935,"num_write":41,"num_write_kb":15366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"2.1c","version":"0'0","reported_seq":2024,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155743+0000","last_change":"2024-09-20T19:38:26.111416+0000","last_active":"2024-09-20T19:39:20.155743+0000","last_peered":"2024-09-20T19:39:20.155743+0000","last_clean":"2024-09-20T19:39:20.155743+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:39:20.155743+0000","last_undegraded":"2024-09-20T19:39:20.155743+0000","last_fullsized":"2024-09-20T19:39:20.155743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:26.111300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.111300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:33.731393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"987'21","reported_seq":1956,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.497162+0000","last_change":"2024-09-20T19:38:30.744236+0000","last_active":"2024-09-20T19:39:23.497162+0000","last_peered":"2024-09-20T19:39:23.497162+0000","last_clean":"2024-09-20T19:39:23.497162+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:39:23.497162+0000","last_undegraded":"2024-09-20T19:39:23.497162+0000","last_fullsized":"2024-09-20T19:39:23.497162+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.744141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.744141+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.937516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2019,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155127+0000","last_change":"2024-09-20T19:36:23.582321+0000","last_active":"2024-09-20T19:39:20.155127+0000","last_peered":"2024-09-20T19:39:20.155127+0000","last_clean":"2024-09-20T19:39:20.155127+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:39:20.155127+0000","last_undegraded":"2024-09-20T19:39:20.155127+0000","last_fullsized":"2024-09-20T19:39:20.155127+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.582265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.377154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":996,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.142532+0000","last_change":"2024-09-20T19:37:24.118390+0000","last_active":"2024-09-20T19:39:20.142532+0000","last_peered":"2024-09-20T19:39:20.142532+0000","last_clean":"2024-09-20T19:39:20.142532+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:39:20.142532+0000","last_undegraded":"2024-09-20T19:39:20.142532+0000","last_fullsized":"2024-09-20T19:39:20.142532+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:24.118312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:24.118312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1656,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.149686+0000","last_change":"2024-09-20T19:38:15.047645+0000","last_active":"2024-09-20T19:39:20.149686+0000","last_peered":"2024-09-20T19:39:20.149686+0000","last_clean":"2024-09-20T19:39:20.149686+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:39:20.149686+0000","last_undegraded":"2024-09-20T19:39:20.149686+0000","last_fullsized":"2024-09-20T19:39:20.149686+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:15.047486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:15.047486+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1021'427","reported_seq":5075,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.500207+0000","last_change":"2024-09-20T19:39:18.933022+0000","last_active":"2024-09-20T19:39:23.500207+0000","last_peered":"2024-09-20T19:39:23.500207+0000","last_clean":"2024-09-20T19:39:23.500207+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:39:23.500207+0000","last_undegraded":"2024-09-20T19:39:23.500207+0000","last_fullsized":"2024-09-20T19:39:23.500207+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"1021'427","last_scrub_stamp":"2024-09-20T19:39:18.932933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:18.932933+0000","objects_scrubbed":1,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.120268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038900900000000001,"stat_sum":{"num_bytes":2235661,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":27202,"num_write":27,"num_write_kb":10018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b3","length":"1"},{"start":"1b5","length":"2"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.d","version":"987'45","reported_seq":1910,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499318+0000","last_change":"2024-09-20T19:38:14.828944+0000","last_active":"2024-09-20T19:39:23.499318+0000","last_peered":"2024-09-20T19:39:23.499318+0000","last_clean":"2024-09-20T19:39:23.499318+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:39:23.499318+0000","last_undegraded":"2024-09-20T19:39:23.499318+0000","last_fullsized":"2024-09-20T19:39:23.499318+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:38:14.828850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.828850+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2028,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154026+0000","last_change":"2024-09-20T19:39:17.990988+0000","last_active":"2024-09-20T19:39:20.154026+0000","last_peered":"2024-09-20T19:39:20.154026+0000","last_clean":"2024-09-20T19:39:20.154026+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:39:20.154026+0000","last_undegraded":"2024-09-20T19:39:20.154026+0000","last_fullsized":"2024-09-20T19:39:20.154026+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:17.990944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.990944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.758835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":2048,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.682578+0000","last_change":"2024-09-20T19:38:38.590561+0000","last_active":"2024-09-20T19:39:19.682578+0000","last_peered":"2024-09-20T19:39:19.682578+0000","last_clean":"2024-09-20T19:39:19.682578+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:39:19.682578+0000","last_undegraded":"2024-09-20T19:39:19.682578+0000","last_fullsized":"2024-09-20T19:39:19.682578+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:38.590498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:38.590498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.175561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":608,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148575+0000","last_change":"2024-09-20T19:37:21.611924+0000","last_active":"2024-09-20T19:39:20.148575+0000","last_peered":"2024-09-20T19:39:20.148575+0000","last_clean":"2024-09-20T19:39:20.148575+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:39:20.148575+0000","last_undegraded":"2024-09-20T19:39:20.148575+0000","last_fullsized":"2024-09-20T19:39:20.148575+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.611767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.611767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":2114,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152986+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:39:20.152986+0000","last_peered":"2024-09-20T19:39:20.152986+0000","last_clean":"2024-09-20T19:39:20.152986+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:39:20.152986+0000","last_undegraded":"2024-09-20T19:39:20.152986+0000","last_fullsized":"2024-09-20T19:39:20.152986+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:45.070277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1046'711","reported_seq":5121,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144003+0000","last_change":"2024-09-20T19:39:18.609861+0000","last_active":"2024-09-20T19:39:20.144003+0000","last_peered":"2024-09-20T19:39:20.144003+0000","last_clean":"2024-09-20T19:39:20.144003+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:39:20.144003+0000","last_undegraded":"2024-09-20T19:39:20.144003+0000","last_fullsized":"2024-09-20T19:39:20.144003+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'687","last_scrub_stamp":"2024-09-20T19:38:24.221337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.221337+0000","objects_scrubbed":1,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.197485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.092474009999999995,"stat_sum":{"num_bytes":2622222,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":35389,"num_write":68,"num_write_kb":28356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.c","version":"987'30","reported_seq":1955,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.498475+0000","last_change":"2024-09-20T19:38:29.725721+0000","last_active":"2024-09-20T19:39:23.498475+0000","last_peered":"2024-09-20T19:39:23.498475+0000","last_clean":"2024-09-20T19:39:23.498475+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:39:23.498475+0000","last_undegraded":"2024-09-20T19:39:23.498475+0000","last_fullsized":"2024-09-20T19:39:23.498475+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:29.725634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:29.725634+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.585770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":2064,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.681448+0000","last_change":"2024-09-20T19:38:18.676904+0000","last_active":"2024-09-20T19:39:19.681448+0000","last_peered":"2024-09-20T19:39:19.681448+0000","last_clean":"2024-09-20T19:39:19.681448+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:39:19.681448+0000","last_undegraded":"2024-09-20T19:39:19.681448+0000","last_fullsized":"2024-09-20T19:39:19.681448+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:18.676841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:18.676841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:22.424549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":2312,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148486+0000","last_change":"2024-09-20T19:39:06.910897+0000","last_active":"2024-09-20T19:39:20.148486+0000","last_peered":"2024-09-20T19:39:20.148486+0000","last_clean":"2024-09-20T19:39:20.148486+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:39:20.148486+0000","last_undegraded":"2024-09-20T19:39:20.148486+0000","last_fullsized":"2024-09-20T19:39:20.148486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:39:06.910817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:06.910817+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.382750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1879,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.500647+0000","last_change":"2024-09-20T19:39:05.797046+0000","last_active":"2024-09-20T19:39:23.500647+0000","last_peered":"2024-09-20T19:39:23.500647+0000","last_clean":"2024-09-20T19:39:23.500647+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:39:23.500647+0000","last_undegraded":"2024-09-20T19:39:23.500647+0000","last_fullsized":"2024-09-20T19:39:23.500647+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.796915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.796915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:13.456865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1627,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151962+0000","last_change":"2024-09-20T19:38:17.061687+0000","last_active":"2024-09-20T19:39:20.151962+0000","last_peered":"2024-09-20T19:39:20.151962+0000","last_clean":"2024-09-20T19:39:20.151962+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:39:20.151962+0000","last_undegraded":"2024-09-20T19:39:20.151962+0000","last_fullsized":"2024-09-20T19:39:20.151962+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:17.061616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.061616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.609939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1041'961","reported_seq":5687,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154761+0000","last_change":"2024-09-20T19:39:18.525089+0000","last_active":"2024-09-20T19:39:20.154761+0000","last_peered":"2024-09-20T19:39:20.154761+0000","last_clean":"2024-09-20T19:39:20.154761+0000","last_became_active":"2024-09-20T19:35:34.395504+0000","last_became_peered":"2024-09-20T19:35:34.395504+0000","last_unstale":"2024-09-20T19:39:20.154761+0000","last_undegraded":"2024-09-20T19:39:20.154761+0000","last_fullsized":"2024-09-20T19:39:20.154761+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":826,"parent":"0.0","parent_split_bits":0,"last_scrub":"1041'961","last_scrub_stamp":"2024-09-20T19:39:13.078655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.078655+0000","objects_scrubbed":1,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.542664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001892148,"stat_sum":{"num_bytes":3025557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":55890,"num_write":39,"num_write_kb":17667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.f","version":"987'39","reported_seq":2149,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.684910+0000","last_change":"2024-09-20T19:38:26.665615+0000","last_active":"2024-09-20T19:39:19.684910+0000","last_peered":"2024-09-20T19:39:19.684910+0000","last_clean":"2024-09-20T19:39:19.684910+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:39:19.684910+0000","last_undegraded":"2024-09-20T19:39:19.684910+0000","last_fullsized":"2024-09-20T19:39:19.684910+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:38:26.665526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.665526+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:32.401246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":2064,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.680475+0000","last_change":"2024-09-20T19:38:17.643987+0000","last_active":"2024-09-20T19:39:19.680475+0000","last_peered":"2024-09-20T19:39:19.680475+0000","last_clean":"2024-09-20T19:39:19.680475+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:39:19.680475+0000","last_undegraded":"2024-09-20T19:39:19.680475+0000","last_fullsized":"2024-09-20T19:39:19.680475+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:17.643862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.643862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:31.220331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":2100,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.501219+0000","last_change":"2024-09-20T19:38:35.798379+0000","last_active":"2024-09-20T19:39:23.501219+0000","last_peered":"2024-09-20T19:39:23.501219+0000","last_clean":"2024-09-20T19:39:23.501219+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:39:23.501219+0000","last_undegraded":"2024-09-20T19:39:23.501219+0000","last_fullsized":"2024-09-20T19:39:23.501219+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:38:35.798250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.798250+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.803640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2040,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150520+0000","last_change":"2024-09-20T19:38:35.252514+0000","last_active":"2024-09-20T19:39:20.150520+0000","last_peered":"2024-09-20T19:39:20.150520+0000","last_clean":"2024-09-20T19:39:20.150520+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:39:20.150520+0000","last_undegraded":"2024-09-20T19:39:20.150520+0000","last_fullsized":"2024-09-20T19:39:20.150520+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.252424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.252424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.236879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1045'656","reported_seq":5332,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155228+0000","last_change":"2024-09-20T19:39:18.539326+0000","last_active":"2024-09-20T19:39:20.155228+0000","last_peered":"2024-09-20T19:39:20.155228+0000","last_clean":"2024-09-20T19:39:20.155228+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:39:20.155228+0000","last_undegraded":"2024-09-20T19:39:20.155228+0000","last_fullsized":"2024-09-20T19:39:20.155228+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'630","last_scrub_stamp":"2024-09-20T19:38:23.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.703700+0000","objects_scrubbed":1,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:26.170658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004543638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":20016,"num_write":45,"num_write_kb":14853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.e","version":"987'27","reported_seq":1933,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.501708+0000","last_change":"2024-09-20T19:37:37.778246+0000","last_active":"2024-09-20T19:39:23.501708+0000","last_peered":"2024-09-20T19:39:23.501708+0000","last_clean":"2024-09-20T19:39:23.501708+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:39:23.501708+0000","last_undegraded":"2024-09-20T19:39:23.501708+0000","last_fullsized":"2024-09-20T19:39:23.501708+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:37:37.778165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:37.778165+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":693,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.149614+0000","last_change":"2024-09-20T19:36:37.827773+0000","last_active":"2024-09-20T19:39:20.149614+0000","last_peered":"2024-09-20T19:39:20.149614+0000","last_clean":"2024-09-20T19:39:20.149614+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:39:20.149614+0000","last_undegraded":"2024-09-20T19:39:20.149614+0000","last_fullsized":"2024-09-20T19:39:20.149614+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:37.827682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:37.827682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2036,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157489+0000","last_change":"2024-09-20T19:39:14.526537+0000","last_active":"2024-09-20T19:39:20.157489+0000","last_peered":"2024-09-20T19:39:20.157489+0000","last_clean":"2024-09-20T19:39:20.157489+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:39:20.157489+0000","last_undegraded":"2024-09-20T19:39:20.157489+0000","last_fullsized":"2024-09-20T19:39:20.157489+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:14.526473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:14.526473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:19.566605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1503,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151761+0000","last_change":"2024-09-20T19:37:21.733870+0000","last_active":"2024-09-20T19:39:20.151761+0000","last_peered":"2024-09-20T19:39:20.151761+0000","last_clean":"2024-09-20T19:39:20.151761+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:39:20.151761+0000","last_undegraded":"2024-09-20T19:39:20.151761+0000","last_fullsized":"2024-09-20T19:39:20.151761+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.733745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.733745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":864,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155705+0000","last_change":"2024-09-20T19:39:15.494274+0000","last_active":"2024-09-20T19:39:20.155705+0000","last_peered":"2024-09-20T19:39:20.155705+0000","last_clean":"2024-09-20T19:39:20.155705+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:39:20.155705+0000","last_undegraded":"2024-09-20T19:39:20.155705+0000","last_fullsized":"2024-09-20T19:39:20.155705+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:15.494138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.494138+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:43.249820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":4370,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.679589+0000","last_change":"2024-09-20T19:39:18.520725+0000","last_active":"2024-09-20T19:39:19.679589+0000","last_peered":"2024-09-20T19:39:19.679589+0000","last_clean":"2024-09-20T19:39:19.679589+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:39:19.679589+0000","last_undegraded":"2024-09-20T19:39:19.679589+0000","last_fullsized":"2024-09-20T19:39:19.679589+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:37:42.138190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.138190+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011805349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.1","version":"987'42","reported_seq":1912,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.497208+0000","last_change":"2024-09-20T19:38:36.778506+0000","last_active":"2024-09-20T19:39:23.497208+0000","last_peered":"2024-09-20T19:39:23.497208+0000","last_clean":"2024-09-20T19:39:23.497208+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:39:23.497208+0000","last_undegraded":"2024-09-20T19:39:23.497208+0000","last_fullsized":"2024-09-20T19:39:23.497208+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:38:36.778420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.778420+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.256363+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2026,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158384+0000","last_change":"2024-09-20T19:37:25.730287+0000","last_active":"2024-09-20T19:39:20.158384+0000","last_peered":"2024-09-20T19:39:20.158384+0000","last_clean":"2024-09-20T19:39:20.158384+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:39:20.158384+0000","last_undegraded":"2024-09-20T19:39:20.158384+0000","last_fullsized":"2024-09-20T19:39:20.158384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.730226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.730226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":974,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.678524+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:39:19.678524+0000","last_peered":"2024-09-20T19:39:19.678524+0000","last_clean":"2024-09-20T19:39:19.678524+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:39:19.678524+0000","last_undegraded":"2024-09-20T19:39:19.678524+0000","last_fullsized":"2024-09-20T19:39:19.678524+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:03.844414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2014,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.153120+0000","last_change":"2024-09-20T19:39:16.001753+0000","last_active":"2024-09-20T19:39:20.153120+0000","last_peered":"2024-09-20T19:39:20.153120+0000","last_clean":"2024-09-20T19:39:20.153120+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:39:20.153120+0000","last_undegraded":"2024-09-20T19:39:20.153120+0000","last_fullsized":"2024-09-20T19:39:20.153120+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:16.001693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.001693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.004172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1446,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499775+0000","last_change":"2024-09-20T19:38:26.684327+0000","last_active":"2024-09-20T19:39:23.499775+0000","last_peered":"2024-09-20T19:39:23.499775+0000","last_clean":"2024-09-20T19:39:23.499775+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:39:23.499775+0000","last_undegraded":"2024-09-20T19:39:23.499775+0000","last_fullsized":"2024-09-20T19:39:23.499775+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:26.684257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.684257+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.590882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1050'508","reported_seq":4982,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147462+0000","last_change":"2024-09-20T19:39:18.550463+0000","last_active":"2024-09-20T19:39:20.147462+0000","last_peered":"2024-09-20T19:39:20.147462+0000","last_clean":"2024-09-20T19:39:20.147462+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:39:20.147462+0000","last_undegraded":"2024-09-20T19:39:20.147462+0000","last_fullsized":"2024-09-20T19:39:20.147462+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'507","last_scrub_stamp":"2024-09-20T19:39:15.008559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.008559+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:17.958113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001225196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":7626,"num_write":23,"num_write_kb":5767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.0","version":"987'42","reported_seq":1770,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150290+0000","last_change":"2024-09-20T19:37:30.646115+0000","last_active":"2024-09-20T19:39:20.150290+0000","last_peered":"2024-09-20T19:39:20.150290+0000","last_clean":"2024-09-20T19:39:20.150290+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:39:20.150290+0000","last_undegraded":"2024-09-20T19:39:20.150290+0000","last_fullsized":"2024-09-20T19:39:20.150290+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:37:30.645963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:30.645963+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":2042,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.684861+0000","last_change":"2024-09-20T19:37:26.984507+0000","last_active":"2024-09-20T19:39:19.684861+0000","last_peered":"2024-09-20T19:39:19.684861+0000","last_clean":"2024-09-20T19:39:19.684861+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:39:19.684861+0000","last_undegraded":"2024-09-20T19:39:19.684861+0000","last_fullsized":"2024-09-20T19:39:19.684861+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:37:26.984377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:26.984377+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1858,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.496229+0000","last_change":"2024-09-20T19:39:22.924959+0000","last_active":"2024-09-20T19:39:23.496229+0000","last_peered":"2024-09-20T19:39:23.496229+0000","last_clean":"2024-09-20T19:39:23.496229+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:39:23.496229+0000","last_undegraded":"2024-09-20T19:39:23.496229+0000","last_fullsized":"2024-09-20T19:39:23.496229+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:22.924793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.924793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:35.342839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1804,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.497557+0000","last_change":"2024-09-20T19:36:48.027651+0000","last_active":"2024-09-20T19:39:23.497557+0000","last_peered":"2024-09-20T19:39:23.497557+0000","last_clean":"2024-09-20T19:39:23.497557+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:39:23.497557+0000","last_undegraded":"2024-09-20T19:39:23.497557+0000","last_fullsized":"2024-09-20T19:39:23.497557+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:48.027589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.027589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.737059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1756,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154359+0000","last_change":"2024-09-20T19:38:31.650903+0000","last_active":"2024-09-20T19:39:20.154359+0000","last_peered":"2024-09-20T19:39:20.154359+0000","last_clean":"2024-09-20T19:39:20.154359+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:39:20.154359+0000","last_undegraded":"2024-09-20T19:39:20.154359+0000","last_fullsized":"2024-09-20T19:39:20.154359+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:31.650776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.650776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.460344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1040'1014","reported_seq":5859,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154972+0000","last_change":"2024-09-20T19:39:18.543146+0000","last_active":"2024-09-20T19:39:20.154972+0000","last_peered":"2024-09-20T19:39:20.154972+0000","last_clean":"2024-09-20T19:39:20.154972+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:39:20.154972+0000","last_undegraded":"2024-09-20T19:39:20.154972+0000","last_fullsized":"2024-09-20T19:39:20.154972+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'991","last_scrub_stamp":"2024-09-20T19:38:33.283803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.283803+0000","objects_scrubbed":1,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.656768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023544490000000002,"stat_sum":{"num_bytes":3294469,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":88272,"num_write":58,"num_write_kb":15972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.3","version":"987'39","reported_seq":1860,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143423+0000","last_change":"2024-09-20T19:36:24.022617+0000","last_active":"2024-09-20T19:39:20.143423+0000","last_peered":"2024-09-20T19:39:20.143423+0000","last_clean":"2024-09-20T19:39:20.143423+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:39:20.143423+0000","last_undegraded":"2024-09-20T19:39:20.143423+0000","last_fullsized":"2024-09-20T19:39:20.143423+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:36:24.022527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:24.022527+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1992,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148419+0000","last_change":"2024-09-20T19:36:31.559020+0000","last_active":"2024-09-20T19:39:20.148419+0000","last_peered":"2024-09-20T19:39:20.148419+0000","last_clean":"2024-09-20T19:39:20.148419+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:39:20.148419+0000","last_undegraded":"2024-09-20T19:39:20.148419+0000","last_fullsized":"2024-09-20T19:39:20.148419+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.558923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.558923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.839139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2038,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.156013+0000","last_change":"2024-09-20T19:38:30.630442+0000","last_active":"2024-09-20T19:39:20.156013+0000","last_peered":"2024-09-20T19:39:20.156013+0000","last_clean":"2024-09-20T19:39:20.156013+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:39:20.156013+0000","last_undegraded":"2024-09-20T19:39:20.156013+0000","last_fullsized":"2024-09-20T19:39:20.156013+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.630318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.630318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.617338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":2033,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.679589+0000","last_change":"2024-09-20T19:39:08.849859+0000","last_active":"2024-09-20T19:39:19.679589+0000","last_peered":"2024-09-20T19:39:19.679589+0000","last_clean":"2024-09-20T19:39:19.679589+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:39:19.679589+0000","last_undegraded":"2024-09-20T19:39:19.679589+0000","last_fullsized":"2024-09-20T19:39:19.679589+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:08.849741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:08.849741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:27.397133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1741,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157279+0000","last_change":"2024-09-20T19:38:22.733902+0000","last_active":"2024-09-20T19:39:20.157279+0000","last_peered":"2024-09-20T19:39:20.157279+0000","last_clean":"2024-09-20T19:39:20.157279+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:39:20.157279+0000","last_undegraded":"2024-09-20T19:39:20.157279+0000","last_fullsized":"2024-09-20T19:39:20.157279+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:22.733746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.733746+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:23.387040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":5164,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.496690+0000","last_change":"2024-09-20T19:39:18.587767+0000","last_active":"2024-09-20T19:39:23.496690+0000","last_peered":"2024-09-20T19:39:23.496690+0000","last_clean":"2024-09-20T19:39:23.496690+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:39:23.496690+0000","last_undegraded":"2024-09-20T19:39:23.496690+0000","last_fullsized":"2024-09-20T19:39:23.496690+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:39:12.816955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.816955+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:41.759335+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0031973679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b3","length":"1"},{"start":"1b5","length":"2"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.2","version":"987'30","reported_seq":2094,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.680985+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:39:19.680985+0000","last_peered":"2024-09-20T19:39:19.680985+0000","last_clean":"2024-09-20T19:39:19.680985+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:39:19.680985+0000","last_undegraded":"2024-09-20T19:39:19.680985+0000","last_fullsized":"2024-09-20T19:39:19.680985+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.824143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2042,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683708+0000","last_change":"2024-09-20T19:39:13.877345+0000","last_active":"2024-09-20T19:39:19.683708+0000","last_peered":"2024-09-20T19:39:19.683708+0000","last_clean":"2024-09-20T19:39:19.683708+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:39:19.683708+0000","last_undegraded":"2024-09-20T19:39:19.683708+0000","last_fullsized":"2024-09-20T19:39:19.683708+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:13.877275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.877275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.110921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":2017,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144055+0000","last_change":"2024-09-20T19:38:19.343965+0000","last_active":"2024-09-20T19:39:20.144055+0000","last_peered":"2024-09-20T19:39:20.144055+0000","last_clean":"2024-09-20T19:39:20.144055+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:39:20.144055+0000","last_undegraded":"2024-09-20T19:39:20.144055+0000","last_fullsized":"2024-09-20T19:39:20.144055+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.343828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.343828+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.328086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1830,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.142963+0000","last_change":"2024-09-20T19:35:47.661521+0000","last_active":"2024-09-20T19:39:20.142963+0000","last_peered":"2024-09-20T19:39:20.142963+0000","last_clean":"2024-09-20T19:39:20.142963+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:39:20.142963+0000","last_undegraded":"2024-09-20T19:39:20.142963+0000","last_fullsized":"2024-09-20T19:39:20.142963+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.661410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.661410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1630,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157460+0000","last_change":"2024-09-20T19:38:12.969468+0000","last_active":"2024-09-20T19:39:20.157460+0000","last_peered":"2024-09-20T19:39:20.157460+0000","last_clean":"2024-09-20T19:39:20.157460+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:39:20.157460+0000","last_undegraded":"2024-09-20T19:39:20.157460+0000","last_fullsized":"2024-09-20T19:39:20.157460+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:12.969339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.969339+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:22.236227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1046'726","reported_seq":5490,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158783+0000","last_change":"2024-09-20T19:39:18.539524+0000","last_active":"2024-09-20T19:39:20.158783+0000","last_peered":"2024-09-20T19:39:20.158783+0000","last_clean":"2024-09-20T19:39:20.158783+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:39:20.158783+0000","last_undegraded":"2024-09-20T19:39:20.158783+0000","last_fullsized":"2024-09-20T19:39:20.158783+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'654","last_scrub_stamp":"2024-09-20T19:37:44.764602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:44.764602+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0046037159999999999,"stat_sum":{"num_bytes":2723064,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":186,"num_read_kb":136098,"num_write":135,"num_write_kb":47643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.4","version":"987'57","reported_seq":2444,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683520+0000","last_change":"2024-09-20T19:39:14.900337+0000","last_active":"2024-09-20T19:39:19.683520+0000","last_peered":"2024-09-20T19:39:19.683520+0000","last_clean":"2024-09-20T19:39:19.683520+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:39:19.683520+0000","last_undegraded":"2024-09-20T19:39:19.683520+0000","last_fullsized":"2024-09-20T19:39:19.683520+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'57","last_scrub_stamp":"2024-09-20T19:39:14.900185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:14.900185+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.992844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2050,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.153903+0000","last_change":"2024-09-20T19:38:32.270500+0000","last_active":"2024-09-20T19:39:20.153903+0000","last_peered":"2024-09-20T19:39:20.153903+0000","last_clean":"2024-09-20T19:39:20.153903+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:39:20.153903+0000","last_undegraded":"2024-09-20T19:39:20.153903+0000","last_fullsized":"2024-09-20T19:39:20.153903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:32.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:32.270373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.192387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1820,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143458+0000","last_change":"2024-09-20T19:35:49.735749+0000","last_active":"2024-09-20T19:39:20.143458+0000","last_peered":"2024-09-20T19:39:20.143458+0000","last_clean":"2024-09-20T19:39:20.143458+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:39:20.143458+0000","last_undegraded":"2024-09-20T19:39:20.143458+0000","last_fullsized":"2024-09-20T19:39:20.143458+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:49.735628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.735628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1485,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.500735+0000","last_change":"2024-09-20T19:36:23.041088+0000","last_active":"2024-09-20T19:39:23.500735+0000","last_peered":"2024-09-20T19:39:23.500735+0000","last_clean":"2024-09-20T19:39:23.500735+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:39:23.500735+0000","last_undegraded":"2024-09-20T19:39:23.500735+0000","last_fullsized":"2024-09-20T19:39:23.500735+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.040967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.040967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.759148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":2100,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158612+0000","last_change":"2024-09-20T19:38:41.037160+0000","last_active":"2024-09-20T19:39:20.158612+0000","last_peered":"2024-09-20T19:39:20.158612+0000","last_clean":"2024-09-20T19:39:20.158612+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:39:20.158612+0000","last_undegraded":"2024-09-20T19:39:20.158612+0000","last_fullsized":"2024-09-20T19:39:20.158612+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:41.037037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.037037+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.295442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1048'1200","reported_seq":5522,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.376885+0000","last_change":"2024-09-20T19:39:20.376885+0000","last_active":"2024-09-20T19:39:20.376885+0000","last_peered":"2024-09-20T19:39:20.376885+0000","last_clean":"2024-09-20T19:39:20.376885+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:39:20.376885+0000","last_undegraded":"2024-09-20T19:39:20.376885+0000","last_fullsized":"2024-09-20T19:39:20.376885+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'1200","last_scrub_stamp":"2024-09-20T19:39:20.376813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.376813+0000","objects_scrubbed":2,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.287158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.15482644300000001,"stat_sum":{"num_bytes":5799009,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":159971,"num_write":135,"num_write_kb":55283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.7","version":"987'21","reported_seq":2008,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148727+0000","last_change":"2024-09-20T19:36:23.692393+0000","last_active":"2024-09-20T19:39:20.148727+0000","last_peered":"2024-09-20T19:39:20.148727+0000","last_clean":"2024-09-20T19:39:20.148727+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:39:20.148727+0000","last_undegraded":"2024-09-20T19:39:20.148727+0000","last_fullsized":"2024-09-20T19:39:20.148727+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:36:23.692257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.692257+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.833506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":3311,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.447288+0000","last_change":"2024-09-20T19:39:16.927632+0000","last_active":"2024-09-20T19:39:20.447288+0000","last_peered":"2024-09-20T19:39:20.447288+0000","last_clean":"2024-09-20T19:39:20.447288+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:39:20.447288+0000","last_undegraded":"2024-09-20T19:39:20.447288+0000","last_fullsized":"2024-09-20T19:39:20.447288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:39:16.927571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.927571+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:17.759811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1256,"num_read_kb":1251,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2062,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147707+0000","last_change":"2024-09-20T19:37:39.514447+0000","last_active":"2024-09-20T19:39:20.147707+0000","last_peered":"2024-09-20T19:39:20.147707+0000","last_clean":"2024-09-20T19:39:20.147707+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:39:20.147707+0000","last_undegraded":"2024-09-20T19:39:20.147707+0000","last_fullsized":"2024-09-20T19:39:20.147707+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:39.514392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:39.514392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":616,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148618+0000","last_change":"2024-09-20T19:37:34.496871+0000","last_active":"2024-09-20T19:39:20.148618+0000","last_peered":"2024-09-20T19:39:20.148618+0000","last_clean":"2024-09-20T19:39:20.148618+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:39:20.148618+0000","last_undegraded":"2024-09-20T19:39:20.148618+0000","last_fullsized":"2024-09-20T19:39:20.148618+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:34.496752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:34.496752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2024,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.159584+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:39:20.159584+0000","last_peered":"2024-09-20T19:39:20.159584+0000","last_clean":"2024-09-20T19:39:20.159584+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:39:20.159584+0000","last_undegraded":"2024-09-20T19:39:20.159584+0000","last_fullsized":"2024-09-20T19:39:20.159584+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.132300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1633,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.685006+0000","last_change":"2024-09-20T19:38:14.535923+0000","last_active":"2024-09-20T19:39:19.685006+0000","last_peered":"2024-09-20T19:39:19.685006+0000","last_clean":"2024-09-20T19:39:19.685006+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:39:19.685006+0000","last_undegraded":"2024-09-20T19:39:19.685006+0000","last_fullsized":"2024-09-20T19:39:19.685006+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:14.535715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.535715+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:27.281420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1050'1256","reported_seq":5979,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.159276+0000","last_change":"2024-09-20T19:39:18.525631+0000","last_active":"2024-09-20T19:39:20.159276+0000","last_peered":"2024-09-20T19:39:20.159276+0000","last_clean":"2024-09-20T19:39:20.159276+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:39:20.159276+0000","last_undegraded":"2024-09-20T19:39:20.159276+0000","last_fullsized":"2024-09-20T19:39:20.159276+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"892'1099","last_scrub_stamp":"2024-09-20T19:36:40.791824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.791824+0000","objects_scrubbed":12,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003200708,"stat_sum":{"num_bytes":12714313,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":271,"num_read_kb":207585,"num_write":180,"num_write_kb":60667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.6","version":"987'36","reported_seq":1878,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144346+0000","last_change":"2024-09-20T19:36:33.028171+0000","last_active":"2024-09-20T19:39:20.144346+0000","last_peered":"2024-09-20T19:39:20.144346+0000","last_clean":"2024-09-20T19:39:20.144346+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:39:20.144346+0000","last_undegraded":"2024-09-20T19:39:20.144346+0000","last_fullsized":"2024-09-20T19:39:20.144346+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:36:33.028057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:33.028057+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1774,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143700+0000","last_change":"2024-09-20T19:38:25.204605+0000","last_active":"2024-09-20T19:39:20.143700+0000","last_peered":"2024-09-20T19:39:20.143700+0000","last_clean":"2024-09-20T19:39:20.143700+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:39:20.143700+0000","last_undegraded":"2024-09-20T19:39:20.143700+0000","last_fullsized":"2024-09-20T19:39:20.143700+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:38:25.204541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.204541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.668929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2037,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151119+0000","last_change":"2024-09-20T19:38:25.143232+0000","last_active":"2024-09-20T19:39:20.151119+0000","last_peered":"2024-09-20T19:39:20.151119+0000","last_clean":"2024-09-20T19:39:20.151119+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:39:20.151119+0000","last_undegraded":"2024-09-20T19:39:20.151119+0000","last_fullsized":"2024-09-20T19:39:20.151119+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.143087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.143087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:32.092810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2036,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157718+0000","last_change":"2024-09-20T19:38:20.052968+0000","last_active":"2024-09-20T19:39:20.157718+0000","last_peered":"2024-09-20T19:39:20.157718+0000","last_clean":"2024-09-20T19:39:20.157718+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:39:20.157718+0000","last_undegraded":"2024-09-20T19:39:20.157718+0000","last_fullsized":"2024-09-20T19:39:20.157718+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.052811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.052811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.621676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2061,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.681297+0000","last_change":"2024-09-20T19:37:25.979434+0000","last_active":"2024-09-20T19:39:19.681297+0000","last_peered":"2024-09-20T19:39:19.681297+0000","last_clean":"2024-09-20T19:39:19.681297+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:39:19.681297+0000","last_undegraded":"2024-09-20T19:39:19.681297+0000","last_fullsized":"2024-09-20T19:39:19.681297+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.979338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.979338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":4575,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144452+0000","last_change":"2024-09-20T19:39:19.369908+0000","last_active":"2024-09-20T19:39:20.144452+0000","last_peered":"2024-09-20T19:39:20.144452+0000","last_clean":"2024-09-20T19:39:20.144452+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:39:20.144452+0000","last_undegraded":"2024-09-20T19:39:20.144452+0000","last_fullsized":"2024-09-20T19:39:20.144452+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:39:19.369830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:19.369830+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:20.862550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.091894885999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.5","version":"987'42","reported_seq":2348,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155980+0000","last_change":"2024-09-20T19:36:48.949647+0000","last_active":"2024-09-20T19:39:20.155980+0000","last_peered":"2024-09-20T19:39:20.155980+0000","last_clean":"2024-09-20T19:39:20.155980+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:39:20.155980+0000","last_undegraded":"2024-09-20T19:39:20.155980+0000","last_fullsized":"2024-09-20T19:39:20.155980+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:36:48.949462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.949462+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.572488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":888,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152947+0000","last_change":"2024-09-20T19:38:22.129174+0000","last_active":"2024-09-20T19:39:20.152947+0000","last_peered":"2024-09-20T19:39:20.152947+0000","last_clean":"2024-09-20T19:39:20.152947+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:39:20.152947+0000","last_undegraded":"2024-09-20T19:39:20.152947+0000","last_fullsized":"2024-09-20T19:39:20.152947+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:38:22.129045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.129045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.992187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2018,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155926+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:39:20.155926+0000","last_peered":"2024-09-20T19:39:20.155926+0000","last_clean":"2024-09-20T19:39:20.155926+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:39:20.155926+0000","last_undegraded":"2024-09-20T19:39:20.155926+0000","last_fullsized":"2024-09-20T19:39:20.155926+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.413613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":2087,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151387+0000","last_change":"2024-09-20T19:39:20.032046+0000","last_active":"2024-09-20T19:39:20.151387+0000","last_peered":"2024-09-20T19:39:20.151387+0000","last_clean":"2024-09-20T19:39:20.151387+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:39:20.151387+0000","last_undegraded":"2024-09-20T19:39:20.151387+0000","last_fullsized":"2024-09-20T19:39:20.151387+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:20.031879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.031879+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:47.216443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":2010,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155474+0000","last_change":"2024-09-20T19:38:30.011404+0000","last_active":"2024-09-20T19:39:20.155474+0000","last_peered":"2024-09-20T19:39:20.155474+0000","last_clean":"2024-09-20T19:39:20.155474+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:39:20.155474+0000","last_undegraded":"2024-09-20T19:39:20.155474+0000","last_fullsized":"2024-09-20T19:39:20.155474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.011277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.011277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.873402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2046,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158333+0000","last_change":"2024-09-20T19:38:34.247348+0000","last_active":"2024-09-20T19:39:20.158333+0000","last_peered":"2024-09-20T19:39:20.158333+0000","last_clean":"2024-09-20T19:39:20.158333+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:39:20.158333+0000","last_undegraded":"2024-09-20T19:39:20.158333+0000","last_fullsized":"2024-09-20T19:39:20.158333+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:34.247292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:34.247292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.593318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"1046'472","reported_seq":4827,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143862+0000","last_change":"2024-09-20T19:39:18.609359+0000","last_active":"2024-09-20T19:39:20.143862+0000","last_peered":"2024-09-20T19:39:20.143862+0000","last_clean":"2024-09-20T19:39:20.143862+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:39:20.143862+0000","last_undegraded":"2024-09-20T19:39:20.143862+0000","last_fullsized":"2024-09-20T19:39:20.143862+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'450","last_scrub_stamp":"2024-09-20T19:38:35.237169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.237169+0000","objects_scrubbed":1,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.156087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.093957481999999995,"stat_sum":{"num_bytes":3944148,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":32971,"num_write":72,"num_write_kb":19085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.a","version":"987'51","reported_seq":2101,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.156571+0000","last_change":"2024-09-20T19:39:18.379050+0000","last_active":"2024-09-20T19:39:20.156571+0000","last_peered":"2024-09-20T19:39:20.156571+0000","last_clean":"2024-09-20T19:39:20.156571+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:39:20.156571+0000","last_undegraded":"2024-09-20T19:39:20.156571+0000","last_fullsized":"2024-09-20T19:39:20.156571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'51","last_scrub_stamp":"2024-09-20T19:39:18.378913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:18.378913+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:46.141156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2063,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148457+0000","last_change":"2024-09-20T19:38:25.535832+0000","last_active":"2024-09-20T19:39:20.148457+0000","last_peered":"2024-09-20T19:39:20.148457+0000","last_clean":"2024-09-20T19:39:20.148457+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:39:20.148457+0000","last_undegraded":"2024-09-20T19:39:20.148457+0000","last_fullsized":"2024-09-20T19:39:20.148457+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.535708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.535708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.031136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":616,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147540+0000","last_change":"2024-09-20T19:38:24.553105+0000","last_active":"2024-09-20T19:39:20.147540+0000","last_peered":"2024-09-20T19:39:20.147540+0000","last_clean":"2024-09-20T19:39:20.147540+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:39:20.147540+0000","last_undegraded":"2024-09-20T19:39:20.147540+0000","last_fullsized":"2024-09-20T19:39:20.147540+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:24.553002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.553002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:27.621618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2015,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151250+0000","last_change":"2024-09-20T19:38:14.008440+0000","last_active":"2024-09-20T19:39:20.151250+0000","last_peered":"2024-09-20T19:39:20.151250+0000","last_clean":"2024-09-20T19:39:20.151250+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:39:20.151250+0000","last_undegraded":"2024-09-20T19:39:20.151250+0000","last_fullsized":"2024-09-20T19:39:20.151250+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:14.008316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.008316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.394501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1045'1673","reported_seq":6288,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.146812+0000","last_change":"2024-09-20T19:39:18.554664+0000","last_active":"2024-09-20T19:39:20.146812+0000","last_peered":"2024-09-20T19:39:20.146812+0000","last_clean":"2024-09-20T19:39:20.146812+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:39:20.146812+0000","last_undegraded":"2024-09-20T19:39:20.146812+0000","last_fullsized":"2024-09-20T19:39:20.146812+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'1669","last_scrub_stamp":"2024-09-20T19:39:07.884701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.884701+0000","objects_scrubbed":4,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.699067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001384714,"stat_sum":{"num_bytes":5983919,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":272,"num_read_kb":211320,"num_write":187,"num_write_kb":73661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.b","version":"987'21","reported_seq":1844,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143887+0000","last_change":"2024-09-20T19:35:50.754361+0000","last_active":"2024-09-20T19:39:20.143887+0000","last_peered":"2024-09-20T19:39:20.143887+0000","last_clean":"2024-09-20T19:39:20.143887+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:39:20.143887+0000","last_undegraded":"2024-09-20T19:39:20.143887+0000","last_fullsized":"2024-09-20T19:39:20.143887+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:50.754272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:50.754272+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":2062,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.684142+0000","last_change":"2024-09-20T19:36:34.027338+0000","last_active":"2024-09-20T19:39:19.684142+0000","last_peered":"2024-09-20T19:39:19.684142+0000","last_clean":"2024-09-20T19:39:19.684142+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:39:19.684142+0000","last_undegraded":"2024-09-20T19:39:19.684142+0000","last_fullsized":"2024-09-20T19:39:19.684142+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:34.027257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:34.027257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.639569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2069,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147345+0000","last_change":"2024-09-20T19:38:31.728072+0000","last_active":"2024-09-20T19:39:20.147345+0000","last_peered":"2024-09-20T19:39:20.147345+0000","last_clean":"2024-09-20T19:39:20.147345+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:39:20.147345+0000","last_undegraded":"2024-09-20T19:39:20.147345+0000","last_fullsized":"2024-09-20T19:39:20.147345+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.728014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.728014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.454777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1836,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143823+0000","last_change":"2024-09-20T19:35:51.726463+0000","last_active":"2024-09-20T19:39:20.143823+0000","last_peered":"2024-09-20T19:39:20.143823+0000","last_clean":"2024-09-20T19:39:20.143823+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:39:20.143823+0000","last_undegraded":"2024-09-20T19:39:20.143823+0000","last_fullsized":"2024-09-20T19:39:20.143823+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:35:51.726405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:51.726405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"1050'682","reported_seq":5333,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.680013+0000","last_change":"2024-09-20T19:39:18.520140+0000","last_active":"2024-09-20T19:39:19.680013+0000","last_peered":"2024-09-20T19:39:19.680013+0000","last_clean":"2024-09-20T19:39:19.680013+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:39:19.680013+0000","last_undegraded":"2024-09-20T19:39:19.680013+0000","last_fullsized":"2024-09-20T19:39:19.680013+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'666","last_scrub_stamp":"2024-09-20T19:38:31.665799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.665799+0000","objects_scrubbed":2,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:37.291094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00078118399999999998,"stat_sum":{"num_bytes":6598448,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":155184,"num_write":61,"num_write_kb":17827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.8","version":"987'39","reported_seq":2098,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.149996+0000","last_change":"2024-09-20T19:39:07.498428+0000","last_active":"2024-09-20T19:39:20.149996+0000","last_peered":"2024-09-20T19:39:20.149996+0000","last_clean":"2024-09-20T19:39:20.149996+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:39:20.149996+0000","last_undegraded":"2024-09-20T19:39:20.149996+0000","last_fullsized":"2024-09-20T19:39:20.149996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:07.498283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.498283+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:11.846306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":620,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.148722+0000","last_change":"2024-09-20T19:37:32.482435+0000","last_active":"2024-09-20T19:39:20.148722+0000","last_peered":"2024-09-20T19:39:20.148722+0000","last_clean":"2024-09-20T19:39:20.148722+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:39:20.148722+0000","last_undegraded":"2024-09-20T19:39:20.148722+0000","last_fullsized":"2024-09-20T19:39:20.148722+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:37:32.482370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.482370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.892648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":2288,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152289+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:39:20.152289+0000","last_peered":"2024-09-20T19:39:20.152289+0000","last_clean":"2024-09-20T19:39:20.152289+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:39:20.152289+0000","last_undegraded":"2024-09-20T19:39:20.152289+0000","last_fullsized":"2024-09-20T19:39:20.152289+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.039100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2032,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152246+0000","last_change":"2024-09-20T19:39:05.442773+0000","last_active":"2024-09-20T19:39:20.152246+0000","last_peered":"2024-09-20T19:39:20.152246+0000","last_clean":"2024-09-20T19:39:20.152246+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:39:20.152246+0000","last_undegraded":"2024-09-20T19:39:20.152246+0000","last_fullsized":"2024-09-20T19:39:20.152246+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.442716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.442716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:22.768089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":4726,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147047+0000","last_change":"2024-09-20T19:39:18.552320+0000","last_active":"2024-09-20T19:39:20.147047+0000","last_peered":"2024-09-20T19:39:20.147047+0000","last_clean":"2024-09-20T19:39:20.147047+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:39:20.147047+0000","last_undegraded":"2024-09-20T19:39:20.147047+0000","last_fullsized":"2024-09-20T19:39:20.147047+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:39:09.971030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.971030+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.776858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017643380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.9","version":"987'24","reported_seq":1919,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.497186+0000","last_change":"2024-09-20T19:36:40.974050+0000","last_active":"2024-09-20T19:39:23.497186+0000","last_peered":"2024-09-20T19:39:23.497186+0000","last_clean":"2024-09-20T19:39:23.497186+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:39:23.497186+0000","last_undegraded":"2024-09-20T19:39:23.497186+0000","last_fullsized":"2024-09-20T19:39:23.497186+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:36:40.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.973907+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.768011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1905,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.500044+0000","last_change":"2024-09-20T19:38:31.783351+0000","last_active":"2024-09-20T19:39:23.500044+0000","last_peered":"2024-09-20T19:39:23.500044+0000","last_clean":"2024-09-20T19:39:23.500044+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:39:23.500044+0000","last_undegraded":"2024-09-20T19:39:23.500044+0000","last_fullsized":"2024-09-20T19:39:23.500044+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:38:31.783222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.783222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:48.587593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2047,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147684+0000","last_change":"2024-09-20T19:38:36.786857+0000","last_active":"2024-09-20T19:39:20.147684+0000","last_peered":"2024-09-20T19:39:20.147684+0000","last_clean":"2024-09-20T19:39:20.147684+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:39:20.147684+0000","last_undegraded":"2024-09-20T19:39:20.147684+0000","last_fullsized":"2024-09-20T19:39:20.147684+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:36.786803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.786803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:39.996355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1520,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152893+0000","last_change":"2024-09-20T19:37:32.585551+0000","last_active":"2024-09-20T19:39:20.152893+0000","last_peered":"2024-09-20T19:39:20.152893+0000","last_clean":"2024-09-20T19:39:20.152893+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:39:20.152893+0000","last_undegraded":"2024-09-20T19:39:20.152893+0000","last_fullsized":"2024-09-20T19:39:20.152893+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:32.585494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.585494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"1045'1372","reported_seq":6403,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.159949+0000","last_change":"2024-09-20T19:39:18.526014+0000","last_active":"2024-09-20T19:39:20.159949+0000","last_peered":"2024-09-20T19:39:20.159949+0000","last_clean":"2024-09-20T19:39:20.159949+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:39:20.159949+0000","last_undegraded":"2024-09-20T19:39:20.159949+0000","last_fullsized":"2024-09-20T19:39:20.159949+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"1045'1372","last_scrub_stamp":"2024-09-20T19:39:17.039956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.039956+0000","objects_scrubbed":4,"log_size":1372,"log_dups_size":0,"ondisk_log_size":1372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:42.691540+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012545200000000001,"stat_sum":{"num_bytes":9026679,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":216218,"num_write":228,"num_write_kb":91119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.16","version":"987'21","reported_seq":1845,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144144+0000","last_change":"2024-09-20T19:37:23.121422+0000","last_active":"2024-09-20T19:39:20.144144+0000","last_peered":"2024-09-20T19:39:20.144144+0000","last_clean":"2024-09-20T19:39:20.144144+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:39:20.144144+0000","last_undegraded":"2024-09-20T19:39:20.144144+0000","last_fullsized":"2024-09-20T19:39:20.144144+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:37:23.121282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.121282+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2079,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147761+0000","last_change":"2024-09-20T19:38:30.686760+0000","last_active":"2024-09-20T19:39:20.147761+0000","last_peered":"2024-09-20T19:39:20.147761+0000","last_clean":"2024-09-20T19:39:20.147761+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:39:20.147761+0000","last_undegraded":"2024-09-20T19:39:20.147761+0000","last_fullsized":"2024-09-20T19:39:20.147761+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.686621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.686621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.603556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1523,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.684734+0000","last_change":"2024-09-20T19:39:15.910802+0000","last_active":"2024-09-20T19:39:19.684734+0000","last_peered":"2024-09-20T19:39:19.684734+0000","last_clean":"2024-09-20T19:39:19.684734+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:39:19.684734+0000","last_undegraded":"2024-09-20T19:39:19.684734+0000","last_fullsized":"2024-09-20T19:39:19.684734+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:15.910687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.910687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:35.477531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2035,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147585+0000","last_change":"2024-09-20T19:37:37.478973+0000","last_active":"2024-09-20T19:39:20.147585+0000","last_peered":"2024-09-20T19:39:20.147585+0000","last_clean":"2024-09-20T19:39:20.147585+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:39:20.147585+0000","last_undegraded":"2024-09-20T19:39:20.147585+0000","last_fullsized":"2024-09-20T19:39:20.147585+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:37.478823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:37.478823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.857932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1048'1653","reported_seq":6328,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.147463+0000","last_change":"2024-09-20T19:39:18.551000+0000","last_active":"2024-09-20T19:39:20.147463+0000","last_peered":"2024-09-20T19:39:20.147463+0000","last_clean":"2024-09-20T19:39:20.147463+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:39:20.147463+0000","last_undegraded":"2024-09-20T19:39:20.147463+0000","last_fullsized":"2024-09-20T19:39:20.147463+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"986'1609","last_scrub_stamp":"2024-09-20T19:38:22.531869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.531869+0000","objects_scrubbed":3,"log_size":1653,"log_dups_size":0,"ondisk_log_size":1653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.814084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00070473599999999997,"stat_sum":{"num_bytes":10832612,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":313,"num_read_kb":203037,"num_write":237,"num_write_kb":87419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"2.11","version":"0'0","reported_seq":2029,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152905+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:39:20.152905+0000","last_peered":"2024-09-20T19:39:20.152905+0000","last_clean":"2024-09-20T19:39:20.152905+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:39:20.152905+0000","last_undegraded":"2024-09-20T19:39:20.152905+0000","last_fullsized":"2024-09-20T19:39:20.152905+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.535576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"987'18","reported_seq":1697,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157686+0000","last_change":"2024-09-20T19:36:39.739712+0000","last_active":"2024-09-20T19:39:20.157686+0000","last_peered":"2024-09-20T19:39:20.157686+0000","last_clean":"2024-09-20T19:39:20.157686+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:39:20.157686+0000","last_undegraded":"2024-09-20T19:39:20.157686+0000","last_fullsized":"2024-09-20T19:39:20.157686+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:36:39.739615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:39.739615+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2050,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154322+0000","last_change":"2024-09-20T19:37:43.848462+0000","last_active":"2024-09-20T19:39:20.154322+0000","last_peered":"2024-09-20T19:39:20.154322+0000","last_clean":"2024-09-20T19:39:20.154322+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:39:20.154322+0000","last_undegraded":"2024-09-20T19:39:20.154322+0000","last_fullsized":"2024-09-20T19:39:20.154322+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:43.848400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:43.848400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2015,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:22.039805+0000","last_change":"2024-09-20T19:39:22.039805+0000","last_active":"2024-09-20T19:39:22.039805+0000","last_peered":"2024-09-20T19:39:22.039805+0000","last_clean":"2024-09-20T19:39:22.039805+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:39:22.039805+0000","last_undegraded":"2024-09-20T19:39:22.039805+0000","last_fullsized":"2024-09-20T19:39:22.039805+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:22.039740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.039740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:44.971668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1020'831","reported_seq":5382,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.158464+0000","last_change":"2024-09-20T19:39:18.527861+0000","last_active":"2024-09-20T19:39:20.158464+0000","last_peered":"2024-09-20T19:39:20.158464+0000","last_clean":"2024-09-20T19:39:20.158464+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:39:20.158464+0000","last_undegraded":"2024-09-20T19:39:20.158464+0000","last_fullsized":"2024-09-20T19:39:20.158464+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"988'810","last_scrub_stamp":"2024-09-20T19:38:21.101246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:21.101246+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:23.762659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001718083,"stat_sum":{"num_bytes":2118012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":145,"num_read_kb":108369,"num_write":90,"num_write_kb":32667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.14","version":"987'30","reported_seq":1720,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683015+0000","last_change":"2024-09-20T19:36:36.023059+0000","last_active":"2024-09-20T19:39:19.683015+0000","last_peered":"2024-09-20T19:39:19.683015+0000","last_clean":"2024-09-20T19:39:19.683015+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:39:19.683015+0000","last_undegraded":"2024-09-20T19:39:19.683015+0000","last_fullsized":"2024-09-20T19:39:19.683015+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:36:36.022993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:36.022993+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:51.646412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":2059,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.153445+0000","last_change":"2024-09-20T19:38:31.227463+0000","last_active":"2024-09-20T19:39:20.153445+0000","last_peered":"2024-09-20T19:39:20.153445+0000","last_clean":"2024-09-20T19:39:20.153445+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:39:20.153445+0000","last_undegraded":"2024-09-20T19:39:20.153445+0000","last_fullsized":"2024-09-20T19:39:20.153445+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.227391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.227391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.357562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":2303,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150950+0000","last_change":"2024-09-20T19:38:19.049146+0000","last_active":"2024-09-20T19:39:20.150950+0000","last_peered":"2024-09-20T19:39:20.150950+0000","last_clean":"2024-09-20T19:39:20.150950+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:39:20.150950+0000","last_undegraded":"2024-09-20T19:39:20.150950+0000","last_fullsized":"2024-09-20T19:39:20.150950+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.049091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.049091+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:31.883013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1808,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144591+0000","last_change":"2024-09-20T19:35:56.751414+0000","last_active":"2024-09-20T19:39:20.144591+0000","last_peered":"2024-09-20T19:39:20.144591+0000","last_clean":"2024-09-20T19:39:20.144591+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:39:20.144591+0000","last_undegraded":"2024-09-20T19:39:20.144591+0000","last_fullsized":"2024-09-20T19:39:20.144591+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:56.751367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:56.751367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1045'761","reported_seq":5130,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.502108+0000","last_change":"2024-09-20T19:39:18.589826+0000","last_active":"2024-09-20T19:39:23.502108+0000","last_peered":"2024-09-20T19:39:23.502108+0000","last_clean":"2024-09-20T19:39:23.502108+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:39:23.502108+0000","last_undegraded":"2024-09-20T19:39:23.502108+0000","last_fullsized":"2024-09-20T19:39:23.502108+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"994'754","last_scrub_stamp":"2024-09-20T19:38:40.806420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.806420+0000","objects_scrubbed":1,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.595373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0033661839999999999,"stat_sum":{"num_bytes":3656437,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":187,"num_read_kb":132046,"num_write":137,"num_write_kb":52044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b3","length":"1"},{"start":"1b5","length":"2"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.15","version":"987'21","reported_seq":2105,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151217+0000","last_change":"2024-09-20T19:38:30.190106+0000","last_active":"2024-09-20T19:39:20.151217+0000","last_peered":"2024-09-20T19:39:20.151217+0000","last_clean":"2024-09-20T19:39:20.151217+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:39:20.151217+0000","last_undegraded":"2024-09-20T19:39:20.151217+0000","last_fullsized":"2024-09-20T19:39:20.151217+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.190020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.190020+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.116547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1781,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144139+0000","last_change":"2024-09-20T19:39:12.256710+0000","last_active":"2024-09-20T19:39:20.144139+0000","last_peered":"2024-09-20T19:39:20.144139+0000","last_clean":"2024-09-20T19:39:20.144139+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:39:20.144139+0000","last_undegraded":"2024-09-20T19:39:20.144139+0000","last_fullsized":"2024-09-20T19:39:20.144139+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:12.256601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.256601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.448150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":754,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150693+0000","last_change":"2024-09-20T19:38:27.671255+0000","last_active":"2024-09-20T19:39:20.150693+0000","last_peered":"2024-09-20T19:39:20.150693+0000","last_clean":"2024-09-20T19:39:20.150693+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:39:20.150693+0000","last_undegraded":"2024-09-20T19:39:20.150693+0000","last_fullsized":"2024-09-20T19:39:20.150693+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'17","last_scrub_stamp":"2024-09-20T19:38:27.671197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.671197+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.930015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1867,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.499580+0000","last_change":"2024-09-20T19:36:49.011748+0000","last_active":"2024-09-20T19:39:23.499580+0000","last_peered":"2024-09-20T19:39:23.499580+0000","last_clean":"2024-09-20T19:39:23.499580+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:39:23.499580+0000","last_undegraded":"2024-09-20T19:39:23.499580+0000","last_fullsized":"2024-09-20T19:39:23.499580+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:49.011625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:49.011625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.796400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1045'626","reported_seq":5279,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.149852+0000","last_change":"2024-09-20T19:39:18.524851+0000","last_active":"2024-09-20T19:39:20.149852+0000","last_peered":"2024-09-20T19:39:20.149852+0000","last_clean":"2024-09-20T19:39:20.149852+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:39:20.149852+0000","last_undegraded":"2024-09-20T19:39:20.149852+0000","last_fullsized":"2024-09-20T19:39:20.149852+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"870'563","last_scrub_stamp":"2024-09-20T19:36:28.907373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:28.907373+0000","objects_scrubbed":6,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048363049999999999,"stat_sum":{"num_bytes":2437345,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":70,"num_read_kb":54472,"num_write":52,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.12","version":"987'21","reported_seq":2068,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.680699+0000","last_change":"2024-09-20T19:35:49.126133+0000","last_active":"2024-09-20T19:39:19.680699+0000","last_peered":"2024-09-20T19:39:19.680699+0000","last_clean":"2024-09-20T19:39:19.680699+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:39:19.680699+0000","last_undegraded":"2024-09-20T19:39:19.680699+0000","last_fullsized":"2024-09-20T19:39:19.680699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:49.126032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.126032+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.626720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2028,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152231+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:39:20.152231+0000","last_peered":"2024-09-20T19:39:20.152231+0000","last_clean":"2024-09-20T19:39:20.152231+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:39:20.152231+0000","last_undegraded":"2024-09-20T19:39:20.152231+0000","last_fullsized":"2024-09-20T19:39:20.152231+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.462527+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1537,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.142959+0000","last_change":"2024-09-20T19:37:20.195581+0000","last_active":"2024-09-20T19:39:20.142959+0000","last_peered":"2024-09-20T19:39:20.142959+0000","last_clean":"2024-09-20T19:39:20.142959+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:39:20.142959+0000","last_undegraded":"2024-09-20T19:39:20.142959+0000","last_fullsized":"2024-09-20T19:39:20.142959+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:20.195458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:20.195458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1783,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143208+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:39:20.143208+0000","last_peered":"2024-09-20T19:39:20.143208+0000","last_clean":"2024-09-20T19:39:20.143208+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:39:20.143208+0000","last_undegraded":"2024-09-20T19:39:20.143208+0000","last_fullsized":"2024-09-20T19:39:20.143208+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.129003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1006'762","reported_seq":5371,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.146794+0000","last_change":"2024-09-20T19:39:18.553777+0000","last_active":"2024-09-20T19:39:20.146794+0000","last_peered":"2024-09-20T19:39:20.146794+0000","last_clean":"2024-09-20T19:39:20.146794+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:39:20.146794+0000","last_undegraded":"2024-09-20T19:39:20.146794+0000","last_fullsized":"2024-09-20T19:39:20.146794+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"979'756","last_scrub_stamp":"2024-09-20T19:38:12.650090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.650090+0000","objects_scrubbed":1,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:27.006004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062587600000000004,"stat_sum":{"num_bytes":3966749,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":86291,"num_write":62,"num_write_kb":17826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.13","version":"987'27","reported_seq":1886,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.497461+0000","last_change":"2024-09-20T19:38:17.754780+0000","last_active":"2024-09-20T19:39:23.497461+0000","last_peered":"2024-09-20T19:39:23.497461+0000","last_clean":"2024-09-20T19:39:23.497461+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:39:23.497461+0000","last_undegraded":"2024-09-20T19:39:23.497461+0000","last_fullsized":"2024-09-20T19:39:23.497461+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.754717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.754717+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":2056,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683885+0000","last_change":"2024-09-20T19:39:09.812428+0000","last_active":"2024-09-20T19:39:19.683885+0000","last_peered":"2024-09-20T19:39:19.683885+0000","last_clean":"2024-09-20T19:39:19.683885+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:39:19.683885+0000","last_undegraded":"2024-09-20T19:39:19.683885+0000","last_fullsized":"2024-09-20T19:39:19.683885+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:09.812317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.812317+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.550714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":2051,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.680264+0000","last_change":"2024-09-20T19:37:23.932269+0000","last_active":"2024-09-20T19:39:19.680264+0000","last_peered":"2024-09-20T19:39:19.680264+0000","last_clean":"2024-09-20T19:39:19.680264+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:39:19.680264+0000","last_undegraded":"2024-09-20T19:39:19.680264+0000","last_fullsized":"2024-09-20T19:39:19.680264+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:23.932204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.932204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1541,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143560+0000","last_change":"2024-09-20T19:38:40.086970+0000","last_active":"2024-09-20T19:39:20.143560+0000","last_peered":"2024-09-20T19:39:20.143560+0000","last_clean":"2024-09-20T19:39:20.143560+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:39:20.143560+0000","last_undegraded":"2024-09-20T19:39:20.143560+0000","last_fullsized":"2024-09-20T19:39:20.143560+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.086844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.086844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:00.305793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"1048'549","reported_seq":5172,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143218+0000","last_change":"2024-09-20T19:39:18.609629+0000","last_active":"2024-09-20T19:39:20.143218+0000","last_peered":"2024-09-20T19:39:20.143218+0000","last_clean":"2024-09-20T19:39:20.143218+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:39:20.143218+0000","last_undegraded":"2024-09-20T19:39:20.143218+0000","last_fullsized":"2024-09-20T19:39:20.143218+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'542","last_scrub_stamp":"2024-09-20T19:38:30.128571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.128571+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.248720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.091911614000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":8,"num_write":50,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.10","version":"987'12","reported_seq":1703,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.679968+0000","last_change":"2024-09-20T19:39:04.841388+0000","last_active":"2024-09-20T19:39:19.679968+0000","last_peered":"2024-09-20T19:39:19.679968+0000","last_clean":"2024-09-20T19:39:19.679968+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:39:19.679968+0000","last_undegraded":"2024-09-20T19:39:19.679968+0000","last_fullsized":"2024-09-20T19:39:19.679968+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12","last_scrub_stamp":"2024-09-20T19:39:04.841305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:04.841305+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.241113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":2038,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.154678+0000","last_change":"2024-09-20T19:37:36.911119+0000","last_active":"2024-09-20T19:39:20.154678+0000","last_peered":"2024-09-20T19:39:20.154678+0000","last_clean":"2024-09-20T19:39:20.154678+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:39:20.154678+0000","last_undegraded":"2024-09-20T19:39:20.154678+0000","last_fullsized":"2024-09-20T19:39:20.154678+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:36.910997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:36.910997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2029,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.151758+0000","last_change":"2024-09-20T19:39:12.350157+0000","last_active":"2024-09-20T19:39:20.151758+0000","last_peered":"2024-09-20T19:39:20.151758+0000","last_clean":"2024-09-20T19:39:20.151758+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:39:20.151758+0000","last_undegraded":"2024-09-20T19:39:20.151758+0000","last_fullsized":"2024-09-20T19:39:20.151758+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:12.350095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.350095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:36.679981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2000,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157168+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:39:20.157168+0000","last_peered":"2024-09-20T19:39:20.157168+0000","last_clean":"2024-09-20T19:39:20.157168+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:39:20.157168+0000","last_undegraded":"2024-09-20T19:39:20.157168+0000","last_fullsized":"2024-09-20T19:39:20.157168+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.047922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":4674,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155399+0000","last_change":"2024-09-20T19:39:18.538535+0000","last_active":"2024-09-20T19:39:20.155399+0000","last_peered":"2024-09-20T19:39:20.155399+0000","last_clean":"2024-09-20T19:39:20.155399+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:39:20.155399+0000","last_undegraded":"2024-09-20T19:39:20.155399+0000","last_fullsized":"2024-09-20T19:39:20.155399+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:38:16.787693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:16.787693+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:26.141527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0057277079999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"4.11","version":"987'27","reported_seq":916,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.153127+0000","last_change":"2024-09-20T19:38:17.128241+0000","last_active":"2024-09-20T19:39:20.153127+0000","last_peered":"2024-09-20T19:39:20.153127+0000","last_clean":"2024-09-20T19:39:20.153127+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:39:20.153127+0000","last_undegraded":"2024-09-20T19:39:20.153127+0000","last_fullsized":"2024-09-20T19:39:20.153127+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.128056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.128056+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:20.660206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2033,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:21.325794+0000","last_change":"2024-09-20T19:39:21.325794+0000","last_active":"2024-09-20T19:39:21.325794+0000","last_peered":"2024-09-20T19:39:21.325794+0000","last_clean":"2024-09-20T19:39:21.325794+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:39:21.325794+0000","last_undegraded":"2024-09-20T19:39:21.325794+0000","last_fullsized":"2024-09-20T19:39:21.325794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:21.325640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.325640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:50.322842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":2028,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.149872+0000","last_change":"2024-09-20T19:38:35.028245+0000","last_active":"2024-09-20T19:39:20.149872+0000","last_peered":"2024-09-20T19:39:20.149872+0000","last_clean":"2024-09-20T19:39:20.149872+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:39:20.149872+0000","last_undegraded":"2024-09-20T19:39:20.149872+0000","last_fullsized":"2024-09-20T19:39:20.149872+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.028101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.028101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:56.459400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1549,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143762+0000","last_change":"2024-09-20T19:36:47.076088+0000","last_active":"2024-09-20T19:39:20.143762+0000","last_peered":"2024-09-20T19:39:20.143762+0000","last_clean":"2024-09-20T19:39:20.143762+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:39:20.143762+0000","last_undegraded":"2024-09-20T19:39:20.143762+0000","last_fullsized":"2024-09-20T19:39:20.143762+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:47.075960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:47.075960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"1045'846","reported_seq":5490,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.681889+0000","last_change":"2024-09-20T19:39:18.521298+0000","last_active":"2024-09-20T19:39:19.681889+0000","last_peered":"2024-09-20T19:39:19.681889+0000","last_clean":"2024-09-20T19:39:19.681889+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:39:19.681889+0000","last_undegraded":"2024-09-20T19:39:19.681889+0000","last_fullsized":"2024-09-20T19:39:19.681889+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"999'811","last_scrub_stamp":"2024-09-20T19:38:37.609506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:37.609506+0000","objects_scrubbed":1,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:39.484493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00069737399999999995,"stat_sum":{"num_bytes":5983919,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":99462,"num_write":87,"num_write_kb":36359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"6.1c","version":"77'1","reported_seq":2006,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.155211+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:39:20.155211+0000","last_peered":"2024-09-20T19:39:20.155211+0000","last_clean":"2024-09-20T19:39:20.155211+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:39:20.155211+0000","last_undegraded":"2024-09-20T19:39:20.155211+0000","last_fullsized":"2024-09-20T19:39:20.155211+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.063854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":2043,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.150264+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:39:20.150264+0000","last_peered":"2024-09-20T19:39:20.150264+0000","last_clean":"2024-09-20T19:39:20.150264+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:39:20.150264+0000","last_undegraded":"2024-09-20T19:39:20.150264+0000","last_fullsized":"2024-09-20T19:39:20.150264+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.325845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"987'30","reported_seq":1851,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144277+0000","last_change":"2024-09-20T19:38:41.053008+0000","last_active":"2024-09-20T19:39:20.144277+0000","last_peered":"2024-09-20T19:39:20.144277+0000","last_clean":"2024-09-20T19:39:20.144277+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:39:20.144277+0000","last_undegraded":"2024-09-20T19:39:20.144277+0000","last_fullsized":"2024-09-20T19:39:20.144277+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:41.052945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.052945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.592511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":2301,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157641+0000","last_change":"2024-09-20T19:38:12.764542+0000","last_active":"2024-09-20T19:39:20.157641+0000","last_peered":"2024-09-20T19:39:20.157641+0000","last_clean":"2024-09-20T19:39:20.157641+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:39:20.157641+0000","last_undegraded":"2024-09-20T19:39:20.157641+0000","last_fullsized":"2024-09-20T19:39:20.157641+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:38:12.764391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.764391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.592596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"1047'1125","reported_seq":5962,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:21.556275+0000","last_change":"2024-09-20T19:39:21.556275+0000","last_active":"2024-09-20T19:39:21.556275+0000","last_peered":"2024-09-20T19:39:21.556275+0000","last_clean":"2024-09-20T19:39:21.556275+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:39:21.556275+0000","last_undegraded":"2024-09-20T19:39:21.556275+0000","last_fullsized":"2024-09-20T19:39:21.556275+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"1047'1125","last_scrub_stamp":"2024-09-20T19:39:21.556194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.556194+0000","objects_scrubbed":4,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.079860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016752500000000001,"stat_sum":{"num_bytes":11770346,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":145,"num_read_kb":111219,"num_write":87,"num_write_kb":32412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"6.1d","version":"0'0","reported_seq":2028,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.681400+0000","last_change":"2024-09-20T19:36:23.037110+0000","last_active":"2024-09-20T19:39:19.681400+0000","last_peered":"2024-09-20T19:39:19.681400+0000","last_clean":"2024-09-20T19:39:19.681400+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:39:19.681400+0000","last_undegraded":"2024-09-20T19:39:19.681400+0000","last_fullsized":"2024-09-20T19:39:19.681400+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.036982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.036982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.561071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":874,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.152354+0000","last_change":"2024-09-20T19:39:17.377907+0000","last_active":"2024-09-20T19:39:20.152354+0000","last_peered":"2024-09-20T19:39:20.152354+0000","last_clean":"2024-09-20T19:39:20.152354+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:39:20.152354+0000","last_undegraded":"2024-09-20T19:39:20.152354+0000","last_fullsized":"2024-09-20T19:39:20.152354+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:17.377850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.377850+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:25.442121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"987'27","reported_seq":2088,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157911+0000","last_change":"2024-09-20T19:38:15.768347+0000","last_active":"2024-09-20T19:39:20.157911+0000","last_peered":"2024-09-20T19:39:20.157911+0000","last_clean":"2024-09-20T19:39:20.157911+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:39:20.157911+0000","last_undegraded":"2024-09-20T19:39:20.157911+0000","last_fullsized":"2024-09-20T19:39:20.157911+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:38:15.768276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:15.768276+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.350115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1821,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.144442+0000","last_change":"2024-09-20T19:35:55.754020+0000","last_active":"2024-09-20T19:39:20.144442+0000","last_peered":"2024-09-20T19:39:20.144442+0000","last_clean":"2024-09-20T19:39:20.144442+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:39:20.144442+0000","last_undegraded":"2024-09-20T19:39:20.144442+0000","last_fullsized":"2024-09-20T19:39:20.144442+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:55.753897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:35:55.753897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"1040'740","reported_seq":5185,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157140+0000","last_change":"2024-09-20T19:39:18.538793+0000","last_active":"2024-09-20T19:39:20.157140+0000","last_peered":"2024-09-20T19:39:20.157140+0000","last_clean":"2024-09-20T19:39:20.157140+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:39:20.157140+0000","last_undegraded":"2024-09-20T19:39:20.157140+0000","last_fullsized":"2024-09-20T19:39:20.157140+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"1040'740","last_scrub_stamp":"2024-09-20T19:39:16.382726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.382726+0000","objects_scrubbed":2,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:45.765741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044108000000000003,"stat_sum":{"num_bytes":5950305,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":59453,"num_write":41,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":1849,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.500940+0000","last_change":"2024-09-20T19:36:27.046038+0000","last_active":"2024-09-20T19:39:23.500940+0000","last_peered":"2024-09-20T19:39:23.500940+0000","last_clean":"2024-09-20T19:39:23.500940+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:39:23.500940+0000","last_undegraded":"2024-09-20T19:39:23.500940+0000","last_fullsized":"2024-09-20T19:39:23.500940+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:27.045955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:27.045955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.803434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2044,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.157138+0000","last_change":"2024-09-20T19:39:13.318256+0000","last_active":"2024-09-20T19:39:20.157138+0000","last_peered":"2024-09-20T19:39:20.157138+0000","last_clean":"2024-09-20T19:39:20.157138+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:39:20.157138+0000","last_undegraded":"2024-09-20T19:39:20.157138+0000","last_fullsized":"2024-09-20T19:39:20.157138+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:13.318186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.318186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.633019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"987'39","reported_seq":657,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:21.055408+0000","last_change":"2024-09-20T19:39:21.055408+0000","last_active":"2024-09-20T19:39:21.055408+0000","last_peered":"2024-09-20T19:39:21.055408+0000","last_clean":"2024-09-20T19:39:21.055408+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:39:21.055408+0000","last_undegraded":"2024-09-20T19:39:21.055408+0000","last_fullsized":"2024-09-20T19:39:21.055408+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:21.055265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.055265+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.619827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":2047,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.682845+0000","last_change":"2024-09-20T19:38:33.621829+0000","last_active":"2024-09-20T19:39:19.682845+0000","last_peered":"2024-09-20T19:39:19.682845+0000","last_clean":"2024-09-20T19:39:19.682845+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:39:19.682845+0000","last_undegraded":"2024-09-20T19:39:19.682845+0000","last_fullsized":"2024-09-20T19:39:19.682845+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:33.621779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.621779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.572657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":4873,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683557+0000","last_change":"2024-09-20T19:39:18.520779+0000","last_active":"2024-09-20T19:39:19.683557+0000","last_peered":"2024-09-20T19:39:19.683557+0000","last_clean":"2024-09-20T19:39:19.683557+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:39:19.683557+0000","last_undegraded":"2024-09-20T19:39:19.683557+0000","last_fullsized":"2024-09-20T19:39:19.683557+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:38:28.709181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.709181+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:37.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076011000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b0","length":"1"},{"start":"1b2","length":"5"},{"start":"1b8","length":"1"},{"start":"1ba","length":"1"}]},{"pgid":"6.1f","version":"0'0","reported_seq":674,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.143674+0000","last_change":"2024-09-20T19:38:23.240112+0000","last_active":"2024-09-20T19:39:20.143674+0000","last_peered":"2024-09-20T19:39:20.143674+0000","last_clean":"2024-09-20T19:39:20.143674+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:39:20.143674+0000","last_undegraded":"2024-09-20T19:39:20.143674+0000","last_fullsized":"2024-09-20T19:39:20.143674+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:23.240001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.240001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.870584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1513,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:20.153765+0000","last_change":"2024-09-20T19:34:51.148618+0000","last_active":"2024-09-20T19:39:20.153765+0000","last_peered":"2024-09-20T19:39:20.153765+0000","last_clean":"2024-09-20T19:39:20.153765+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:39:20.153765+0000","last_undegraded":"2024-09-20T19:39:20.153765+0000","last_fullsized":"2024-09-20T19:39:20.153765+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.148494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.148494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"987'24","reported_seq":926,"reported_epoch":1051,"state":"active+clean","last_fresh":"2024-09-20T19:39:19.683356+0000","last_change":"2024-09-20T19:37:28.963853+0000","last_active":"2024-09-20T19:39:19.683356+0000","last_peered":"2024-09-20T19:39:19.683356+0000","last_clean":"2024-09-20T19:39:19.683356+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:39:19.683356+0000","last_undegraded":"2024-09-20T19:39:19.683356+0000","last_fullsized":"2024-09-20T19:39:19.683356+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:37:28.963655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:28.963655+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1862,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-09-20T19:39:23.498650+0000","last_change":"2024-09-20T19:38:20.716172+0000","last_active":"2024-09-20T19:39:23.498650+0000","last_peered":"2024-09-20T19:39:23.498650+0000","last_clean":"2024-09-20T19:39:23.498650+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:39:23.498650+0000","last_undegraded":"2024-09-20T19:39:23.498650+0000","last_fullsized":"2024-09-20T19:39:23.498650+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.716039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.716039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":131855404,"num_objects":54,"num_object_clones":14,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":2,"num_read":3555,"num_read_kb":2772580,"num_write":2479,"num_write_kb":934517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368209920,"data_stored":426625972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":27130,"ondisk_log_size":27130,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1256,"num_read_kb":1251,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907208,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132816,"kb_used_data":61368,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93612336,"statfs":{"total":95995035648,"available":95859032064,"internally_reserved":0,"allocated":62840832,"data_stored":67863562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234259,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107532,"kb_used_data":31920,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93637620,"statfs":{"total":95995035648,"available":95884922880,"internally_reserved":0,"allocated":32686080,"data_stored":35026632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561309,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96100,"kb_used_data":42708,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93649052,"statfs":{"total":95995035648,"available":95896629248,"internally_reserved":0,"allocated":43732992,"data_stored":49446873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928620,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89164,"kb_used_data":30776,"kb_used_omap":35,"kb_used_meta":58332,"kb_avail":93655988,"statfs":{"total":95995035648,"available":95903731712,"internally_reserved":0,"allocated":31514624,"data_stored":35423111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":59732913},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012902,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378945,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114604,"kb_used_data":39000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93630548,"statfs":{"total":95995035648,"available":95877681152,"internally_reserved":0,"allocated":39936000,"data_stored":41164246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120856,"kb_used_data":49416,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93624296,"statfs":{"total":95995035648,"available":95871279104,"internally_reserved":0,"allocated":50601984,"data_stored":52682179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648112,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103992,"kb_used_data":52836,"kb_used_omap":49,"kb_used_meta":51086,"kb_avail":93641160,"statfs":{"total":95995035648,"available":95888547840,"internally_reserved":0,"allocated":54104064,"data_stored":58708646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":52312617},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53010432,"data_stored":58085852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49508352,"data_stored":52058963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38572032,"data_stored":40278593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30416896,"data_stored":34794053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":42053632,"data_stored":48233440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":33813276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61427712,"data_stored":66913464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:39:29.332 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T19:39:29.332 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T19:39:29.332 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-20T19:39:29.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-20T19:39:29.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:29.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:29.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:29.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:29.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:29.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:29.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.900 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.900 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.900 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.900 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:29.900 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:29.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:29.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.385666+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.387968+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.504557+0000 mgr.y (mgr.24491) 805 : cluster [DBG] pgmap v1581: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 464 KiB/s wr, 6 op/s 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:28.731390+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:28.733180+0000 mon.a (mon.0) 2728 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.950421+0000 osd.1 (osd.1) 373 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:28.951885+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: audit 2024-09-20T19:39:29.085888+0000 mon.a (mon.0) 2729 : audit [DBG] from='client.? v1:172.21.15.42:0/4113927247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.376641+0000 osd.1 (osd.1) 375 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:29.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:29 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.377630+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:30.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:30.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:39:29 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:30.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:30.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:30.245 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:39:30.245 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":1054,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:39:29.726947+0000","last_up_change":"2024-09-20T19:30:58.350676+0000","last_in_change":"2024-09-20T19:33:42.415728+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"863","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"259","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9000000953674316,"score_stable":2.9000000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.5499999523162842,"raw_score_stable":2.5499999523162842,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-20T19:18:05.791948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.4900000095367432,"score_stable":1.4900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-20T19:18:06.729842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6700000762939453,"score_stable":2.6700000762939453,"optimal_score":0.75,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T19:18:07.353972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.2400000095367432,"score_stable":2.2400000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.9700000286102295,"raw_score_stable":1.9700000286102295,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T19:18:09.310321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.4900000095367432,"score_stable":1.4900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T19:18:11.438339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":7,"pool_name":"unique_pool_0","create_time":"2024-09-20T19:21:28.700742+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1051","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"698","auid":0,"snap_mode":"selfmanaged","snap_seq":443,"snap_epoch":1051,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5499999523162842,"score_stable":1.5499999523162842,"optimal_score":0.87999999523162842,"raw_score_acting":1.3600000143051147,"raw_score_stable":1.3600000143051147,"primary_affinity_weighted":0.98441123962402344,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":393,"up_from":500,"up_thru":789,"down_at":394,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":701330554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":701330554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":701330554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":701330554}]},"public_addr":"172.21.15.42:6801/701330554","cluster_addr":"172.21.15.42:6802/701330554","heartbeat_back_addr":"172.21.15.42:6804/701330554","heartbeat_front_addr":"172.21.15.42:6803/701330554","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":757,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":757,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":28,"last_clean_end":479,"up_from":550,"up_thru":704,"down_at":479,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1278675080}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1278675080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1278675080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1278675080}]},"public_addr":"172.21.15.42:6813/1278675080","cluster_addr":"172.21.15.42:6814/1278675080","heartbeat_back_addr":"172.21.15.42:6816/1278675080","heartbeat_front_addr":"172.21.15.42:6815/1278675080","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":443,"up_from":522,"up_thru":757,"down_at":444,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":449450686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":449450686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":449450686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":449450686}]},"public_addr":"172.21.15.162:6800/449450686","cluster_addr":"172.21.15.162:6801/449450686","heartbeat_back_addr":"172.21.15.162:6803/449450686","heartbeat_front_addr":"172.21.15.162:6802/449450686","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":760,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":0.8752899169921875,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":757,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":825,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:38:22.006453+0000","dead_epoch":393},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:29.377634+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:03.657474+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-20T19:38:09.712715+0000","dead_epoch":478},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:38:02.642441+0000","dead_epoch":443},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:38:21.062676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:38:33.637356+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:38:58.733007+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.8","osds":[2,3,1]}],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"2.10","mappings":[{"from":2,"to":0}]},{"pgid":"2.1f","mappings":[{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.14","mappings":[{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":6,"to":0}]},{"pgid":"7.e","mappings":[{"from":2,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/1743761075":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/3764180564":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3982292550":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/2784207196":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3505349280":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/368716713":"2024-09-21T19:18:55.853462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:39:30.268 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:30.287 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:30.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:30.417 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:30.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:30.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:30.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:30.668 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:30.761 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is set 2024-09-20T19:39:30.771 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:30.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:30.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:30.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:30.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.789 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:30.790 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:30.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:30.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:30.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:30.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:30.791 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:30 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:30.899 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:30.923 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:30.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:30.943 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:39:31.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.736372+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:29.837594+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.42:0/2815167349' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:29.838180+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26183 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.927414+0000 osd.7 (osd.7) 255 : cluster [DBG] 7.11 scrub starts 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:31.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.929458+0000 osd.7 (osd.7) 256 : cluster [DBG] 7.11 scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.962878+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:29.964646+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.227563+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.229471+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: audit 2024-09-20T19:39:30.241565+0000 mon.a (mon.0) 2732 : audit [DBG] from='client.? v1:172.21.15.42:0/3140366161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.683599+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:30 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.684590+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:31.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:31.183 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:31.286 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:31.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:31.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:31.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:31.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:31.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:31 smithi162 bash[24557]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:31.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:31.943 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[22706]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: audit 2024-09-20T19:39:29.983806+0000 mgr.y (mgr.24491) 806 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.416029+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.418163+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:39:31.945 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.505266+0000 mgr.y (mgr.24491) 807 : cluster [DBG] pgmap v1583: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 729 B/s rd, 0 op/s 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.742676+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: audit 2024-09-20T19:39:30.749997+0000 mon.a (mon.0) 2734 : audit [INF] from='client.26183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.751936+0000 osd.4 (osd.4) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.753798+0000 osd.4 (osd.4) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.946 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:31 smithi042 bash[32075]: cluster 2024-09-20T19:39:30.755503+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2024-09-20T19:39:31.960 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:31.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:32.063 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:32.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:32.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:32.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:32.353 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:32.455 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:32.485 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:32.588 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.627 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:32.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:32.687 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:39:32.687 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:39:32.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:32.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:32.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:32.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:32.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:32.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:32.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:32.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[22706]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:32.809 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T19:39:32.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:32.877 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:39:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:32.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:32.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:33.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:32 smithi162 bash[24557]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:33.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:33.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:33.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.240162+0000 osd.5 (osd.5) 339 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:33.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:33.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.241740+0000 osd.5 (osd.5) 340 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:33.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:33.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:32 smithi042 bash[32075]: cluster 2024-09-20T19:39:31.765157+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2024-09-20T19:39:33.244 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:33.272 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:33.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:33.397 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:33.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T19:39:33.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:33.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:33.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:33.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:33.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:33.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:34.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:34.031 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:33 smithi162 bash[24557]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.031 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:33 smithi162 bash[24557]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.031 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:33 smithi162 bash[24557]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.031 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:33 smithi162 bash[24557]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:34.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:34.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[32075]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[32075]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[32075]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[32075]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[22706]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[22706]: cluster 2024-09-20T19:39:32.506414+0000 mgr.y (mgr.24491) 808 : cluster [DBG] pgmap v1586: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[22706]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:33 smithi042 bash[22706]: cluster 2024-09-20T19:39:33.502953+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2024-09-20T19:39:34.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.304 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:34.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:34.434 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:34.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:34.565 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:34.667 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:34.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:34.883 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T19:39:34.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:34 smithi042 bash[32075]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:34.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:34 smithi042 bash[32075]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:34.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:34 smithi042 bash[22706]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:34.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:34 smithi042 bash[22706]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:34.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:34.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:34.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:34 smithi162 bash[24557]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:34 smithi162 bash[24557]: cluster 2024-09-20T19:39:34.507390+0000 mgr.y (mgr.24491) 809 : cluster [DBG] pgmap v1588: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:39:35.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.088 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:35.191 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:35.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:35.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:35.395 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:35.447 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:35.468 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:35.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:35.598 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:35.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:35.730 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:35.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:35.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.991 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:35.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:36.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:36.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:36.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:36.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:36.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:36.381 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:36.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:36.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:36.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:36.626 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:36.641 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:36.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:36.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:36.900 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:36.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:37.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:37.031 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:37.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:37.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:37.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:37.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:37.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:37.419 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:37.421 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:37.423 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:37.423 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:37.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:37.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:37.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:37.677 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:37.714 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:37 smithi042 bash[22706]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:37.714 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:37 smithi042 bash[22706]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:37.780 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:37.806 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:37 smithi162 bash[24557]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:37.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:37 smithi162 bash[24557]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:37.809 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:37.913 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:37.942 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:37.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:38.045 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:38.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:37 smithi042 bash[32075]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:38.057 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:37 smithi042 bash[32075]: cluster 2024-09-20T19:39:36.508181+0000 mgr.y (mgr.24491) 810 : cluster [DBG] pgmap v1589: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:39:38.073 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:38.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:38.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:38.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:38.464 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:38.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:38.591 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:38.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:38.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:38.824 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:38.853 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:38.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:38.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:38.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:39.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-20T19:39:39.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:39.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:39.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:39.245 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:39.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:39.375 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:39.460 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:39.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:39.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:39.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:39.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:39.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:39 smithi162 bash[24557]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:39 smithi162 bash[24557]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.814 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:39 smithi042 bash[32075]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.814 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:39 smithi042 bash[32075]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:39 smithi042 bash[22706]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:39 smithi042 bash[22706]: cluster 2024-09-20T19:39:38.509634+0000 mgr.y (mgr.24491) 811 : cluster [DBG] pgmap v1590: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:39:39.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:39.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:39.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:40.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:40.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:40.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:40.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:39:39 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:40.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:40.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:40.438 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:40.565 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:40.593 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:40.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:40.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:40.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:40.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:40.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:40.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:40.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:41.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:41.217 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:41.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:41.378 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:41.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:41.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:41.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:41.664 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:39:41.664 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:39:41.682 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:39:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:39:41.699 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:39:41.700 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:39:41.717 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:39:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:39:41.738 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:39:41.738 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:39:41.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:41 smithi162 bash[24557]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:41 smithi162 bash[24557]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:41 smithi162 bash[24557]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.752 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:41 smithi162 bash[24557]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.758 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:39:41.759 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:39:41.767 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:41.776 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:39:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:39:41.797 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:39:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[32075]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[32075]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[32075]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[32075]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[22706]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[22706]: audit 2024-09-20T19:39:39.992132+0000 mgr.y (mgr.24491) 812 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[22706]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:41 smithi042 bash[22706]: cluster 2024-09-20T19:39:40.510502+0000 mgr.y (mgr.24491) 813 : cluster [DBG] pgmap v1591: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:41.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:41.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:42.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:42.132 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:42.224 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-20T19:39:42.224 INFO:teuthology.orchestra.run.smithi042.stdout: "ops": [], 2024-09-20T19:39:42.224 INFO:teuthology.orchestra.run.smithi042.stdout: "num_ops": 0 2024-09-20T19:39:42.224 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-20T19:39:42.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:42.282 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:42.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:42.413 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:42.516 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:42.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:42 smithi042 bash[22706]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.591 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:42 smithi042 bash[22706]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:42 smithi042 bash[32075]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.591 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:42 smithi042 bash[32075]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:42.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:42.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:42.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:42 smithi162 bash[24557]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:42 smithi162 bash[24557]: audit 2024-09-20T19:39:41.644947+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.42:0/3223423018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:42.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:42.896 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:39:42.899 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:39:42.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:42.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:43.043 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.068 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:43.170 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:43.196 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:43.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:43.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:43.430 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is set 2024-09-20T19:39:43.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:43.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[32075]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[32075]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[32075]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[32075]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[22706]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[22706]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[22706]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:43 smithi042 bash[22706]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:43.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:43 smithi162 bash[24557]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:43 smithi162 bash[24557]: cluster 2024-09-20T19:39:42.512007+0000 mgr.y (mgr.24491) 814 : cluster [DBG] pgmap v1592: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:43.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:43 smithi162 bash[24557]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:43 smithi162 bash[24557]: audit 2024-09-20T19:39:42.989684+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:39:43.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.840 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:43.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.971 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:43.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:44.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.101 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:44.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:44.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:44.460 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:44.593 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.621 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:44.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:44.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:44 smithi162 bash[24557]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:44.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:44.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.889 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[32075]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.890 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:42.882271+0000 mgr.y (mgr.24491) 815 : audit [DBG] from='client.26207 v1:172.21.15.42:0/4053737979' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.001920+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.002990+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.529689+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.537438+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? v1:172.21.15.42:0/493705038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: cluster 2024-09-20T19:39:43.542483+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.761337+0000 mon.a (mon.0) 2742 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:39:44.891 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.892 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:44 smithi042 bash[22706]: audit 2024-09-20T19:39:43.762698+0000 mon.a (mon.0) 2743 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:44.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:45.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:45.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:45.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:45.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:45.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:45.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:45.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:45.538 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:45.564 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T19:39:45.642 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:45.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:45.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:45.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.939 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:45.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:46.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:45 smithi162 bash[24557]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:45 smithi162 bash[24557]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.069 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:46.161 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T19:39:46.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:45 smithi042 bash[32075]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.174 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:45 smithi042 bash[32075]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.174 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:45 smithi042 bash[22706]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.174 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:45 smithi042 bash[22706]: cluster 2024-09-20T19:39:44.512939+0000 mgr.y (mgr.24491) 816 : cluster [DBG] pgmap v1594: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:46.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:46.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:46.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:46.360 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1592,"stamp":"2024-09-20T19:39:42.510749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132451911,"num_objects":253,"num_object_clones":14,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":2,"num_read":6376,"num_read_kb":2775153,"num_write":3487,"num_write_kb":934562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":122190271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29098,"ondisk_log_size":29098,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":765568,"kb_used_data":308516,"kb_used_omap":92,"kb_used_meta":456803,"kb_avail":655450496,"statfs":{"total":671965249536,"available":671181307904,"internally_reserved":0,"allocated":315920384,"data_stored":340610850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94535,"internal_metadata":467766969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005157"},"pg_stats":[{"pgid":"7.23","version":"697'409","reported_seq":4822,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092235+0000","last_change":"2024-09-20T19:39:18.538563+0000","last_active":"2024-09-20T19:39:34.092235+0000","last_peered":"2024-09-20T19:39:34.092235+0000","last_clean":"2024-09-20T19:39:34.092235+0000","last_became_active":"2024-09-20T19:33:46.372997+0000","last_became_peered":"2024-09-20T19:33:46.372997+0000","last_unstale":"2024-09-20T19:39:34.092235+0000","last_undegraded":"2024-09-20T19:39:34.092235+0000","last_fullsized":"2024-09-20T19:39:34.092235+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"697'409","last_scrub_stamp":"2024-09-20T19:38:17.790503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.790503+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0057066840000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"995'368","reported_seq":4626,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506486+0000","last_change":"2024-09-20T19:39:27.944329+0000","last_active":"2024-09-20T19:39:33.506486+0000","last_peered":"2024-09-20T19:39:33.506486+0000","last_clean":"2024-09-20T19:39:33.506486+0000","last_became_active":"2024-09-20T19:33:33.351048+0000","last_became_peered":"2024-09-20T19:33:33.351048+0000","last_unstale":"2024-09-20T19:39:33.506486+0000","last_undegraded":"2024-09-20T19:39:33.506486+0000","last_fullsized":"2024-09-20T19:39:33.506486+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"995'368","last_scrub_stamp":"2024-09-20T19:39:27.944212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:27.944212+0000","objects_scrubbed":1,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.806876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090287879999999994,"stat_sum":{"num_bytes":3344890,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":40573,"num_write":37,"num_write_kb":14578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"1050'849","reported_seq":5589,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712525+0000","last_change":"2024-09-20T19:39:22.542022+0000","last_active":"2024-09-20T19:39:34.712525+0000","last_peered":"2024-09-20T19:39:34.712525+0000","last_clean":"2024-09-20T19:39:34.712525+0000","last_became_active":"2024-09-20T19:33:46.372311+0000","last_became_peered":"2024-09-20T19:33:46.372311+0000","last_unstale":"2024-09-20T19:39:34.712525+0000","last_undegraded":"2024-09-20T19:39:34.712525+0000","last_fullsized":"2024-09-20T19:39:34.712525+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"1050'849","last_scrub_stamp":"2024-09-20T19:39:22.541865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.541865+0000","objects_scrubbed":4,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.932519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.29602658100000001,"stat_sum":{"num_bytes":9237240,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":145,"num_read_kb":155075,"num_write":64,"num_write_kb":35558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.20","version":"1014'991","reported_seq":5630,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.500242+0000","last_change":"2024-09-20T19:39:18.554134+0000","last_active":"2024-09-20T19:39:33.500242+0000","last_peered":"2024-09-20T19:39:33.500242+0000","last_clean":"2024-09-20T19:39:33.500242+0000","last_became_active":"2024-09-20T19:33:33.349019+0000","last_became_peered":"2024-09-20T19:33:33.349019+0000","last_unstale":"2024-09-20T19:39:33.500242+0000","last_undegraded":"2024-09-20T19:39:33.500242+0000","last_fullsized":"2024-09-20T19:39:33.500242+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'982","last_scrub_stamp":"2024-09-20T19:38:13.700013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:13.700013+0000","objects_scrubbed":1,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:31.185044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077880600000000005,"stat_sum":{"num_bytes":3815486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":67428,"num_write":32,"num_write_kb":14158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1a","version":"1050'798","reported_seq":5200,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506902+0000","last_change":"2024-09-20T19:39:21.003139+0000","last_active":"2024-09-20T19:39:33.506902+0000","last_peered":"2024-09-20T19:39:33.506902+0000","last_clean":"2024-09-20T19:39:33.506902+0000","last_became_active":"2024-09-20T19:33:33.354499+0000","last_became_peered":"2024-09-20T19:33:33.354499+0000","last_unstale":"2024-09-20T19:39:33.506902+0000","last_undegraded":"2024-09-20T19:39:33.506902+0000","last_fullsized":"2024-09-20T19:39:33.506902+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":5,"last_scrub":"1050'798","last_scrub_stamp":"2024-09-20T19:39:21.003036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.003036+0000","objects_scrubbed":3,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.152836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0091316239999999996,"stat_sum":{"num_bytes":4908304,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":116514,"num_write":96,"num_write_kb":37050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1519,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.517820+0000","last_change":"2024-09-20T19:36:31.860620+0000","last_active":"2024-09-20T19:39:33.517820+0000","last_peered":"2024-09-20T19:39:33.517820+0000","last_clean":"2024-09-20T19:39:33.517820+0000","last_became_active":"2024-09-20T19:33:43.387021+0000","last_became_peered":"2024-09-20T19:33:43.387021+0000","last_unstale":"2024-09-20T19:39:33.517820+0000","last_undegraded":"2024-09-20T19:39:33.517820+0000","last_fullsized":"2024-09-20T19:39:33.517820+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.860530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.860530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1839,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747765+0000","last_change":"2024-09-20T19:37:25.135773+0000","last_active":"2024-09-20T19:39:34.747765+0000","last_peered":"2024-09-20T19:39:34.747765+0000","last_clean":"2024-09-20T19:39:34.747765+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:39:34.747765+0000","last_undegraded":"2024-09-20T19:39:34.747765+0000","last_fullsized":"2024-09-20T19:39:34.747765+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:25.135646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:25.135646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"987'39","reported_seq":1787,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501004+0000","last_change":"2024-09-20T19:39:22.057603+0000","last_active":"2024-09-20T19:39:33.501004+0000","last_peered":"2024-09-20T19:39:33.501004+0000","last_clean":"2024-09-20T19:39:33.501004+0000","last_became_active":"2024-09-20T19:33:43.364402+0000","last_became_peered":"2024-09-20T19:33:43.364402+0000","last_unstale":"2024-09-20T19:39:33.501004+0000","last_undegraded":"2024-09-20T19:39:33.501004+0000","last_fullsized":"2024-09-20T19:39:33.501004+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:22.057437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.057437+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.474404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1817,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503319+0000","last_change":"2024-09-20T19:38:28.745755+0000","last_active":"2024-09-20T19:39:33.503319+0000","last_peered":"2024-09-20T19:39:33.503319+0000","last_clean":"2024-09-20T19:39:33.503319+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:39:33.503319+0000","last_undegraded":"2024-09-20T19:39:33.503319+0000","last_fullsized":"2024-09-20T19:39:33.503319+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:28.745668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.745668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.949968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"695'481","reported_seq":5099,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746895+0000","last_change":"2024-09-20T19:39:18.608387+0000","last_active":"2024-09-20T19:39:34.746895+0000","last_peered":"2024-09-20T19:39:34.746895+0000","last_clean":"2024-09-20T19:39:34.746895+0000","last_became_active":"2024-09-20T19:33:46.384543+0000","last_became_peered":"2024-09-20T19:33:46.384543+0000","last_unstale":"2024-09-20T19:39:34.746895+0000","last_undegraded":"2024-09-20T19:39:34.746895+0000","last_fullsized":"2024-09-20T19:39:34.746895+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'481","last_scrub_stamp":"2024-09-20T19:38:14.315644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.315644+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.091726462999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1574,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747509+0000","last_change":"2024-09-20T19:37:21.156138+0000","last_active":"2024-09-20T19:39:34.747509+0000","last_peered":"2024-09-20T19:39:34.747509+0000","last_clean":"2024-09-20T19:39:34.747509+0000","last_became_active":"2024-09-20T19:33:43.367493+0000","last_became_peered":"2024-09-20T19:33:43.367493+0000","last_unstale":"2024-09-20T19:39:34.747509+0000","last_undegraded":"2024-09-20T19:39:34.747509+0000","last_fullsized":"2024-09-20T19:39:34.747509+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:21.156013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:21.156013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"987'21","reported_seq":1930,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503579+0000","last_change":"2024-09-20T19:36:44.033709+0000","last_active":"2024-09-20T19:39:33.503579+0000","last_peered":"2024-09-20T19:39:33.503579+0000","last_clean":"2024-09-20T19:39:33.503579+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:39:33.503579+0000","last_undegraded":"2024-09-20T19:39:33.503579+0000","last_fullsized":"2024-09-20T19:39:33.503579+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:36:44.033568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:44.033568+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.776685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":2057,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501588+0000","last_change":"2024-09-20T19:39:20.933373+0000","last_active":"2024-09-20T19:39:33.501588+0000","last_peered":"2024-09-20T19:39:33.501588+0000","last_clean":"2024-09-20T19:39:33.501588+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:39:33.501588+0000","last_undegraded":"2024-09-20T19:39:33.501588+0000","last_fullsized":"2024-09-20T19:39:33.501588+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:20.933250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.933250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.308194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1879,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.507094+0000","last_change":"2024-09-20T19:38:27.709303+0000","last_active":"2024-09-20T19:39:33.507094+0000","last_peered":"2024-09-20T19:39:33.507094+0000","last_clean":"2024-09-20T19:39:33.507094+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:39:33.507094+0000","last_undegraded":"2024-09-20T19:39:33.507094+0000","last_fullsized":"2024-09-20T19:39:33.507094+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:27.709247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.709247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.024307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"1030'1284","reported_seq":5879,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.745806+0000","last_change":"2024-09-20T19:39:18.608130+0000","last_active":"2024-09-20T19:39:34.745806+0000","last_peered":"2024-09-20T19:39:34.745806+0000","last_clean":"2024-09-20T19:39:34.745806+0000","last_became_active":"2024-09-20T19:30:12.797520+0000","last_became_peered":"2024-09-20T19:30:12.797520+0000","last_unstale":"2024-09-20T19:39:34.745806+0000","last_undegraded":"2024-09-20T19:39:34.745806+0000","last_fullsized":"2024-09-20T19:39:34.745806+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"895'1218","last_scrub_stamp":"2024-09-20T19:36:46.063473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:46.063473+0000","objects_scrubbed":6,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:33.449275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.092706255000000001,"stat_sum":{"num_bytes":5866270,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":128956,"num_write":108,"num_write_kb":39081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":15476484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":2047,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518895+0000","last_change":"2024-09-20T19:38:40.080992+0000","last_active":"2024-09-20T19:39:33.518895+0000","last_peered":"2024-09-20T19:39:33.518895+0000","last_clean":"2024-09-20T19:39:33.518895+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:39:33.518895+0000","last_undegraded":"2024-09-20T19:39:33.518895+0000","last_fullsized":"2024-09-20T19:39:33.518895+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.080865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.080865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.792937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"987'9","reported_seq":1935,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505806+0000","last_change":"2024-09-20T19:39:29.964713+0000","last_active":"2024-09-20T19:39:33.505806+0000","last_peered":"2024-09-20T19:39:33.505806+0000","last_clean":"2024-09-20T19:39:33.505806+0000","last_became_active":"2024-09-20T19:34:26.408346+0000","last_became_peered":"2024-09-20T19:34:26.408346+0000","last_unstale":"2024-09-20T19:39:33.505806+0000","last_undegraded":"2024-09-20T19:39:33.505806+0000","last_fullsized":"2024-09-20T19:39:33.505806+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'9","last_scrub_stamp":"2024-09-20T19:39:29.964651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:29.964651+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:47.290343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":2039,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520774+0000","last_change":"2024-09-20T19:39:21.057305+0000","last_active":"2024-09-20T19:39:33.520774+0000","last_peered":"2024-09-20T19:39:33.520774+0000","last_clean":"2024-09-20T19:39:33.520774+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:39:33.520774+0000","last_undegraded":"2024-09-20T19:39:33.520774+0000","last_fullsized":"2024-09-20T19:39:33.520774+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:21.057247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.057247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.649619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2050,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713617+0000","last_change":"2024-09-20T19:36:22.545289+0000","last_active":"2024-09-20T19:39:34.713617+0000","last_peered":"2024-09-20T19:39:34.713617+0000","last_clean":"2024-09-20T19:39:34.713617+0000","last_became_active":"2024-09-20T19:33:43.445966+0000","last_became_peered":"2024-09-20T19:33:43.445966+0000","last_unstale":"2024-09-20T19:39:34.713617+0000","last_undegraded":"2024-09-20T19:39:34.713617+0000","last_fullsized":"2024-09-20T19:39:34.713617+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:22.545231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:22.545231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.331104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"950'482","reported_seq":5129,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091001+0000","last_change":"2024-09-20T19:39:23.315521+0000","last_active":"2024-09-20T19:39:34.091001+0000","last_peered":"2024-09-20T19:39:34.091001+0000","last_clean":"2024-09-20T19:39:34.091001+0000","last_became_active":"2024-09-20T19:28:48.220955+0000","last_became_peered":"2024-09-20T19:28:48.220955+0000","last_unstale":"2024-09-20T19:39:34.091001+0000","last_undegraded":"2024-09-20T19:39:34.091001+0000","last_fullsized":"2024-09-20T19:39:34.091001+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"950'482","last_scrub_stamp":"2024-09-20T19:39:23.315340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:23.315340+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:34.103109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049883319999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":55935,"num_write":41,"num_write_kb":15366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2036,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.519071+0000","last_change":"2024-09-20T19:38:26.111416+0000","last_active":"2024-09-20T19:39:33.519071+0000","last_peered":"2024-09-20T19:39:33.519071+0000","last_clean":"2024-09-20T19:39:33.519071+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:39:33.519071+0000","last_undegraded":"2024-09-20T19:39:33.519071+0000","last_fullsized":"2024-09-20T19:39:33.519071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:26.111300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.111300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:33.731393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"987'21","reported_seq":1966,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506500+0000","last_change":"2024-09-20T19:38:30.744236+0000","last_active":"2024-09-20T19:39:33.506500+0000","last_peered":"2024-09-20T19:39:33.506500+0000","last_clean":"2024-09-20T19:39:33.506500+0000","last_became_active":"2024-09-20T19:34:26.408282+0000","last_became_peered":"2024-09-20T19:34:26.408282+0000","last_unstale":"2024-09-20T19:39:33.506500+0000","last_undegraded":"2024-09-20T19:39:33.506500+0000","last_fullsized":"2024-09-20T19:39:33.506500+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.744141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.744141+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.937516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2031,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713681+0000","last_change":"2024-09-20T19:36:23.582321+0000","last_active":"2024-09-20T19:39:34.713681+0000","last_peered":"2024-09-20T19:39:34.713681+0000","last_clean":"2024-09-20T19:39:34.713681+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:39:34.713681+0000","last_undegraded":"2024-09-20T19:39:34.713681+0000","last_fullsized":"2024-09-20T19:39:34.713681+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.582265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.377154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1008,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.745660+0000","last_change":"2024-09-20T19:37:24.118390+0000","last_active":"2024-09-20T19:39:34.745660+0000","last_peered":"2024-09-20T19:39:34.745660+0000","last_clean":"2024-09-20T19:39:34.745660+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:39:34.745660+0000","last_undegraded":"2024-09-20T19:39:34.745660+0000","last_fullsized":"2024-09-20T19:39:34.745660+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:24.118312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:24.118312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1676,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712476+0000","last_change":"2024-09-20T19:39:27.527926+0000","last_active":"2024-09-20T19:39:34.712476+0000","last_peered":"2024-09-20T19:39:34.712476+0000","last_clean":"2024-09-20T19:39:34.712476+0000","last_became_active":"2024-09-20T19:33:43.370688+0000","last_became_peered":"2024-09-20T19:33:43.370688+0000","last_unstale":"2024-09-20T19:39:34.712476+0000","last_undegraded":"2024-09-20T19:39:34.712476+0000","last_fullsized":"2024-09-20T19:39:34.712476+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:27.527753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:27.527753+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.410303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1021'427","reported_seq":5085,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504994+0000","last_change":"2024-09-20T19:39:18.933022+0000","last_active":"2024-09-20T19:39:33.504994+0000","last_peered":"2024-09-20T19:39:33.504994+0000","last_clean":"2024-09-20T19:39:33.504994+0000","last_became_active":"2024-09-20T19:34:00.374259+0000","last_became_peered":"2024-09-20T19:34:00.374259+0000","last_unstale":"2024-09-20T19:39:33.504994+0000","last_undegraded":"2024-09-20T19:39:33.504994+0000","last_fullsized":"2024-09-20T19:39:33.504994+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"1021'427","last_scrub_stamp":"2024-09-20T19:39:18.932933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:18.932933+0000","objects_scrubbed":1,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.120268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038900900000000001,"stat_sum":{"num_bytes":2235661,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":27202,"num_write":27,"num_write_kb":10018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"987'45","reported_seq":1921,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504879+0000","last_change":"2024-09-20T19:38:14.828944+0000","last_active":"2024-09-20T19:39:33.504879+0000","last_peered":"2024-09-20T19:39:33.504879+0000","last_clean":"2024-09-20T19:39:33.504879+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:39:33.504879+0000","last_undegraded":"2024-09-20T19:39:33.504879+0000","last_fullsized":"2024-09-20T19:39:33.504879+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'31","last_scrub_stamp":"2024-09-20T19:38:14.828850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.828850+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.912036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2040,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518423+0000","last_change":"2024-09-20T19:39:17.990988+0000","last_active":"2024-09-20T19:39:33.518423+0000","last_peered":"2024-09-20T19:39:33.518423+0000","last_clean":"2024-09-20T19:39:33.518423+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:39:33.518423+0000","last_undegraded":"2024-09-20T19:39:33.518423+0000","last_fullsized":"2024-09-20T19:39:33.518423+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:17.990944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.990944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.758835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":2060,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504869+0000","last_change":"2024-09-20T19:38:38.590561+0000","last_active":"2024-09-20T19:39:33.504869+0000","last_peered":"2024-09-20T19:39:33.504869+0000","last_clean":"2024-09-20T19:39:33.504869+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:39:33.504869+0000","last_undegraded":"2024-09-20T19:39:33.504869+0000","last_fullsized":"2024-09-20T19:39:33.504869+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:38.590498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:38.590498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.175561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":628,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503941+0000","last_change":"2024-09-20T19:39:24.143772+0000","last_active":"2024-09-20T19:39:33.503941+0000","last_peered":"2024-09-20T19:39:33.503941+0000","last_clean":"2024-09-20T19:39:33.503941+0000","last_became_active":"2024-09-20T19:34:26.401646+0000","last_became_peered":"2024-09-20T19:34:26.401646+0000","last_unstale":"2024-09-20T19:39:33.503941+0000","last_undegraded":"2024-09-20T19:39:33.503941+0000","last_fullsized":"2024-09-20T19:39:33.503941+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:24.143702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:24.143702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:40.453303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":2126,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520486+0000","last_change":"2024-09-20T19:34:26.405887+0000","last_active":"2024-09-20T19:39:33.520486+0000","last_peered":"2024-09-20T19:39:33.520486+0000","last_clean":"2024-09-20T19:39:33.520486+0000","last_became_active":"2024-09-20T19:34:26.405232+0000","last_became_peered":"2024-09-20T19:34:26.405232+0000","last_unstale":"2024-09-20T19:39:33.520486+0000","last_undegraded":"2024-09-20T19:39:33.520486+0000","last_fullsized":"2024-09-20T19:39:33.520486+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:33:02.955367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.955367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:45.070277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1046'711","reported_seq":5133,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747629+0000","last_change":"2024-09-20T19:39:18.609861+0000","last_active":"2024-09-20T19:39:34.747629+0000","last_peered":"2024-09-20T19:39:34.747629+0000","last_clean":"2024-09-20T19:39:34.747629+0000","last_became_active":"2024-09-20T19:33:46.373812+0000","last_became_peered":"2024-09-20T19:33:46.373812+0000","last_unstale":"2024-09-20T19:39:34.747629+0000","last_undegraded":"2024-09-20T19:39:34.747629+0000","last_fullsized":"2024-09-20T19:39:34.747629+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'687","last_scrub_stamp":"2024-09-20T19:38:24.221337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.221337+0000","objects_scrubbed":1,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.197485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.092474009999999995,"stat_sum":{"num_bytes":2622222,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":35389,"num_write":68,"num_write_kb":28356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"987'30","reported_seq":1966,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505043+0000","last_change":"2024-09-20T19:38:29.725721+0000","last_active":"2024-09-20T19:39:33.505043+0000","last_peered":"2024-09-20T19:39:33.505043+0000","last_clean":"2024-09-20T19:39:33.505043+0000","last_became_active":"2024-09-20T19:33:43.442232+0000","last_became_peered":"2024-09-20T19:33:43.442232+0000","last_unstale":"2024-09-20T19:39:33.505043+0000","last_undegraded":"2024-09-20T19:39:33.505043+0000","last_fullsized":"2024-09-20T19:39:33.505043+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:29.725634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:29.725634+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.585770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":2076,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503120+0000","last_change":"2024-09-20T19:38:18.676904+0000","last_active":"2024-09-20T19:39:33.503120+0000","last_peered":"2024-09-20T19:39:33.503120+0000","last_clean":"2024-09-20T19:39:33.503120+0000","last_became_active":"2024-09-20T19:33:43.445324+0000","last_became_peered":"2024-09-20T19:33:43.445324+0000","last_unstale":"2024-09-20T19:39:33.503120+0000","last_undegraded":"2024-09-20T19:39:33.503120+0000","last_fullsized":"2024-09-20T19:39:33.503120+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:18.676841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:18.676841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":2328,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504980+0000","last_change":"2024-09-20T19:39:06.910897+0000","last_active":"2024-09-20T19:39:33.504980+0000","last_peered":"2024-09-20T19:39:33.504980+0000","last_clean":"2024-09-20T19:39:33.504980+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:39:33.504980+0000","last_undegraded":"2024-09-20T19:39:33.504980+0000","last_fullsized":"2024-09-20T19:39:33.504980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:39:06.910817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:06.910817+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.382750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1890,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505627+0000","last_change":"2024-09-20T19:39:05.797046+0000","last_active":"2024-09-20T19:39:33.505627+0000","last_peered":"2024-09-20T19:39:33.505627+0000","last_clean":"2024-09-20T19:39:33.505627+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:39:33.505627+0000","last_undegraded":"2024-09-20T19:39:33.505627+0000","last_fullsized":"2024-09-20T19:39:33.505627+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.796915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.796915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:13.456865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1639,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712789+0000","last_change":"2024-09-20T19:38:17.061687+0000","last_active":"2024-09-20T19:39:34.712789+0000","last_peered":"2024-09-20T19:39:34.712789+0000","last_clean":"2024-09-20T19:39:34.712789+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:39:34.712789+0000","last_undegraded":"2024-09-20T19:39:34.712789+0000","last_fullsized":"2024-09-20T19:39:34.712789+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:17.061616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.061616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.609939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1041'961","reported_seq":5699,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518308+0000","last_change":"2024-09-20T19:39:18.525089+0000","last_active":"2024-09-20T19:39:33.518308+0000","last_peered":"2024-09-20T19:39:33.518308+0000","last_clean":"2024-09-20T19:39:33.518308+0000","last_became_active":"2024-09-20T19:35:34.395504+0000","last_became_peered":"2024-09-20T19:35:34.395504+0000","last_unstale":"2024-09-20T19:39:33.518308+0000","last_undegraded":"2024-09-20T19:39:33.518308+0000","last_fullsized":"2024-09-20T19:39:33.518308+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":826,"parent":"0.0","parent_split_bits":0,"last_scrub":"1041'961","last_scrub_stamp":"2024-09-20T19:39:13.078655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.078655+0000","objects_scrubbed":1,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.542664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001892148,"stat_sum":{"num_bytes":3025557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":55890,"num_write":39,"num_write_kb":17667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"987'39","reported_seq":2161,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505862+0000","last_change":"2024-09-20T19:38:26.665615+0000","last_active":"2024-09-20T19:39:33.505862+0000","last_peered":"2024-09-20T19:39:33.505862+0000","last_clean":"2024-09-20T19:39:33.505862+0000","last_became_active":"2024-09-20T19:33:43.448306+0000","last_became_peered":"2024-09-20T19:33:43.448306+0000","last_unstale":"2024-09-20T19:39:33.505862+0000","last_undegraded":"2024-09-20T19:39:33.505862+0000","last_fullsized":"2024-09-20T19:39:33.505862+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:38:26.665526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.665526+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":2076,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502205+0000","last_change":"2024-09-20T19:38:17.643987+0000","last_active":"2024-09-20T19:39:33.502205+0000","last_peered":"2024-09-20T19:39:33.502205+0000","last_clean":"2024-09-20T19:39:33.502205+0000","last_became_active":"2024-09-20T19:33:43.446249+0000","last_became_peered":"2024-09-20T19:33:43.446249+0000","last_unstale":"2024-09-20T19:39:33.502205+0000","last_undegraded":"2024-09-20T19:39:33.502205+0000","last_fullsized":"2024-09-20T19:39:33.502205+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:17.643862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.643862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":2114,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506121+0000","last_change":"2024-09-20T19:38:35.798379+0000","last_active":"2024-09-20T19:39:33.506121+0000","last_peered":"2024-09-20T19:39:33.506121+0000","last_clean":"2024-09-20T19:39:33.506121+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:39:33.506121+0000","last_undegraded":"2024-09-20T19:39:33.506121+0000","last_fullsized":"2024-09-20T19:39:33.506121+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:38:35.798250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.798250+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.803640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2052,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714127+0000","last_change":"2024-09-20T19:38:35.252514+0000","last_active":"2024-09-20T19:39:34.714127+0000","last_peered":"2024-09-20T19:39:34.714127+0000","last_clean":"2024-09-20T19:39:34.714127+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:39:34.714127+0000","last_undegraded":"2024-09-20T19:39:34.714127+0000","last_fullsized":"2024-09-20T19:39:34.714127+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.252424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.252424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.236879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1045'656","reported_seq":5345,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092100+0000","last_change":"2024-09-20T19:39:18.539326+0000","last_active":"2024-09-20T19:39:34.092100+0000","last_peered":"2024-09-20T19:39:34.092100+0000","last_clean":"2024-09-20T19:39:34.092100+0000","last_became_active":"2024-09-20T19:33:27.389104+0000","last_became_peered":"2024-09-20T19:33:27.389104+0000","last_unstale":"2024-09-20T19:39:34.092100+0000","last_undegraded":"2024-09-20T19:39:34.092100+0000","last_fullsized":"2024-09-20T19:39:34.092100+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'630","last_scrub_stamp":"2024-09-20T19:38:23.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.703700+0000","objects_scrubbed":1,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004543638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":20016,"num_write":45,"num_write_kb":14853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"987'27","reported_seq":1952,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506408+0000","last_change":"2024-09-20T19:39:26.955183+0000","last_active":"2024-09-20T19:39:33.506408+0000","last_peered":"2024-09-20T19:39:33.506408+0000","last_clean":"2024-09-20T19:39:33.506408+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:39:33.506408+0000","last_undegraded":"2024-09-20T19:39:33.506408+0000","last_fullsized":"2024-09-20T19:39:33.506408+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:39:26.955049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:26.955049+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:47.920732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":705,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.519207+0000","last_change":"2024-09-20T19:36:37.827773+0000","last_active":"2024-09-20T19:39:33.519207+0000","last_peered":"2024-09-20T19:39:33.519207+0000","last_clean":"2024-09-20T19:39:33.519207+0000","last_became_active":"2024-09-20T19:33:43.387558+0000","last_became_peered":"2024-09-20T19:33:43.387558+0000","last_unstale":"2024-09-20T19:39:33.519207+0000","last_undegraded":"2024-09-20T19:39:33.519207+0000","last_fullsized":"2024-09-20T19:39:33.519207+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:37.827682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:37.827682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2048,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.715369+0000","last_change":"2024-09-20T19:39:14.526537+0000","last_active":"2024-09-20T19:39:34.715369+0000","last_peered":"2024-09-20T19:39:34.715369+0000","last_clean":"2024-09-20T19:39:34.715369+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:39:34.715369+0000","last_undegraded":"2024-09-20T19:39:34.715369+0000","last_fullsized":"2024-09-20T19:39:34.715369+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:14.526473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:14.526473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:19.566605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1523,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092130+0000","last_change":"2024-09-20T19:39:22.314997+0000","last_active":"2024-09-20T19:39:34.092130+0000","last_peered":"2024-09-20T19:39:34.092130+0000","last_clean":"2024-09-20T19:39:34.092130+0000","last_became_active":"2024-09-20T19:33:43.352179+0000","last_became_peered":"2024-09-20T19:33:43.352179+0000","last_unstale":"2024-09-20T19:39:34.092130+0000","last_undegraded":"2024-09-20T19:39:34.092130+0000","last_fullsized":"2024-09-20T19:39:34.092130+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:22.314869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.314869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:35.340395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":876,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713715+0000","last_change":"2024-09-20T19:39:15.494274+0000","last_active":"2024-09-20T19:39:34.713715+0000","last_peered":"2024-09-20T19:39:34.713715+0000","last_clean":"2024-09-20T19:39:34.713715+0000","last_became_active":"2024-09-20T19:33:43.356263+0000","last_became_peered":"2024-09-20T19:33:43.356263+0000","last_unstale":"2024-09-20T19:39:34.713715+0000","last_undegraded":"2024-09-20T19:39:34.713715+0000","last_fullsized":"2024-09-20T19:39:34.713715+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:15.494138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.494138+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:43.249820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"667'196","reported_seq":4382,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502167+0000","last_change":"2024-09-20T19:39:18.520725+0000","last_active":"2024-09-20T19:39:33.502167+0000","last_peered":"2024-09-20T19:39:33.502167+0000","last_clean":"2024-09-20T19:39:33.502167+0000","last_became_active":"2024-09-20T19:33:46.368593+0000","last_became_peered":"2024-09-20T19:33:46.368593+0000","last_unstale":"2024-09-20T19:39:33.502167+0000","last_undegraded":"2024-09-20T19:39:33.502167+0000","last_fullsized":"2024-09-20T19:39:33.502167+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'196","last_scrub_stamp":"2024-09-20T19:37:42.138190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:37:42.138190+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011805349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"987'42","reported_seq":1922,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504097+0000","last_change":"2024-09-20T19:38:36.778506+0000","last_active":"2024-09-20T19:39:33.504097+0000","last_peered":"2024-09-20T19:39:33.504097+0000","last_clean":"2024-09-20T19:39:33.504097+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:39:33.504097+0000","last_undegraded":"2024-09-20T19:39:33.504097+0000","last_fullsized":"2024-09-20T19:39:33.504097+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:38:36.778420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.778420+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.256363+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2046,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.093126+0000","last_change":"2024-09-20T19:39:25.267412+0000","last_active":"2024-09-20T19:39:34.093126+0000","last_peered":"2024-09-20T19:39:34.093126+0000","last_clean":"2024-09-20T19:39:34.093126+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:39:34.093126+0000","last_undegraded":"2024-09-20T19:39:34.093126+0000","last_fullsized":"2024-09-20T19:39:34.093126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:25.267325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:25.267325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:26.854816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"715'16","reported_seq":990,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505178+0000","last_change":"2024-09-20T19:34:08.200032+0000","last_active":"2024-09-20T19:39:33.505178+0000","last_peered":"2024-09-20T19:39:33.505178+0000","last_clean":"2024-09-20T19:39:33.505178+0000","last_became_active":"2024-09-20T19:33:43.439666+0000","last_became_peered":"2024-09-20T19:33:43.439666+0000","last_unstale":"2024-09-20T19:39:33.505178+0000","last_undegraded":"2024-09-20T19:39:33.505178+0000","last_fullsized":"2024-09-20T19:39:33.505178+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"715'16","last_scrub_stamp":"2024-09-20T19:34:08.199971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:34:08.199971+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:03.844414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2026,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.517248+0000","last_change":"2024-09-20T19:39:16.001753+0000","last_active":"2024-09-20T19:39:33.517248+0000","last_peered":"2024-09-20T19:39:33.517248+0000","last_clean":"2024-09-20T19:39:33.517248+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:39:33.517248+0000","last_undegraded":"2024-09-20T19:39:33.517248+0000","last_fullsized":"2024-09-20T19:39:33.517248+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:16.001693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.001693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.004172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1456,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505099+0000","last_change":"2024-09-20T19:38:26.684327+0000","last_active":"2024-09-20T19:39:33.505099+0000","last_peered":"2024-09-20T19:39:33.505099+0000","last_clean":"2024-09-20T19:39:33.505099+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:39:33.505099+0000","last_undegraded":"2024-09-20T19:39:33.505099+0000","last_fullsized":"2024-09-20T19:39:33.505099+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:26.684257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:26.684257+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.590882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1050'508","reported_seq":4994,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502435+0000","last_change":"2024-09-20T19:39:18.550463+0000","last_active":"2024-09-20T19:39:33.502435+0000","last_peered":"2024-09-20T19:39:33.502435+0000","last_clean":"2024-09-20T19:39:33.502435+0000","last_became_active":"2024-09-20T19:33:46.367854+0000","last_became_peered":"2024-09-20T19:33:46.367854+0000","last_unstale":"2024-09-20T19:39:33.502435+0000","last_undegraded":"2024-09-20T19:39:33.502435+0000","last_fullsized":"2024-09-20T19:39:33.502435+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'507","last_scrub_stamp":"2024-09-20T19:39:15.008559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.008559+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:17.958113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001225196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":7626,"num_write":23,"num_write_kb":5767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"987'42","reported_seq":1782,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.521102+0000","last_change":"2024-09-20T19:37:30.646115+0000","last_active":"2024-09-20T19:39:33.521102+0000","last_peered":"2024-09-20T19:39:33.521102+0000","last_clean":"2024-09-20T19:39:33.521102+0000","last_became_active":"2024-09-20T19:33:43.449221+0000","last_became_peered":"2024-09-20T19:33:43.449221+0000","last_unstale":"2024-09-20T19:39:33.521102+0000","last_undegraded":"2024-09-20T19:39:33.521102+0000","last_fullsized":"2024-09-20T19:39:33.521102+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:37:30.645963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:30.645963+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":2062,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505089+0000","last_change":"2024-09-20T19:39:28.951954+0000","last_active":"2024-09-20T19:39:33.505089+0000","last_peered":"2024-09-20T19:39:33.505089+0000","last_clean":"2024-09-20T19:39:33.505089+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:39:33.505089+0000","last_undegraded":"2024-09-20T19:39:33.505089+0000","last_fullsized":"2024-09-20T19:39:33.505089+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:28.951890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:28.951890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:54.433483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1868,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503358+0000","last_change":"2024-09-20T19:39:22.924959+0000","last_active":"2024-09-20T19:39:33.503358+0000","last_peered":"2024-09-20T19:39:33.503358+0000","last_clean":"2024-09-20T19:39:33.503358+0000","last_became_active":"2024-09-20T19:33:43.375059+0000","last_became_peered":"2024-09-20T19:33:43.375059+0000","last_unstale":"2024-09-20T19:39:33.503358+0000","last_undegraded":"2024-09-20T19:39:33.503358+0000","last_fullsized":"2024-09-20T19:39:33.503358+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:22.924793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.924793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:35.342839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1814,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506681+0000","last_change":"2024-09-20T19:36:48.027651+0000","last_active":"2024-09-20T19:39:33.506681+0000","last_peered":"2024-09-20T19:39:33.506681+0000","last_clean":"2024-09-20T19:39:33.506681+0000","last_became_active":"2024-09-20T19:33:43.365449+0000","last_became_peered":"2024-09-20T19:33:43.365449+0000","last_unstale":"2024-09-20T19:39:33.506681+0000","last_undegraded":"2024-09-20T19:39:33.506681+0000","last_fullsized":"2024-09-20T19:39:33.506681+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:48.027589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.027589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.737059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1768,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092442+0000","last_change":"2024-09-20T19:38:31.650903+0000","last_active":"2024-09-20T19:39:34.092442+0000","last_peered":"2024-09-20T19:39:34.092442+0000","last_clean":"2024-09-20T19:39:34.092442+0000","last_became_active":"2024-09-20T19:33:43.353146+0000","last_became_peered":"2024-09-20T19:33:43.353146+0000","last_unstale":"2024-09-20T19:39:34.092442+0000","last_undegraded":"2024-09-20T19:39:34.092442+0000","last_fullsized":"2024-09-20T19:39:34.092442+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:31.650776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.650776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.460344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1040'1014","reported_seq":5872,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712361+0000","last_change":"2024-09-20T19:39:18.543146+0000","last_active":"2024-09-20T19:39:34.712361+0000","last_peered":"2024-09-20T19:39:34.712361+0000","last_clean":"2024-09-20T19:39:34.712361+0000","last_became_active":"2024-09-20T19:34:29.376353+0000","last_became_peered":"2024-09-20T19:34:29.376353+0000","last_unstale":"2024-09-20T19:39:34.712361+0000","last_undegraded":"2024-09-20T19:39:34.712361+0000","last_fullsized":"2024-09-20T19:39:34.712361+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"996'991","last_scrub_stamp":"2024-09-20T19:38:33.283803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.283803+0000","objects_scrubbed":1,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.656768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023544490000000002,"stat_sum":{"num_bytes":3294469,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":88272,"num_write":58,"num_write_kb":15972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":5414681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"987'39","reported_seq":1872,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746733+0000","last_change":"2024-09-20T19:36:24.022617+0000","last_active":"2024-09-20T19:39:34.746733+0000","last_peered":"2024-09-20T19:39:34.746733+0000","last_clean":"2024-09-20T19:39:34.746733+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:39:34.746733+0000","last_undegraded":"2024-09-20T19:39:34.746733+0000","last_fullsized":"2024-09-20T19:39:34.746733+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-20T19:36:24.022527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:24.022527+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2004,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504303+0000","last_change":"2024-09-20T19:36:31.559020+0000","last_active":"2024-09-20T19:39:33.504303+0000","last_peered":"2024-09-20T19:39:33.504303+0000","last_clean":"2024-09-20T19:39:33.504303+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:39:33.504303+0000","last_undegraded":"2024-09-20T19:39:33.504303+0000","last_fullsized":"2024-09-20T19:39:33.504303+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:31.558923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:31.558923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:55.839139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2050,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091429+0000","last_change":"2024-09-20T19:38:30.630442+0000","last_active":"2024-09-20T19:39:34.091429+0000","last_peered":"2024-09-20T19:39:34.091429+0000","last_clean":"2024-09-20T19:39:34.091429+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:39:34.091429+0000","last_undegraded":"2024-09-20T19:39:34.091429+0000","last_fullsized":"2024-09-20T19:39:34.091429+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.630318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.630318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.617338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":2045,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501285+0000","last_change":"2024-09-20T19:39:08.849859+0000","last_active":"2024-09-20T19:39:33.501285+0000","last_peered":"2024-09-20T19:39:33.501285+0000","last_clean":"2024-09-20T19:39:33.501285+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:39:33.501285+0000","last_undegraded":"2024-09-20T19:39:33.501285+0000","last_fullsized":"2024-09-20T19:39:33.501285+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:08.849741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:08.849741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:27.397133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1761,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092706+0000","last_change":"2024-09-20T19:39:27.274822+0000","last_active":"2024-09-20T19:39:34.092706+0000","last_peered":"2024-09-20T19:39:34.092706+0000","last_clean":"2024-09-20T19:39:34.092706+0000","last_became_active":"2024-09-20T19:33:43.351689+0000","last_became_peered":"2024-09-20T19:33:43.351689+0000","last_unstale":"2024-09-20T19:39:34.092706+0000","last_undegraded":"2024-09-20T19:39:34.092706+0000","last_fullsized":"2024-09-20T19:39:34.092706+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:27.274755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:27.274755+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:51.035850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"695'610","reported_seq":5174,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503811+0000","last_change":"2024-09-20T19:39:18.587767+0000","last_active":"2024-09-20T19:39:33.503811+0000","last_peered":"2024-09-20T19:39:33.503811+0000","last_clean":"2024-09-20T19:39:33.503811+0000","last_became_active":"2024-09-20T19:33:46.371775+0000","last_became_peered":"2024-09-20T19:33:46.371775+0000","last_unstale":"2024-09-20T19:39:33.503811+0000","last_undegraded":"2024-09-20T19:39:33.503811+0000","last_fullsized":"2024-09-20T19:39:33.503811+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'610","last_scrub_stamp":"2024-09-20T19:39:12.816955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.816955+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:41.759335+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0031973679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"987'30","reported_seq":2106,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502840+0000","last_change":"2024-09-20T19:34:11.179943+0000","last_active":"2024-09-20T19:39:33.502840+0000","last_peered":"2024-09-20T19:39:33.502840+0000","last_clean":"2024-09-20T19:39:33.502840+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:39:33.502840+0000","last_undegraded":"2024-09-20T19:39:33.502840+0000","last_fullsized":"2024-09-20T19:39:33.502840+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:34:11.179855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:34:11.179855+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.824143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2054,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506123+0000","last_change":"2024-09-20T19:39:13.877345+0000","last_active":"2024-09-20T19:39:33.506123+0000","last_peered":"2024-09-20T19:39:33.506123+0000","last_clean":"2024-09-20T19:39:33.506123+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:39:33.506123+0000","last_undegraded":"2024-09-20T19:39:33.506123+0000","last_fullsized":"2024-09-20T19:39:33.506123+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:13.877275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.877275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.110921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":2032,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747025+0000","last_change":"2024-09-20T19:38:19.343965+0000","last_active":"2024-09-20T19:39:34.747025+0000","last_peered":"2024-09-20T19:39:34.747025+0000","last_clean":"2024-09-20T19:39:34.747025+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:39:34.747025+0000","last_undegraded":"2024-09-20T19:39:34.747025+0000","last_fullsized":"2024-09-20T19:39:34.747025+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.343828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.343828+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.328086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1842,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746025+0000","last_change":"2024-09-20T19:35:47.661521+0000","last_active":"2024-09-20T19:39:34.746025+0000","last_peered":"2024-09-20T19:39:34.746025+0000","last_clean":"2024-09-20T19:39:34.746025+0000","last_became_active":"2024-09-20T19:34:26.391383+0000","last_became_peered":"2024-09-20T19:34:26.391383+0000","last_unstale":"2024-09-20T19:39:34.746025+0000","last_undegraded":"2024-09-20T19:39:34.746025+0000","last_fullsized":"2024-09-20T19:39:34.746025+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:35:47.661410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:35:47.661410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:31.417002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1642,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712906+0000","last_change":"2024-09-20T19:38:12.969468+0000","last_active":"2024-09-20T19:39:34.712906+0000","last_peered":"2024-09-20T19:39:34.712906+0000","last_clean":"2024-09-20T19:39:34.712906+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:39:34.712906+0000","last_undegraded":"2024-09-20T19:39:34.712906+0000","last_fullsized":"2024-09-20T19:39:34.712906+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:12.969339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.969339+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:31.552030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1046'726","reported_seq":5510,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092855+0000","last_change":"2024-09-20T19:39:26.300018+0000","last_active":"2024-09-20T19:39:34.092855+0000","last_peered":"2024-09-20T19:39:34.092855+0000","last_clean":"2024-09-20T19:39:34.092855+0000","last_became_active":"2024-09-20T19:33:46.371475+0000","last_became_peered":"2024-09-20T19:33:46.371475+0000","last_unstale":"2024-09-20T19:39:34.092855+0000","last_undegraded":"2024-09-20T19:39:34.092855+0000","last_fullsized":"2024-09-20T19:39:34.092855+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"1046'726","last_scrub_stamp":"2024-09-20T19:39:26.299883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:26.299883+0000","objects_scrubbed":1,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:50.134650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0046037159999999999,"stat_sum":{"num_bytes":2723064,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":186,"num_read_kb":136098,"num_write":135,"num_write_kb":47643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8676382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"987'57","reported_seq":2460,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505947+0000","last_change":"2024-09-20T19:39:14.900337+0000","last_active":"2024-09-20T19:39:33.505947+0000","last_peered":"2024-09-20T19:39:33.505947+0000","last_clean":"2024-09-20T19:39:33.505947+0000","last_became_active":"2024-09-20T19:33:43.450422+0000","last_became_peered":"2024-09-20T19:33:43.450422+0000","last_unstale":"2024-09-20T19:39:33.505947+0000","last_undegraded":"2024-09-20T19:39:33.505947+0000","last_fullsized":"2024-09-20T19:39:33.505947+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'57","last_scrub_stamp":"2024-09-20T19:39:14.900185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:14.900185+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:24.992844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2062,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714110+0000","last_change":"2024-09-20T19:38:32.270500+0000","last_active":"2024-09-20T19:39:34.714110+0000","last_peered":"2024-09-20T19:39:34.714110+0000","last_clean":"2024-09-20T19:39:34.714110+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:39:34.714110+0000","last_undegraded":"2024-09-20T19:39:34.714110+0000","last_fullsized":"2024-09-20T19:39:34.714110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:32.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:32.270373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.192387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1840,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747070+0000","last_change":"2024-09-20T19:39:30.418319+0000","last_active":"2024-09-20T19:39:34.747070+0000","last_peered":"2024-09-20T19:39:34.747070+0000","last_clean":"2024-09-20T19:39:34.747070+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:39:34.747070+0000","last_undegraded":"2024-09-20T19:39:34.747070+0000","last_fullsized":"2024-09-20T19:39:34.747070+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:30.418174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:30.418174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:33.387323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1495,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505814+0000","last_change":"2024-09-20T19:36:23.041088+0000","last_active":"2024-09-20T19:39:33.505814+0000","last_peered":"2024-09-20T19:39:33.505814+0000","last_clean":"2024-09-20T19:39:33.505814+0000","last_became_active":"2024-09-20T19:33:43.347039+0000","last_became_peered":"2024-09-20T19:33:43.347039+0000","last_unstale":"2024-09-20T19:39:33.505814+0000","last_undegraded":"2024-09-20T19:39:33.505814+0000","last_fullsized":"2024-09-20T19:39:33.505814+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.040967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.040967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.759148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":2112,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.521121+0000","last_change":"2024-09-20T19:38:41.037160+0000","last_active":"2024-09-20T19:39:33.521121+0000","last_peered":"2024-09-20T19:39:33.521121+0000","last_clean":"2024-09-20T19:39:33.521121+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:39:33.521121+0000","last_undegraded":"2024-09-20T19:39:33.521121+0000","last_fullsized":"2024-09-20T19:39:33.521121+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:41.037037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.037037+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:57.295442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1048'1200","reported_seq":5534,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746895+0000","last_change":"2024-09-20T19:39:20.376885+0000","last_active":"2024-09-20T19:39:34.746895+0000","last_peered":"2024-09-20T19:39:34.746895+0000","last_clean":"2024-09-20T19:39:34.746895+0000","last_became_active":"2024-09-20T19:30:31.372282+0000","last_became_peered":"2024-09-20T19:30:31.372282+0000","last_unstale":"2024-09-20T19:39:34.746895+0000","last_undegraded":"2024-09-20T19:39:34.746895+0000","last_fullsized":"2024-09-20T19:39:34.746895+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'1200","last_scrub_stamp":"2024-09-20T19:39:20.376813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.376813+0000","objects_scrubbed":2,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.287158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.15482644300000001,"stat_sum":{"num_bytes":5799009,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":159971,"num_write":135,"num_write_kb":55283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"987'21","reported_seq":2021,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.508116+0000","last_change":"2024-09-20T19:36:23.692393+0000","last_active":"2024-09-20T19:39:33.508116+0000","last_peered":"2024-09-20T19:39:33.508116+0000","last_clean":"2024-09-20T19:39:33.508116+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:39:33.508116+0000","last_undegraded":"2024-09-20T19:39:33.508116+0000","last_fullsized":"2024-09-20T19:39:33.508116+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'17","last_scrub_stamp":"2024-09-20T19:36:23.692257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.692257+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.833506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":3343,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:40.490052+0000","last_change":"2024-09-20T19:39:16.927632+0000","last_active":"2024-09-20T19:39:40.490052+0000","last_peered":"2024-09-20T19:39:40.490052+0000","last_clean":"2024-09-20T19:39:40.490052+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:39:40.490052+0000","last_undegraded":"2024-09-20T19:39:40.490052+0000","last_fullsized":"2024-09-20T19:39:40.490052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:39:16.927571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.927571+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:17.759811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2082,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501296+0000","last_change":"2024-09-20T19:39:25.111475+0000","last_active":"2024-09-20T19:39:33.501296+0000","last_peered":"2024-09-20T19:39:33.501296+0000","last_clean":"2024-09-20T19:39:33.501296+0000","last_became_active":"2024-09-20T19:33:43.443146+0000","last_became_peered":"2024-09-20T19:33:43.443146+0000","last_unstale":"2024-09-20T19:39:33.501296+0000","last_undegraded":"2024-09-20T19:39:33.501296+0000","last_fullsized":"2024-09-20T19:39:33.501296+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:25.111343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:25.111343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.595189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":636,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506955+0000","last_change":"2024-09-20T19:39:30.229551+0000","last_active":"2024-09-20T19:39:33.506955+0000","last_peered":"2024-09-20T19:39:33.506955+0000","last_clean":"2024-09-20T19:39:33.506955+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:39:33.506955+0000","last_undegraded":"2024-09-20T19:39:33.506955+0000","last_fullsized":"2024-09-20T19:39:33.506955+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:30.229478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:30.229478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:53.169434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2036,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518342+0000","last_change":"2024-09-20T19:33:43.365945+0000","last_active":"2024-09-20T19:39:33.518342+0000","last_peered":"2024-09-20T19:39:33.518342+0000","last_clean":"2024-09-20T19:39:33.518342+0000","last_became_active":"2024-09-20T19:33:43.365621+0000","last_became_peered":"2024-09-20T19:33:43.365621+0000","last_unstale":"2024-09-20T19:39:33.518342+0000","last_undegraded":"2024-09-20T19:39:33.518342+0000","last_fullsized":"2024-09-20T19:39:33.518342+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:21.799252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:21.799252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.132300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1645,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502282+0000","last_change":"2024-09-20T19:38:14.535923+0000","last_active":"2024-09-20T19:39:33.502282+0000","last_peered":"2024-09-20T19:39:33.502282+0000","last_clean":"2024-09-20T19:39:33.502282+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:39:33.502282+0000","last_undegraded":"2024-09-20T19:39:33.502282+0000","last_fullsized":"2024-09-20T19:39:33.502282+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:38:14.535715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.535715+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1050'1256","reported_seq":5991,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518022+0000","last_change":"2024-09-20T19:39:18.525631+0000","last_active":"2024-09-20T19:39:33.518022+0000","last_peered":"2024-09-20T19:39:33.518022+0000","last_clean":"2024-09-20T19:39:33.518022+0000","last_became_active":"2024-09-20T19:30:09.792041+0000","last_became_peered":"2024-09-20T19:30:09.792041+0000","last_unstale":"2024-09-20T19:39:33.518022+0000","last_undegraded":"2024-09-20T19:39:33.518022+0000","last_fullsized":"2024-09-20T19:39:33.518022+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"892'1099","last_scrub_stamp":"2024-09-20T19:36:40.791824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.791824+0000","objects_scrubbed":12,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003200708,"stat_sum":{"num_bytes":12714313,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":271,"num_read_kb":207585,"num_write":180,"num_write_kb":60667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"987'36","reported_seq":1890,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747397+0000","last_change":"2024-09-20T19:36:33.028171+0000","last_active":"2024-09-20T19:39:34.747397+0000","last_peered":"2024-09-20T19:39:34.747397+0000","last_clean":"2024-09-20T19:39:34.747397+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:39:34.747397+0000","last_undegraded":"2024-09-20T19:39:34.747397+0000","last_fullsized":"2024-09-20T19:39:34.747397+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'24","last_scrub_stamp":"2024-09-20T19:36:33.028057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:33.028057+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.454887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1786,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746292+0000","last_change":"2024-09-20T19:38:25.204605+0000","last_active":"2024-09-20T19:39:34.746292+0000","last_peered":"2024-09-20T19:39:34.746292+0000","last_clean":"2024-09-20T19:39:34.746292+0000","last_became_active":"2024-09-20T19:33:43.362716+0000","last_became_peered":"2024-09-20T19:33:43.362716+0000","last_unstale":"2024-09-20T19:39:34.746292+0000","last_undegraded":"2024-09-20T19:39:34.746292+0000","last_fullsized":"2024-09-20T19:39:34.746292+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:38:25.204541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.204541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.668929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2049,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520155+0000","last_change":"2024-09-20T19:38:25.143232+0000","last_active":"2024-09-20T19:39:33.520155+0000","last_peered":"2024-09-20T19:39:33.520155+0000","last_clean":"2024-09-20T19:39:33.520155+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:39:33.520155+0000","last_undegraded":"2024-09-20T19:39:33.520155+0000","last_fullsized":"2024-09-20T19:39:33.520155+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.143087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.143087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2048,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.715487+0000","last_change":"2024-09-20T19:38:20.052968+0000","last_active":"2024-09-20T19:39:34.715487+0000","last_peered":"2024-09-20T19:39:34.715487+0000","last_clean":"2024-09-20T19:39:34.715487+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:39:34.715487+0000","last_undegraded":"2024-09-20T19:39:34.715487+0000","last_fullsized":"2024-09-20T19:39:34.715487+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.052811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.052811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.621676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2081,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504576+0000","last_change":"2024-09-20T19:39:23.951885+0000","last_active":"2024-09-20T19:39:33.504576+0000","last_peered":"2024-09-20T19:39:33.504576+0000","last_clean":"2024-09-20T19:39:33.504576+0000","last_became_active":"2024-09-20T19:33:43.441638+0000","last_became_peered":"2024-09-20T19:33:43.441638+0000","last_unstale":"2024-09-20T19:39:33.504576+0000","last_undegraded":"2024-09-20T19:39:33.504576+0000","last_fullsized":"2024-09-20T19:39:33.504576+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:23.951769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:23.951769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:46.265724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"447'254","reported_seq":4587,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747545+0000","last_change":"2024-09-20T19:39:19.369908+0000","last_active":"2024-09-20T19:39:34.747545+0000","last_peered":"2024-09-20T19:39:34.747545+0000","last_clean":"2024-09-20T19:39:34.747545+0000","last_became_active":"2024-09-20T19:33:46.380931+0000","last_became_peered":"2024-09-20T19:33:46.380931+0000","last_unstale":"2024-09-20T19:39:34.747545+0000","last_undegraded":"2024-09-20T19:39:34.747545+0000","last_fullsized":"2024-09-20T19:39:34.747545+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'254","last_scrub_stamp":"2024-09-20T19:39:19.369830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:19.369830+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:20.862550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.091894885999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"987'42","reported_seq":2364,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092348+0000","last_change":"2024-09-20T19:36:48.949647+0000","last_active":"2024-09-20T19:39:34.092348+0000","last_peered":"2024-09-20T19:39:34.092348+0000","last_clean":"2024-09-20T19:39:34.092348+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:39:34.092348+0000","last_undegraded":"2024-09-20T19:39:34.092348+0000","last_fullsized":"2024-09-20T19:39:34.092348+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'30","last_scrub_stamp":"2024-09-20T19:36:48.949462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:48.949462+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.572488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":900,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713503+0000","last_change":"2024-09-20T19:38:22.129174+0000","last_active":"2024-09-20T19:39:34.713503+0000","last_peered":"2024-09-20T19:39:34.713503+0000","last_clean":"2024-09-20T19:39:34.713503+0000","last_became_active":"2024-09-20T19:33:43.343499+0000","last_became_peered":"2024-09-20T19:33:43.343499+0000","last_unstale":"2024-09-20T19:39:34.713503+0000","last_undegraded":"2024-09-20T19:39:34.713503+0000","last_fullsized":"2024-09-20T19:39:34.713503+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:38:22.129045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.129045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.992187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2030,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714143+0000","last_change":"2024-09-20T19:34:12.051323+0000","last_active":"2024-09-20T19:39:34.714143+0000","last_peered":"2024-09-20T19:39:34.714143+0000","last_clean":"2024-09-20T19:39:34.714143+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:39:34.714143+0000","last_undegraded":"2024-09-20T19:39:34.714143+0000","last_fullsized":"2024-09-20T19:39:34.714143+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:12.051203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:12.051203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:04.413613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":2100,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520617+0000","last_change":"2024-09-20T19:39:20.032046+0000","last_active":"2024-09-20T19:39:33.520617+0000","last_peered":"2024-09-20T19:39:33.520617+0000","last_clean":"2024-09-20T19:39:33.520617+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:39:33.520617+0000","last_undegraded":"2024-09-20T19:39:33.520617+0000","last_fullsized":"2024-09-20T19:39:33.520617+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:39:20.031879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:39:20.031879+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:47.216443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":2022,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518852+0000","last_change":"2024-09-20T19:38:30.011404+0000","last_active":"2024-09-20T19:39:33.518852+0000","last_peered":"2024-09-20T19:39:33.518852+0000","last_clean":"2024-09-20T19:39:33.518852+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:39:33.518852+0000","last_undegraded":"2024-09-20T19:39:33.518852+0000","last_fullsized":"2024-09-20T19:39:33.518852+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.011277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.011277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:59.873402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2058,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.715765+0000","last_change":"2024-09-20T19:38:34.247348+0000","last_active":"2024-09-20T19:39:34.715765+0000","last_peered":"2024-09-20T19:39:34.715765+0000","last_clean":"2024-09-20T19:39:34.715765+0000","last_became_active":"2024-09-20T19:33:43.446212+0000","last_became_peered":"2024-09-20T19:33:43.446212+0000","last_unstale":"2024-09-20T19:39:34.715765+0000","last_undegraded":"2024-09-20T19:39:34.715765+0000","last_fullsized":"2024-09-20T19:39:34.715765+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:34.247292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:34.247292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.593318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"1046'472","reported_seq":4839,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746189+0000","last_change":"2024-09-20T19:39:18.609359+0000","last_active":"2024-09-20T19:39:34.746189+0000","last_peered":"2024-09-20T19:39:34.746189+0000","last_clean":"2024-09-20T19:39:34.746189+0000","last_became_active":"2024-09-20T19:30:31.348343+0000","last_became_peered":"2024-09-20T19:30:31.348343+0000","last_unstale":"2024-09-20T19:39:34.746189+0000","last_undegraded":"2024-09-20T19:39:34.746189+0000","last_fullsized":"2024-09-20T19:39:34.746189+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'450","last_scrub_stamp":"2024-09-20T19:38:35.237169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.237169+0000","objects_scrubbed":1,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.156087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.093957481999999995,"stat_sum":{"num_bytes":3944148,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":32971,"num_write":72,"num_write_kb":19085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"987'51","reported_seq":2113,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091712+0000","last_change":"2024-09-20T19:39:18.379050+0000","last_active":"2024-09-20T19:39:34.091712+0000","last_peered":"2024-09-20T19:39:34.091712+0000","last_clean":"2024-09-20T19:39:34.091712+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:39:34.091712+0000","last_undegraded":"2024-09-20T19:39:34.091712+0000","last_fullsized":"2024-09-20T19:39:34.091712+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'51","last_scrub_stamp":"2024-09-20T19:39:18.378913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:18.378913+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:46.141156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2075,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506258+0000","last_change":"2024-09-20T19:38:25.535832+0000","last_active":"2024-09-20T19:39:33.506258+0000","last_peered":"2024-09-20T19:39:33.506258+0000","last_clean":"2024-09-20T19:39:33.506258+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:39:33.506258+0000","last_undegraded":"2024-09-20T19:39:33.506258+0000","last_fullsized":"2024-09-20T19:39:33.506258+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:25.535708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:25.535708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.031136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":628,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502969+0000","last_change":"2024-09-20T19:38:24.553105+0000","last_active":"2024-09-20T19:39:33.502969+0000","last_peered":"2024-09-20T19:39:33.502969+0000","last_clean":"2024-09-20T19:39:33.502969+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:39:33.502969+0000","last_undegraded":"2024-09-20T19:39:33.502969+0000","last_fullsized":"2024-09-20T19:39:33.502969+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:24.553002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:24.553002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2027,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520334+0000","last_change":"2024-09-20T19:38:14.008440+0000","last_active":"2024-09-20T19:39:33.520334+0000","last_peered":"2024-09-20T19:39:33.520334+0000","last_clean":"2024-09-20T19:39:33.520334+0000","last_became_active":"2024-09-20T19:33:43.369087+0000","last_became_peered":"2024-09-20T19:33:43.369087+0000","last_unstale":"2024-09-20T19:39:33.520334+0000","last_undegraded":"2024-09-20T19:39:33.520334+0000","last_fullsized":"2024-09-20T19:39:33.520334+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:14.008316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:14.008316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:40.394501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1045'1673","reported_seq":6300,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.499791+0000","last_change":"2024-09-20T19:39:18.554664+0000","last_active":"2024-09-20T19:39:33.499791+0000","last_peered":"2024-09-20T19:39:33.499791+0000","last_clean":"2024-09-20T19:39:33.499791+0000","last_became_active":"2024-09-20T19:30:31.347466+0000","last_became_peered":"2024-09-20T19:30:31.347466+0000","last_unstale":"2024-09-20T19:39:33.499791+0000","last_undegraded":"2024-09-20T19:39:33.499791+0000","last_fullsized":"2024-09-20T19:39:33.499791+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'1669","last_scrub_stamp":"2024-09-20T19:39:07.884701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.884701+0000","objects_scrubbed":4,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.699067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001384714,"stat_sum":{"num_bytes":5983919,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":272,"num_read_kb":211320,"num_write":187,"num_write_kb":73661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"987'21","reported_seq":1864,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747026+0000","last_change":"2024-09-20T19:39:27.366877+0000","last_active":"2024-09-20T19:39:34.747026+0000","last_peered":"2024-09-20T19:39:34.747026+0000","last_clean":"2024-09-20T19:39:34.747026+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:39:34.747026+0000","last_undegraded":"2024-09-20T19:39:34.747026+0000","last_fullsized":"2024-09-20T19:39:34.747026+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:39:27.366735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:27.366735+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:55.151855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":2074,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503733+0000","last_change":"2024-09-20T19:36:34.027338+0000","last_active":"2024-09-20T19:39:33.503733+0000","last_peered":"2024-09-20T19:39:33.503733+0000","last_clean":"2024-09-20T19:39:33.503733+0000","last_became_active":"2024-09-20T19:34:26.405717+0000","last_became_peered":"2024-09-20T19:34:26.405717+0000","last_unstale":"2024-09-20T19:39:33.503733+0000","last_undegraded":"2024-09-20T19:39:33.503733+0000","last_fullsized":"2024-09-20T19:39:33.503733+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:34.027257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:36:34.027257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.639569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2081,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501133+0000","last_change":"2024-09-20T19:38:31.728072+0000","last_active":"2024-09-20T19:39:33.501133+0000","last_peered":"2024-09-20T19:39:33.501133+0000","last_clean":"2024-09-20T19:39:33.501133+0000","last_became_active":"2024-09-20T19:33:43.358969+0000","last_became_peered":"2024-09-20T19:33:43.358969+0000","last_unstale":"2024-09-20T19:39:33.501133+0000","last_undegraded":"2024-09-20T19:39:33.501133+0000","last_fullsized":"2024-09-20T19:39:33.501133+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.728014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.728014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:34.454777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1856,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746433+0000","last_change":"2024-09-20T19:39:26.372914+0000","last_active":"2024-09-20T19:39:34.746433+0000","last_peered":"2024-09-20T19:39:34.746433+0000","last_clean":"2024-09-20T19:39:34.746433+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:39:34.746433+0000","last_undegraded":"2024-09-20T19:39:34.746433+0000","last_fullsized":"2024-09-20T19:39:34.746433+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:39:26.372831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:26.372831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:34.455172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"1050'682","reported_seq":5346,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501300+0000","last_change":"2024-09-20T19:39:18.520140+0000","last_active":"2024-09-20T19:39:33.501300+0000","last_peered":"2024-09-20T19:39:33.501300+0000","last_clean":"2024-09-20T19:39:33.501300+0000","last_became_active":"2024-09-20T19:33:27.382044+0000","last_became_peered":"2024-09-20T19:33:27.382044+0000","last_unstale":"2024-09-20T19:39:33.501300+0000","last_undegraded":"2024-09-20T19:39:33.501300+0000","last_fullsized":"2024-09-20T19:39:33.501300+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'666","last_scrub_stamp":"2024-09-20T19:38:31.665799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.665799+0000","objects_scrubbed":2,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:37.291094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00078118399999999998,"stat_sum":{"num_bytes":6598448,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":155184,"num_write":61,"num_write_kb":17827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"987'39","reported_seq":2110,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712762+0000","last_change":"2024-09-20T19:39:07.498428+0000","last_active":"2024-09-20T19:39:34.712762+0000","last_peered":"2024-09-20T19:39:34.712762+0000","last_clean":"2024-09-20T19:39:34.712762+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:39:34.712762+0000","last_undegraded":"2024-09-20T19:39:34.712762+0000","last_fullsized":"2024-09-20T19:39:34.712762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:07.498283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:07.498283+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:11.846306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":632,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.507096+0000","last_change":"2024-09-20T19:37:32.482435+0000","last_active":"2024-09-20T19:39:33.507096+0000","last_peered":"2024-09-20T19:39:33.507096+0000","last_clean":"2024-09-20T19:39:33.507096+0000","last_became_active":"2024-09-20T19:34:26.389974+0000","last_became_peered":"2024-09-20T19:34:26.389974+0000","last_unstale":"2024-09-20T19:39:33.507096+0000","last_undegraded":"2024-09-20T19:39:33.507096+0000","last_fullsized":"2024-09-20T19:39:33.507096+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:37:32.482370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.482370+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:02.892648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":2304,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520128+0000","last_change":"2024-09-20T19:32:23.747205+0000","last_active":"2024-09-20T19:39:33.520128+0000","last_peered":"2024-09-20T19:39:33.520128+0000","last_clean":"2024-09-20T19:39:33.520128+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:39:33.520128+0000","last_undegraded":"2024-09-20T19:39:33.520128+0000","last_fullsized":"2024-09-20T19:39:33.520128+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:32:23.747056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:32:23.747056+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.039100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2044,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713160+0000","last_change":"2024-09-20T19:39:05.442773+0000","last_active":"2024-09-20T19:39:34.713160+0000","last_peered":"2024-09-20T19:39:34.713160+0000","last_clean":"2024-09-20T19:39:34.713160+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:39:34.713160+0000","last_undegraded":"2024-09-20T19:39:34.713160+0000","last_fullsized":"2024-09-20T19:39:34.713160+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:05.442716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:05.442716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:22.768089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"695'460","reported_seq":4738,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.500207+0000","last_change":"2024-09-20T19:39:18.552320+0000","last_active":"2024-09-20T19:39:33.500207+0000","last_peered":"2024-09-20T19:39:33.500207+0000","last_clean":"2024-09-20T19:39:33.500207+0000","last_became_active":"2024-09-20T19:33:46.388457+0000","last_became_peered":"2024-09-20T19:33:46.388457+0000","last_unstale":"2024-09-20T19:39:33.500207+0000","last_undegraded":"2024-09-20T19:39:33.500207+0000","last_fullsized":"2024-09-20T19:39:33.500207+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'460","last_scrub_stamp":"2024-09-20T19:39:09.971030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.971030+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:29.776858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017643380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"987'24","reported_seq":1929,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504034+0000","last_change":"2024-09-20T19:36:40.974050+0000","last_active":"2024-09-20T19:39:33.504034+0000","last_peered":"2024-09-20T19:39:33.504034+0000","last_clean":"2024-09-20T19:39:33.504034+0000","last_became_active":"2024-09-20T19:33:43.345822+0000","last_became_peered":"2024-09-20T19:33:43.345822+0000","last_unstale":"2024-09-20T19:39:33.504034+0000","last_undegraded":"2024-09-20T19:39:33.504034+0000","last_fullsized":"2024-09-20T19:39:33.504034+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:36:40.973907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:40.973907+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.768011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":1916,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505737+0000","last_change":"2024-09-20T19:38:31.783351+0000","last_active":"2024-09-20T19:39:33.505737+0000","last_peered":"2024-09-20T19:39:33.505737+0000","last_clean":"2024-09-20T19:39:33.505737+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:39:33.505737+0000","last_undegraded":"2024-09-20T19:39:33.505737+0000","last_fullsized":"2024-09-20T19:39:33.505737+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:38:31.783222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.783222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:48.587593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2059,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.500517+0000","last_change":"2024-09-20T19:38:36.786857+0000","last_active":"2024-09-20T19:39:33.500517+0000","last_peered":"2024-09-20T19:39:33.500517+0000","last_clean":"2024-09-20T19:39:33.500517+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:39:33.500517+0000","last_undegraded":"2024-09-20T19:39:33.500517+0000","last_fullsized":"2024-09-20T19:39:33.500517+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:36.786803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:36.786803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:39.996355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1532,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.517037+0000","last_change":"2024-09-20T19:37:32.585551+0000","last_active":"2024-09-20T19:39:33.517037+0000","last_peered":"2024-09-20T19:39:33.517037+0000","last_clean":"2024-09-20T19:39:33.517037+0000","last_became_active":"2024-09-20T19:33:43.447017+0000","last_became_peered":"2024-09-20T19:33:43.447017+0000","last_unstale":"2024-09-20T19:39:33.517037+0000","last_undegraded":"2024-09-20T19:39:33.517037+0000","last_fullsized":"2024-09-20T19:39:33.517037+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:32.585494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:32.585494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"1045'1372","reported_seq":6415,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518751+0000","last_change":"2024-09-20T19:39:18.526014+0000","last_active":"2024-09-20T19:39:33.518751+0000","last_peered":"2024-09-20T19:39:33.518751+0000","last_clean":"2024-09-20T19:39:33.518751+0000","last_became_active":"2024-09-20T19:34:29.374158+0000","last_became_peered":"2024-09-20T19:34:29.374158+0000","last_unstale":"2024-09-20T19:39:33.518751+0000","last_undegraded":"2024-09-20T19:39:33.518751+0000","last_fullsized":"2024-09-20T19:39:33.518751+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"1045'1372","last_scrub_stamp":"2024-09-20T19:39:17.039956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.039956+0000","objects_scrubbed":4,"log_size":1372,"log_dups_size":0,"ondisk_log_size":1372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:42.691540+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012545200000000001,"stat_sum":{"num_bytes":9026679,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":293,"num_read_kb":216218,"num_write":228,"num_write_kb":91119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":40162717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"987'21","reported_seq":1857,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747243+0000","last_change":"2024-09-20T19:37:23.121422+0000","last_active":"2024-09-20T19:39:34.747243+0000","last_peered":"2024-09-20T19:39:34.747243+0000","last_clean":"2024-09-20T19:39:34.747243+0000","last_became_active":"2024-09-20T19:33:43.365136+0000","last_became_peered":"2024-09-20T19:33:43.365136+0000","last_unstale":"2024-09-20T19:39:34.747243+0000","last_undegraded":"2024-09-20T19:39:34.747243+0000","last_fullsized":"2024-09-20T19:39:34.747243+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:37:23.121282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:23.121282+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2091,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.500950+0000","last_change":"2024-09-20T19:38:30.686760+0000","last_active":"2024-09-20T19:39:33.500950+0000","last_peered":"2024-09-20T19:39:33.500950+0000","last_clean":"2024-09-20T19:39:33.500950+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:39:33.500950+0000","last_undegraded":"2024-09-20T19:39:33.500950+0000","last_fullsized":"2024-09-20T19:39:33.500950+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:30.686621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.686621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:35.603556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1535,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505221+0000","last_change":"2024-09-20T19:39:15.910802+0000","last_active":"2024-09-20T19:39:33.505221+0000","last_peered":"2024-09-20T19:39:33.505221+0000","last_clean":"2024-09-20T19:39:33.505221+0000","last_became_active":"2024-09-20T19:33:43.448978+0000","last_became_peered":"2024-09-20T19:33:43.448978+0000","last_unstale":"2024-09-20T19:39:33.505221+0000","last_undegraded":"2024-09-20T19:39:33.505221+0000","last_fullsized":"2024-09-20T19:39:33.505221+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:15.910687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:15.910687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:35.477531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2047,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505077+0000","last_change":"2024-09-20T19:37:37.478973+0000","last_active":"2024-09-20T19:39:33.505077+0000","last_peered":"2024-09-20T19:39:33.505077+0000","last_clean":"2024-09-20T19:39:33.505077+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:39:33.505077+0000","last_undegraded":"2024-09-20T19:39:33.505077+0000","last_fullsized":"2024-09-20T19:39:33.505077+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:37.478823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:37:37.478823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:44.857932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1048'1653","reported_seq":6340,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504786+0000","last_change":"2024-09-20T19:39:18.551000+0000","last_active":"2024-09-20T19:39:33.504786+0000","last_peered":"2024-09-20T19:39:33.504786+0000","last_clean":"2024-09-20T19:39:33.504786+0000","last_became_active":"2024-09-20T19:33:46.391752+0000","last_became_peered":"2024-09-20T19:33:46.391752+0000","last_unstale":"2024-09-20T19:39:33.504786+0000","last_undegraded":"2024-09-20T19:39:33.504786+0000","last_fullsized":"2024-09-20T19:39:33.504786+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"986'1609","last_scrub_stamp":"2024-09-20T19:38:22.531869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:22.531869+0000","objects_scrubbed":3,"log_size":1653,"log_dups_size":0,"ondisk_log_size":1653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.814084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00070473599999999997,"stat_sum":{"num_bytes":10832612,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":313,"num_read_kb":203037,"num_write":237,"num_write_kb":87419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":26383389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":2041,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091519+0000","last_change":"2024-09-20T19:34:57.465453+0000","last_active":"2024-09-20T19:39:34.091519+0000","last_peered":"2024-09-20T19:39:34.091519+0000","last_clean":"2024-09-20T19:39:34.091519+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:39:34.091519+0000","last_undegraded":"2024-09-20T19:39:34.091519+0000","last_fullsized":"2024-09-20T19:39:34.091519+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:57.465382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:57.465382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:54.535576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"987'18","reported_seq":1709,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520196+0000","last_change":"2024-09-20T19:36:39.739712+0000","last_active":"2024-09-20T19:39:33.520196+0000","last_peered":"2024-09-20T19:39:33.520196+0000","last_clean":"2024-09-20T19:39:33.520196+0000","last_became_active":"2024-09-20T19:33:43.447188+0000","last_became_peered":"2024-09-20T19:33:43.447188+0000","last_unstale":"2024-09-20T19:39:33.520196+0000","last_undegraded":"2024-09-20T19:39:33.520196+0000","last_fullsized":"2024-09-20T19:39:33.520196+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'12","last_scrub_stamp":"2024-09-20T19:36:39.739615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:39.739615+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2070,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.715024+0000","last_change":"2024-09-20T19:39:23.545803+0000","last_active":"2024-09-20T19:39:34.715024+0000","last_peered":"2024-09-20T19:39:34.715024+0000","last_clean":"2024-09-20T19:39:34.715024+0000","last_became_active":"2024-09-20T19:33:43.446174+0000","last_became_peered":"2024-09-20T19:33:43.446174+0000","last_unstale":"2024-09-20T19:39:34.715024+0000","last_undegraded":"2024-09-20T19:39:34.715024+0000","last_fullsized":"2024-09-20T19:39:34.715024+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:23.545688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:23.545688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:51.444512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2027,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518199+0000","last_change":"2024-09-20T19:39:22.039805+0000","last_active":"2024-09-20T19:39:33.518199+0000","last_peered":"2024-09-20T19:39:33.518199+0000","last_clean":"2024-09-20T19:39:33.518199+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:39:33.518199+0000","last_undegraded":"2024-09-20T19:39:33.518199+0000","last_fullsized":"2024-09-20T19:39:33.518199+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:22.039740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:22.039740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:44.971668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1020'831","reported_seq":5394,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520620+0000","last_change":"2024-09-20T19:39:18.527861+0000","last_active":"2024-09-20T19:39:33.520620+0000","last_peered":"2024-09-20T19:39:33.520620+0000","last_clean":"2024-09-20T19:39:33.520620+0000","last_became_active":"2024-09-20T19:28:48.212760+0000","last_became_peered":"2024-09-20T19:28:48.212760+0000","last_unstale":"2024-09-20T19:39:33.520620+0000","last_undegraded":"2024-09-20T19:39:33.520620+0000","last_fullsized":"2024-09-20T19:39:33.520620+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"988'810","last_scrub_stamp":"2024-09-20T19:38:21.101246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:21.101246+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001718083,"stat_sum":{"num_bytes":2118012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":145,"num_read_kb":108369,"num_write":90,"num_write_kb":32667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"987'30","reported_seq":1732,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505322+0000","last_change":"2024-09-20T19:36:36.023059+0000","last_active":"2024-09-20T19:39:33.505322+0000","last_peered":"2024-09-20T19:39:33.505322+0000","last_clean":"2024-09-20T19:39:33.505322+0000","last_became_active":"2024-09-20T19:33:43.446556+0000","last_became_peered":"2024-09-20T19:33:43.446556+0000","last_unstale":"2024-09-20T19:39:33.505322+0000","last_undegraded":"2024-09-20T19:39:33.505322+0000","last_fullsized":"2024-09-20T19:39:33.505322+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'20","last_scrub_stamp":"2024-09-20T19:36:36.022993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:36:36.022993+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:51.646412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":2071,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.713842+0000","last_change":"2024-09-20T19:38:31.227463+0000","last_active":"2024-09-20T19:39:34.713842+0000","last_peered":"2024-09-20T19:39:34.713842+0000","last_clean":"2024-09-20T19:39:34.713842+0000","last_became_active":"2024-09-20T19:33:43.374115+0000","last_became_peered":"2024-09-20T19:33:43.374115+0000","last_unstale":"2024-09-20T19:39:34.713842+0000","last_undegraded":"2024-09-20T19:39:34.713842+0000","last_fullsized":"2024-09-20T19:39:34.713842+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:31.227391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:38:31.227391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.357562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":2319,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714575+0000","last_change":"2024-09-20T19:38:19.049146+0000","last_active":"2024-09-20T19:39:34.714575+0000","last_peered":"2024-09-20T19:39:34.714575+0000","last_clean":"2024-09-20T19:39:34.714575+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:39:34.714575+0000","last_undegraded":"2024-09-20T19:39:34.714575+0000","last_fullsized":"2024-09-20T19:39:34.714575+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:38:19.049091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:19.049091+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:34.591338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1828,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747709+0000","last_change":"2024-09-20T19:39:28.388101+0000","last_active":"2024-09-20T19:39:34.747709+0000","last_peered":"2024-09-20T19:39:34.747709+0000","last_clean":"2024-09-20T19:39:34.747709+0000","last_became_active":"2024-09-20T19:33:43.367032+0000","last_became_peered":"2024-09-20T19:33:43.367032+0000","last_unstale":"2024-09-20T19:39:34.747709+0000","last_undegraded":"2024-09-20T19:39:34.747709+0000","last_fullsized":"2024-09-20T19:39:34.747709+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:28.387977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:39:28.387977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:48.239216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1045'761","reported_seq":5140,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506244+0000","last_change":"2024-09-20T19:39:18.589826+0000","last_active":"2024-09-20T19:39:33.506244+0000","last_peered":"2024-09-20T19:39:33.506244+0000","last_clean":"2024-09-20T19:39:33.506244+0000","last_became_active":"2024-09-20T19:30:34.244620+0000","last_became_peered":"2024-09-20T19:30:34.244620+0000","last_unstale":"2024-09-20T19:39:33.506244+0000","last_undegraded":"2024-09-20T19:39:33.506244+0000","last_fullsized":"2024-09-20T19:39:33.506244+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"994'754","last_scrub_stamp":"2024-09-20T19:38:40.806420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.806420+0000","objects_scrubbed":1,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.595373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0033661839999999999,"stat_sum":{"num_bytes":3656437,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":187,"num_read_kb":132046,"num_write":137,"num_write_kb":52044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"987'21","reported_seq":2117,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714593+0000","last_change":"2024-09-20T19:38:30.190106+0000","last_active":"2024-09-20T19:39:34.714593+0000","last_peered":"2024-09-20T19:39:34.714593+0000","last_clean":"2024-09-20T19:39:34.714593+0000","last_became_active":"2024-09-20T19:33:43.356673+0000","last_became_peered":"2024-09-20T19:33:43.356673+0000","last_unstale":"2024-09-20T19:39:34.714593+0000","last_undegraded":"2024-09-20T19:39:34.714593+0000","last_fullsized":"2024-09-20T19:39:34.714593+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:38:30.190020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.190020+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:49.116547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1794,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747185+0000","last_change":"2024-09-20T19:39:12.256710+0000","last_active":"2024-09-20T19:39:34.747185+0000","last_peered":"2024-09-20T19:39:34.747185+0000","last_clean":"2024-09-20T19:39:34.747185+0000","last_became_active":"2024-09-20T19:33:43.369596+0000","last_became_peered":"2024-09-20T19:33:43.369596+0000","last_unstale":"2024-09-20T19:39:34.747185+0000","last_undegraded":"2024-09-20T19:39:34.747185+0000","last_fullsized":"2024-09-20T19:39:34.747185+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:12.256601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.256601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:28.448150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"759'17","reported_seq":770,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091770+0000","last_change":"2024-09-20T19:38:27.671255+0000","last_active":"2024-09-20T19:39:34.091770+0000","last_peered":"2024-09-20T19:39:34.091770+0000","last_clean":"2024-09-20T19:39:34.091770+0000","last_became_active":"2024-09-20T19:34:26.401967+0000","last_became_peered":"2024-09-20T19:34:26.401967+0000","last_unstale":"2024-09-20T19:39:34.091770+0000","last_undegraded":"2024-09-20T19:39:34.091770+0000","last_fullsized":"2024-09-20T19:39:34.091770+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'17","last_scrub_stamp":"2024-09-20T19:38:27.671197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:27.671197+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1877,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505410+0000","last_change":"2024-09-20T19:36:49.011748+0000","last_active":"2024-09-20T19:39:33.505410+0000","last_peered":"2024-09-20T19:39:33.505410+0000","last_clean":"2024-09-20T19:39:33.505410+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:39:33.505410+0000","last_undegraded":"2024-09-20T19:39:33.505410+0000","last_fullsized":"2024-09-20T19:39:33.505410+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:49.011625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:49.011625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:53.796400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1045'626","reported_seq":5299,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.519265+0000","last_change":"2024-09-20T19:39:29.929524+0000","last_active":"2024-09-20T19:39:33.519265+0000","last_peered":"2024-09-20T19:39:33.519265+0000","last_clean":"2024-09-20T19:39:33.519265+0000","last_became_active":"2024-09-20T19:33:46.374957+0000","last_became_peered":"2024-09-20T19:33:46.374957+0000","last_unstale":"2024-09-20T19:39:33.519265+0000","last_undegraded":"2024-09-20T19:39:33.519265+0000","last_fullsized":"2024-09-20T19:39:33.519265+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"1045'626","last_scrub_stamp":"2024-09-20T19:39:29.929463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:29.929463+0000","objects_scrubbed":2,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:55.309790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0048363049999999999,"stat_sum":{"num_bytes":2437345,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":70,"num_read_kb":54472,"num_write":52,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":5198758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"987'21","reported_seq":2080,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502681+0000","last_change":"2024-09-20T19:35:49.126133+0000","last_active":"2024-09-20T19:39:33.502681+0000","last_peered":"2024-09-20T19:39:33.502681+0000","last_clean":"2024-09-20T19:39:33.502681+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:39:33.502681+0000","last_undegraded":"2024-09-20T19:39:33.502681+0000","last_fullsized":"2024-09-20T19:39:33.502681+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:35:49.126032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:35:49.126032+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:50.626720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2040,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092454+0000","last_change":"2024-09-20T19:34:51.523032+0000","last_active":"2024-09-20T19:39:34.092454+0000","last_peered":"2024-09-20T19:39:34.092454+0000","last_clean":"2024-09-20T19:39:34.092454+0000","last_became_active":"2024-09-20T19:33:43.351728+0000","last_became_peered":"2024-09-20T19:33:43.351728+0000","last_unstale":"2024-09-20T19:39:34.092454+0000","last_undegraded":"2024-09-20T19:39:34.092454+0000","last_fullsized":"2024-09-20T19:39:34.092454+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:51.522920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:51.522920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.462527+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1549,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746516+0000","last_change":"2024-09-20T19:37:20.195581+0000","last_active":"2024-09-20T19:39:34.746516+0000","last_peered":"2024-09-20T19:39:34.746516+0000","last_clean":"2024-09-20T19:39:34.746516+0000","last_became_active":"2024-09-20T19:33:43.446950+0000","last_became_peered":"2024-09-20T19:33:43.446950+0000","last_unstale":"2024-09-20T19:39:34.746516+0000","last_undegraded":"2024-09-20T19:39:34.746516+0000","last_fullsized":"2024-09-20T19:39:34.746516+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:37:20.195458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:37:20.195458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1796,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746138+0000","last_change":"2024-09-20T19:33:02.559086+0000","last_active":"2024-09-20T19:39:34.746138+0000","last_peered":"2024-09-20T19:39:34.746138+0000","last_clean":"2024-09-20T19:39:34.746138+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:39:34.746138+0000","last_undegraded":"2024-09-20T19:39:34.746138+0000","last_fullsized":"2024-09-20T19:39:34.746138+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:33:02.558937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:02.558937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:01.129003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1006'762","reported_seq":5383,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.499481+0000","last_change":"2024-09-20T19:39:18.553777+0000","last_active":"2024-09-20T19:39:33.499481+0000","last_peered":"2024-09-20T19:39:33.499481+0000","last_clean":"2024-09-20T19:39:33.499481+0000","last_became_active":"2024-09-20T19:33:46.371344+0000","last_became_peered":"2024-09-20T19:33:46.371344+0000","last_unstale":"2024-09-20T19:39:33.499481+0000","last_undegraded":"2024-09-20T19:39:33.499481+0000","last_fullsized":"2024-09-20T19:39:33.499481+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"979'756","last_scrub_stamp":"2024-09-20T19:38:12.650090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.650090+0000","objects_scrubbed":1,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:33.199380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062587600000000004,"stat_sum":{"num_bytes":3966749,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":86291,"num_write":62,"num_write_kb":17826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6030431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"987'27","reported_seq":1896,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.505607+0000","last_change":"2024-09-20T19:38:17.754780+0000","last_active":"2024-09-20T19:39:33.505607+0000","last_peered":"2024-09-20T19:39:33.505607+0000","last_clean":"2024-09-20T19:39:33.505607+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:39:33.505607+0000","last_undegraded":"2024-09-20T19:39:33.505607+0000","last_fullsized":"2024-09-20T19:39:33.505607+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.754717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.754717+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":2068,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503574+0000","last_change":"2024-09-20T19:39:09.812428+0000","last_active":"2024-09-20T19:39:33.503574+0000","last_peered":"2024-09-20T19:39:33.503574+0000","last_clean":"2024-09-20T19:39:33.503574+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:39:33.503574+0000","last_undegraded":"2024-09-20T19:39:33.503574+0000","last_fullsized":"2024-09-20T19:39:33.503574+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:09.812317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:09.812317+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.550714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":2071,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502530+0000","last_change":"2024-09-20T19:39:27.929372+0000","last_active":"2024-09-20T19:39:33.502530+0000","last_peered":"2024-09-20T19:39:33.502530+0000","last_clean":"2024-09-20T19:39:33.502530+0000","last_became_active":"2024-09-20T19:33:43.445406+0000","last_became_peered":"2024-09-20T19:33:43.445406+0000","last_unstale":"2024-09-20T19:39:33.502530+0000","last_undegraded":"2024-09-20T19:39:33.502530+0000","last_fullsized":"2024-09-20T19:39:33.502530+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:27.929219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:27.929219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:55.589653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1553,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746326+0000","last_change":"2024-09-20T19:38:40.086970+0000","last_active":"2024-09-20T19:39:34.746326+0000","last_peered":"2024-09-20T19:39:34.746326+0000","last_clean":"2024-09-20T19:39:34.746326+0000","last_became_active":"2024-09-20T19:33:43.375289+0000","last_became_peered":"2024-09-20T19:33:43.375289+0000","last_unstale":"2024-09-20T19:39:34.746326+0000","last_undegraded":"2024-09-20T19:39:34.746326+0000","last_fullsized":"2024-09-20T19:39:34.746326+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:40.086844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:40.086844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:00.305793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"1048'549","reported_seq":5185,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746611+0000","last_change":"2024-09-20T19:39:18.609629+0000","last_active":"2024-09-20T19:39:34.746611+0000","last_peered":"2024-09-20T19:39:34.746611+0000","last_clean":"2024-09-20T19:39:34.746611+0000","last_became_active":"2024-09-20T19:34:29.368391+0000","last_became_peered":"2024-09-20T19:34:29.368391+0000","last_unstale":"2024-09-20T19:39:34.746611+0000","last_undegraded":"2024-09-20T19:39:34.746611+0000","last_fullsized":"2024-09-20T19:39:34.746611+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'542","last_scrub_stamp":"2024-09-20T19:38:30.128571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:30.128571+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.248720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.091911614000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":8,"num_write":50,"num_write_kb":18704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":14581313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"987'12","reported_seq":1715,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506241+0000","last_change":"2024-09-20T19:39:04.841388+0000","last_active":"2024-09-20T19:39:33.506241+0000","last_peered":"2024-09-20T19:39:33.506241+0000","last_clean":"2024-09-20T19:39:33.506241+0000","last_became_active":"2024-09-20T19:33:43.355421+0000","last_became_peered":"2024-09-20T19:33:43.355421+0000","last_unstale":"2024-09-20T19:39:33.506241+0000","last_undegraded":"2024-09-20T19:39:33.506241+0000","last_fullsized":"2024-09-20T19:39:33.506241+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12","last_scrub_stamp":"2024-09-20T19:39:04.841305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:04.841305+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:30.241113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":2058,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.715203+0000","last_change":"2024-09-20T19:39:24.553000+0000","last_active":"2024-09-20T19:39:34.715203+0000","last_peered":"2024-09-20T19:39:34.715203+0000","last_clean":"2024-09-20T19:39:34.715203+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:39:34.715203+0000","last_undegraded":"2024-09-20T19:39:34.715203+0000","last_fullsized":"2024-09-20T19:39:34.715203+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:24.552929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:24.552929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:36.000003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2041,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091257+0000","last_change":"2024-09-20T19:39:12.350157+0000","last_active":"2024-09-20T19:39:34.091257+0000","last_peered":"2024-09-20T19:39:34.091257+0000","last_clean":"2024-09-20T19:39:34.091257+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:39:34.091257+0000","last_undegraded":"2024-09-20T19:39:34.091257+0000","last_fullsized":"2024-09-20T19:39:34.091257+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:12.350095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:12.350095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:36.679981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2012,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.519342+0000","last_change":"2024-09-20T19:32:30.835674+0000","last_active":"2024-09-20T19:39:33.519342+0000","last_peered":"2024-09-20T19:39:33.519342+0000","last_clean":"2024-09-20T19:39:33.519342+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:39:33.519342+0000","last_undegraded":"2024-09-20T19:39:33.519342+0000","last_fullsized":"2024-09-20T19:39:33.519342+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:32:30.835550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:32:30.835550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.047922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"450'149","reported_seq":4694,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091508+0000","last_change":"2024-09-20T19:39:28.241896+0000","last_active":"2024-09-20T19:39:34.091508+0000","last_peered":"2024-09-20T19:39:34.091508+0000","last_clean":"2024-09-20T19:39:34.091508+0000","last_became_active":"2024-09-20T19:30:12.803423+0000","last_became_peered":"2024-09-20T19:30:12.803423+0000","last_unstale":"2024-09-20T19:39:34.091508+0000","last_undegraded":"2024-09-20T19:39:34.091508+0000","last_fullsized":"2024-09-20T19:39:34.091508+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"450'149","last_scrub_stamp":"2024-09-20T19:39:28.241807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:28.241807+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:34.134322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0057277079999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"987'27","reported_seq":928,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.520696+0000","last_change":"2024-09-20T19:38:17.128241+0000","last_active":"2024-09-20T19:39:33.520696+0000","last_peered":"2024-09-20T19:39:33.520696+0000","last_clean":"2024-09-20T19:39:33.520696+0000","last_became_active":"2024-09-20T19:33:43.351548+0000","last_became_peered":"2024-09-20T19:33:43.351548+0000","last_unstale":"2024-09-20T19:39:33.520696+0000","last_undegraded":"2024-09-20T19:39:33.520696+0000","last_fullsized":"2024-09-20T19:39:33.520696+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:38:17.128056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:17.128056+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2045,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.093054+0000","last_change":"2024-09-20T19:39:21.325794+0000","last_active":"2024-09-20T19:39:34.093054+0000","last_peered":"2024-09-20T19:39:34.093054+0000","last_clean":"2024-09-20T19:39:34.093054+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:39:34.093054+0000","last_undegraded":"2024-09-20T19:39:34.093054+0000","last_fullsized":"2024-09-20T19:39:34.093054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:21.325640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.325640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:50.322842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":2040,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.519568+0000","last_change":"2024-09-20T19:38:35.028245+0000","last_active":"2024-09-20T19:39:33.519568+0000","last_peered":"2024-09-20T19:39:33.519568+0000","last_clean":"2024-09-20T19:39:33.519568+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:39:33.519568+0000","last_undegraded":"2024-09-20T19:39:33.519568+0000","last_fullsized":"2024-09-20T19:39:33.519568+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:35.028101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:35.028101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:56.459400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1561,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.746744+0000","last_change":"2024-09-20T19:36:47.076088+0000","last_active":"2024-09-20T19:39:34.746744+0000","last_peered":"2024-09-20T19:39:34.746744+0000","last_clean":"2024-09-20T19:39:34.746744+0000","last_became_active":"2024-09-20T19:33:43.367417+0000","last_became_peered":"2024-09-20T19:33:43.367417+0000","last_unstale":"2024-09-20T19:39:34.746744+0000","last_undegraded":"2024-09-20T19:39:34.746744+0000","last_fullsized":"2024-09-20T19:39:34.746744+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:47.075960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:47.075960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"1045'846","reported_seq":5502,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502294+0000","last_change":"2024-09-20T19:39:18.521298+0000","last_active":"2024-09-20T19:39:33.502294+0000","last_peered":"2024-09-20T19:39:33.502294+0000","last_clean":"2024-09-20T19:39:33.502294+0000","last_became_active":"2024-09-20T19:33:30.371073+0000","last_became_peered":"2024-09-20T19:33:30.371073+0000","last_unstale":"2024-09-20T19:39:33.502294+0000","last_undegraded":"2024-09-20T19:39:33.502294+0000","last_fullsized":"2024-09-20T19:39:33.502294+0000","mapping_epoch":701,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":5,"last_scrub":"999'811","last_scrub_stamp":"2024-09-20T19:38:37.609506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:37.609506+0000","objects_scrubbed":1,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:39.484493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00069737399999999995,"stat_sum":{"num_bytes":5983919,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":99462,"num_write":87,"num_write_kb":36359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":2018,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518700+0000","last_change":"2024-09-20T19:33:54.210148+0000","last_active":"2024-09-20T19:39:33.518700+0000","last_peered":"2024-09-20T19:39:33.518700+0000","last_clean":"2024-09-20T19:39:33.518700+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:39:33.518700+0000","last_undegraded":"2024-09-20T19:39:33.518700+0000","last_fullsized":"2024-09-20T19:39:33.518700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:33:54.210011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:33:54.210011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:47.063854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":2055,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.712868+0000","last_change":"2024-09-20T19:34:44.070441+0000","last_active":"2024-09-20T19:39:34.712868+0000","last_peered":"2024-09-20T19:39:34.712868+0000","last_clean":"2024-09-20T19:39:34.712868+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:39:34.712868+0000","last_undegraded":"2024-09-20T19:39:34.712868+0000","last_fullsized":"2024-09-20T19:39:34.712868+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:34:44.070328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:34:44.070328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:52.325845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"987'30","reported_seq":1863,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747376+0000","last_change":"2024-09-20T19:38:41.053008+0000","last_active":"2024-09-20T19:39:34.747376+0000","last_peered":"2024-09-20T19:39:34.747376+0000","last_clean":"2024-09-20T19:39:34.747376+0000","last_became_active":"2024-09-20T19:33:43.446858+0000","last_became_peered":"2024-09-20T19:33:43.446858+0000","last_unstale":"2024-09-20T19:39:34.747376+0000","last_undegraded":"2024-09-20T19:39:34.747376+0000","last_fullsized":"2024-09-20T19:39:34.747376+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:38:41.052945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:41.052945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:42.592511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":2317,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.093040+0000","last_change":"2024-09-20T19:38:12.764542+0000","last_active":"2024-09-20T19:39:34.093040+0000","last_peered":"2024-09-20T19:39:34.093040+0000","last_clean":"2024-09-20T19:39:34.093040+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:39:34.093040+0000","last_undegraded":"2024-09-20T19:39:34.093040+0000","last_fullsized":"2024-09-20T19:39:34.093040+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:38:12.764391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:12.764391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.592596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"1047'1125","reported_seq":5974,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.714392+0000","last_change":"2024-09-20T19:39:21.556275+0000","last_active":"2024-09-20T19:39:34.714392+0000","last_peered":"2024-09-20T19:39:34.714392+0000","last_clean":"2024-09-20T19:39:34.714392+0000","last_became_active":"2024-09-20T19:33:46.363878+0000","last_became_peered":"2024-09-20T19:33:46.363878+0000","last_unstale":"2024-09-20T19:39:34.714392+0000","last_undegraded":"2024-09-20T19:39:34.714392+0000","last_fullsized":"2024-09-20T19:39:34.714392+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"1047'1125","last_scrub_stamp":"2024-09-20T19:39:21.556194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.556194+0000","objects_scrubbed":4,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:39.079860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016752500000000001,"stat_sum":{"num_bytes":11770346,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":145,"num_read_kb":111219,"num_write":87,"num_write_kb":32412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2040,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.501976+0000","last_change":"2024-09-20T19:36:23.037110+0000","last_active":"2024-09-20T19:39:33.501976+0000","last_peered":"2024-09-20T19:39:33.501976+0000","last_clean":"2024-09-20T19:39:33.501976+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:39:33.501976+0000","last_undegraded":"2024-09-20T19:39:33.501976+0000","last_fullsized":"2024-09-20T19:39:33.501976+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:23.036982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:23.036982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:41.561071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":886,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091448+0000","last_change":"2024-09-20T19:39:17.377907+0000","last_active":"2024-09-20T19:39:34.091448+0000","last_peered":"2024-09-20T19:39:34.091448+0000","last_clean":"2024-09-20T19:39:34.091448+0000","last_became_active":"2024-09-20T19:33:43.365485+0000","last_became_peered":"2024-09-20T19:33:43.365485+0000","last_unstale":"2024-09-20T19:39:34.091448+0000","last_undegraded":"2024-09-20T19:39:34.091448+0000","last_fullsized":"2024-09-20T19:39:34.091448+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:39:17.377850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:17.377850+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:25.442121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"987'27","reported_seq":2100,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092957+0000","last_change":"2024-09-20T19:38:15.768347+0000","last_active":"2024-09-20T19:39:34.092957+0000","last_peered":"2024-09-20T19:39:34.092957+0000","last_clean":"2024-09-20T19:39:34.092957+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:39:34.092957+0000","last_undegraded":"2024-09-20T19:39:34.092957+0000","last_fullsized":"2024-09-20T19:39:34.092957+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'19","last_scrub_stamp":"2024-09-20T19:38:15.768276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:38:15.768276+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:38.350115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1841,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747620+0000","last_change":"2024-09-20T19:39:23.360560+0000","last_active":"2024-09-20T19:39:34.747620+0000","last_peered":"2024-09-20T19:39:34.747620+0000","last_clean":"2024-09-20T19:39:34.747620+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:39:34.747620+0000","last_undegraded":"2024-09-20T19:39:34.747620+0000","last_fullsized":"2024-09-20T19:39:34.747620+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:23.360497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:39:23.360497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.068129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"1040'740","reported_seq":5197,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.092598+0000","last_change":"2024-09-20T19:39:18.538793+0000","last_active":"2024-09-20T19:39:34.092598+0000","last_peered":"2024-09-20T19:39:34.092598+0000","last_clean":"2024-09-20T19:39:34.092598+0000","last_became_active":"2024-09-20T19:33:46.369895+0000","last_became_peered":"2024-09-20T19:33:46.369895+0000","last_unstale":"2024-09-20T19:39:34.092598+0000","last_undegraded":"2024-09-20T19:39:34.092598+0000","last_fullsized":"2024-09-20T19:39:34.092598+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"1040'740","last_scrub_stamp":"2024-09-20T19:39:16.382726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:39:16.382726+0000","objects_scrubbed":2,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:45.765741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044108000000000003,"stat_sum":{"num_bytes":5950305,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":59453,"num_write":41,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1859,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.506000+0000","last_change":"2024-09-20T19:36:27.046038+0000","last_active":"2024-09-20T19:39:33.506000+0000","last_peered":"2024-09-20T19:39:33.506000+0000","last_clean":"2024-09-20T19:39:33.506000+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:39:33.506000+0000","last_undegraded":"2024-09-20T19:39:33.506000+0000","last_fullsized":"2024-09-20T19:39:33.506000+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:36:27.045955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:36:27.045955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.803434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2056,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.091870+0000","last_change":"2024-09-20T19:39:13.318256+0000","last_active":"2024-09-20T19:39:34.091870+0000","last_peered":"2024-09-20T19:39:34.091870+0000","last_clean":"2024-09-20T19:39:34.091870+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:39:34.091870+0000","last_undegraded":"2024-09-20T19:39:34.091870+0000","last_fullsized":"2024-09-20T19:39:34.091870+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:13.318186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:13.318186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.633019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"987'39","reported_seq":670,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503578+0000","last_change":"2024-09-20T19:39:21.055408+0000","last_active":"2024-09-20T19:39:33.503578+0000","last_peered":"2024-09-20T19:39:33.503578+0000","last_clean":"2024-09-20T19:39:33.503578+0000","last_became_active":"2024-09-20T19:34:26.384789+0000","last_became_peered":"2024-09-20T19:34:26.384789+0000","last_unstale":"2024-09-20T19:39:33.503578+0000","last_undegraded":"2024-09-20T19:39:33.503578+0000","last_fullsized":"2024-09-20T19:39:33.503578+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:39:21.055265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:39:21.055265+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.619827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":2059,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.502887+0000","last_change":"2024-09-20T19:38:33.621829+0000","last_active":"2024-09-20T19:39:33.502887+0000","last_peered":"2024-09-20T19:39:33.502887+0000","last_clean":"2024-09-20T19:39:33.502887+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:39:33.502887+0000","last_undegraded":"2024-09-20T19:39:33.502887+0000","last_fullsized":"2024-09-20T19:39:33.502887+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:33.621779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:33.621779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:43.572657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"695'248","reported_seq":4886,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504391+0000","last_change":"2024-09-20T19:39:18.520779+0000","last_active":"2024-09-20T19:39:33.504391+0000","last_peered":"2024-09-20T19:39:33.504391+0000","last_clean":"2024-09-20T19:39:33.504391+0000","last_became_active":"2024-09-20T19:33:46.364253+0000","last_became_peered":"2024-09-20T19:33:46.364253+0000","last_unstale":"2024-09-20T19:39:33.504391+0000","last_undegraded":"2024-09-20T19:39:33.504391+0000","last_fullsized":"2024-09-20T19:39:33.504391+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":698,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'248","last_scrub_stamp":"2024-09-20T19:38:28.709181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:21:29.215168+0000","last_clean_scrub_stamp":"2024-09-20T19:38:28.709181+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076011000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":686,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:34.747468+0000","last_change":"2024-09-20T19:38:23.240112+0000","last_active":"2024-09-20T19:39:34.747468+0000","last_peered":"2024-09-20T19:39:34.747468+0000","last_clean":"2024-09-20T19:39:34.747468+0000","last_became_active":"2024-09-20T19:33:57.480005+0000","last_became_peered":"2024-09-20T19:33:57.480005+0000","last_unstale":"2024-09-20T19:39:34.747468+0000","last_undegraded":"2024-09-20T19:39:34.747468+0000","last_fullsized":"2024-09-20T19:39:34.747468+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:23.240001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:38:23.240001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:39:46.870584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1533,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.518133+0000","last_change":"2024-09-20T19:39:24.961285+0000","last_active":"2024-09-20T19:39:33.518133+0000","last_peered":"2024-09-20T19:39:33.518133+0000","last_clean":"2024-09-20T19:39:33.518133+0000","last_became_active":"2024-09-20T19:33:43.344198+0000","last_became_peered":"2024-09-20T19:33:43.344198+0000","last_unstale":"2024-09-20T19:39:33.518133+0000","last_undegraded":"2024-09-20T19:39:33.518133+0000","last_fullsized":"2024-09-20T19:39:33.518133+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:39:24.961220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:39:24.961220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:37.658824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"987'24","reported_seq":938,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.503293+0000","last_change":"2024-09-20T19:37:28.963853+0000","last_active":"2024-09-20T19:39:33.503293+0000","last_peered":"2024-09-20T19:39:33.503293+0000","last_clean":"2024-09-20T19:39:33.503293+0000","last_became_active":"2024-09-20T19:33:43.440729+0000","last_became_peered":"2024-09-20T19:33:43.440729+0000","last_unstale":"2024-09-20T19:39:33.503293+0000","last_undegraded":"2024-09-20T19:39:33.503293+0000","last_fullsized":"2024-09-20T19:39:33.503293+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'18","last_scrub_stamp":"2024-09-20T19:37:28.963655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:37:28.963655+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:40:32.875136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1872,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-20T19:39:33.504858+0000","last_change":"2024-09-20T19:38:20.716172+0000","last_active":"2024-09-20T19:39:33.504858+0000","last_peered":"2024-09-20T19:39:33.504858+0000","last_clean":"2024-09-20T19:39:33.504858+0000","last_became_active":"2024-09-20T19:33:43.364247+0000","last_became_peered":"2024-09-20T19:33:43.364247+0000","last_unstale":"2024-09-20T19:39:33.504858+0000","last_undegraded":"2024-09-20T19:39:33.504858+0000","last_fullsized":"2024-09-20T19:39:33.504858+0000","mapping_epoch":714,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:38:20.716039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:38:20.716039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":131855404,"num_objects":54,"num_object_clones":14,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":2,"num_read":3555,"num_read_kb":2772580,"num_write":2479,"num_write_kb":934517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":121924155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368209920,"data_stored":426625972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},"log_size":27130,"ondisk_log_size":27130,"up":108,"acting":108,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907212,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132884,"kb_used_data":61440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93612268,"statfs":{"total":95995035648,"available":95858962432,"internally_reserved":0,"allocated":62914560,"data_stored":67906836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234263,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107600,"kb_used_data":31992,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93637552,"statfs":{"total":95995035648,"available":95884853248,"internally_reserved":0,"allocated":32759808,"data_stored":35069906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561313,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96180,"kb_used_data":42780,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93648972,"statfs":{"total":95995035648,"available":95896547328,"internally_reserved":0,"allocated":43806720,"data_stored":49490147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928623,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89232,"kb_used_data":30836,"kb_used_omap":35,"kb_used_meta":58332,"kb_avail":93655920,"statfs":{"total":95995035648,"available":95903662080,"internally_reserved":0,"allocated":31576064,"data_stored":35459068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":59732913},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2362232012906,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378949,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114676,"kb_used_data":39072,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93630476,"statfs":{"total":95995035648,"available":95877607424,"internally_reserved":0,"allocated":40009728,"data_stored":41207520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575185,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120932,"kb_used_data":49488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93624220,"statfs":{"total":95995035648,"available":95871201280,"internally_reserved":0,"allocated":50675712,"data_stored":52725453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648115,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104064,"kb_used_data":52908,"kb_used_omap":49,"kb_used_meta":51086,"kb_avail":93641088,"statfs":{"total":95995035648,"available":95888474112,"internally_reserved":0,"allocated":54177792,"data_stored":58751920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":52312617},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53010432,"data_stored":58085852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3121,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49508352,"data_stored":52058963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38572032,"data_stored":40278593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62238720,"data_stored":92448331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30416896,"data_stored":34794053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":42053632,"data_stored":48233440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30982144,"data_stored":33813276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61427712,"data_stored":66913464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:39:46.367 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-20T19:39:46.367 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-20T19:39:46.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:46.436 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:46.455 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:46.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:46.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:46.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:46.712 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:46.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:46.841 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:46.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:46.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:47.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:47.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:47.205 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:47.233 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:47.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:47.364 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:47.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:47.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:47.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:47.625 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:47.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:47.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:47.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:47.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:47.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:47 smithi042 bash[32075]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:47.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:47 smithi042 bash[32075]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:47 smithi042 bash[22706]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:47.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:47 smithi042 bash[22706]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:47.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:48.000 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:47 smithi162 bash[24557]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:48.000 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:47 smithi162 bash[24557]: cluster 2024-09-20T19:39:46.513377+0000 mgr.y (mgr.24491) 817 : cluster [DBG] pgmap v1595: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:48.016 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:48.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:48.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:48.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:48.276 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:48.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:39:48.379 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:48.433 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:48.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:48.556 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:48.659 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:48.680 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:48.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:48.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:48.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:48.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:48.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:49.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:49.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.198 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:49.301 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:49.329 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:49.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-20T19:39:49.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:49.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:49.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:49.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.574 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.576 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:49 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.586 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:49.689 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:48.514522+0000 mgr.y (mgr.24491) 818 : cluster [DBG] pgmap v1596: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.123551+0000 osd.6 (osd.6) 262 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:49 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.125054+0000 osd.6 (osd.6) 263 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:39:49.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:49.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:49.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:50.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:50.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.208 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:39:50 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:39:50.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:50.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:50.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.481 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:50.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.609 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:50.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:50 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.723 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:50 smithi162 bash[24557]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:50.841 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:50.853 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:50 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.853 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:50 smithi042 bash[32075]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:50 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.853 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:50 smithi042 bash[22706]: cluster 2024-09-20T19:39:49.513930+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2024-09-20T19:39:50.869 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:50.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:50.905 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:50.905 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:50.906 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:50.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:51.104 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:51.133 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:51.236 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:51.265 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:51.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:51.399 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:51.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:51.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:51.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:51.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[22706]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[22706]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[22706]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[22706]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[32075]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[32075]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[32075]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:51 smithi042 bash[32075]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.761 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:51 smithi162 bash[24557]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:51 smithi162 bash[24557]: audit 2024-09-20T19:39:50.000120+0000 mgr.y (mgr.24491) 819 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:39:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:51 smithi162 bash[24557]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:51 smithi162 bash[24557]: cluster 2024-09-20T19:39:50.515181+0000 mgr.y (mgr.24491) 820 : cluster [DBG] pgmap v1598: 178 pgs: 178 active+clean; 126 MiB data, 748 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:39:51.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:51.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:51.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:52.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:52.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:52.152 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:52.284 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:52.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:52.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:52.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.570 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:52.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:52.701 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:52.804 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:52.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:52.940 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:52.958 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:52.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:53.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:53.084 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:53.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:53.214 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:53.226 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0: {} 2024-09-20T19:39:53.227 INFO:teuthology.orchestra.run.smithi042.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T19:39:53.244 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1: {} 2024-09-20T19:39:53.244 INFO:teuthology.orchestra.run.smithi042.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T19:39:53.261 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2: {} 2024-09-20T19:39:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T19:39:53.278 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3: {} 2024-09-20T19:39:53.278 INFO:teuthology.orchestra.run.smithi042.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T19:39:53.297 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4: {} 2024-09-20T19:39:53.297 INFO:teuthology.orchestra.run.smithi042.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T19:39:53.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.319 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5: {} 2024-09-20T19:39:53.319 INFO:teuthology.orchestra.run.smithi042.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T19:39:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.339 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6: {} 2024-09-20T19:39:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T19:39:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:53.361 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7: {} 2024-09-20T19:39:53.361 INFO:teuthology.orchestra.run.smithi042.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T19:39:53.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:53.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:53.573 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.590 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:53.617 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:39:53.617 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:39:53.624 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[22706]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[22706]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[22706]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[22706]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[32075]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[32075]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[32075]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.625 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:53 smithi042 bash[32075]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.715 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:53.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:53 smithi162 bash[24557]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:53 smithi162 bash[24557]: cluster 2024-09-20T19:39:52.516732+0000 mgr.y (mgr.24491) 821 : cluster [DBG] pgmap v1599: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:39:53.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:53 smithi162 bash[24557]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:53 smithi162 bash[24557]: audit 2024-09-20T19:39:53.206802+0000 mon.a (mon.0) 2745 : audit [DBG] from='client.? v1:172.21.15.42:0/1151127411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-20T19:39:53.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:53.841 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:53.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:53.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:53.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:54.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:54.102 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:54.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:54.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:54.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:54.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:54.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'unique_pool_0' removed 2024-09-20T19:39:54.590 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[22706]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[22706]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[22706]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[22706]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[32075]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[32075]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[32075]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.600 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:54 smithi042 bash[32075]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.611 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:54.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:54.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:54 smithi162 bash[24557]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:54 smithi162 bash[24557]: audit 2024-09-20T19:39:54.437789+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.42:0/110375290' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:54 smithi162 bash[24557]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.725 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:54 smithi162 bash[24557]: audit 2024-09-20T19:39:54.438281+0000 mon.a (mon.0) 2746 : audit [INF] from='client.26231 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-20T19:39:54.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:54.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:54.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:54.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:54.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7 2024-09-20T19:39:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.7' 2024-09-20T19:39:55.104 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:55.139 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0 2024-09-20T19:39:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.0' 2024-09-20T19:39:55.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:55.269 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:55.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.401 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:55.504 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.530 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:55.633 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.645 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[32075]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:55 smithi042 bash[22706]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:55.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.793 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: cluster 2024-09-20T19:39:54.517719+0000 mgr.y (mgr.24491) 822 : cluster [DBG] pgmap v1600: 178 pgs: 178 active+clean; 126 MiB data, 747 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: audit 2024-09-20T19:39:54.536037+0000 mon.a (mon.0) 2747 : audit [INF] from='client.26231 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:55 smithi162 bash[24557]: cluster 2024-09-20T19:39:54.542620+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2024-09-20T19:39:55.896 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.924 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:55.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:56.027 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3 2024-09-20T19:39:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.3' 2024-09-20T19:39:56.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:56.185 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:56.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:56.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:56.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2 2024-09-20T19:39:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.2' 2024-09-20T19:39:56.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:56.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5 2024-09-20T19:39:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.5' 2024-09-20T19:39:56.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:39:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:39:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:39:56.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:56.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6 2024-09-20T19:39:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.6' 2024-09-20T19:39:56.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4 2024-09-20T19:39:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.4' 2024-09-20T19:39:56.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:56.972 INFO:teuthology.orchestra.run.smithi042.stderr:Error response from daemon: Cannot kill container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1 2024-09-20T19:39:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:39:56.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi042 with status 1: 'sudo docker kill -s 1 ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd.1' 2024-09-20T19:39:57.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T19:39:57.067 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-20T19:39:57.084 INFO:tasks.thrashosds:joining thrashosds 2024-09-20T19:39:57.522 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:39:57.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:57 smithi042 bash[22706]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:57.544 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:57 smithi042 bash[22706]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:57.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:57 smithi162 bash[24557]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:57.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:57 smithi162 bash[24557]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:57.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:57 smithi042 bash[32075]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:57.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:57 smithi042 bash[32075]: cluster 2024-09-20T19:39:56.518393+0000 mgr.y (mgr.24491) 823 : cluster [DBG] pgmap v1602: 142 pgs: 142 active+clean; 583 KiB data, 747 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:39:58.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:58 smithi162 bash[24557]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:58.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:58 smithi162 bash[24557]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:58.807 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:39:58 smithi162 bash[51183]: logger=cleanup t=2024-09-20T19:39:58.470286711Z level=info msg="Completed cleanup jobs" duration=77.758139ms 2024-09-20T19:39:58.808 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:39:58 smithi162 bash[51183]: logger=plugins.update.checker t=2024-09-20T19:39:58.540304477Z level=info msg="Update check succeeded" duration=68.457071ms 2024-09-20T19:39:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:58 smithi042 bash[32075]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:58.862 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:58 smithi042 bash[32075]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:58.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:58 smithi042 bash[22706]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:58.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:58 smithi042 bash[22706]: audit 2024-09-20T19:39:57.618994+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:39:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:39:59 smithi162 bash[24557]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[32075]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.519859+0000 mgr.y (mgr.24491) 824 : cluster [DBG] pgmap v1603: 142 pgs: 142 active+clean; 583 KiB data, 483 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.543331+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: audit 2024-09-20T19:39:58.554729+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? v1:172.21.15.42:0/3725453315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.558906+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.659074+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.661545+0000 osd.4 (osd.4) 208 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:39:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:39:59 smithi042 bash[22706]: audit 2024-09-20T19:39:58.777634+0000 mon.a (mon.0) 2753 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:00.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:00.408 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:00.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:00.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.911396+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:58.913645+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.357727+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.359626+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.625791+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.628128+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.664307+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.666517+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.000143+0000 mon.a (mon.0) 2754 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.000180+0000 mon.a (mon.0) 2755 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-20T19:40:01.756 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[32075]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:01 smithi042 bash[22706]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:02.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.960604+0000 osd.1 (osd.1) 379 : cluster [DBG] 4.f scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:39:59.965196+0000 osd.1 (osd.1) 380 : cluster [DBG] 4.f scrub ok 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: audit 2024-09-20T19:40:00.008862+0000 mgr.y (mgr.24491) 825 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.327083+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.331311+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.520700+0000 mgr.y (mgr.24491) 826 : cluster [DBG] pgmap v1605: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.598670+0000 osd.4 (osd.4) 211 : cluster [DBG] 2.f scrub starts 2024-09-20T19:40:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:01 smithi162 bash[24557]: cluster 2024-09-20T19:40:00.601851+0000 osd.4 (osd.4) 212 : cluster [DBG] 2.f scrub ok 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[22706]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:02.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:02.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:02.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:02 smithi042 bash[32075]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:03.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:03.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.569694+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.c scrub starts 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.573566+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.c scrub ok 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.702343+0000 osd.5 (osd.5) 343 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.705929+0000 osd.5 (osd.5) 344 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.914918+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:02 smithi162 bash[24557]: cluster 2024-09-20T19:40:01.916946+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:40:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:03.854 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:03.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:03.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:03.855 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:04.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:04.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:04.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:04.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:03 smithi162 bash[24557]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[22706]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.272708+0000 osd.0 (osd.0) 149 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.274469+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:03 smithi042 bash[32075]: cluster 2024-09-20T19:40:02.522125+0000 mgr.y (mgr.24491) 827 : cluster [DBG] pgmap v1606: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail 2024-09-20T19:40:04.293 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:04 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:05.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.267857+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.270715+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.730959+0000 osd.5 (osd.5) 345 : cluster [DBG] 6.d scrub starts 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:03.733285+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.d scrub ok 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.054039+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:04 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.056552+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:40:05.833 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:05.834 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:05.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[22706]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:05.845 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:05 smithi042 bash[32075]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:06.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:06.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.523156+0000 mgr.y (mgr.24491) 828 : cluster [DBG] pgmap v1607: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:40:06.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.584868+0000 osd.4 (osd.4) 215 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.589566+0000 osd.4 (osd.4) 216 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.706101+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:05 smithi162 bash[24557]: cluster 2024-09-20T19:40:04.708281+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:40:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:06.589 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:06 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:07.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:05.599488+0000 osd.4 (osd.4) 217 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:05.602255+0000 osd.4 (osd.4) 218 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.045183+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:06 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.052209+0000 osd.7 (osd.7) 260 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:40:07.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T19:40:07.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[22706]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:07.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:07.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:07 smithi042 bash[32075]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.523864+0000 mgr.y (mgr.24491) 829 : cluster [DBG] pgmap v1608: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.565742+0000 osd.4 (osd.4) 219 : cluster [DBG] 5.e scrub starts 2024-09-20T19:40:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:07 smithi162 bash[24557]: cluster 2024-09-20T19:40:06.568809+0000 osd.4 (osd.4) 220 : cluster [DBG] 5.e scrub ok 2024-09-20T19:40:08.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-20T19:40:08.696 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[32075]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:08.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:08.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:08.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:08.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:08.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:08.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:08 smithi042 bash[22706]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:09.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:09.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.418168+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.421298+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.744988+0000 osd.5 (osd.5) 349 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:08 smithi162 bash[24557]: cluster 2024-09-20T19:40:07.747348+0000 osd.5 (osd.5) 350 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:40:09.603 INFO:teuthology.orchestra.run.smithi042.stderr:noscrub is unset 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[22706]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:09 smithi042 bash[32075]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.238845+0000 osd.0 (osd.0) 153 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.241262+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:10.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: cluster 2024-09-20T19:40:08.525335+0000 mgr.y (mgr.24491) 830 : cluster [DBG] pgmap v1609: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:10.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:10.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:09 smithi162 bash[24557]: audit 2024-09-20T19:40:09.071173+0000 mon.a (mon.0) 2756 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-20T19:40:10.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:10.564 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:10.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:10.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:10 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:11.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: audit 2024-09-20T19:40:09.602272+0000 mon.a (mon.0) 2757 : audit [INF] from='client.? v1:172.21.15.42:0/3626413220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:09.609331+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.065891+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:10 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.068731+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:40:11.877 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:11 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: audit 2024-09-20T19:40:10.020364+0000 mgr.y (mgr.24491) 831 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:10.525840+0000 mgr.y (mgr.24491) 832 : cluster [DBG] pgmap v1611: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.065157+0000 osd.7 (osd.7) 261 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:40:11.978 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:11.979 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:11 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.069057+0000 osd.7 (osd.7) 262 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:40:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[32075]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:12 smithi042 bash[22706]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:13.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:13.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.586595+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:13.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:13.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.590890+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:13.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:12 smithi162 bash[24557]: cluster 2024-09-20T19:40:11.639242+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2024-09-20T19:40:13.461 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:13.461 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[32075]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:13.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:13 smithi042 bash[22706]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:14.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.527118+0000 mgr.y (mgr.24491) 833 : cluster [DBG] pgmap v1613: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.740771+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:12.743743+0000 osd.5 (osd.5) 352 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:13.075360+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:13 smithi162 bash[24557]: cluster 2024-09-20T19:40:13.077265+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:40:14.092 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:14.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T19:40:14.656 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:14.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:14.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:14.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:14.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:14.657 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:14 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: audit 2024-09-20T19:40:13.807709+0000 mon.a (mon.0) 2760 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:15.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.086458+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:40:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:14 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.089006+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:40:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:15.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:15 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:15.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:15 smithi042 bash[32075]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:15 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:15.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:15 smithi042 bash[22706]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:15 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:15 smithi162 bash[24557]: cluster 2024-09-20T19:40:14.528123+0000 mgr.y (mgr.24491) 834 : cluster [DBG] pgmap v1614: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:16.614 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:17.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-20T19:40:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:nodeep-scrub is unset 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:17.678 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[32075]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:17.679 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:17.680 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:17 smithi042 bash[22706]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.528545+0000 mgr.y (mgr.24491) 835 : cluster [DBG] pgmap v1615: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.979326+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:16.980815+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.018405+0000 osd.2 (osd.2) 347 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.021744+0000 osd.2 (osd.2) 348 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.073056+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:40:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.075181+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:40:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:17 smithi162 bash[24557]: audit 2024-09-20T19:40:17.184753+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[22706]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:18.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:18.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:18 smithi042 bash[32075]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:18.946 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.592586+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.e scrub starts 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.594835+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.e scrub ok 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.655131+0000 mon.a (mon.0) 2762 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.655158+0000 mon.a (mon.0) 2763 : cluster [INF] Cluster is now healthy 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:19.050 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: audit 2024-09-20T19:40:17.656865+0000 mon.a (mon.0) 2764 : audit [INF] from='client.? v1:172.21.15.42:0/3901237370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T19:40:19.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:19.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: cluster 2024-09-20T19:40:17.660883+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2024-09-20T19:40:19.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:19.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:18 smithi162 bash[24557]: audit 2024-09-20T19:40:18.046097+0000 mon.a (mon.0) 2766 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[32075]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:19.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:19.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:19 smithi042 bash[22706]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.529342+0000 mgr.y (mgr.24491) 836 : cluster [DBG] pgmap v1617: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.774977+0000 mon.a (mon.0) 2767 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.775919+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: audit 2024-09-20T19:40:18.783699+0000 mon.a (mon.0) 2769 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.982289+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.c scrub starts 2024-09-20T19:40:20.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:20.031 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:19 smithi162 bash[24557]: cluster 2024-09-20T19:40:18.984301+0000 osd.1 (osd.1) 386 : cluster [DBG] 5.c scrub ok 2024-09-20T19:40:20.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:20 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:20.312 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:20.312 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:21.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T19:40:21.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[32075]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[32075]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[32075]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[32075]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[22706]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[22706]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[22706]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:21 smithi042 bash[22706]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:22.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:21 smithi162 bash[24557]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:22.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:21 smithi162 bash[24557]: audit 2024-09-20T19:40:20.027456+0000 mgr.y (mgr.24491) 837 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:22.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:21 smithi162 bash[24557]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:22.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:21 smithi162 bash[24557]: cluster 2024-09-20T19:40:20.530243+0000 mgr.y (mgr.24491) 838 : cluster [DBG] pgmap v1618: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:40:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:23.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:22 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:21.830031+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub starts 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:21.832249+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub ok 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.114245+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:22 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.116375+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:40:23.939 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:23.940 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:23.941 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:23 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:24.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:24.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.531548+0000 mgr.y (mgr.24491) 839 : cluster [DBG] pgmap v1619: 142 pgs: 142 active+clean; 583 KiB data, 454 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.965446+0000 osd.1 (osd.1) 387 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:22.968317+0000 osd.1 (osd.1) 388 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.151343+0000 osd.7 (osd.7) 273 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:23 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.153554+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:40:24.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-20T19:40:25.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:24 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:24 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:24 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:24 smithi162 bash[24557]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[32075]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.103785+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:24 smithi042 bash[22706]: cluster 2024-09-20T19:40:23.107168+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:40:25.934 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:25 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.135338+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.137659+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.532541+0000 mgr.y (mgr.24491) 840 : cluster [DBG] pgmap v1620: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.834057+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub starts 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:24.836301+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.a scrub ok 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.489438+0000 osd.2 (osd.2) 349 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:25 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.494246+0000 osd.2 (osd.2) 350 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:40:26.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:26 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.968243+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:25.971614+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:40:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.032861+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:40:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:26 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.035901+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:40:27.237 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:27.237 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:27.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T19:40:27.958 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:27 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:27.958 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:27 smithi042 bash[22706]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:27.958 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:27 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:27.958 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:27 smithi042 bash[32075]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:28.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:27 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:28.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:27 smithi162 bash[24557]: cluster 2024-09-20T19:40:26.533219+0000 mgr.y (mgr.24491) 841 : cluster [DBG] pgmap v1621: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:29.520 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:29.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:29 smithi162 bash[24557]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[22706]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:27.956952+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:27.959447+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.534878+0000 mgr.y (mgr.24491) 842 : cluster [DBG] pgmap v1622: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: audit 2024-09-20T19:40:28.844018+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:29.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:29 smithi042 bash[32075]: audit 2024-09-20T19:40:28.846257+0000 mon.a (mon.0) 2771 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:30.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:30 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:30.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:30 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:30 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:30 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:30.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:30 smithi162 bash[24557]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:30.873 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.943 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.943 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:30.943 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[32075]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:30.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.954098+0000 osd.1 (osd.1) 393 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:40:30.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:30.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:30 smithi042 bash[22706]: cluster 2024-09-20T19:40:28.956502+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:31.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:31 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:31.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.810 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:31.811 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: audit 2024-09-20T19:40:30.039029+0000 mgr.y (mgr.24491) 843 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:30.535898+0000 mgr.y (mgr.24491) 844 : cluster [DBG] pgmap v1623: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.109366+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.111426+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.152459+0000 osd.7 (osd.7) 275 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:31.812 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:31 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.154843+0000 osd.7 (osd.7) 276 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:40:32.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:32 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:32 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:32 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:32.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:32 smithi162 bash[24557]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:32.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[22706]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.887635+0000 osd.5 (osd.5) 357 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:32 smithi042 bash[32075]: cluster 2024-09-20T19:40:31.890201+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:40:33.314 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:33.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:33 smithi042 bash[32075]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:33.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:33 smithi042 bash[32075]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:33 smithi042 bash[22706]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:33.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:33 smithi042 bash[22706]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:34.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:33 smithi162 bash[24557]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:34.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:33 smithi162 bash[24557]: cluster 2024-09-20T19:40:32.537477+0000 mgr.y (mgr.24491) 845 : cluster [DBG] pgmap v1624: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:34.860 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[32075]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:34.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:34.862 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:34 smithi042 bash[22706]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:34.993 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:34.994 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.074449+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.076887+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.514326+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.d scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.517719+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.d scrub ok 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.579176+0000 osd.4 (osd.4) 223 : cluster [DBG] 4.d scrub starts 2024-09-20T19:40:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:34 smithi162 bash[24557]: cluster 2024-09-20T19:40:33.584388+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.d scrub ok 2024-09-20T19:40:35.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:35.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[32075]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:35.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:35 smithi042 bash[22706]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.538271+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.538407+0000 mgr.y (mgr.24491) 846 : cluster [DBG] pgmap v1625: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.540562+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.980487+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:40:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:35 smithi162 bash[24557]: cluster 2024-09-20T19:40:34.983338+0000 osd.1 (osd.1) 396 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:40:36.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:36.813 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-20T19:40:36.813 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-20T19:40:36.814 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd in 3 2024-09-20T19:40:37.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:37.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:37.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:37 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:38.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.513026+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b scrub starts 2024-09-20T19:40:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.515612+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.b scrub ok 2024-09-20T19:40:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:37 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.539133+0000 mgr.y (mgr.24491) 847 : cluster [DBG] pgmap v1626: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:38.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:38.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:38.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:38.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:38.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:38.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:38.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:38 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:39.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:39.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.927616+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:40:39.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:39.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:36.930411+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:40:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.014390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.a scrub starts 2024-09-20T19:40:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:38 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.016899+0000 osd.1 (osd.1) 398 : cluster [DBG] 2.a scrub ok 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:39.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:39 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:40.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:40.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:38.540659+0000 mgr.y (mgr.24491) 848 : cluster [DBG] pgmap v1627: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:40.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:40.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.109586+0000 osd.6 (osd.6) 270 : cluster [DBG] 6.c scrub starts 2024-09-20T19:40:40.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:40.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:39 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.111717+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.c scrub ok 2024-09-20T19:40:40.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:40.615 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:40.616 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:40.922 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:40.923 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:40.924 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:40 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.566212+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.570308+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.069070+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.072455+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.088936+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.091186+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.534267+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:40 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.536838+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[32075]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:41.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:41 smithi042 bash[22706]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.939151+0000 osd.5 (osd.5) 361 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:39.942007+0000 osd.5 (osd.5) 362 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: audit 2024-09-20T19:40:40.050452+0000 mgr.y (mgr.24491) 849 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.541636+0000 mgr.y (mgr.24491) 850 : cluster [DBG] pgmap v1628: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.581972+0000 osd.4 (osd.4) 227 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:40:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:41 smithi162 bash[24557]: cluster 2024-09-20T19:40:40.584202+0000 osd.4 (osd.4) 228 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:40:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:42.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:42.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:42.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:42 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:42.957 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:42.957 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.153212+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.156321+0000 osd.6 (osd.6) 275 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.555432+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:42 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.557674+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:40:43.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: debug 2024-09-20T19:40:43.204+0000 7fc23bce4640 -1 mon.a@0(leader).osd e1064 definitely_dead 0 2024-09-20T19:40:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:marked in osd.3. 2024-09-20T19:40:43.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T19:40:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:43.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:43.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:43.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:43.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:43 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:44.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:44.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:41.947412+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub starts 2024-09-20T19:40:44.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:44.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:41.949940+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.a scrub ok 2024-09-20T19:40:44.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:42.543038+0000 mgr.y (mgr.24491) 851 : cluster [DBG] pgmap v1629: 142 pgs: 142 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.116356+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.118565+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: audit 2024-09-20T19:40:43.205873+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.42:0/3577947718' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: audit 2024-09-20T19:40:43.206472+0000 mon.a (mon.0) 2772 : audit [DBG] from='client.26285 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.586117+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:43 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.588367+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:40:44.742 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-20T19:40:44.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:44.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:44.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:44.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:44.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:44 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.122440+0000 osd.0 (osd.0) 161 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:40:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.126715+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.241235+0000 osd.4 (osd.4) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.244128+0000 osd.4 (osd.4) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: audit 2024-09-20T19:40:43.635302+0000 mon.a (mon.0) 2773 : audit [INF] from='client.26285 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:43.643093+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e1065: 8 total, 8 up, 8 in 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: audit 2024-09-20T19:40:43.875912+0000 mon.a (mon.0) 2775 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.040042+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:40:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.042347+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.171355+0000 osd.7 (osd.7) 277 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.173373+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:45.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:44 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.642120+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e1066: 8 total, 8 up, 8 in 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:45 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:44.543871+0000 mgr.y (mgr.24491) 852 : cluster [DBG] pgmap v1631: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:40:46.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.212823+0000 osd.7 (osd.7) 279 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:40:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:45 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.215263+0000 osd.7 (osd.7) 280 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:40:46.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:47.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:46 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.088288+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:45.092231+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.055989+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.d scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.058517+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.d scrub ok 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.210320+0000 osd.6 (osd.6) 278 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.215306+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.223696+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.226027+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.587978+0000 osd.2 (osd.2) 363 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:47.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:46 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.589239+0000 osd.2 (osd.2) 364 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:40:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:47 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.544397+0000 mgr.y (mgr.24491) 853 : cluster [DBG] pgmap v1633: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 453 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.687404+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.691138+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:40:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.272679+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:40:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:47 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.274974+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:40:48.554 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:48.554 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:48 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.985950+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:46.988344+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.068468+0000 osd.0 (osd.0) 165 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.070754+0000 osd.0 (osd.0) 166 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.283536+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:47.286657+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.062232+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.064099+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.568025+0000 osd.2 (osd.2) 365 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:49.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:48 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.569594+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:40:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:49 smithi162 bash[24557]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[22706]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.311460+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.313856+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:40:50.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:48.545919+0000 mgr.y (mgr.24491) 854 : cluster [DBG] pgmap v1634: 142 pgs: 7 peering, 135 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 9 objects/s recovering 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.091169+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.093331+0000 osd.1 (osd.1) 408 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.215611+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.219513+0000 osd.6 (osd.6) 281 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.296471+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:49 smithi042 bash[32075]: cluster 2024-09-20T19:40:49.300595+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:40:50.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:40:50.882 INFO:teuthology.orchestra.run.smithi042.stdout:ERROR: (22) Invalid argument 2024-09-20T19:40:50.882 INFO:teuthology.orchestra.run.smithi042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T19:40:50.899 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a: {} 2024-09-20T19:40:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:40:50.914 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b: {} 2024-09-20T19:40:50.914 INFO:teuthology.orchestra.run.smithi042.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:40:50.922 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c: {} 2024-09-20T19:40:50.923 INFO:teuthology.orchestra.run.smithi042.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T19:40:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:50 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.105 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.106 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.122174+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.125075+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.265489+0000 osd.6 (osd.6) 282 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.267969+0000 osd.6 (osd.6) 283 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:40:51.107 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.310460+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.312905+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.605163+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.f scrub starts 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:50 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.607409+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.f scrub ok 2024-09-20T19:40:51.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:51.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:51.730 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:51.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:51.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:51.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:51.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:51.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:51.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:51.734 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[22706]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:52.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:52.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:52.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:52.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:52.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:51 smithi162 bash[24557]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: audit 2024-09-20T19:40:50.060845+0000 mgr.y (mgr.24491) 855 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:50.546862+0000 mgr.y (mgr.24491) 856 : cluster [DBG] pgmap v1635: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: audit 2024-09-20T19:40:50.886428+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.42:0/2654405204' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:51.216976+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:40:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:52.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:51 smithi042 bash[32075]: cluster 2024-09-20T19:40:51.219889+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:40:52.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:52 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:53.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.048260+0000 osd.6 (osd.6) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.050885+0000 osd.6 (osd.6) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.196282+0000 osd.6 (osd.6) 288 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:53.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:52 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.198600+0000 osd.6 (osd.6) 289 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:40:54.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:53 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.054664+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.057049+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.079008+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.083880+0000 osd.0 (osd.0) 168 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.548216+0000 mgr.y (mgr.24491) 857 : cluster [DBG] pgmap v1636: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 65 KiB/s, 10 objects/s recovering 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:52.744284+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e1067: 8 total, 8 up, 8 in 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.232706+0000 osd.1 (osd.1) 411 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:54.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:53 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.236322+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:54 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.087034+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.089383+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:40:55.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.753626+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1068: 8 total, 8 up, 8 in 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.878864+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:53.881380+0000 osd.3 (osd.3) 62 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.204657+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.a scrub starts 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:55.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:54 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.210834+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.a scrub ok 2024-09-20T19:40:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:55 smithi162 bash[24557]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[22706]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.084285+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.087453+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.549216+0000 mgr.y (mgr.24491) 858 : cluster [DBG] pgmap v1639: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 72 KiB/s, 11 objects/s recovering 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.741521+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:40:56.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.745429+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:40:56.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:55 smithi042 bash[32075]: cluster 2024-09-20T19:40:54.756030+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e1069: 8 total, 8 up, 8 in 2024-09-20T19:40:56.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:40:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:40:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:56 smithi162 bash[24557]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:56 smithi162 bash[24557]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:56 smithi162 bash[24557]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:56 smithi162 bash[24557]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[32075]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[32075]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[32075]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[32075]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[22706]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[22706]: cluster 2024-09-20T19:40:55.113138+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[22706]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:57.179 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:56 smithi042 bash[22706]: cluster 2024-09-20T19:40:55.117455+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:57 smithi162 bash[24557]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[32075]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.054588+0000 osd.0 (osd.0) 173 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.056994+0000 osd.0 (osd.0) 174 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:56.549800+0000 mgr.y (mgr.24491) 859 : cluster [DBG] pgmap v1641: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:57.109861+0000 osd.1 (osd.1) 413 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:57 smithi042 bash[22706]: cluster 2024-09-20T19:40:57.112087+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:40:59.532 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T19:40:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:40:59.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:40:59 smithi162 bash[24557]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[32075]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.100242+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.102608+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.105220+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.109289+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: cluster 2024-09-20T19:40:58.551951+0000 mgr.y (mgr.24491) 860 : cluster [DBG] pgmap v1642: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: audit 2024-09-20T19:40:58.902259+0000 mon.a (mon.0) 2780 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:40:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:40:59 smithi042 bash[22706]: audit 2024-09-20T19:40:59.067078+0000 mon.a (mon.0) 2781 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:00.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-20T19:41:00.546 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:00.685 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:00.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.147248+0000 osd.1 (osd.1) 417 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.149886+0000 osd.1 (osd.1) 418 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.389421+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.391805+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.432230+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.434761+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: audit 2024-09-20T19:40:59.530684+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? v1:172.21.15.42:0/974648136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.536282+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e1070: 8 total, 8 up, 8 in 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.541581+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:40:59.545627+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.155000+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:41:00.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.157517+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:41:00.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.237134+0000 osd.7 (osd.7) 285 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:41:00.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:00.811 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.239849+0000 osd.7 (osd.7) 286 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:41:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:01.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:01.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[22706]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:01.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:01.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:01.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:01.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:01.937 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:01 smithi042 bash[32075]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.059211+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.063642+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: audit 2024-09-20T19:41:00.072211+0000 mgr.y (mgr.24491) 861 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.111888+0000 osd.0 (osd.0) 177 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.116372+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.396959+0000 osd.3 (osd.3) 65 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.398418+0000 osd.3 (osd.3) 66 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:41:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.552498+0000 mgr.y (mgr.24491) 862 : cluster [DBG] pgmap v1644: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.558206+0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1071: 8 total, 8 up, 8 in 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.706021+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:02.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:01 smithi162 bash[24557]: cluster 2024-09-20T19:41:00.714004+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:02 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.017466+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.020003+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.148202+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:01.151001+0000 osd.0 (osd.0) 180 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.189194+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:02 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.191508+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:03.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:03 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.037066+0000 osd.5 (osd.5) 377 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.040346+0000 osd.5 (osd.5) 378 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.366684+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.b scrub starts 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.369068+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.b scrub ok 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:02.553827+0000 mgr.y (mgr.24491) 863 : cluster [DBG] pgmap v1646: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.234872+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:03 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.237700+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:04 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.168605+0000 osd.0 (osd.0) 181 : cluster [DBG] 4.b scrub starts 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:03.171702+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.b scrub ok 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.185423+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.b scrub starts 2024-09-20T19:41:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:04 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.187690+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.b scrub ok 2024-09-20T19:41:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:05.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[22706]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:05.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:05 smithi042 bash[32075]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.168486+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.170719+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.554826+0000 mgr.y (mgr.24491) 864 : cluster [DBG] pgmap v1647: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.824485+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:04.826716+0000 osd.4 (osd.4) 238 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:05.195248+0000 osd.6 (osd.6) 294 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:05 smithi162 bash[24557]: cluster 2024-09-20T19:41:05.197663+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:41:06.593 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:06.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:06 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:06 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:06 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.184321+0000 osd.6 (osd.6) 296 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:41:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:06 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:06 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.186662+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:41:07.595 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[32075]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[32075]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[32075]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[22706]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[22706]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:07.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:07 smithi042 bash[22706]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:07 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:07 smithi162 bash[24557]: cluster 2024-09-20T19:41:06.555576+0000 mgr.y (mgr.24491) 865 : cluster [DBG] pgmap v1648: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:07 smithi162 bash[24557]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:07 smithi162 bash[24557]: audit 2024-09-20T19:41:07.057881+0000 mon.a (mon.0) 2785 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-20T19:41:08.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T19:41:08.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:08.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:08.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:08.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:08.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:08.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:08 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.406715+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:41:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.409077+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.423655+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.425782+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: audit 2024-09-20T19:41:07.594028+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? v1:172.21.15.42:0/872603242' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.600112+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1072: 8 total, 8 up, 8 in 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.889631+0000 osd.4 (osd.4) 239 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:07.891941+0000 osd.4 (osd.4) 240 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.183345+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:08 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.185733+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:09.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[22706]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:09 smithi042 bash[32075]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.430775+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.c scrub starts 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.432888+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.c scrub ok 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.556373+0000 mgr.y (mgr.24491) 866 : cluster [DBG] pgmap v1650: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.877136+0000 osd.4 (osd.4) 241 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:41:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:09 smithi162 bash[24557]: cluster 2024-09-20T19:41:08.881306+0000 osd.4 (osd.4) 242 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:41:10.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:10.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:10.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:10 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.049619+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.053097+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.116890+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:41:11.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:11.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:10 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.119222+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:41:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:11.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:11.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:11.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:11.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:11.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:11 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: audit 2024-09-20T19:41:10.083601+0000 mgr.y (mgr.24491) 867 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.382831+0000 osd.2 (osd.2) 375 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.387515+0000 osd.2 (osd.2) 376 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:10.557366+0000 mgr.y (mgr.24491) 868 : cluster [DBG] pgmap v1651: 142 pgs: 142 active+clean; 583 KiB data, 506 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.057111+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.f scrub starts 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:11 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.061788+0000 osd.1 (osd.1) 424 : cluster [DBG] 4.f scrub ok 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:12.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:12 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.403798+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:41:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:11.406224+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:41:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.263530+0000 osd.7 (osd.7) 295 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:41:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:12 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.266011+0000 osd.7 (osd.7) 296 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:41:13.250 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[22706]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:13.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:13.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:13 smithi042 bash[32075]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.558886+0000 mgr.y (mgr.24491) 869 : cluster [DBG] pgmap v1652: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.801658+0000 osd.4 (osd.4) 243 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:13 smithi162 bash[24557]: cluster 2024-09-20T19:41:12.803986+0000 osd.4 (osd.4) 244 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:41:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:14.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[22706]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:14.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:14.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:14 smithi042 bash[32075]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.182118+0000 osd.5 (osd.5) 379 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.184319+0000 osd.5 (osd.5) 380 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.799936+0000 osd.4 (osd.4) 245 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:41:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: cluster 2024-09-20T19:41:13.803272+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:41:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:14 smithi162 bash[24557]: audit 2024-09-20T19:41:13.920926+0000 mon.a (mon.0) 2788 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:15.651 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[32075]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:15.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:15.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:15.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:15 smithi042 bash[22706]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: cluster 2024-09-20T19:41:14.559982+0000 mgr.y (mgr.24491) 870 : cluster [DBG] pgmap v1653: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: audit 2024-09-20T19:41:15.074171+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.42:0/2695838878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:15 smithi162 bash[24557]: audit 2024-09-20T19:41:15.074618+0000 mon.a (mon.0) 2789 : audit [INF] from='client.26315 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:16.488 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-20T19:41:16.663 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:16.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:16.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:16.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:16 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.149231+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.151496+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: audit 2024-09-20T19:41:15.641158+0000 mon.a (mon.0) 2790 : audit [INF] from='client.26315 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.649005+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1073: 8 total, 8 up, 8 in 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.718325+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.e scrub starts 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:15.722379+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.e scrub ok 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.109667+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:16 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.111810+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[32075]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:17.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:17.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:17.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:17.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:17 smithi042 bash[22706]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.560792+0000 mgr.y (mgr.24491) 871 : cluster [DBG] pgmap v1655: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.701510+0000 osd.4 (osd.4) 249 : cluster [DBG] 2.f scrub starts 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:17 smithi162 bash[24557]: cluster 2024-09-20T19:41:16.704747+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub ok 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:18.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:18.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:18 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:17.712406+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:17.717118+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.299732+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:41:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:19.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:18 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.301967+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:19.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[32075]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:19 smithi042 bash[22706]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.193214+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:41:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.197363+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:41:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: cluster 2024-09-20T19:41:18.562937+0000 mgr.y (mgr.24491) 872 : cluster [DBG] pgmap v1656: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:19 smithi162 bash[24557]: audit 2024-09-20T19:41:19.216082+0000 mon.a (mon.0) 2792 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:41:20.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:20 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:20.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[32075]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[32075]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[32075]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[32075]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[22706]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[22706]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[22706]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:20 smithi042 bash[22706]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:20 smithi162 bash[24557]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:20 smithi162 bash[24557]: cluster 2024-09-20T19:41:19.453137+0000 osd.3 (osd.3) 71 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:41:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:20 smithi162 bash[24557]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:20 smithi162 bash[24557]: cluster 2024-09-20T19:41:19.455814+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:41:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:21.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[22706]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:21.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:21 smithi042 bash[32075]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:22.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:22.246 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: audit 2024-09-20T19:41:20.095076+0000 mgr.y (mgr.24491) 873 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:22.246 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.123371+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.125689+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.484335+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.486809+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:22.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:21 smithi162 bash[24557]: cluster 2024-09-20T19:41:20.563863+0000 mgr.y (mgr.24491) 874 : cluster [DBG] pgmap v1657: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:22 smithi162 bash[24557]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[32075]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.122313+0000 osd.5 (osd.5) 385 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.124622+0000 osd.5 (osd.5) 386 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.493582+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:22 smithi042 bash[22706]: cluster 2024-09-20T19:41:21.496177+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:41:23.692 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-20T19:41:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:23 smithi162 bash[24557]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[32075]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.108212+0000 osd.5 (osd.5) 387 : cluster [DBG] 6.d scrub starts 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.110624+0000 osd.5 (osd.5) 388 : cluster [DBG] 6.d scrub ok 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.479795+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.484053+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: cluster 2024-09-20T19:41:22.565346+0000 mgr.y (mgr.24491) 875 : cluster [DBG] pgmap v1658: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: audit 2024-09-20T19:41:23.191245+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.42:0/1919090936' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:23 smithi042 bash[22706]: audit 2024-09-20T19:41:23.191991+0000 mon.a (mon.0) 2793 : audit [INF] from='client.26321 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-20T19:41:24.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:24.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[22706]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:24.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:24.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:24 smithi042 bash[32075]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.111018+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.113144+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.687346+0000 mon.a (mon.0) 2794 : audit [INF] from='client.26321 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.690804+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1074: 8 total, 8 up, 8 in 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.763871+0000 osd.4 (osd.4) 253 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:41:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: cluster 2024-09-20T19:41:23.765817+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:41:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.859526+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:24 smithi162 bash[24557]: audit 2024-09-20T19:41:23.865413+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:25 smithi162 bash[24557]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[32075]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: cluster 2024-09-20T19:41:24.566030+0000 mgr.y (mgr.24491) 876 : cluster [DBG] pgmap v1660: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: audit 2024-09-20T19:41:25.194554+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:25 smithi042 bash[22706]: audit 2024-09-20T19:41:25.205050+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:26.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:26 smithi162 bash[24557]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[22706]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.860602+0000 mon.a (mon.0) 2800 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:41:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.862280+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:41:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:26 smithi042 bash[32075]: audit 2024-09-20T19:41:25.873414+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:41:28.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:27 smithi162 bash[24557]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:28.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:27 smithi162 bash[24557]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:28.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:27 smithi042 bash[32075]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:28.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:27 smithi042 bash[32075]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:27 smithi042 bash[22706]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:28.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:27 smithi042 bash[22706]: cluster 2024-09-20T19:41:26.566728+0000 mgr.y (mgr.24491) 877 : cluster [DBG] pgmap v1661: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:28 smithi162 bash[24557]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[32075]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.240624+0000 osd.0 (osd.0) 185 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.244624+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.803747+0000 osd.4 (osd.4) 255 : cluster [DBG] 4.c scrub starts 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:28 smithi042 bash[22706]: cluster 2024-09-20T19:41:27.807549+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.c scrub ok 2024-09-20T19:41:29.360 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:29.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[22706]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:29.709 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[22706]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:30.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:29 smithi162 bash[24557]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:30.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:29 smithi162 bash[24557]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:30.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:29 smithi162 bash[24557]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:29 smithi162 bash[24557]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[22706]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[22706]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[32075]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:30.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[32075]: cluster 2024-09-20T19:41:28.568459+0000 mgr.y (mgr.24491) 878 : cluster [DBG] pgmap v1662: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:30.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[32075]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:29 smithi042 bash[32075]: audit 2024-09-20T19:41:28.950349+0000 mon.a (mon.0) 2803 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:30.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:31.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:30 smithi162 bash[24557]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[22706]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.087863+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.091304+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:41:31.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.359440+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:41:31.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:30 smithi042 bash[32075]: cluster 2024-09-20T19:41:29.362957+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:41:31.728 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T19:41:32.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:31 smithi162 bash[24557]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[32075]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:30.106598+0000 mgr.y (mgr.24491) 879 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.258250+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:41:32.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.260545+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: cluster 2024-09-20T19:41:30.569375+0000 mgr.y (mgr.24491) 880 : cluster [DBG] pgmap v1663: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:31.107092+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.42:0/649254728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:31 smithi042 bash[22706]: audit 2024-09-20T19:41:31.107658+0000 mon.a (mon.0) 2804 : audit [INF] from='client.26327 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:32.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[32075]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[32075]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[32075]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[32075]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[22706]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[22706]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:32.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[22706]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:32.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:32 smithi042 bash[22706]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:33.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:32 smithi162 bash[24557]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:33.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:32 smithi162 bash[24557]: audit 2024-09-20T19:41:31.717422+0000 mon.a (mon.0) 2805 : audit [INF] from='client.26327 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:33.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:32 smithi162 bash[24557]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:33.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:32 smithi162 bash[24557]: cluster 2024-09-20T19:41:31.726270+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e1075: 8 total, 8 up, 8 in 2024-09-20T19:41:34.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:33 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:32.570828+0000 mgr.y (mgr.24491) 881 : cluster [DBG] pgmap v1665: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.239988+0000 osd.7 (osd.7) 299 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.242323+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.600987+0000 osd.2 (osd.2) 379 : cluster [DBG] 2.e scrub starts 2024-09-20T19:41:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:33 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.604538+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.e scrub ok 2024-09-20T19:41:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:34 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:34 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:34 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:35.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:34 smithi162 bash[24557]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:35.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[22706]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.768558+0000 osd.4 (osd.4) 257 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:35.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:34 smithi042 bash[32075]: cluster 2024-09-20T19:41:33.770896+0000 osd.4 (osd.4) 258 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:35 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:34.571775+0000 mgr.y (mgr.24491) 882 : cluster [DBG] pgmap v1666: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.355139+0000 osd.3 (osd.3) 81 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:41:36.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:35 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.357146+0000 osd.3 (osd.3) 82 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:41:36.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:37.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:36 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.305122+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.307469+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.836606+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:35.838906+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.351961+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:36 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.354311+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:41:37.373 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:37 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:38.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:37 smithi162 bash[24557]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:38.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:37 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:38.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:37 smithi042 bash[22706]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:37 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:37 smithi042 bash[32075]: cluster 2024-09-20T19:41:36.572531+0000 mgr.y (mgr.24491) 883 : cluster [DBG] pgmap v1667: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:39.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:38 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.681470+0000 osd.7 (osd.7) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.685598+0000 osd.7 (osd.7) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.777116+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e1076: 8 total, 8 up, 8 in 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.866136+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.e scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:37.869784+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.e scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.245171+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.247431+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.342946+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:38 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.347853+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:41:39.784 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:39 smithi162 bash[24557]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[22706]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: cluster 2024-09-20T19:41:38.573220+0000 mgr.y (mgr.24491) 884 : cluster [DBG] pgmap v1669: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:40.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: audit 2024-09-20T19:41:39.170899+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.42:0/3534666702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:39 smithi042 bash[32075]: audit 2024-09-20T19:41:39.171452+0000 mon.a (mon.0) 2808 : audit [INF] from='client.26333 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-20T19:41:40.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:40 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:40.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[32075]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[32075]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[32075]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[32075]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[22706]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[22706]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[22706]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:40.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:40 smithi042 bash[22706]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:40 smithi162 bash[24557]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:40 smithi162 bash[24557]: audit 2024-09-20T19:41:39.774331+0000 mon.a (mon.0) 2809 : audit [INF] from='client.26333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T19:41:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:40 smithi162 bash[24557]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:40 smithi162 bash[24557]: cluster 2024-09-20T19:41:39.781229+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e1077: 8 total, 8 up, 8 in 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:41 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.034810+0000 osd.5 (osd.5) 393 : cluster [DBG] 5.f scrub starts 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.037010+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.f scrub ok 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: audit 2024-09-20T19:41:40.117146+0000 mgr.y (mgr.24491) 885 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.573897+0000 mgr.y (mgr.24491) 886 : cluster [DBG] pgmap v1671: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.872008+0000 osd.1 (osd.1) 427 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:40.873608+0000 osd.1 (osd.1) 428 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.304417+0000 osd.7 (osd.7) 305 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.306937+0000 osd.7 (osd.7) 306 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.542736+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.d scrub starts 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:42.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:41 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.545528+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.d scrub ok 2024-09-20T19:41:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:43.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:42 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:43.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.000653+0000 osd.5 (osd.5) 395 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.003148+0000 osd.5 (osd.5) 396 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.902568+0000 osd.4 (osd.4) 263 : cluster [DBG] 5.e scrub starts 2024-09-20T19:41:43.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:41.905573+0000 osd.4 (osd.4) 264 : cluster [DBG] 5.e scrub ok 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.308146+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.310547+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:41:43.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:43.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:42 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.575388+0000 mgr.y (mgr.24491) 887 : cluster [DBG] pgmap v1672: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:44.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:43 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:43 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:43 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:44.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:43 smithi162 bash[24557]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:44.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[22706]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:44.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.958718+0000 osd.5 (osd.5) 397 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:41:44.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:44.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:43 smithi042 bash[32075]: cluster 2024-09-20T19:41:42.962717+0000 osd.5 (osd.5) 398 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:41:45.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:45.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:44 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:45.426 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:45.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:45.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: audit 2024-09-20T19:41:43.979399+0000 mon.a (mon.0) 2811 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.073074+0000 osd.2 (osd.2) 383 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.076775+0000 osd.2 (osd.2) 384 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.345930+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.348973+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:45.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:44 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.576368+0000 mgr.y (mgr.24491) 888 : cluster [DBG] pgmap v1673: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:46.249 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.249 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.249 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.249 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.249 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.250 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:45 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:44.978709+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e1078: 8 total, 8 up, 8 in 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.241419+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:45 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.245675+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:41:46.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:46 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.803257+0000 osd.1 (osd.1) 429 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:41:47.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.805882+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.969590+0000 osd.5 (osd.5) 399 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:45.972767+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.235164+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.236869+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:46 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.577031+0000 mgr.y (mgr.24491) 889 : cluster [DBG] pgmap v1675: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:41:47.992 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T19:41:48.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:47 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.565723+0000 osd.4 (osd.4) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.568587+0000 osd.4 (osd.4) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.979647+0000 osd.5 (osd.5) 401 : cluster [DBG] 1.a scrub starts 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.983120+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e1079: 8 total, 8 up, 8 in 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.434 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:46.984366+0000 osd.5 (osd.5) 402 : cluster [DBG] 1.a scrub ok 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.225225+0000 osd.3 (osd.3) 89 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.229969+0000 osd.3 (osd.3) 90 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: audit 2024-09-20T19:41:47.273017+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.42:0/246521802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: audit 2024-09-20T19:41:47.273434+0000 mon.a (mon.0) 2814 : audit [INF] from='client.26339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.435 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:47 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.388399+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:41:48.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-20T19:41:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[32075]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:49.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:48 smithi042 bash[22706]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.308 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.390931+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: audit 2024-09-20T19:41:47.981423+0000 mon.a (mon.0) 2815 : audit [INF] from='client.26339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:47.989314+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e1080: 8 total, 8 up, 8 in 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.219758+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.222318+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:49.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:48 smithi162 bash[24557]: cluster 2024-09-20T19:41:48.577755+0000 mgr.y (mgr.24491) 890 : cluster [DBG] pgmap v1678: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:50.549 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:41:50 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:41:50.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:50 smithi162 bash[24557]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:50 smithi162 bash[24557]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:50 smithi162 bash[24557]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:50.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:50 smithi162 bash[24557]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[32075]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[32075]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[32075]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[32075]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[22706]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[22706]: cluster 2024-09-20T19:41:49.013190+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.a scrub starts 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[22706]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:50.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:50 smithi042 bash[22706]: cluster 2024-09-20T19:41:49.015368+0000 osd.5 (osd.5) 404 : cluster [DBG] 6.a scrub ok 2024-09-20T19:41:51.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:51.809 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:51 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: audit 2024-09-20T19:41:50.125092+0000 mgr.y (mgr.24491) 891 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:50.578671+0000 mgr.y (mgr.24491) 892 : cluster [DBG] pgmap v1679: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.365430+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:51 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.367507+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:41:52.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:52 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:52 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:52 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.168909+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:41:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:52 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:52 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.172185+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:41:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:53.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:53 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:54.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.716594+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.718884+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.820767+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.a scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:51.823029+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.a scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.187684+0000 osd.0 (osd.0) 195 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.190208+0000 osd.0 (osd.0) 196 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:53 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.580099+0000 mgr.y (mgr.24491) 893 : cluster [DBG] pgmap v1680: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:54.920 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[32075]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:54.921 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:54 smithi042 bash[22706]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.687565+0000 osd.2 (osd.2) 387 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:41:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:55.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:52.689867+0000 osd.2 (osd.2) 388 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:41:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:53.203108+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:41:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:54 smithi162 bash[24557]: cluster 2024-09-20T19:41:53.207603+0000 osd.0 (osd.0) 198 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:41:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:55.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[22706]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:55 smithi042 bash[32075]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.119375+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.122513+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.581091+0000 mgr.y (mgr.24491) 894 : cluster [DBG] pgmap v1681: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:55 smithi162 bash[24557]: audit 2024-09-20T19:41:55.297939+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-20T19:41:56.402 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T19:41:56.599 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:41:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:41:56.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:56.680 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[32075]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:56.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:56.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:56.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:56.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:56 smithi042 bash[22706]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.667080+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:41:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:54.668917+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:41:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: audit 2024-09-20T19:41:55.576606+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? v1:172.21.15.42:0/1017502352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T19:41:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:56 smithi162 bash[24557]: cluster 2024-09-20T19:41:55.584457+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e1081: 8 total, 8 up, 8 in 2024-09-20T19:41:57.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:57 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:57.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:57 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:57.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:57 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:57.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:57 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:57 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:57 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.581745+0000 mgr.y (mgr.24491) 895 : cluster [DBG] pgmap v1683: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-09-20T19:41:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:58 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:59.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.637957+0000 osd.2 (osd.2) 391 : cluster [DBG] 5.b scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.639216+0000 osd.2 (osd.2) 392 : cluster [DBG] 5.b scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.937309+0000 osd.1 (osd.1) 433 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:56.940866+0000 osd.1 (osd.1) 434 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.207456+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.212607+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.792986+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:41:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:57.797197+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:41:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.110861+0000 osd.1 (osd.1) 435 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:41:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:58 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.114024+0000 osd.1 (osd.1) 436 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:41:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[32075]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:41:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:41:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:41:59 smithi042 bash[22706]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:00.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:42:00.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.058089+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.061631+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.583241+0000 mgr.y (mgr.24491) 896 : cluster [DBG] pgmap v1684: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.621575+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1082: 8 total, 8 up, 8 in 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.798580+0000 osd.4 (osd.4) 269 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: cluster 2024-09-20T19:41:58.801043+0000 osd.4 (osd.4) 270 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:41:59 smithi162 bash[24557]: audit 2024-09-20T19:41:59.008857+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:00.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:00.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:59.866598+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:42:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:41:59.869030+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:42:01.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.211076+0000 osd.7 (osd.7) 315 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:42:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.213584+0000 osd.7 (osd.7) 316 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:42:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:01 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: audit 2024-09-20T19:42:00.127824+0000 mgr.y (mgr.24491) 897 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.584168+0000 mgr.y (mgr.24491) 898 : cluster [DBG] pgmap v1686: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.857729+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:42:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:01 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.863261+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:42:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[32075]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:02.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:02.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:02 smithi042 bash[22706]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.991506+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:00.994350+0000 osd.3 (osd.3) 98 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.241676+0000 osd.0 (osd.0) 201 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.243459+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.614211+0000 osd.2 (osd.2) 393 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.616750+0000 osd.2 (osd.2) 394 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.821490+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.d scrub starts 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:02 smithi162 bash[24557]: cluster 2024-09-20T19:42:01.826755+0000 osd.4 (osd.4) 272 : cluster [DBG] 4.d scrub ok 2024-09-20T19:42:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T19:42:03.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:03.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[22706]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:03.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:03 smithi042 bash[32075]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.021978+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.026247+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.585008+0000 mgr.y (mgr.24491) 899 : cluster [DBG] pgmap v1687: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.825265+0000 osd.1 (osd.1) 441 : cluster [DBG] 5.c scrub starts 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: cluster 2024-09-20T19:42:02.827268+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub ok 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: audit 2024-09-20T19:42:02.960870+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.42:0/2697174871' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:03 smithi162 bash[24557]: audit 2024-09-20T19:42:02.961556+0000 mon.a (mon.0) 2822 : audit [INF] from='client.26351 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:04.496 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-20T19:42:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:04.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:04.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:04.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:04 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: audit 2024-09-20T19:42:03.637224+0000 mon.a (mon.0) 2823 : audit [INF] from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:05.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.645445+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e1083: 8 total, 8 up, 8 in 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.791952+0000 osd.1 (osd.1) 443 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:03.794795+0000 osd.1 (osd.1) 444 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.371904+0000 osd.6 (osd.6) 300 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:05.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:04 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.376643+0000 osd.6 (osd.6) 301 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[32075]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:05.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:05.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:05.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:05 smithi042 bash[22706]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:06.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.410981+0000 osd.5 (osd.5) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.413530+0000 osd.5 (osd.5) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.585782+0000 mgr.y (mgr.24491) 900 : cluster [DBG] pgmap v1689: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.666639+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e1084: 8 total, 8 up, 8 in 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.818029+0000 osd.1 (osd.1) 445 : cluster [DBG] 2.d scrub starts 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:05 smithi162 bash[24557]: cluster 2024-09-20T19:42:04.820182+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.d scrub ok 2024-09-20T19:42:06.672 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:06.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:06.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:06.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:06.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:06.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:06 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.398956+0000 osd.6 (osd.6) 302 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.403523+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.613353+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:06 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.618013+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:42:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:07.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:07.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:07.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:07.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:07 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:08.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.115575+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.118983+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.586597+0000 mgr.y (mgr.24491) 901 : cluster [DBG] pgmap v1691: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.836772+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:06.839171+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.379027+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:07 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.381049+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:08.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:08.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:08 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.132672+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.c scrub starts 2024-09-20T19:42:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:09.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:07.134947+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.c scrub ok 2024-09-20T19:42:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.193007+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.b scrub starts 2024-09-20T19:42:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:08 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.195236+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.b scrub ok 2024-09-20T19:42:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:09.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:09.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:09.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:09.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:09 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.218956+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.222135+0000 osd.0 (osd.0) 204 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.588134+0000 mgr.y (mgr.24491) 902 : cluster [DBG] pgmap v1692: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.653507+0000 osd.2 (osd.2) 397 : cluster [DBG] 5.a scrub starts 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.655902+0000 osd.2 (osd.2) 398 : cluster [DBG] 5.a scrub ok 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:10.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.785268+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:08.787491+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.165914+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.168077+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.399996+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:10.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:09 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.402173+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:42:10.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:10 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.149150+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.151728+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.174508+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.177119+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.398460+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.400732+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.596002+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:10 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.598669+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:42:11.688 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-20T19:42:12.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:11 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.861803+0000 osd.0 (osd.0) 205 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:09.866136+0000 osd.0 (osd.0) 206 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: audit 2024-09-20T19:42:10.139280+0000 mgr.y (mgr.24491) 903 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.588965+0000 mgr.y (mgr.24491) 904 : cluster [DBG] pgmap v1693: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:10.693828+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e1085: 8 total, 8 up, 8 in 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: audit 2024-09-20T19:42:11.133961+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.583040+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:11 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.585197+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:42:12.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:12.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[32075]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:12.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:12.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:12 smithi042 bash[22706]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.135103+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:42:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.136589+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: audit 2024-09-20T19:42:11.686595+0000 mon.a (mon.0) 2828 : audit [INF] from='client.? v1:172.21.15.42:0/1067456845' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.693544+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e1086: 8 total, 8 up, 8 in 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.939437+0000 osd.4 (osd.4) 273 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:12 smithi162 bash[24557]: cluster 2024-09-20T19:42:11.943746+0000 osd.4 (osd.4) 274 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:42:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:13 smithi162 bash[24557]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:14.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:13 smithi162 bash[24557]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:14.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:13 smithi042 bash[32075]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:14.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:13 smithi042 bash[32075]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:14.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:13 smithi042 bash[22706]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:14.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:13 smithi042 bash[22706]: cluster 2024-09-20T19:42:12.590408+0000 mgr.y (mgr.24491) 905 : cluster [DBG] pgmap v1696: 142 pgs: 142 active+clean; 583 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:14 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:13.767118+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:13.769473+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: audit 2024-09-20T19:42:14.038170+0000 mon.a (mon.0) 2830 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.413848+0000 osd.6 (osd.6) 312 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.416003+0000 osd.6 (osd.6) 313 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.552963+0000 osd.2 (osd.2) 403 : cluster [DBG] 6.f scrub starts 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:15.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:14 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.555345+0000 osd.2 (osd.2) 404 : cluster [DBG] 6.f scrub ok 2024-09-20T19:42:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:15 smithi162 bash[24557]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[32075]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.163215+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.167274+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.591360+0000 mgr.y (mgr.24491) 906 : cluster [DBG] pgmap v1697: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.735306+0000 osd.5 (osd.5) 413 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:42:16.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:15 smithi042 bash[22706]: cluster 2024-09-20T19:42:14.737595+0000 osd.5 (osd.5) 414 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:42:16.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:16 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:15.990927+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:15.993299+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.669147+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:16 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.671970+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:17.361 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:18.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:17 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:18.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.592082+0000 mgr.y (mgr.24491) 907 : cluster [DBG] pgmap v1698: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.669156+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.672327+0000 osd.5 (osd.5) 416 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.737381+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e1087: 8 total, 8 up, 8 in 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.777109+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:16.780481+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.031883+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:42:18.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:18.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:17 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.034338+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:42:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:18 smithi162 bash[24557]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[32075]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.391386+0000 osd.0 (osd.0) 207 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:42:19.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.393284+0000 osd.0 (osd.0) 208 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:42:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.806251+0000 osd.1 (osd.1) 449 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:42:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:18 smithi042 bash[22706]: cluster 2024-09-20T19:42:17.809067+0000 osd.1 (osd.1) 450 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:42:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T19:42:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:19 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.593622+0000 mgr.y (mgr.24491) 908 : cluster [DBG] pgmap v1700: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.845404+0000 osd.1 (osd.1) 451 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:18.850319+0000 osd.1 (osd.1) 452 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.138100+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.140387+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: audit 2024-09-20T19:42:19.242584+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.42:0/3248294275' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: audit 2024-09-20T19:42:19.243117+0000 mon.a (mon.0) 2832 : audit [INF] from='client.26363 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.420262+0000 osd.6 (osd.6) 314 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:19 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.422706+0000 osd.6 (osd.6) 315 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:42:20.539 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-20T19:42:20.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[22706]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:20.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:20 smithi042 bash[32075]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.704109+0000 osd.4 (osd.4) 275 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:42:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.706451+0000 osd.4 (osd.4) 276 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:42:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: audit 2024-09-20T19:42:19.734633+0000 mon.a (mon.0) 2833 : audit [INF] from='client.26363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:20 smithi162 bash[24557]: cluster 2024-09-20T19:42:19.742190+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1088: 8 total, 8 up, 8 in 2024-09-20T19:42:22.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:21 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: audit 2024-09-20T19:42:20.144815+0000 mgr.y (mgr.24491) 909 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.594168+0000 mgr.y (mgr.24491) 910 : cluster [DBG] pgmap v1702: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:42:22.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.896570+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:20.899438+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.380088+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:22.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:21 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.382296+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:22 smithi162 bash[24557]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[32075]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.613060+0000 osd.5 (osd.5) 417 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.615321+0000 osd.5 (osd.5) 418 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.712063+0000 osd.4 (osd.4) 277 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.714456+0000 osd.4 (osd.4) 278 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:42:23.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.914348+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:42:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:23.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:22 smithi042 bash[22706]: cluster 2024-09-20T19:42:21.918234+0000 osd.1 (osd.1) 456 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:42:24.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:23 smithi162 bash[24557]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[22706]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.595463+0000 mgr.y (mgr.24491) 911 : cluster [DBG] pgmap v1703: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.940918+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:22.943996+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:23.301428+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:24.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:23 smithi042 bash[32075]: cluster 2024-09-20T19:42:23.303721+0000 osd.6 (osd.6) 319 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:42:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:24 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:24 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:24 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:24 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.105741+0000 osd.7 (osd.7) 327 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:24 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.108661+0000 osd.7 (osd.7) 328 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:42:25.361 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:25 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:24.596428+0000 mgr.y (mgr.24491) 912 : cluster [DBG] pgmap v1704: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.096760+0000 osd.7 (osd.7) 329 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:42:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:25 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.099825+0000 osd.7 (osd.7) 330 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:42:26.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:27.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:26 smithi162 bash[24557]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:27.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[32075]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.281518+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.283980+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.642783+0000 osd.5 (osd.5) 419 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.644883+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.948071+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:25.950545+0000 osd.1 (osd.1) 460 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.138353+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.140711+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.179283+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.181672+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:27.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:26 smithi042 bash[22706]: audit 2024-09-20T19:42:26.345607+0000 mon.a (mon.0) 2835 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:27 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.097 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-20T19:42:28.116 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.597073+0000 mgr.y (mgr.24491) 913 : cluster [DBG] pgmap v1705: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.752866+0000 osd.4 (osd.4) 279 : cluster [DBG] 2.f scrub starts 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:26.755676+0000 osd.4 (osd.4) 280 : cluster [DBG] 2.f scrub ok 2024-09-20T19:42:28.118 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.087730+0000 mon.a (mon.0) 2836 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.088942+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.096424+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: audit 2024-09-20T19:42:27.344704+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.411744+0000 osd.6 (osd.6) 320 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.119 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:27 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.414630+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:42:28.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T19:42:29.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[32075]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:29.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:29 smithi042 bash[22706]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.801449+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:27.805056+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: audit 2024-09-20T19:42:28.095837+0000 mon.a (mon.0) 2840 : audit [INF] from='client.? v1:172.21.15.42:0/2131744466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T19:42:29.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.104850+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e1089: 8 total, 8 up, 8 in 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.257507+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.260255+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.532322+0000 osd.2 (osd.2) 405 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.534708+0000 osd.2 (osd.2) 406 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.597777+0000 mgr.y (mgr.24491) 914 : cluster [DBG] pgmap v1707: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:29.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:29 smithi162 bash[24557]: audit 2024-09-20T19:42:29.066775+0000 mon.a (mon.0) 2842 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[32075]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:30.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:30 smithi042 bash[22706]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:30.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:30 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:30.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:30 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:30 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.757734+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:42:30.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:30 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:30.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:30 smithi162 bash[24557]: cluster 2024-09-20T19:42:28.762352+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:42:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[32075]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:31.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:31.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:31 smithi042 bash[22706]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.008671+0000 osd.1 (osd.1) 461 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.011307+0000 osd.1 (osd.1) 462 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: audit 2024-09-20T19:42:30.153491+0000 mgr.y (mgr.24491) 915 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.183426+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.b scrub starts 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.185642+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.b scrub ok 2024-09-20T19:42:31.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.549466+0000 osd.5 (osd.5) 421 : cluster [DBG] 6.d scrub starts 2024-09-20T19:42:31.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.552061+0000 osd.5 (osd.5) 422 : cluster [DBG] 6.d scrub ok 2024-09-20T19:42:31.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:31.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:31 smithi162 bash[24557]: cluster 2024-09-20T19:42:30.598669+0000 mgr.y (mgr.24491) 916 : cluster [DBG] pgmap v1708: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[32075]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[32075]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[32075]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[32075]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[22706]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[22706]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[22706]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:32.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:32 smithi042 bash[22706]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:32.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:32 smithi162 bash[24557]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:32 smithi162 bash[24557]: cluster 2024-09-20T19:42:31.193360+0000 osd.3 (osd.3) 117 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:42:32.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:32 smithi162 bash[24557]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:32.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:32 smithi162 bash[24557]: cluster 2024-09-20T19:42:31.196019+0000 osd.3 (osd.3) 118 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:42:33.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:33 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:33 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:33 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:33 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:33 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:33 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.600062+0000 mgr.y (mgr.24491) 917 : cluster [DBG] pgmap v1709: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:33.761 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:34.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:34.184 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:34 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:34.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.810307+0000 osd.4 (osd.4) 285 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:32.812633+0000 osd.4 (osd.4) 286 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.235069+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.237733+0000 osd.3 (osd.3) 120 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.429723+0000 osd.0 (osd.0) 213 : cluster [DBG] 4.b scrub starts 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.433645+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.b scrub ok 2024-09-20T19:42:34.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.623732+0000 osd.5 (osd.5) 423 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:42:34.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:34.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:34 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.626167+0000 osd.5 (osd.5) 424 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:42:35.430 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:35.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[22706]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:35.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:35 smithi042 bash[32075]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.987438+0000 osd.7 (osd.7) 333 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:42:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:33.989458+0000 osd.7 (osd.7) 334 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:42:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.071094+0000 osd.1 (osd.1) 463 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.073253+0000 osd.1 (osd.1) 464 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.540277+0000 osd.2 (osd.2) 407 : cluster [DBG] 2.c scrub starts 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.542632+0000 osd.2 (osd.2) 408 : cluster [DBG] 2.c scrub ok 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:35.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:35 smithi162 bash[24557]: cluster 2024-09-20T19:42:34.601001+0000 mgr.y (mgr.24491) 918 : cluster [DBG] pgmap v1710: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:36.143 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T19:42:36.430 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[22706]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:36 smithi042 bash[32075]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.121585+0000 osd.1 (osd.1) 465 : cluster [DBG] 4.f scrub starts 2024-09-20T19:42:36.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.125926+0000 osd.1 (osd.1) 466 : cluster [DBG] 4.f scrub ok 2024-09-20T19:42:36.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: audit 2024-09-20T19:42:35.679583+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:36.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.737259+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:42:36.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:36 smithi162 bash[24557]: cluster 2024-09-20T19:42:35.739721+0000 osd.4 (osd.4) 288 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:42:36.933 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-20T19:42:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.431 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[32075]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.434 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:37.435 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:37 smithi042 bash[22706]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: audit 2024-09-20T19:42:36.141615+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? v1:172.21.15.42:0/3400343822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.150679+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e1090: 8 total, 8 up, 8 in 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.260733+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.262349+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.339873+0000 osd.6 (osd.6) 322 : cluster [DBG] 4.a scrub starts 2024-09-20T19:42:37.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.344599+0000 osd.6 (osd.6) 323 : cluster [DBG] 4.a scrub ok 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.422980+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.425335+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:37.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:37 smithi162 bash[24557]: cluster 2024-09-20T19:42:36.601707+0000 mgr.y (mgr.24491) 919 : cluster [DBG] pgmap v1712: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[22706]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:38.432 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:38 smithi042 bash[32075]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:38.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.375469+0000 osd.0 (osd.0) 217 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.378579+0000 osd.0 (osd.0) 218 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.670340+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.c scrub starts 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:38.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:38 smithi162 bash[24557]: cluster 2024-09-20T19:42:37.674320+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub ok 2024-09-20T19:42:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[32075]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:39 smithi042 bash[22706]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:40.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.603376+0000 mgr.y (mgr.24491) 920 : cluster [DBG] pgmap v1713: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.663415+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:38.667350+0000 osd.4 (osd.4) 292 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:39.324222+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:39 smithi162 bash[24557]: cluster 2024-09-20T19:42:39.326479+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:42:40.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:40 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:40 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:40 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:40 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:40 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.567974+0000 osd.2 (osd.2) 409 : cluster [DBG] 4.1c scrub starts 2024-09-20T19:42:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:40 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.572382+0000 osd.2 (osd.2) 410 : cluster [DBG] 4.1c scrub ok 2024-09-20T19:42:41.752 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:42.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:41 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: audit 2024-09-20T19:42:40.164821+0000 mgr.y (mgr.24491) 921 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.604324+0000 mgr.y (mgr.24491) 922 : cluster [DBG] pgmap v1714: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.686072+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:40.688940+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.328289+0000 osd.6 (osd.6) 326 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:42.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:41 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.330776+0000 osd.6 (osd.6) 327 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:42:43.025 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.025 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[32075]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.026 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.027 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.027 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:42 smithi042 bash[22706]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.238586+0000 osd.3 (osd.3) 123 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.240438+0000 osd.3 (osd.3) 124 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.301220+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:42 smithi162 bash[24557]: cluster 2024-09-20T19:42:41.303545+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:42:43.699 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:43 smithi162 bash[24557]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[32075]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.260838+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.0 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.265082+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.0 scrub ok 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.605747+0000 mgr.y (mgr.24491) 923 : cluster [DBG] pgmap v1715: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.953442+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:42.955909+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.098158+0000 osd.1 (osd.1) 467 : cluster [DBG] 6.1 scrub starts 2024-09-20T19:42:44.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.099593+0000 osd.1 (osd.1) 468 : cluster [DBG] 6.1 scrub ok 2024-09-20T19:42:44.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:43 smithi042 bash[22706]: audit 2024-09-20T19:42:43.393875+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-20T19:42:44.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:44.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[32075]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:44.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:44.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:44 smithi042 bash[22706]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:45.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.653981+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.656255+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: audit 2024-09-20T19:42:43.697363+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? v1:172.21.15.42:0/2409543270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: cluster 2024-09-20T19:42:43.704594+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e1091: 8 total, 8 up, 8 in 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:44 smithi162 bash[24557]: audit 2024-09-20T19:42:44.094667+0000 mon.a (mon.0) 2849 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:45 smithi162 bash[24557]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[22706]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.306330+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.309540+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:45 smithi042 bash[32075]: cluster 2024-09-20T19:42:44.606572+0000 mgr.y (mgr.24491) 924 : cluster [DBG] pgmap v1717: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:42:46.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:46 smithi162 bash[24557]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[32075]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.330146+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1f scrub starts 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.332461+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.1f scrub ok 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.691569+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.e scrub starts 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.695456+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.e scrub ok 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.929739+0000 osd.7 (osd.7) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:47.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:46 smithi042 bash[22706]: cluster 2024-09-20T19:42:45.933936+0000 osd.7 (osd.7) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:47 smithi162 bash[24557]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:47 smithi162 bash[24557]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:47 smithi162 bash[24557]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:47 smithi162 bash[24557]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[32075]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[32075]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[32075]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[32075]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[22706]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[22706]: cluster 2024-09-20T19:42:46.607224+0000 mgr.y (mgr.24491) 925 : cluster [DBG] pgmap v1718: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[22706]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:48.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:47 smithi042 bash[22706]: cluster 2024-09-20T19:42:46.737441+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e1092: 8 total, 8 up, 8 in 2024-09-20T19:42:49.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:48 smithi162 bash[24557]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[32075]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.293820+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.1e scrub starts 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.297932+0000 osd.0 (osd.0) 222 : cluster [DBG] 4.1e scrub ok 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.635405+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.638337+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:48 smithi042 bash[22706]: cluster 2024-09-20T19:42:47.745005+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e1093: 8 total, 8 up, 8 in 2024-09-20T19:42:49.382 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:49.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[22706]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:49.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:49 smithi042 bash[32075]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.608186+0000 mgr.y (mgr.24491) 926 : cluster [DBG] pgmap v1721: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:42:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.761116+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:42:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:50.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:49 smithi162 bash[24557]: cluster 2024-09-20T19:42:48.764266+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:42:50.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:42:50 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:42:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:50 smithi162 bash[24557]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:50 smithi162 bash[24557]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:50 smithi162 bash[24557]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:50 smithi162 bash[24557]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[32075]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[32075]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[32075]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[32075]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[22706]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[22706]: cluster 2024-09-20T19:42:49.721957+0000 osd.5 (osd.5) 427 : cluster [DBG] 4.15 scrub starts 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[22706]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:50 smithi042 bash[22706]: cluster 2024-09-20T19:42:49.725561+0000 osd.5 (osd.5) 428 : cluster [DBG] 4.15 scrub ok 2024-09-20T19:42:51.763 INFO:teuthology.orchestra.run.smithi042.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:51 smithi162 bash[24557]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[32075]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:50.176101+0000 mgr.y (mgr.24491) 927 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: cluster 2024-09-20T19:42:50.609098+0000 mgr.y (mgr.24491) 928 : cluster [DBG] pgmap v1722: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:51.180926+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.42:0/2822759238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:51 smithi042 bash[22706]: audit 2024-09-20T19:42:51.181312+0000 mon.a (mon.0) 2852 : audit [INF] from='client.26387 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:52.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[32075]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:52.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:52.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:52.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:52 smithi042 bash[22706]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:53.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.700756+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1b scrub starts 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.704285+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1b scrub ok 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.737953+0000 osd.5 (osd.5) 429 : cluster [DBG] 2.12 scrub starts 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.739942+0000 osd.5 (osd.5) 430 : cluster [DBG] 2.12 scrub ok 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: audit 2024-09-20T19:42:51.752364+0000 mon.a (mon.0) 2853 : audit [INF] from='client.26387 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:42:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:53.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:52 smithi162 bash[24557]: cluster 2024-09-20T19:42:51.759382+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e1094: 8 total, 8 up, 8 in 2024-09-20T19:42:54.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:53 smithi162 bash[24557]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[22706]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:52.609997+0000 mgr.y (mgr.24491) 929 : cluster [DBG] pgmap v1724: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.014139+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.a scrub starts 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.016273+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.a scrub ok 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.543696+0000 osd.2 (osd.2) 411 : cluster [DBG] 5.d scrub starts 2024-09-20T19:42:54.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:54.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:53 smithi042 bash[32075]: cluster 2024-09-20T19:42:53.546633+0000 osd.2 (osd.2) 412 : cluster [DBG] 5.d scrub ok 2024-09-20T19:42:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:55 smithi162 bash[24557]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[32075]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.610919+0000 mgr.y (mgr.24491) 930 : cluster [DBG] pgmap v1725: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.647325+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.e scrub starts 2024-09-20T19:42:56.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:55 smithi042 bash[22706]: cluster 2024-09-20T19:42:54.650004+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.e scrub ok 2024-09-20T19:42:56.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:42:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:42:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:56 smithi162 bash[24557]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[32075]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.628222+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.e scrub starts 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.630677+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.e scrub ok 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.690425+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.7 scrub starts 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:56 smithi042 bash[22706]: cluster 2024-09-20T19:42:55.692581+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.7 scrub ok 2024-09-20T19:42:57.375 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:42:58.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:57 smithi162 bash[24557]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[32075]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:58.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.611606+0000 mgr.y (mgr.24491) 931 : cluster [DBG] pgmap v1726: 142 pgs: 142 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.677507+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub starts 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:56.682187+0000 osd.5 (osd.5) 434 : cluster [DBG] 4.8 scrub ok 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:57.418307+0000 osd.3 (osd.3) 131 : cluster [DBG] 4.11 scrub starts 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:58.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:57 smithi042 bash[22706]: cluster 2024-09-20T19:42:57.421637+0000 osd.3 (osd.3) 132 : cluster [DBG] 4.11 scrub ok 2024-09-20T19:42:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:42:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[32075]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:42:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:42:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:42:59 smithi042 bash[22706]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.418450+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.14 scrub starts 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.421628+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.14 scrub ok 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:58.613148+0000 mgr.y (mgr.24491) 932 : cluster [DBG] pgmap v1727: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.080458+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:43:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.082689+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.139828+0000 mon.a (mon.0) 2855 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.276125+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.42:0/1335173190' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:43:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:42:59 smithi162 bash[24557]: audit 2024-09-20T19:42:59.276566+0000 mon.a (mon.0) 2856 : audit [INF] from='client.26393 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-20T19:43:00.373 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-20T19:43:00.373 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-20T19:43:00.373 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-20T19:43:00.373 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-20T19:43:00.373 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T19:43:00.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:43:00.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:00.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:00.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:00.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:00.684 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.066872+0000 osd.1 (osd.1) 471 : cluster [DBG] 5.1d scrub starts 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.068273+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub ok 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.513714+0000 osd.2 (osd.2) 413 : cluster [DBG] 2.e scrub starts 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.516849+0000 osd.2 (osd.2) 414 : cluster [DBG] 2.e scrub ok 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: audit 2024-09-20T19:42:59.573434+0000 mon.a (mon.0) 2857 : audit [INF] from='client.26393 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T19:43:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:42:59.582129+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e1095: 8 total, 8 up, 8 in 2024-09-20T19:43:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.100649+0000 osd.7 (osd.7) 341 : cluster [DBG] 1.2 scrub starts 2024-09-20T19:43:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:01.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.103099+0000 osd.7 (osd.7) 342 : cluster [DBG] 1.2 scrub ok 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[22706]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:01 smithi042 bash[32075]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: audit 2024-09-20T19:43:00.187496+0000 mgr.y (mgr.24491) 933 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.498914+0000 osd.2 (osd.2) 415 : cluster [DBG] 5.b scrub starts 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.500626+0000 osd.2 (osd.2) 416 : cluster [DBG] 5.b scrub ok 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:01 smithi162 bash[24557]: cluster 2024-09-20T19:43:00.613890+0000 mgr.y (mgr.24491) 934 : cluster [DBG] pgmap v1729: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:02 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:01.733829+0000 osd.5 (osd.5) 435 : cluster [DBG] 1.1 scrub starts 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:01.736575+0000 osd.5 (osd.5) 436 : cluster [DBG] 1.1 scrub ok 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.101052+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1d scrub starts 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:02 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.103192+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.1d scrub ok 2024-09-20T19:43:03.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:03 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:03.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:03 smithi042 bash[22706]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:03 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:03 smithi042 bash[32075]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:03 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:04.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:03 smithi162 bash[24557]: cluster 2024-09-20T19:43:02.615435+0000 mgr.y (mgr.24491) 935 : cluster [DBG] pgmap v1730: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:05.175 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:05.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[32075]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:05.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:05 smithi042 bash[22706]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.231245+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.10 scrub starts 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.233655+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.10 scrub ok 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.354354+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.358403+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-09-20T19:43:06.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:06.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:05 smithi162 bash[24557]: cluster 2024-09-20T19:43:04.616425+0000 mgr.y (mgr.24491) 936 : cluster [DBG] pgmap v1731: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:06.637 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:43:07.002 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":1095,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:42:59.570583+0000","last_up_change":"2024-09-20T19:30:58.350676+0000","last_in_change":"2024-09-20T19:40:43.632213+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"863","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"259","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-20T19:18:05.791948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-20T19:18:06.729842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T19:18:07.353972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T19:18:09.310321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T19:18:11.438339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":393,"up_from":500,"up_thru":1065,"down_at":394,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":701330554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":701330554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":701330554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":701330554}]},"public_addr":"172.21.15.42:6801/701330554","cluster_addr":"172.21.15.42:6802/701330554","heartbeat_back_addr":"172.21.15.42:6804/701330554","heartbeat_front_addr":"172.21.15.42:6803/701330554","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":28,"last_clean_end":479,"up_from":550,"up_thru":1065,"down_at":479,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1278675080}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1278675080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1278675080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1278675080}]},"public_addr":"172.21.15.42:6813/1278675080","cluster_addr":"172.21.15.42:6814/1278675080","heartbeat_back_addr":"172.21.15.42:6816/1278675080","heartbeat_front_addr":"172.21.15.42:6815/1278675080","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":443,"up_from":522,"up_thru":1065,"down_at":444,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":449450686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":449450686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":449450686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":449450686}]},"public_addr":"172.21.15.162:6800/449450686","cluster_addr":"172.21.15.162:6801/449450686","heartbeat_back_addr":"172.21.15.162:6803/449450686","heartbeat_front_addr":"172.21.15.162:6802/449450686","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:09.866142+0000","dead_epoch":393},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:58.114028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:44.076780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:16.671976+0000","dead_epoch":478},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:47.638345+0000","dead_epoch":443},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:04.413535+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:10.177124+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:45.933941+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.8","osds":[2,3,1]}],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"2.10","mappings":[{"from":2,"to":0}]},{"pgid":"2.1f","mappings":[{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.14","mappings":[{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/1743761075":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/3764180564":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3982292550":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/2784207196":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3505349280":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/368716713":"2024-09-21T19:18:55.853462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:43:07.829 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T19:43:07.830 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:07.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[22706]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:07.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:07.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:07.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:07 smithi042 bash[32075]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.183305+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub starts 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.185845+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.0 scrub ok 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.309977+0000 osd.3 (osd.3) 137 : cluster [DBG] 6.11 scrub starts 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.312222+0000 osd.3 (osd.3) 138 : cluster [DBG] 6.11 scrub ok 2024-09-20T19:43:08.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.617117+0000 mgr.y (mgr.24491) 937 : cluster [DBG] pgmap v1732: 142 pgs: 142 active+clean; 583 KiB data, 508 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.733700+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.18 scrub starts 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.735892+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.18 scrub ok 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.754950+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.f scrub starts 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: cluster 2024-09-20T19:43:06.757171+0000 osd.5 (osd.5) 438 : cluster [DBG] 5.f scrub ok 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:08.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:07 smithi162 bash[24557]: audit 2024-09-20T19:43:06.997950+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? v1:172.21.15.42:0/3576508193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:09.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:08 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:07.189785+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:07.192680+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-20T19:43:09.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.232253+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.234500+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.392163+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.1 scrub starts 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:09.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:08 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.394583+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.1 scrub ok 2024-09-20T19:43:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:09 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:09 smithi162 bash[24557]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.180 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:09 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:09 smithi042 bash[22706]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:09 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:09 smithi042 bash[32075]: cluster 2024-09-20T19:43:08.618817+0000 mgr.y (mgr.24491) 938 : cluster [DBG] pgmap v1733: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:10.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:10 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:11.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:10 smithi162 bash[24557]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:10 smithi162 bash[24557]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:10 smithi162 bash[24557]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:11.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:10 smithi162 bash[24557]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[32075]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[32075]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[32075]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[32075]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[22706]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[22706]: cluster 2024-09-20T19:43:09.283444+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.19 scrub starts 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[22706]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:11.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:10 smithi042 bash[22706]: cluster 2024-09-20T19:43:09.288251+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.19 scrub ok 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:11 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: audit 2024-09-20T19:43:10.198903+0000 mgr.y (mgr.24491) 939 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.266360+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.c scrub starts 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.268976+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.c scrub ok 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:10.619423+0000 mgr.y (mgr.24491) 940 : cluster [DBG] pgmap v1734: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.366157+0000 osd.2 (osd.2) 419 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:11 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.369664+0000 osd.2 (osd.2) 420 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:12.634 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:13.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:12 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.251641+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.2 scrub starts 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.254727+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.2 scrub ok 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:11.712567+0000 mon.a (mon.0) 2859 : cluster [DBG] osdmap e1096: 8 total, 8 up, 8 in 2024-09-20T19:43:13.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.086936+0000 osd.1 (osd.1) 473 : cluster [DBG] 3.0 scrub starts 2024-09-20T19:43:13.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:12 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.090123+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.0 scrub ok 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:13.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:13 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.615329+0000 osd.5 (osd.5) 439 : cluster [DBG] 5.15 scrub starts 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.619894+0000 osd.5 (osd.5) 440 : cluster [DBG] 5.15 scrub ok 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:12.620951+0000 mgr.y (mgr.24491) 941 : cluster [DBG] pgmap v1736: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.261989+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub starts 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:14.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:13 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.264234+0000 osd.7 (osd.7) 348 : cluster [DBG] 5.10 scrub ok 2024-09-20T19:43:14.312 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:43:14.312 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":1096,"fsid":"49cb5aae-7783-11ef-baf5-efdc52797490","created":"2024-09-20T19:07:00.852407+0000","modified":"2024-09-20T19:43:11.695124+0000","last_up_change":"2024-09-20T19:30:58.350676+0000","last_in_change":"2024-09-20T19:40:43.632213+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:12:46.311971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"863","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"259","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-20T19:18:05.791948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-20T19:18:06.729842+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T19:18:07.353972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T19:18:09.310321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T19:18:11.438339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb59ab36-8bbc-4e10-bc0f-b0cca84b317b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":393,"up_from":500,"up_thru":1065,"down_at":394,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6801","nonce":701330554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6802","nonce":701330554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6804","nonce":701330554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6803","nonce":701330554}]},"public_addr":"172.21.15.42:6801/701330554","cluster_addr":"172.21.15.42:6802/701330554","heartbeat_back_addr":"172.21.15.42:6804/701330554","heartbeat_front_addr":"172.21.15.42:6803/701330554","state":["exists","up"]},{"osd":1,"uuid":"1bee6f4f-6377-4f7f-94a1-bc49b0a33d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6805","nonce":3513258758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6806","nonce":3513258758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6808","nonce":3513258758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6807","nonce":3513258758}]},"public_addr":"172.21.15.42:6805/3513258758","cluster_addr":"172.21.15.42:6806/3513258758","heartbeat_back_addr":"172.21.15.42:6808/3513258758","heartbeat_front_addr":"172.21.15.42:6807/3513258758","state":["exists","up"]},{"osd":2,"uuid":"12b15f1c-ba47-49cb-aefe-d99ff84d926d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6809","nonce":2620956901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6810","nonce":2620956901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6812","nonce":2620956901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6811","nonce":2620956901}]},"public_addr":"172.21.15.42:6809/2620956901","cluster_addr":"172.21.15.42:6810/2620956901","heartbeat_back_addr":"172.21.15.42:6812/2620956901","heartbeat_front_addr":"172.21.15.42:6811/2620956901","state":["exists","up"]},{"osd":3,"uuid":"f0b69ee0-b93b-4ab6-8a35-783b18b9c73e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":28,"last_clean_end":479,"up_from":550,"up_thru":1065,"down_at":479,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6813","nonce":1278675080}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6814","nonce":1278675080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6816","nonce":1278675080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.42:6815","nonce":1278675080}]},"public_addr":"172.21.15.42:6813/1278675080","cluster_addr":"172.21.15.42:6814/1278675080","heartbeat_back_addr":"172.21.15.42:6816/1278675080","heartbeat_front_addr":"172.21.15.42:6815/1278675080","state":["exists","up"]},{"osd":4,"uuid":"90c5738d-14a9-415f-9249-eac0455635ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":443,"up_from":522,"up_thru":1065,"down_at":444,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":449450686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":449450686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":449450686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":449450686}]},"public_addr":"172.21.15.162:6800/449450686","cluster_addr":"172.21.15.162:6801/449450686","heartbeat_back_addr":"172.21.15.162:6803/449450686","heartbeat_front_addr":"172.21.15.162:6802/449450686","state":["exists","up"]},{"osd":5,"uuid":"e7648e52-3260-47ec-8778-35d4fa70ee27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4269525575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4269525575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4269525575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4269525575}]},"public_addr":"172.21.15.162:6804/4269525575","cluster_addr":"172.21.15.162:6805/4269525575","heartbeat_back_addr":"172.21.15.162:6807/4269525575","heartbeat_front_addr":"172.21.15.162:6806/4269525575","state":["exists","up"]},{"osd":6,"uuid":"34b6c8df-9cb5-4fcb-955f-69fd6520465a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":1785075253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":1785075253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":1785075253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":1785075253}]},"public_addr":"172.21.15.162:6808/1785075253","cluster_addr":"172.21.15.162:6809/1785075253","heartbeat_back_addr":"172.21.15.162:6811/1785075253","heartbeat_front_addr":"172.21.15.162:6810/1785075253","state":["exists","up"]},{"osd":7,"uuid":"6046a58c-ccf6-41ce-9888-b0c7a771205e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":1845916231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":1845916231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":1845916231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":1845916231}]},"public_addr":"172.21.15.162:6812/1845916231","cluster_addr":"172.21.15.162:6813/1845916231","heartbeat_back_addr":"172.21.15.162:6815/1845916231","heartbeat_front_addr":"172.21.15.162:6814/1845916231","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:09.866142+0000","dead_epoch":393},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:58.114028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:43:11.369669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:16.671976+0000","dead_epoch":478},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:47.638345+0000","dead_epoch":443},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:04.413535+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:10.177124+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:42:45.933941+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.8","osds":[2,3,1]}],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"2.10","mappings":[{"from":2,"to":0}]},{"pgid":"2.1f","mappings":[{"from":7,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.14","mappings":[{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:0/1743761075":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3575489582":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2546869704":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/1361717385":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/2117630454":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/409595637":"2024-09-21T19:07:29.716210+0000","172.21.15.42:6800/2206296983":"2024-09-21T19:08:07.715726+0000","172.21.15.42:0/3240017151":"2024-09-21T19:08:07.715726+0000","172.21.15.42:6800/3764180564":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3774737654":"2024-09-21T19:07:29.716210+0000","172.21.15.42:0/3982292550":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/2784207196":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/3505349280":"2024-09-21T19:18:55.853462+0000","172.21.15.42:0/368716713":"2024-09-21T19:18:55.853462+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:43:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:14 smithi162 bash[24557]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.121 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.122 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.123 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.123 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[32075]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.193377+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.19 scrub starts 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.196282+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.19 scrub ok 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.124 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.498237+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.1f scrub starts 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:13.501910+0000 osd.6 (osd.6) 329 : cluster [DBG] 4.1f scrub ok 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: audit 2024-09-20T19:43:14.167840+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.233916+0000 osd.7 (osd.7) 349 : cluster [DBG] 5.4 scrub starts 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.236207+0000 osd.7 (osd.7) 350 : cluster [DBG] 5.4 scrub ok 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.125 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:14 smithi042 bash[22706]: audit 2024-09-20T19:43:14.308503+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.42:0/4117749339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-20T19:43:15.125 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T19:43:15.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T19:43:15.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T19:43:15.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T19:43:15.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T19:43:15.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T19:43:15.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T19:43:15.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T19:43:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:15 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:15 smithi162 bash[24557]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:15 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:15 smithi042 bash[32075]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:15 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:15 smithi042 bash[22706]: cluster 2024-09-20T19:43:14.621921+0000 mgr.y (mgr.24491) 942 : cluster [DBG] pgmap v1737: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:16.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:17.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:16 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:17.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.068678+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.19 scrub starts 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.071065+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.19 scrub ok 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.173033+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.b scrub starts 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:17.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:16 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.175246+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub ok 2024-09-20T19:43:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:18.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:17 smithi162 bash[24557]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[22706]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.622635+0000 mgr.y (mgr.24491) 943 : cluster [DBG] pgmap v1738: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.751709+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.13 scrub starts 2024-09-20T19:43:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:18.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:17 smithi042 bash[32075]: cluster 2024-09-20T19:43:16.756369+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.13 scrub ok 2024-09-20T19:43:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:19.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[22706]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:19.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:19.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:19.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:19 smithi042 bash[32075]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:20.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.329452+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.3 scrub starts 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.334146+0000 osd.0 (osd.0) 230 : cluster [DBG] 4.3 scrub ok 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.624256+0000 mgr.y (mgr.24491) 944 : cluster [DBG] pgmap v1739: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.634014+0000 osd.5 (osd.5) 441 : cluster [DBG] 5.1b scrub starts 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:20.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:19 smithi162 bash[24557]: cluster 2024-09-20T19:43:18.636409+0000 osd.5 (osd.5) 442 : cluster [DBG] 5.1b scrub ok 2024-09-20T19:43:20.076 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.076 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.076 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:20.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:20 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:20.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:20 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:20 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:20 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.382847+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.2 scrub starts 2024-09-20T19:43:21.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:20 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:21.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:20 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.385053+0000 osd.6 (osd.6) 331 : cluster [DBG] 5.2 scrub ok 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:21.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:21.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:21.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:21.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:21.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:21.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:21.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:21 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: audit 2024-09-20T19:43:20.210354+0000 mgr.y (mgr.24491) 945 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.272979+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.13 scrub starts 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.275613+0000 osd.0 (osd.0) 232 : cluster [DBG] 2.13 scrub ok 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.307932+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.10 scrub starts 2024-09-20T19:43:22.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.311468+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.10 scrub ok 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.625188+0000 mgr.y (mgr.24491) 946 : cluster [DBG] pgmap v1740: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.682612+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.a scrub starts 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:20.685099+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.a scrub ok 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.142523+0000 osd.7 (osd.7) 353 : cluster [DBG] 5.1a scrub starts 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:22.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:21 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.144827+0000 osd.7 (osd.7) 354 : cluster [DBG] 5.1a scrub ok 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[32075]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:22.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:22 smithi042 bash[22706]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:22 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:23.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:22 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.668263+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.d scrub starts 2024-09-20T19:43:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:22 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:23.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:22 smithi162 bash[24557]: cluster 2024-09-20T19:43:21.672864+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.d scrub ok 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:23.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:23.826 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:23 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.306987+0000 osd.2 (osd.2) 423 : cluster [DBG] 5.a scrub starts 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.309261+0000 osd.2 (osd.2) 424 : cluster [DBG] 5.a scrub ok 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:22.626255+0000 mgr.y (mgr.24491) 947 : cluster [DBG] pgmap v1741: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.074669+0000 osd.7 (osd.7) 355 : cluster [DBG] 1.9 scrub starts 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:24.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:23 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.076702+0000 osd.7 (osd.7) 356 : cluster [DBG] 1.9 scrub ok 2024-09-20T19:43:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:24.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:24.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[22706]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:24.683 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:24.684 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:24 smithi042 bash[32075]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.103663+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.d scrub starts 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.105332+0000 osd.1 (osd.1) 478 : cluster [DBG] 2.d scrub ok 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.180185+0000 osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.182314+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.351833+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.8 scrub starts 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.354114+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.8 scrub ok 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.624622+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e1097: 8 total, 8 up, 8 in 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.679812+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1c scrub starts 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:24 smithi162 bash[24557]: cluster 2024-09-20T19:43:23.681845+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1c scrub ok 2024-09-20T19:43:25.442 INFO:teuthology.orchestra.run.smithi042.stdout:2241972928668 2024-09-20T19:43:25.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:25.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:25.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:25.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:25 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.491722+0000 osd.5 (osd.5) 445 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.494276+0000 osd.5 (osd.5) 446 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.627777+0000 mgr.y (mgr.24491) 948 : cluster [DBG] pgmap v1743: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.639686+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e1098: 8 total, 8 up, 8 in 2024-09-20T19:43:26.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.670081+0000 osd.5 (osd.5) 447 : cluster [DBG] 1.a scrub starts 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:24.673980+0000 osd.5 (osd.5) 448 : cluster [DBG] 1.a scrub ok 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.325609+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:26.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:25 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.328188+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:26.595 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:26.596 INFO:teuthology.orchestra.run.smithi042.stdout:2362232012951 2024-09-20T19:43:26.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[22706]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:26.848 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:26.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:26.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:26.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:26.849 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:26 smithi042 bash[32075]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:27.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.240400+0000 osd.0 (osd.0) 233 : cluster [DBG] 6.9 scrub starts 2024-09-20T19:43:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.260350+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.9 scrub ok 2024-09-20T19:43:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:27.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:26 smithi162 bash[24557]: cluster 2024-09-20T19:43:25.639673+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1099: 8 total, 8 up, 8 in 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:27.781 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[22706]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:27.782 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:27 smithi042 bash[32075]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:28.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.628323+0000 mgr.y (mgr.24491) 949 : cluster [DBG] pgmap v1746: 142 pgs: 142 active+clean; 583 KiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.715345+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.9 scrub starts 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: cluster 2024-09-20T19:43:26.719212+0000 osd.4 (osd.4) 316 : cluster [DBG] 4.9 scrub ok 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:28.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:27 smithi162 bash[24557]: audit 2024-09-20T19:43:27.541940+0000 mon.a (mon.0) 2864 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:43:28.185 INFO:teuthology.orchestra.run.smithi042.stdout:55834575230 2024-09-20T19:43:28.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:28.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:28.949 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:28.950 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:28 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:27.738594+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.2 scrub starts 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:27.740669+0000 osd.5 (osd.5) 450 : cluster [DBG] 2.2 scrub ok 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.261922+0000 mon.a (mon.0) 2865 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.262876+0000 mon.a (mon.0) 2866 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: audit 2024-09-20T19:43:28.269894+0000 mon.a (mon.0) 2867 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' 2024-09-20T19:43:29.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:29.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.387535+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.8 scrub starts 2024-09-20T19:43:29.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:29.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:28 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.388533+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.8 scrub ok 2024-09-20T19:43:29.104 INFO:teuthology.orchestra.run.smithi042.stdout:188978561358 2024-09-20T19:43:29.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T19:43:29.905 INFO:teuthology.orchestra.run.smithi042.stdout:2147483648161 2024-09-20T19:43:29.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T19:43:29.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:29.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:29.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:29.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:29.914 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[32075]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:29.915 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:29.916 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:29 smithi042 bash[22706]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:30.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.140393+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.141966+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: cluster 2024-09-20T19:43:28.629727+0000 mgr.y (mgr.24491) 950 : cluster [DBG] pgmap v1747: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T19:43:30.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:30.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:29 smithi162 bash[24557]: audit 2024-09-20T19:43:29.194461+0000 mon.a (mon.0) 2868 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:30.335 INFO:teuthology.orchestra.run.smithi042.stdout:231928234308 2024-09-20T19:43:30.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T19:43:30.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:30.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:30.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:30.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:30.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:30 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.096890+0000 osd.3 (osd.3) 149 : cluster [DBG] 6.6 scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.098509+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.6 scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.163913+0000 osd.0 (osd.0) 235 : cluster [DBG] 4.6 scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:29.166685+0000 osd.0 (osd.0) 236 : cluster [DBG] 4.6 scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.171240+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub starts 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:31.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.172831+0000 osd.1 (osd.1) 482 : cluster [DBG] 6.1d scrub ok 2024-09-20T19:43:31.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:31.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.425557+0000 osd.2 (osd.2) 429 : cluster [DBG] 6.f scrub starts 2024-09-20T19:43:31.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:31.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:30 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.427571+0000 osd.2 (osd.2) 430 : cluster [DBG] 6.f scrub ok 2024-09-20T19:43:31.562 INFO:teuthology.orchestra.run.smithi042.stdout:81604378995 2024-09-20T19:43:31.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T19:43:31.586 INFO:teuthology.orchestra.run.smithi042.stdout:274877907258 2024-09-20T19:43:31.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:31.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:31.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[32075]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:31.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:31.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:31.936 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:31 smithi042 bash[22706]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.110600+0000 osd.3 (osd.3) 151 : cluster [DBG] 4.17 scrub starts 2024-09-20T19:43:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.112933+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.17 scrub ok 2024-09-20T19:43:32.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.152661+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub starts 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.154014+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.16 scrub ok 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.220152+0000 osd.0 (osd.0) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: audit 2024-09-20T19:43:30.221711+0000 mgr.y (mgr.24491) 951 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.224750+0000 osd.0 (osd.0) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.630529+0000 mgr.y (mgr.24491) 952 : cluster [DBG] pgmap v1748: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-09-20T19:43:32.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:30.668749+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e1100: 8 total, 8 up, 8 in 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.085783+0000 osd.7 (osd.7) 357 : cluster [DBG] 5.9 scrub starts 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.088904+0000 osd.7 (osd.7) 358 : cluster [DBG] 5.9 scrub ok 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.172061+0000 osd.1 (osd.1) 483 : cluster [DBG] 5.c scrub starts 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:32.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:31 smithi162 bash[24557]: cluster 2024-09-20T19:43:31.174221+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.c scrub ok 2024-09-20T19:43:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:33.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:33.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:33.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:33.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:33 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.113591+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.1b scrub starts 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.115256+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1b scrub ok 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:32.631879+0000 mgr.y (mgr.24491) 953 : cluster [DBG] pgmap v1750: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-20T19:43:34.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.191728+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.4 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.196861+0000 osd.1 (osd.1) 486 : cluster [DBG] 4.4 scrub ok 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.426897+0000 osd.6 (osd.6) 334 : cluster [DBG] 1.6 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.429617+0000 osd.6 (osd.6) 335 : cluster [DBG] 1.6 scrub ok 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.441960+0000 osd.2 (osd.2) 431 : cluster [DBG] 6.14 scrub starts 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:34.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:33 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.444404+0000 osd.2 (osd.2) 432 : cluster [DBG] 6.14 scrub ok 2024-09-20T19:43:34.337 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.337 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.339 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.339 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.339 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.340 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:34.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:34.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:34.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:34.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:34 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.981227+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.15 scrub starts 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:33.983599+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub ok 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.242448+0000 osd.1 (osd.1) 487 : cluster [DBG] 4.2 scrub starts 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:35.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:34 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.246530+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.2 scrub ok 2024-09-20T19:43:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:36.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:35 smithi162 bash[24557]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.180 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[32075]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:34.632849+0000 mgr.y (mgr.24491) 954 : cluster [DBG] pgmap v1751: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.181 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:35.419332+0000 osd.2 (osd.2) 433 : cluster [DBG] 2.5 scrub starts 2024-09-20T19:43:36.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:35 smithi042 bash[22706]: cluster 2024-09-20T19:43:35.420879+0000 osd.2 (osd.2) 434 : cluster [DBG] 2.5 scrub ok 2024-09-20T19:43:36.377 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:36.681 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:36 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:37 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:36.633357+0000 mgr.y (mgr.24491) 955 : cluster [DBG] pgmap v1752: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.265018+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.9 scrub starts 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.267310+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.9 scrub ok 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.426885+0000 osd.2 (osd.2) 435 : cluster [DBG] 4.7 scrub starts 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:37 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.431286+0000 osd.2 (osd.2) 436 : cluster [DBG] 4.7 scrub ok 2024-09-20T19:43:38.652 INFO:teuthology.orchestra.run.smithi042.stdout:231928234310 2024-09-20T19:43:38.717 INFO:teuthology.orchestra.run.smithi042.stdout:188978561360 2024-09-20T19:43:38.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:38.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[32075]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:38.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:38.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:38.935 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:38 smithi042 bash[22706]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:38.995 INFO:teuthology.orchestra.run.smithi042.stdout:2147483648163 2024-09-20T19:43:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.563052+0000 osd.5 (osd.5) 451 : cluster [DBG] 5.7 scrub starts 2024-09-20T19:43:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.565248+0000 osd.5 (osd.5) 452 : cluster [DBG] 5.7 scrub ok 2024-09-20T19:43:39.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.639419+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:37.641677+0000 osd.4 (osd.4) 318 : cluster [DBG] 5.1 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.014074+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.3 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.016220+0000 osd.7 (osd.7) 362 : cluster [DBG] 6.3 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.261923+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.15 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.265768+0000 osd.1 (osd.1) 492 : cluster [DBG] 2.15 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.376426+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.5 scrub starts 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:39.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.381152+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.5 scrub ok 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.475596+0000 osd.2 (osd.2) 437 : cluster [DBG] 2.c scrub starts 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.476939+0000 osd.2 (osd.2) 438 : cluster [DBG] 2.c scrub ok 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:39.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:38 smithi162 bash[24557]: audit 2024-09-20T19:43:38.651476+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.42:0/2874247015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-20T19:43:39.560 INFO:teuthology.orchestra.run.smithi042.stdout:55834575232 2024-09-20T19:43:39.620 INFO:teuthology.orchestra.run.smithi042.stdout:2362232012954 2024-09-20T19:43:39.715 INFO:teuthology.orchestra.run.smithi042.stdout:2241972928671 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:39.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[32075]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:39.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:39.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:39 smithi042 bash[22706]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.634871+0000 mgr.y (mgr.24491) 956 : cluster [DBG] pgmap v1753: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.685081+0000 osd.4 (osd.4) 319 : cluster [DBG] 6.1e scrub starts 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: cluster 2024-09-20T19:43:38.688501+0000 osd.4 (osd.4) 320 : cluster [DBG] 6.1e scrub ok 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:38.716313+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? v1:172.21.15.42:0/2365212289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-20T19:43:40.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:40.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:38.994550+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? v1:172.21.15.42:0/3720421382' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-20T19:43:40.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:40.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:39.559277+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? v1:172.21.15.42:0/675886972' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-20T19:43:40.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:40.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:39 smithi162 bash[24557]: audit 2024-09-20T19:43:39.619446+0000 mon.a (mon.0) 2870 : audit [DBG] from='client.? v1:172.21.15.42:0/2497800013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-20T19:43:40.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:40.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[32075]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:40.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:40 smithi042 bash[22706]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:41.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.635709+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-09-20T19:43:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.639354+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-09-20T19:43:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:41.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:40 smithi162 bash[24557]: audit 2024-09-20T19:43:39.714270+0000 mon.a (mon.0) 2871 : audit [DBG] from='client.? v1:172.21.15.42:0/1724217950' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-20T19:43:41.641 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2147483648161 got 2147483648163 for osd.0 2024-09-20T19:43:41.641 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:41.656 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234308 got 231928234310 for osd.6 2024-09-20T19:43:41.656 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:41.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561358 got 188978561360 for osd.5 2024-09-20T19:43:41.657 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:41.765 INFO:teuthology.orchestra.run.smithi042.stdout:274877907260 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:41.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:41.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:41.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:41.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:41 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:41.959 INFO:teuthology.orchestra.run.smithi042.stdout:81604378997 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.922967+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:39.925038+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: audit 2024-09-20T19:43:40.232984+0000 mgr.y (mgr.24491) 957 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.553082+0000 osd.5 (osd.5) 453 : cluster [DBG] 6.19 scrub starts 2024-09-20T19:43:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.555946+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.19 scrub ok 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.635906+0000 mgr.y (mgr.24491) 958 : cluster [DBG] pgmap v1754: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.700197+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e1101: 8 total, 8 up, 8 in 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.928798+0000 osd.7 (osd.7) 363 : cluster [DBG] 1.0 scrub starts 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:40.931423+0000 osd.7 (osd.7) 364 : cluster [DBG] 1.0 scrub ok 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.357689+0000 osd.1 (osd.1) 493 : cluster [DBG] 1.3 scrub starts 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:42.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:41 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.360078+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:42 smithi162 bash[24557]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.190 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.190 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.190 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[22706]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.191 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.232672+0000 osd.3 (osd.3) 157 : cluster [DBG] 5.13 scrub starts 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.236511+0000 osd.3 (osd.3) 158 : cluster [DBG] 5.13 scrub ok 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.509853+0000 osd.5 (osd.5) 455 : cluster [DBG] 2.18 scrub starts 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: cluster 2024-09-20T19:43:41.512076+0000 osd.5 (osd.5) 456 : cluster [DBG] 2.18 scrub ok 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: audit 2024-09-20T19:43:41.764214+0000 mon.a (mon.0) 2873 : audit [DBG] from='client.? v1:172.21.15.42:0/196140726' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.192 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:42 smithi042 bash[32075]: audit 2024-09-20T19:43:41.957653+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? v1:172.21.15.42:0/3160728245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:43.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[32075]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:43.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:43.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:43 smithi042 bash[22706]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:44.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575230 got 55834575232 for osd.1 2024-09-20T19:43:44.035 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.055785+0000 osd.0 (osd.0) 241 : cluster [DBG] 2.1f scrub starts 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.057320+0000 osd.0 (osd.0) 242 : cluster [DBG] 2.1f scrub ok 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.490622+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.8 scrub starts 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.493526+0000 osd.5 (osd.5) 458 : cluster [DBG] 1.8 scrub ok 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:44.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:42.637324+0000 mgr.y (mgr.24491) 959 : cluster [DBG] pgmap v1756: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:43.400335+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub starts 2024-09-20T19:43:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:44.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:43 smithi162 bash[24557]: cluster 2024-09-20T19:43:43.403948+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.6 scrub ok 2024-09-20T19:43:44.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2362232012951 got 2362232012954 for osd.3 2024-09-20T19:43:44.189 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:44.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2241972928668 got 2241972928671 for osd.4 2024-09-20T19:43:44.236 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:44.390 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378995 got 81604378997 for osd.2 2024-09-20T19:43:44.391 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:44.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877907258 got 274877907260 for osd.7 2024-09-20T19:43:44.440 DEBUG:teuthology.parallel:result is None 2024-09-20T19:43:44.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T19:43:44.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:45.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:44 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: audit 2024-09-20T19:43:44.215243+0000 mon.a (mon.0) 2874 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.440083+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.4 scrub starts 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.441893+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.4 scrub ok 2024-09-20T19:43:45.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.485945+0000 osd.2 (osd.2) 439 : cluster [DBG] 5.6 scrub starts 2024-09-20T19:43:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:45.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:44 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.488551+0000 osd.2 (osd.2) 440 : cluster [DBG] 5.6 scrub ok 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:45 smithi162 bash[24557]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[32075]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.088209+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub starts 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.092278+0000 osd.0 (osd.0) 244 : cluster [DBG] 4.b scrub ok 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.537681+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.3 scrub starts 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.539938+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.3 scrub ok 2024-09-20T19:43:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:45 smithi042 bash[22706]: cluster 2024-09-20T19:43:44.638214+0000 mgr.y (mgr.24491) 960 : cluster [DBG] pgmap v1757: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:46.680 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:47.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:47.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:46 smithi162 bash[24557]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[32075]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.492622+0000 osd.5 (osd.5) 461 : cluster [DBG] 6.d scrub starts 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.495308+0000 osd.5 (osd.5) 462 : cluster [DBG] 6.d scrub ok 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.547694+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:47.182 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:46 smithi042 bash[22706]: cluster 2024-09-20T19:43:45.551982+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:48.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:47 smithi162 bash[24557]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.182 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[32075]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.183 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.093688+0000 osd.0 (osd.0) 245 : cluster [DBG] 5.3 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.096706+0000 osd.0 (osd.0) 246 : cluster [DBG] 5.3 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.239242+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.10 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.242415+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.10 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.507690+0000 osd.5 (osd.5) 463 : cluster [DBG] 5.16 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.509968+0000 osd.5 (osd.5) 464 : cluster [DBG] 5.16 scrub ok 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:46.638809+0000 mgr.y (mgr.24491) 961 : cluster [DBG] pgmap v1758: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:47.385548+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-20T19:43:48.184 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:48.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:47 smithi042 bash[22706]: cluster 2024-09-20T19:43:47.387635+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-20T19:43:49.257 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:49 smithi042 bash[22706]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:49.681 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:49 smithi042 bash[22706]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:49.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:49 smithi042 bash[32075]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:49.681 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:49 smithi042 bash[32075]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:49 smithi162 bash[24557]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:50.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:49 smithi162 bash[24557]: cluster 2024-09-20T19:43:48.640315+0000 mgr.y (mgr.24491) 962 : cluster [DBG] pgmap v1759: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:50.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:43:50.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:50.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[32075]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:50.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:50 smithi042 bash[22706]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:50.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:43:50.963 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.260015+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.14 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.263091+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.14 scrub ok 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.477948+0000 osd.1 (osd.1) 499 : cluster [DBG] 6.4 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.480221+0000 osd.1 (osd.1) 500 : cluster [DBG] 6.4 scrub ok 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.558821+0000 osd.4 (osd.4) 325 : cluster [DBG] 1.5 scrub starts 2024-09-20T19:43:51.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:51.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:50 smithi162 bash[24557]: cluster 2024-09-20T19:43:49.561488+0000 osd.4 (osd.4) 326 : cluster [DBG] 1.5 scrub ok 2024-09-20T19:43:51.751 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1760,"stamp":"2024-09-20T19:43:50.640542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3061,"num_read_kb":2813,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":858818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1971,"ondisk_log_size":1971,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":524028,"kb_used_data":7932,"kb_used_omap":143,"kb_used_meta":515824,"kb_avail":749437188,"statfs":{"total":767960285184,"available":767423680512,"internally_reserved":0,"allocated":8122368,"data_stored":5025641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147312,"internal_metadata":528203920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005282"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234874+0000","last_change":"2024-09-20T19:43:32.115321+0000","last_active":"2024-09-20T19:43:41.234874+0000","last_peered":"2024-09-20T19:43:41.234874+0000","last_clean":"2024-09-20T19:43:41.234874+0000","last_became_active":"2024-09-20T19:40:44.720052+0000","last_became_peered":"2024-09-20T19:40:44.720052+0000","last_unstale":"2024-09-20T19:43:41.234874+0000","last_undegraded":"2024-09-20T19:43:41.234874+0000","last_fullsized":"2024-09-20T19:43:41.234874+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:32.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:32.115260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.334406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1952,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:42.057372+0000","last_change":"2024-09-20T19:43:42.057372+0000","last_active":"2024-09-20T19:43:42.057372+0000","last_peered":"2024-09-20T19:43:42.057372+0000","last_clean":"2024-09-20T19:43:42.057372+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:43:42.057372+0000","last_undegraded":"2024-09-20T19:43:42.057372+0000","last_fullsized":"2024-09-20T19:43:42.057372+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:42.057324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:42.057324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.036559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"987'39","reported_seq":113,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.231408+0000","last_change":"2024-09-20T19:43:09.288423+0000","last_active":"2024-09-20T19:43:41.231408+0000","last_peered":"2024-09-20T19:43:41.231408+0000","last_clean":"2024-09-20T19:43:41.231408+0000","last_became_active":"2024-09-20T19:40:44.709477+0000","last_became_peered":"2024-09-20T19:40:44.709477+0000","last_unstale":"2024-09-20T19:43:41.231408+0000","last_undegraded":"2024-09-20T19:43:41.231408+0000","last_fullsized":"2024-09-20T19:43:41.231408+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:43:09.288269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:09.288269+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:14.727998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1931,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699384+0000","last_change":"2024-09-20T19:43:06.736048+0000","last_active":"2024-09-20T19:43:40.699384+0000","last_peered":"2024-09-20T19:43:40.699384+0000","last_clean":"2024-09-20T19:43:40.699384+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:43:40.699384+0000","last_undegraded":"2024-09-20T19:43:40.699384+0000","last_fullsized":"2024-09-20T19:43:40.699384+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.735902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.735902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.049403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236163+0000","last_change":"2024-09-20T19:42:33.237917+0000","last_active":"2024-09-20T19:43:41.236163+0000","last_peered":"2024-09-20T19:43:41.236163+0000","last_clean":"2024-09-20T19:43:41.236163+0000","last_became_active":"2024-09-20T19:40:44.713295+0000","last_became_peered":"2024-09-20T19:40:44.713295+0000","last_unstale":"2024-09-20T19:43:41.236163+0000","last_undegraded":"2024-09-20T19:43:41.236163+0000","last_fullsized":"2024-09-20T19:43:41.236163+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:33.237754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:33.237754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.896189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"987'21","reported_seq":2044,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699793+0000","last_change":"2024-09-20T19:43:39.639518+0000","last_active":"2024-09-20T19:43:40.699793+0000","last_peered":"2024-09-20T19:43:40.699793+0000","last_clean":"2024-09-20T19:43:40.699793+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:43:40.699793+0000","last_undegraded":"2024-09-20T19:43:40.699793+0000","last_fullsized":"2024-09-20T19:43:40.699793+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:39.639367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:39.639367+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:09.093303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":2171,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700471+0000","last_change":"2024-09-20T19:43:16.071198+0000","last_active":"2024-09-20T19:43:40.700471+0000","last_peered":"2024-09-20T19:43:40.700471+0000","last_clean":"2024-09-20T19:43:40.700471+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:43:40.700471+0000","last_undegraded":"2024-09-20T19:43:40.700471+0000","last_fullsized":"2024-09-20T19:43:40.700471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:16.071077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.071077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:30.584638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1993,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700502+0000","last_change":"2024-09-20T19:42:43.656390+0000","last_active":"2024-09-20T19:43:40.700502+0000","last_peered":"2024-09-20T19:43:40.700502+0000","last_clean":"2024-09-20T19:43:40.700502+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:43:40.700502+0000","last_undegraded":"2024-09-20T19:43:40.700502+0000","last_fullsized":"2024-09-20T19:43:40.700502+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:43.656267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:43.656267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:59.187062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":2161,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.870080+0000","last_change":"2024-09-20T19:43:02.103353+0000","last_active":"2024-09-20T19:43:40.870080+0000","last_peered":"2024-09-20T19:43:40.870080+0000","last_clean":"2024-09-20T19:43:40.870080+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:43:40.870080+0000","last_undegraded":"2024-09-20T19:43:40.870080+0000","last_fullsized":"2024-09-20T19:43:40.870080+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:02.103206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:02.103206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.000717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"987'9","reported_seq":2054,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700485+0000","last_change":"2024-09-20T19:42:51.704440+0000","last_active":"2024-09-20T19:43:40.700485+0000","last_peered":"2024-09-20T19:43:40.700485+0000","last_clean":"2024-09-20T19:43:40.700485+0000","last_became_active":"2024-09-20T19:40:44.709660+0000","last_became_peered":"2024-09-20T19:40:44.709660+0000","last_unstale":"2024-09-20T19:43:40.700485+0000","last_undegraded":"2024-09-20T19:43:40.700485+0000","last_fullsized":"2024-09-20T19:43:40.700485+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'9","last_scrub_stamp":"2024-09-20T19:42:51.704299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:51.704299+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:20.045871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":2153,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868763+0000","last_change":"2024-09-20T19:43:21.144988+0000","last_active":"2024-09-20T19:43:40.868763+0000","last_peered":"2024-09-20T19:43:40.868763+0000","last_clean":"2024-09-20T19:43:40.868763+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:43:40.868763+0000","last_undegraded":"2024-09-20T19:43:40.868763+0000","last_fullsized":"2024-09-20T19:43:40.868763+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:21.144862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:21.144862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.664498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2169,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304395+0000","last_change":"2024-09-20T19:43:40.556085+0000","last_active":"2024-09-20T19:43:41.304395+0000","last_peered":"2024-09-20T19:43:41.304395+0000","last_clean":"2024-09-20T19:43:41.304395+0000","last_became_active":"2024-09-20T19:40:44.711101+0000","last_became_peered":"2024-09-20T19:40:44.711101+0000","last_unstale":"2024-09-20T19:43:41.304395+0000","last_undegraded":"2024-09-20T19:43:41.304395+0000","last_fullsized":"2024-09-20T19:43:41.304395+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:40.555955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:40.555955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:48.345498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2150,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869690+0000","last_change":"2024-09-20T19:43:08.234634+0000","last_active":"2024-09-20T19:43:40.869690+0000","last_peered":"2024-09-20T19:43:40.869690+0000","last_clean":"2024-09-20T19:43:40.869690+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:43:40.869690+0000","last_undegraded":"2024-09-20T19:43:40.869690+0000","last_fullsized":"2024-09-20T19:43:40.869690+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:08.234509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:08.234509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.022931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"987'21","reported_seq":2097,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:45.552166+0000","last_change":"2024-09-20T19:43:45.552166+0000","last_active":"2024-09-20T19:43:45.552166+0000","last_peered":"2024-09-20T19:43:45.552166+0000","last_clean":"2024-09-20T19:43:45.552166+0000","last_became_active":"2024-09-20T19:40:44.707865+0000","last_became_peered":"2024-09-20T19:40:44.707865+0000","last_unstale":"2024-09-20T19:43:45.552166+0000","last_undegraded":"2024-09-20T19:43:45.552166+0000","last_fullsized":"2024-09-20T19:43:45.552166+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:45.551993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:45.551993+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:02.939959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2143,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305902+0000","last_change":"2024-09-20T19:43:18.636559+0000","last_active":"2024-09-20T19:43:41.305902+0000","last_peered":"2024-09-20T19:43:41.305902+0000","last_clean":"2024-09-20T19:43:41.305902+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:43:41.305902+0000","last_undegraded":"2024-09-20T19:43:41.305902+0000","last_fullsized":"2024-09-20T19:43:41.305902+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:18.636421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:18.636421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.222169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1121,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923447+0000","last_change":"2024-09-20T19:42:41.303707+0000","last_active":"2024-09-20T19:43:40.923447+0000","last_peered":"2024-09-20T19:43:40.923447+0000","last_clean":"2024-09-20T19:43:40.923447+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:43:40.923447+0000","last_undegraded":"2024-09-20T19:43:40.923447+0000","last_fullsized":"2024-09-20T19:43:40.923447+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:41.303556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:41.303556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:44.445641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1799,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:42.493667+0000","last_change":"2024-09-20T19:43:42.493667+0000","last_active":"2024-09-20T19:43:42.493667+0000","last_peered":"2024-09-20T19:43:42.493667+0000","last_clean":"2024-09-20T19:43:42.493667+0000","last_became_active":"2024-09-20T19:40:44.706211+0000","last_became_peered":"2024-09-20T19:40:44.706211+0000","last_unstale":"2024-09-20T19:43:42.493667+0000","last_undegraded":"2024-09-20T19:43:42.493667+0000","last_fullsized":"2024-09-20T19:43:42.493667+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:42.493535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:42.493535+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:58.459920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"987'45","reported_seq":2034,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700994+0000","last_change":"2024-09-20T19:43:21.673082+0000","last_active":"2024-09-20T19:43:40.700994+0000","last_peered":"2024-09-20T19:43:40.700994+0000","last_clean":"2024-09-20T19:43:40.700994+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:43:40.700994+0000","last_undegraded":"2024-09-20T19:43:40.700994+0000","last_fullsized":"2024-09-20T19:43:40.700994+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'45","last_scrub_stamp":"2024-09-20T19:43:21.672879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:21.672879+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.663091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2154,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.870219+0000","last_change":"2024-09-20T19:43:16.175369+0000","last_active":"2024-09-20T19:43:40.870219+0000","last_peered":"2024-09-20T19:43:40.870219+0000","last_clean":"2024-09-20T19:43:40.870219+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:43:40.870219+0000","last_undegraded":"2024-09-20T19:43:40.870219+0000","last_fullsized":"2024-09-20T19:43:40.870219+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:16.175256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.175256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:17.100865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":2174,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701840+0000","last_change":"2024-09-20T19:43:31.174354+0000","last_active":"2024-09-20T19:43:40.701840+0000","last_peered":"2024-09-20T19:43:40.701840+0000","last_clean":"2024-09-20T19:43:40.701840+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:43:40.701840+0000","last_undegraded":"2024-09-20T19:43:40.701840+0000","last_fullsized":"2024-09-20T19:43:40.701840+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:31.174232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:31.174232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.759389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":748,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701689+0000","last_change":"2024-09-20T19:43:30.427709+0000","last_active":"2024-09-20T19:43:40.701689+0000","last_peered":"2024-09-20T19:43:40.701689+0000","last_clean":"2024-09-20T19:43:40.701689+0000","last_became_active":"2024-09-20T19:40:44.698643+0000","last_became_peered":"2024-09-20T19:40:44.698643+0000","last_unstale":"2024-09-20T19:43:40.701689+0000","last_undegraded":"2024-09-20T19:43:40.701689+0000","last_fullsized":"2024-09-20T19:43:40.701689+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.427581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.427581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:43.576650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":2247,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868011+0000","last_change":"2024-09-20T19:43:23.076869+0000","last_active":"2024-09-20T19:43:40.868011+0000","last_peered":"2024-09-20T19:43:40.868011+0000","last_clean":"2024-09-20T19:43:40.868011+0000","last_became_active":"2024-09-20T19:40:44.703846+0000","last_became_peered":"2024-09-20T19:40:44.703846+0000","last_unstale":"2024-09-20T19:43:40.868011+0000","last_undegraded":"2024-09-20T19:43:40.868011+0000","last_fullsized":"2024-09-20T19:43:40.868011+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:23.076715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.076715+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.917049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"987'30","reported_seq":2098,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700861+0000","last_change":"2024-09-20T19:42:37.674511+0000","last_active":"2024-09-20T19:43:40.700861+0000","last_peered":"2024-09-20T19:43:40.700861+0000","last_clean":"2024-09-20T19:43:40.700861+0000","last_became_active":"2024-09-20T19:40:44.707964+0000","last_became_peered":"2024-09-20T19:40:44.707964+0000","last_unstale":"2024-09-20T19:43:40.700861+0000","last_undegraded":"2024-09-20T19:43:40.700861+0000","last_fullsized":"2024-09-20T19:43:40.700861+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:42:37.674338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:37.674338+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:03.653539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":2197,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701242+0000","last_change":"2024-09-20T19:42:53.016414+0000","last_active":"2024-09-20T19:43:40.701242+0000","last_peered":"2024-09-20T19:43:40.701242+0000","last_clean":"2024-09-20T19:43:40.701242+0000","last_became_active":"2024-09-20T19:40:44.713805+0000","last_became_peered":"2024-09-20T19:40:44.713805+0000","last_unstale":"2024-09-20T19:43:40.701242+0000","last_undegraded":"2024-09-20T19:43:40.701242+0000","last_fullsized":"2024-09-20T19:43:40.701242+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:53.016283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:53.016283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:05.103857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":2490,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701602+0000","last_change":"2024-09-20T19:42:53.546824+0000","last_active":"2024-09-20T19:43:40.701602+0000","last_peered":"2024-09-20T19:43:40.701602+0000","last_clean":"2024-09-20T19:43:40.701602+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:43:40.701602+0000","last_undegraded":"2024-09-20T19:43:40.701602+0000","last_fullsized":"2024-09-20T19:43:40.701602+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:42:53.546647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:53.546647+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:00.137973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2003,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701080+0000","last_change":"2024-09-20T19:42:55.630806+0000","last_active":"2024-09-20T19:43:40.701080+0000","last_peered":"2024-09-20T19:43:40.701080+0000","last_clean":"2024-09-20T19:43:40.701080+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:43:40.701080+0000","last_undegraded":"2024-09-20T19:43:40.701080+0000","last_fullsized":"2024-09-20T19:43:40.701080+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:55.630688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:55.630688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:09.663889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1751,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305604+0000","last_change":"2024-09-20T19:43:24.674112+0000","last_active":"2024-09-20T19:43:41.305604+0000","last_peered":"2024-09-20T19:43:41.305604+0000","last_clean":"2024-09-20T19:43:41.305604+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:43:41.305604+0000","last_undegraded":"2024-09-20T19:43:41.305604+0000","last_fullsized":"2024-09-20T19:43:41.305604+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:24.673991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:24.673991+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.741203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"987'39","reported_seq":2298,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702508+0000","last_change":"2024-09-20T19:42:35.126118+0000","last_active":"2024-09-20T19:43:40.702508+0000","last_peered":"2024-09-20T19:43:40.702508+0000","last_clean":"2024-09-20T19:43:40.702508+0000","last_became_active":"2024-09-20T19:40:44.712608+0000","last_became_peered":"2024-09-20T19:40:44.712608+0000","last_unstale":"2024-09-20T19:43:40.702508+0000","last_undegraded":"2024-09-20T19:43:40.702508+0000","last_fullsized":"2024-09-20T19:43:40.702508+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:35.125939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:35.125939+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.306367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":2205,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700621+0000","last_change":"2024-09-20T19:43:37.267471+0000","last_active":"2024-09-20T19:43:40.700621+0000","last_peered":"2024-09-20T19:43:40.700621+0000","last_clean":"2024-09-20T19:43:40.700621+0000","last_became_active":"2024-09-20T19:40:44.712329+0000","last_became_peered":"2024-09-20T19:40:44.712329+0000","last_unstale":"2024-09-20T19:43:40.700621+0000","last_undegraded":"2024-09-20T19:43:40.700621+0000","last_fullsized":"2024-09-20T19:43:40.700621+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.267320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.267320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.571815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":2277,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:43.543227+0000","last_change":"2024-09-20T19:42:54.650140+0000","last_active":"2024-09-20T19:43:43.543227+0000","last_peered":"2024-09-20T19:43:43.543227+0000","last_clean":"2024-09-20T19:43:43.543227+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:43:43.543227+0000","last_undegraded":"2024-09-20T19:43:43.543227+0000","last_fullsized":"2024-09-20T19:43:43.543227+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:42:54.650016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:54.650016+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:18.726791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2172,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:45.495446+0000","last_change":"2024-09-20T19:43:45.495446+0000","last_active":"2024-09-20T19:43:45.495446+0000","last_peered":"2024-09-20T19:43:45.495446+0000","last_clean":"2024-09-20T19:43:45.495446+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:43:45.495446+0000","last_undegraded":"2024-09-20T19:43:45.495446+0000","last_fullsized":"2024-09-20T19:43:45.495446+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:45.495318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:45.495318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:06.383492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"987'27","reported_seq":2057,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701655+0000","last_change":"2024-09-20T19:42:45.695604+0000","last_active":"2024-09-20T19:43:40.701655+0000","last_peered":"2024-09-20T19:43:40.701655+0000","last_clean":"2024-09-20T19:43:40.701655+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:43:40.701655+0000","last_undegraded":"2024-09-20T19:43:40.701655+0000","last_fullsized":"2024-09-20T19:43:40.701655+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:42:45.695469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:45.695469+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:03.816337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.698947+0000","last_change":"2024-09-20T19:43:28.388629+0000","last_active":"2024-09-20T19:43:40.698947+0000","last_peered":"2024-09-20T19:43:40.698947+0000","last_clean":"2024-09-20T19:43:40.698947+0000","last_became_active":"2024-09-20T19:40:44.685956+0000","last_became_peered":"2024-09-20T19:40:44.685956+0000","last_unstale":"2024-09-20T19:43:40.698947+0000","last_undegraded":"2024-09-20T19:43:40.698947+0000","last_fullsized":"2024-09-20T19:43:40.698947+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:28.388542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:28.388542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.024679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306960+0000","last_change":"2024-09-20T19:43:06.757297+0000","last_active":"2024-09-20T19:43:41.306960+0000","last_peered":"2024-09-20T19:43:41.306960+0000","last_clean":"2024-09-20T19:43:41.306960+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:43:41.306960+0000","last_undegraded":"2024-09-20T19:43:41.306960+0000","last_fullsized":"2024-09-20T19:43:41.306960+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.757180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.757180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:16.939577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.235262+0000","last_change":"2024-09-20T19:43:10.269113+0000","last_active":"2024-09-20T19:43:41.235262+0000","last_peered":"2024-09-20T19:43:41.235262+0000","last_clean":"2024-09-20T19:43:41.235262+0000","last_became_active":"2024-09-20T19:40:44.720237+0000","last_became_peered":"2024-09-20T19:40:44.720237+0000","last_unstale":"2024-09-20T19:43:41.235262+0000","last_undegraded":"2024-09-20T19:43:41.235262+0000","last_fullsized":"2024-09-20T19:43:41.235262+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:10.268986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:10.268986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:20.106838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":101,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.235321+0000","last_change":"2024-09-20T19:42:31.196114+0000","last_active":"2024-09-20T19:43:41.235321+0000","last_peered":"2024-09-20T19:43:41.235321+0000","last_clean":"2024-09-20T19:43:41.235321+0000","last_became_active":"2024-09-20T19:40:44.706650+0000","last_became_peered":"2024-09-20T19:40:44.706650+0000","last_unstale":"2024-09-20T19:43:41.235321+0000","last_undegraded":"2024-09-20T19:43:41.235321+0000","last_fullsized":"2024-09-20T19:43:41.235321+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:42:31.196025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:42:31.196025+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:43.875398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"987'42","reported_seq":2036,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699999+0000","last_change":"2024-09-20T19:42:28.762499+0000","last_active":"2024-09-20T19:43:40.699999+0000","last_peered":"2024-09-20T19:43:40.699999+0000","last_clean":"2024-09-20T19:43:40.699999+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:43:40.699999+0000","last_undegraded":"2024-09-20T19:43:40.699999+0000","last_fullsized":"2024-09-20T19:43:40.699999+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:42:28.762361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:28.762361+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:54.586443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2152,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.930178+0000","last_change":"2024-09-20T19:42:07.381183+0000","last_active":"2024-09-20T19:43:40.930178+0000","last_peered":"2024-09-20T19:43:40.930178+0000","last_clean":"2024-09-20T19:43:40.930178+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:43:40.930178+0000","last_undegraded":"2024-09-20T19:43:40.930178+0000","last_fullsized":"2024-09-20T19:43:40.930178+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:07.381059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:07.381059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1066'17","reported_seq":138,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.230521+0000","last_change":"2024-09-20T19:42:44.309725+0000","last_active":"2024-09-20T19:43:41.230521+0000","last_peered":"2024-09-20T19:43:41.230521+0000","last_clean":"2024-09-20T19:43:41.230521+0000","last_became_active":"2024-09-20T19:40:44.707312+0000","last_became_peered":"2024-09-20T19:40:44.707312+0000","last_unstale":"2024-09-20T19:43:41.230521+0000","last_undegraded":"2024-09-20T19:43:41.230521+0000","last_fullsized":"2024-09-20T19:43:41.230521+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'17","last_scrub_stamp":"2024-09-20T19:42:44.309554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:44.309554+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.482497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2140,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869362+0000","last_change":"2024-09-20T19:43:38.016361+0000","last_active":"2024-09-20T19:43:40.869362+0000","last_peered":"2024-09-20T19:43:40.869362+0000","last_clean":"2024-09-20T19:43:40.869362+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:43:40.869362+0000","last_undegraded":"2024-09-20T19:43:40.869362+0000","last_fullsized":"2024-09-20T19:43:40.869362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.016233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.016233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.872867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1562,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701290+0000","last_change":"2024-09-20T19:42:21.714593+0000","last_active":"2024-09-20T19:43:40.701290+0000","last_peered":"2024-09-20T19:43:40.701290+0000","last_clean":"2024-09-20T19:43:40.701290+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:43:40.701290+0000","last_undegraded":"2024-09-20T19:43:40.701290+0000","last_fullsized":"2024-09-20T19:43:40.701290+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:42:21.714466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:42:21.714466+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:45.987252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"987'42","reported_seq":115,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.233243+0000","last_change":"2024-09-20T19:42:42.265268+0000","last_active":"2024-09-20T19:43:41.233243+0000","last_peered":"2024-09-20T19:43:41.233243+0000","last_clean":"2024-09-20T19:43:41.233243+0000","last_became_active":"2024-09-20T19:40:44.709362+0000","last_became_peered":"2024-09-20T19:40:44.709362+0000","last_unstale":"2024-09-20T19:43:41.233243+0000","last_undegraded":"2024-09-20T19:43:41.233243+0000","last_fullsized":"2024-09-20T19:43:41.233243+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:42:42.265093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:42.265093+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:04.684731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":2176,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:43.404096+0000","last_change":"2024-09-20T19:43:43.404096+0000","last_active":"2024-09-20T19:43:43.404096+0000","last_peered":"2024-09-20T19:43:43.404096+0000","last_clean":"2024-09-20T19:43:43.404096+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:43:43.404096+0000","last_undegraded":"2024-09-20T19:43:43.404096+0000","last_fullsized":"2024-09-20T19:43:43.404096+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:43.403961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:43.403961+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.066903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1989,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699384+0000","last_change":"2024-09-20T19:43:37.641806+0000","last_active":"2024-09-20T19:43:40.699384+0000","last_peered":"2024-09-20T19:43:40.699384+0000","last_clean":"2024-09-20T19:43:40.699384+0000","last_became_active":"2024-09-20T19:40:44.699997+0000","last_became_peered":"2024-09-20T19:40:44.699997+0000","last_unstale":"2024-09-20T19:43:40.699384+0000","last_undegraded":"2024-09-20T19:43:40.699384+0000","last_fullsized":"2024-09-20T19:43:40.699384+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.641686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.641686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.606200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1935,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699863+0000","last_change":"2024-09-20T19:42:48.764388+0000","last_active":"2024-09-20T19:43:40.699863+0000","last_peered":"2024-09-20T19:43:40.699863+0000","last_clean":"2024-09-20T19:43:40.699863+0000","last_became_active":"2024-09-20T19:40:44.708574+0000","last_became_peered":"2024-09-20T19:40:44.708574+0000","last_unstale":"2024-09-20T19:43:40.699863+0000","last_undegraded":"2024-09-20T19:43:40.699863+0000","last_fullsized":"2024-09-20T19:43:40.699863+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:48.764274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:48.764274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:52.833060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1889,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929507+0000","last_change":"2024-09-20T19:43:33.429853+0000","last_active":"2024-09-20T19:43:40.929507+0000","last_peered":"2024-09-20T19:43:40.929507+0000","last_clean":"2024-09-20T19:43:40.929507+0000","last_became_active":"2024-09-20T19:40:44.713745+0000","last_became_peered":"2024-09-20T19:40:44.713745+0000","last_unstale":"2024-09-20T19:43:40.929507+0000","last_undegraded":"2024-09-20T19:43:40.929507+0000","last_fullsized":"2024-09-20T19:43:40.929507+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:33.429639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.429639+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.371654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"987'39","reported_seq":1985,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924355+0000","last_change":"2024-09-20T19:43:18.334308+0000","last_active":"2024-09-20T19:43:40.924355+0000","last_peered":"2024-09-20T19:43:40.924355+0000","last_clean":"2024-09-20T19:43:40.924355+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:43:40.924355+0000","last_undegraded":"2024-09-20T19:43:40.924355+0000","last_fullsized":"2024-09-20T19:43:40.924355+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:43:18.334155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:18.334155+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:22.767119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2117,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701209+0000","last_change":"2024-09-20T19:43:35.420958+0000","last_active":"2024-09-20T19:43:40.701209+0000","last_peered":"2024-09-20T19:43:40.701209+0000","last_clean":"2024-09-20T19:43:40.701209+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:43:40.701209+0000","last_undegraded":"2024-09-20T19:43:40.701209+0000","last_fullsized":"2024-09-20T19:43:40.701209+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:35.420883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:35.420883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.087239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2164,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.927771+0000","last_change":"2024-09-20T19:43:20.385183+0000","last_active":"2024-09-20T19:43:40.927771+0000","last_peered":"2024-09-20T19:43:40.927771+0000","last_clean":"2024-09-20T19:43:40.927771+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:43:40.927771+0000","last_undegraded":"2024-09-20T19:43:40.927771+0000","last_fullsized":"2024-09-20T19:43:40.927771+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.385063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.385063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.325790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":2159,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700317+0000","last_change":"2024-09-20T19:42:43.099657+0000","last_active":"2024-09-20T19:43:40.700317+0000","last_peered":"2024-09-20T19:43:40.700317+0000","last_clean":"2024-09-20T19:43:40.700317+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:43:40.700317+0000","last_undegraded":"2024-09-20T19:43:40.700317+0000","last_fullsized":"2024-09-20T19:43:40.700317+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:43.099599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:43.099599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:12.352694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1874,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929299+0000","last_change":"2024-09-20T19:42:41.330947+0000","last_active":"2024-09-20T19:43:40.929299+0000","last_peered":"2024-09-20T19:43:40.929299+0000","last_clean":"2024-09-20T19:43:40.929299+0000","last_became_active":"2024-09-20T19:40:44.713893+0000","last_became_peered":"2024-09-20T19:40:44.713893+0000","last_unstale":"2024-09-20T19:43:40.929299+0000","last_undegraded":"2024-09-20T19:43:40.929299+0000","last_fullsized":"2024-09-20T19:43:40.929299+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:42:41.330790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:42:41.330790+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:43.745241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"987'30","reported_seq":2220,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701171+0000","last_change":"2024-09-20T19:43:34.246690+0000","last_active":"2024-09-20T19:43:40.701171+0000","last_peered":"2024-09-20T19:43:40.701171+0000","last_clean":"2024-09-20T19:43:40.701171+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:43:40.701171+0000","last_undegraded":"2024-09-20T19:43:40.701171+0000","last_fullsized":"2024-09-20T19:43:40.701171+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:43:34.246546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:34.246546+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:58.942888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2168,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.442004+0000","last_change":"2024-09-20T19:43:44.442004+0000","last_active":"2024-09-20T19:43:44.442004+0000","last_peered":"2024-09-20T19:43:44.442004+0000","last_clean":"2024-09-20T19:43:44.442004+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:43:44.442004+0000","last_undegraded":"2024-09-20T19:43:44.442004+0000","last_fullsized":"2024-09-20T19:43:44.442004+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.441901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.441901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.960477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":2203,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.096849+0000","last_change":"2024-09-20T19:43:46.096849+0000","last_active":"2024-09-20T19:43:46.096849+0000","last_peered":"2024-09-20T19:43:46.096849+0000","last_clean":"2024-09-20T19:43:46.096849+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:43:46.096849+0000","last_undegraded":"2024-09-20T19:43:46.096849+0000","last_fullsized":"2024-09-20T19:43:46.096849+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:43:46.096716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.096716+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:07.301177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1962,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923004+0000","last_change":"2024-09-20T19:43:06.185977+0000","last_active":"2024-09-20T19:43:40.923004+0000","last_peered":"2024-09-20T19:43:40.923004+0000","last_clean":"2024-09-20T19:43:40.923004+0000","last_became_active":"2024-09-20T19:40:44.696294+0000","last_became_peered":"2024-09-20T19:40:44.696294+0000","last_unstale":"2024-09-20T19:43:40.923004+0000","last_undegraded":"2024-09-20T19:43:40.923004+0000","last_fullsized":"2024-09-20T19:43:40.923004+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.185854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.185854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.466406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1754,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304891+0000","last_change":"2024-09-20T19:43:01.736720+0000","last_active":"2024-09-20T19:43:41.304891+0000","last_peered":"2024-09-20T19:43:41.304891+0000","last_clean":"2024-09-20T19:43:41.304891+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:43:41.304891+0000","last_undegraded":"2024-09-20T19:43:41.304891+0000","last_fullsized":"2024-09-20T19:43:41.304891+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:01.736588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:01.736588+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.669744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"1066'58","reported_seq":2653,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702115+0000","last_change":"2024-09-20T19:43:33.197027+0000","last_active":"2024-09-20T19:43:40.702115+0000","last_peered":"2024-09-20T19:43:40.702115+0000","last_clean":"2024-09-20T19:43:40.702115+0000","last_became_active":"2024-09-20T19:40:44.714403+0000","last_became_peered":"2024-09-20T19:40:44.714403+0000","last_unstale":"2024-09-20T19:43:40.702115+0000","last_undegraded":"2024-09-20T19:43:40.702115+0000","last_fullsized":"2024-09-20T19:43:40.702115+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'58","last_scrub_stamp":"2024-09-20T19:43:33.196872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.196872+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.712005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2174,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306153+0000","last_change":"2024-09-20T19:43:27.740801+0000","last_active":"2024-09-20T19:43:41.306153+0000","last_peered":"2024-09-20T19:43:41.306153+0000","last_clean":"2024-09-20T19:43:41.306153+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:43:41.306153+0000","last_undegraded":"2024-09-20T19:43:41.306153+0000","last_fullsized":"2024-09-20T19:43:41.306153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:27.740678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:27.740678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.169334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1945,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928231+0000","last_change":"2024-09-20T19:42:36.425491+0000","last_active":"2024-09-20T19:43:40.928231+0000","last_peered":"2024-09-20T19:43:40.928231+0000","last_clean":"2024-09-20T19:43:40.928231+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:43:40.928231+0000","last_undegraded":"2024-09-20T19:43:40.928231+0000","last_fullsized":"2024-09-20T19:43:40.928231+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:36.425343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.425343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.749302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232172+0000","last_change":"2024-09-20T19:43:29.098576+0000","last_active":"2024-09-20T19:43:41.232172+0000","last_peered":"2024-09-20T19:43:41.232172+0000","last_clean":"2024-09-20T19:43:41.232172+0000","last_became_active":"2024-09-20T19:40:44.707825+0000","last_became_peered":"2024-09-20T19:40:44.707825+0000","last_unstale":"2024-09-20T19:43:41.232172+0000","last_undegraded":"2024-09-20T19:43:41.232172+0000","last_fullsized":"2024-09-20T19:43:41.232172+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:29.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:29.098515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.259043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":2226,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869652+0000","last_change":"2024-09-20T19:43:00.103265+0000","last_active":"2024-09-20T19:43:40.869652+0000","last_peered":"2024-09-20T19:43:40.869652+0000","last_clean":"2024-09-20T19:43:40.869652+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:43:40.869652+0000","last_undegraded":"2024-09-20T19:43:40.869652+0000","last_fullsized":"2024-09-20T19:43:40.869652+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:00.103109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:00.103109+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.133124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"987'21","reported_seq":2134,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702312+0000","last_change":"2024-09-20T19:43:37.431497+0000","last_active":"2024-09-20T19:43:40.702312+0000","last_peered":"2024-09-20T19:43:40.702312+0000","last_clean":"2024-09-20T19:43:40.702312+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:43:40.702312+0000","last_undegraded":"2024-09-20T19:43:40.702312+0000","last_fullsized":"2024-09-20T19:43:40.702312+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:37.431298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.431298+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:52.073391+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":3701,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:45.026178+0000","last_change":"2024-09-20T19:43:12.090271+0000","last_active":"2024-09-20T19:43:45.026178+0000","last_peered":"2024-09-20T19:43:45.026178+0000","last_clean":"2024-09-20T19:43:45.026178+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:43:45.026178+0000","last_undegraded":"2024-09-20T19:43:45.026178+0000","last_fullsized":"2024-09-20T19:43:45.026178+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:43:12.090136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:12.090136+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.493942+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1520,"num_read_kb":1515,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2202,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699335+0000","last_change":"2024-09-20T19:43:08.394716+0000","last_active":"2024-09-20T19:43:40.699335+0000","last_peered":"2024-09-20T19:43:40.699335+0000","last_clean":"2024-09-20T19:43:40.699335+0000","last_became_active":"2024-09-20T19:40:44.686713+0000","last_became_peered":"2024-09-20T19:40:44.686713+0000","last_unstale":"2024-09-20T19:43:40.699335+0000","last_undegraded":"2024-09-20T19:43:40.699335+0000","last_fullsized":"2024-09-20T19:43:40.699335+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:08.394592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:08.394592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:32.209356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":749,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.488692+0000","last_change":"2024-09-20T19:43:44.488692+0000","last_active":"2024-09-20T19:43:44.488692+0000","last_peered":"2024-09-20T19:43:44.488692+0000","last_clean":"2024-09-20T19:43:44.488692+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:43:44.488692+0000","last_undegraded":"2024-09-20T19:43:44.488692+0000","last_fullsized":"2024-09-20T19:43:44.488692+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.488563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:13.360055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2157,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869287+0000","last_change":"2024-09-20T19:42:33.989590+0000","last_active":"2024-09-20T19:43:40.869287+0000","last_peered":"2024-09-20T19:43:40.869287+0000","last_clean":"2024-09-20T19:43:40.869287+0000","last_became_active":"2024-09-20T19:40:44.702203+0000","last_became_peered":"2024-09-20T19:40:44.702203+0000","last_unstale":"2024-09-20T19:43:40.869287+0000","last_undegraded":"2024-09-20T19:43:40.869287+0000","last_fullsized":"2024-09-20T19:43:40.869287+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:33.989467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:33.989467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:48.633237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1759,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.360224+0000","last_change":"2024-09-20T19:43:41.360224+0000","last_active":"2024-09-20T19:43:41.360224+0000","last_peered":"2024-09-20T19:43:41.360224+0000","last_clean":"2024-09-20T19:43:41.360224+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:43:41.360224+0000","last_undegraded":"2024-09-20T19:43:41.360224+0000","last_fullsized":"2024-09-20T19:43:41.360224+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:41.360089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.360089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:51.419942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"987'36","reported_seq":2003,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929064+0000","last_change":"2024-09-20T19:43:29.166809+0000","last_active":"2024-09-20T19:43:40.929064+0000","last_peered":"2024-09-20T19:43:40.929064+0000","last_clean":"2024-09-20T19:43:40.929064+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:43:40.929064+0000","last_undegraded":"2024-09-20T19:43:40.929064+0000","last_fullsized":"2024-09-20T19:43:40.929064+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'36","last_scrub_stamp":"2024-09-20T19:43:29.166694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:29.166694+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.598626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1910,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923000+0000","last_change":"2024-09-20T19:43:07.192824+0000","last_active":"2024-09-20T19:43:40.923000+0000","last_peered":"2024-09-20T19:43:40.923000+0000","last_clean":"2024-09-20T19:43:40.923000+0000","last_became_active":"2024-09-20T19:40:44.715407+0000","last_became_peered":"2024-09-20T19:40:44.715407+0000","last_unstale":"2024-09-20T19:43:40.923000+0000","last_undegraded":"2024-09-20T19:43:40.923000+0000","last_fullsized":"2024-09-20T19:43:40.923000+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:43:07.192693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:07.192693+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:21.949938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2163,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868651+0000","last_change":"2024-09-20T19:42:42.956040+0000","last_active":"2024-09-20T19:43:40.868651+0000","last_peered":"2024-09-20T19:43:40.868651+0000","last_clean":"2024-09-20T19:43:40.868651+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:43:40.868651+0000","last_undegraded":"2024-09-20T19:43:40.868651+0000","last_fullsized":"2024-09-20T19:43:40.868651+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:42.955918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:42.955918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:00.713228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.307033+0000","last_change":"2024-09-20T19:43:37.565380+0000","last_active":"2024-09-20T19:43:41.307033+0000","last_peered":"2024-09-20T19:43:41.307033+0000","last_clean":"2024-09-20T19:43:41.307033+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:43:41.307033+0000","last_undegraded":"2024-09-20T19:43:41.307033+0000","last_fullsized":"2024-09-20T19:43:41.307033+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.565261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.565261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.417913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2194,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701791+0000","last_change":"2024-09-20T19:42:34.073398+0000","last_active":"2024-09-20T19:43:40.701791+0000","last_peered":"2024-09-20T19:43:40.701791+0000","last_clean":"2024-09-20T19:43:40.701791+0000","last_became_active":"2024-09-20T19:40:44.712370+0000","last_became_peered":"2024-09-20T19:40:44.712370+0000","last_unstale":"2024-09-20T19:43:40.701791+0000","last_undegraded":"2024-09-20T19:43:40.701791+0000","last_fullsized":"2024-09-20T19:43:40.701791+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:34.073270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:34.073270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:43.824744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"987'42","reported_seq":2527,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929321+0000","last_change":"2024-09-20T19:43:38.381338+0000","last_active":"2024-09-20T19:43:40.929321+0000","last_peered":"2024-09-20T19:43:40.929321+0000","last_clean":"2024-09-20T19:43:40.929321+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:43:40.929321+0000","last_undegraded":"2024-09-20T19:43:40.929321+0000","last_fullsized":"2024-09-20T19:43:40.929321+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:43:38.381167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.381167+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.532424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":101,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234541+0000","last_change":"2024-09-20T19:43:11.254899+0000","last_active":"2024-09-20T19:43:41.234541+0000","last_peered":"2024-09-20T19:43:41.234541+0000","last_clean":"2024-09-20T19:43:41.234541+0000","last_became_active":"2024-09-20T19:40:44.718297+0000","last_became_peered":"2024-09-20T19:40:44.718297+0000","last_unstale":"2024-09-20T19:43:41.234541+0000","last_undegraded":"2024-09-20T19:43:41.234541+0000","last_fullsized":"2024-09-20T19:43:41.234541+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:43:11.254763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:11.254763+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:36.708223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2150,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.540062+0000","last_change":"2024-09-20T19:43:44.540062+0000","last_active":"2024-09-20T19:43:44.540062+0000","last_peered":"2024-09-20T19:43:44.540062+0000","last_clean":"2024-09-20T19:43:44.540062+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:43:44.540062+0000","last_undegraded":"2024-09-20T19:43:44.540062+0000","last_fullsized":"2024-09-20T19:43:44.540062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.539947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.539947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.862876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":2214,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.931692+0000","last_change":"2024-09-20T19:43:40.931558+0000","last_active":"2024-09-20T19:43:40.931692+0000","last_peered":"2024-09-20T19:43:40.931692+0000","last_clean":"2024-09-20T19:43:40.931692+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:43:40.931692+0000","last_undegraded":"2024-09-20T19:43:40.931692+0000","last_fullsized":"2024-09-20T19:43:40.931692+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:40.931434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:40.931434+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.441953+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":2136,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869114+0000","last_change":"2024-09-20T19:43:14.236338+0000","last_active":"2024-09-20T19:43:40.869114+0000","last_peered":"2024-09-20T19:43:40.869114+0000","last_clean":"2024-09-20T19:43:40.869114+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:43:40.869114+0000","last_undegraded":"2024-09-20T19:43:40.869114+0000","last_fullsized":"2024-09-20T19:43:40.869114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:14.236216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:14.236216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.885413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2177,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304709+0000","last_change":"2024-09-20T19:42:55.692715+0000","last_active":"2024-09-20T19:43:41.304709+0000","last_peered":"2024-09-20T19:43:41.304709+0000","last_clean":"2024-09-20T19:43:41.304709+0000","last_became_active":"2024-09-20T19:40:44.711751+0000","last_became_peered":"2024-09-20T19:40:44.711751+0000","last_unstale":"2024-09-20T19:43:41.304709+0000","last_undegraded":"2024-09-20T19:43:41.304709+0000","last_fullsized":"2024-09-20T19:43:41.304709+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:55.692591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:55.692591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:21.189425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"987'51","reported_seq":2219,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928283+0000","last_change":"2024-09-20T19:42:36.344750+0000","last_active":"2024-09-20T19:43:40.928283+0000","last_peered":"2024-09-20T19:43:40.928283+0000","last_clean":"2024-09-20T19:43:40.928283+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:43:40.928283+0000","last_undegraded":"2024-09-20T19:43:40.928283+0000","last_fullsized":"2024-09-20T19:43:40.928283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'51","last_scrub_stamp":"2024-09-20T19:42:36.344612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.344612+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.398229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2188,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.703221+0000","last_change":"2024-09-20T19:43:38.476998+0000","last_active":"2024-09-20T19:43:40.703221+0000","last_peered":"2024-09-20T19:43:40.703221+0000","last_clean":"2024-09-20T19:43:40.703221+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:43:40.703221+0000","last_undegraded":"2024-09-20T19:43:40.703221+0000","last_fullsized":"2024-09-20T19:43:40.703221+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.476943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.476943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.330919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":741,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.703657+0000","last_change":"2024-09-20T19:43:00.500715+0000","last_active":"2024-09-20T19:43:40.703657+0000","last_peered":"2024-09-20T19:43:40.703657+0000","last_clean":"2024-09-20T19:43:40.703657+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:43:40.703657+0000","last_undegraded":"2024-09-20T19:43:40.703657+0000","last_fullsized":"2024-09-20T19:43:40.703657+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:00.500632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:00.500632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:23.774619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2148,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.867811+0000","last_change":"2024-09-20T19:42:59.082822+0000","last_active":"2024-09-20T19:43:40.867811+0000","last_peered":"2024-09-20T19:43:40.867811+0000","last_clean":"2024-09-20T19:43:40.867811+0000","last_became_active":"2024-09-20T19:40:44.704276+0000","last_became_peered":"2024-09-20T19:40:44.704276+0000","last_unstale":"2024-09-20T19:43:40.867811+0000","last_undegraded":"2024-09-20T19:43:40.867811+0000","last_fullsized":"2024-09-20T19:43:40.867811+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:59.082699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.082699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.506479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"987'21","reported_seq":1977,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.092427+0000","last_change":"2024-09-20T19:43:44.092427+0000","last_active":"2024-09-20T19:43:44.092427+0000","last_peered":"2024-09-20T19:43:44.092427+0000","last_clean":"2024-09-20T19:43:44.092427+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:43:44.092427+0000","last_undegraded":"2024-09-20T19:43:44.092427+0000","last_fullsized":"2024-09-20T19:43:44.092427+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:44.092289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.092289+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.539505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":2195,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702614+0000","last_change":"2024-09-20T19:43:23.105414+0000","last_active":"2024-09-20T19:43:40.702614+0000","last_peered":"2024-09-20T19:43:40.702614+0000","last_clean":"2024-09-20T19:43:40.702614+0000","last_became_active":"2024-09-20T19:40:44.713034+0000","last_became_peered":"2024-09-20T19:40:44.713034+0000","last_unstale":"2024-09-20T19:43:40.702614+0000","last_undegraded":"2024-09-20T19:43:40.702614+0000","last_fullsized":"2024-09-20T19:43:40.702614+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.105343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.105343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.627530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2201,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700469+0000","last_change":"2024-09-20T19:43:22.309399+0000","last_active":"2024-09-20T19:43:40.700469+0000","last_peered":"2024-09-20T19:43:40.700469+0000","last_clean":"2024-09-20T19:43:40.700469+0000","last_became_active":"2024-09-20T19:40:44.687389+0000","last_became_peered":"2024-09-20T19:40:44.687389+0000","last_unstale":"2024-09-20T19:43:40.700469+0000","last_undegraded":"2024-09-20T19:43:40.700469+0000","last_fullsized":"2024-09-20T19:43:40.700469+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:22.309270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:22.309270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:40.515709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1969,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924077+0000","last_change":"2024-09-20T19:43:25.260488+0000","last_active":"2024-09-20T19:43:40.924077+0000","last_peered":"2024-09-20T19:43:40.924077+0000","last_clean":"2024-09-20T19:43:40.924077+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:43:40.924077+0000","last_undegraded":"2024-09-20T19:43:40.924077+0000","last_fullsized":"2024-09-20T19:43:40.924077+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:43:25.260361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:25.260361+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:37.041510+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"987'39","reported_seq":2222,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304890+0000","last_change":"2024-09-20T19:42:56.682400+0000","last_active":"2024-09-20T19:43:41.304890+0000","last_peered":"2024-09-20T19:43:41.304890+0000","last_clean":"2024-09-20T19:43:41.304890+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:43:41.304890+0000","last_undegraded":"2024-09-20T19:43:41.304890+0000","last_fullsized":"2024-09-20T19:43:41.304890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:56.682198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:56.682198+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:09.143416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":756,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701312+0000","last_change":"2024-09-20T19:42:59.517014+0000","last_active":"2024-09-20T19:43:40.701312+0000","last_peered":"2024-09-20T19:43:40.701312+0000","last_clean":"2024-09-20T19:43:40.701312+0000","last_became_active":"2024-09-20T19:40:44.686132+0000","last_became_peered":"2024-09-20T19:40:44.686132+0000","last_unstale":"2024-09-20T19:43:40.701312+0000","last_undegraded":"2024-09-20T19:43:40.701312+0000","last_fullsized":"2024-09-20T19:43:40.701312+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:42:59.516859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.516859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:04.492595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":2467,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868608+0000","last_change":"2024-09-20T19:43:31.089047+0000","last_active":"2024-09-20T19:43:40.868608+0000","last_peered":"2024-09-20T19:43:40.868608+0000","last_clean":"2024-09-20T19:43:40.868608+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:43:40.868608+0000","last_undegraded":"2024-09-20T19:43:40.868608+0000","last_fullsized":"2024-09-20T19:43:40.868608+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:43:31.088914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:31.088914+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.119752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2156,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305692+0000","last_change":"2024-09-20T19:43:20.685241+0000","last_active":"2024-09-20T19:43:41.305692+0000","last_peered":"2024-09-20T19:43:41.305692+0000","last_clean":"2024-09-20T19:43:41.305692+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:43:41.305692+0000","last_undegraded":"2024-09-20T19:43:41.305692+0000","last_fullsized":"2024-09-20T19:43:41.305692+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.685112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.685112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.777487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"987'24","reported_seq":2062,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699926+0000","last_change":"2024-09-20T19:43:26.719370+0000","last_active":"2024-09-20T19:43:40.699926+0000","last_peered":"2024-09-20T19:43:40.699926+0000","last_clean":"2024-09-20T19:43:40.699926+0000","last_became_active":"2024-09-20T19:40:44.723579+0000","last_became_peered":"2024-09-20T19:40:44.723579+0000","last_unstale":"2024-09-20T19:43:40.699926+0000","last_undegraded":"2024-09-20T19:43:40.699926+0000","last_fullsized":"2024-09-20T19:43:40.699926+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'24","last_scrub_stamp":"2024-09-20T19:43:26.719227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:26.719227+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.124353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":2029,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701881+0000","last_change":"2024-09-20T19:42:26.755813+0000","last_active":"2024-09-20T19:43:40.701881+0000","last_peered":"2024-09-20T19:43:40.701881+0000","last_clean":"2024-09-20T19:43:40.701881+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:43:40.701881+0000","last_undegraded":"2024-09-20T19:43:40.701881+0000","last_fullsized":"2024-09-20T19:43:40.701881+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:42:26.755686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:26.755686+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:50.708001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2172,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699328+0000","last_change":"2024-09-20T19:43:23.354259+0000","last_active":"2024-09-20T19:43:40.699328+0000","last_peered":"2024-09-20T19:43:40.699328+0000","last_clean":"2024-09-20T19:43:40.699328+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:43:40.699328+0000","last_undegraded":"2024-09-20T19:43:40.699328+0000","last_fullsized":"2024-09-20T19:43:40.699328+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.354126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.164042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.230887+0000","last_change":"2024-09-20T19:42:30.185762+0000","last_active":"2024-09-20T19:43:41.230887+0000","last_peered":"2024-09-20T19:43:41.230887+0000","last_clean":"2024-09-20T19:43:41.230887+0000","last_became_active":"2024-09-20T19:40:44.706254+0000","last_became_peered":"2024-09-20T19:40:44.706254+0000","last_unstale":"2024-09-20T19:43:41.230887+0000","last_undegraded":"2024-09-20T19:43:41.230887+0000","last_fullsized":"2024-09-20T19:43:41.230887+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:30.185651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:30.185651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.980187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"987'21","reported_seq":1979,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928774+0000","last_change":"2024-09-20T19:42:28.260343+0000","last_active":"2024-09-20T19:43:40.928774+0000","last_peered":"2024-09-20T19:43:40.928774+0000","last_clean":"2024-09-20T19:43:40.928774+0000","last_became_active":"2024-09-20T19:40:44.716453+0000","last_became_peered":"2024-09-20T19:40:44.716453+0000","last_unstale":"2024-09-20T19:43:40.928774+0000","last_undegraded":"2024-09-20T19:43:40.928774+0000","last_fullsized":"2024-09-20T19:43:40.928774+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:28.260260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:28.260260+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:54.559454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2204,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699741+0000","last_change":"2024-09-20T19:43:20.311874+0000","last_active":"2024-09-20T19:43:40.699741+0000","last_peered":"2024-09-20T19:43:40.699741+0000","last_clean":"2024-09-20T19:43:40.699741+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:43:40.699741+0000","last_undegraded":"2024-09-20T19:43:40.699741+0000","last_fullsized":"2024-09-20T19:43:40.699741+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.311486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.311486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.295045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232965+0000","last_change":"2024-09-20T19:42:36.262415+0000","last_active":"2024-09-20T19:43:41.232965+0000","last_peered":"2024-09-20T19:43:41.232965+0000","last_clean":"2024-09-20T19:43:41.232965+0000","last_became_active":"2024-09-20T19:40:44.708011+0000","last_became_peered":"2024-09-20T19:40:44.708011+0000","last_unstale":"2024-09-20T19:43:41.232965+0000","last_undegraded":"2024-09-20T19:43:41.232965+0000","last_fullsized":"2024-09-20T19:43:41.232965+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:36.262354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.262354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.124471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2159,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701957+0000","last_change":"2024-09-20T19:43:33.444532+0000","last_active":"2024-09-20T19:43:40.701957+0000","last_peered":"2024-09-20T19:43:40.701957+0000","last_clean":"2024-09-20T19:43:40.701957+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:43:40.701957+0000","last_undegraded":"2024-09-20T19:43:40.701957+0000","last_fullsized":"2024-09-20T19:43:40.701957+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:33.444414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.444414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.223120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":2147,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928393+0000","last_change":"2024-09-20T19:42:39.326616+0000","last_active":"2024-09-20T19:43:40.928393+0000","last_peered":"2024-09-20T19:43:40.928393+0000","last_clean":"2024-09-20T19:43:40.928393+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:43:40.928393+0000","last_undegraded":"2024-09-20T19:43:40.928393+0000","last_fullsized":"2024-09-20T19:43:40.928393+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:39.326492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:39.326492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:48.431680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"987'18","reported_seq":113,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232909+0000","last_change":"2024-09-20T19:43:30.112997+0000","last_active":"2024-09-20T19:43:41.232909+0000","last_peered":"2024-09-20T19:43:41.232909+0000","last_clean":"2024-09-20T19:43:41.232909+0000","last_became_active":"2024-09-20T19:40:44.708528+0000","last_became_peered":"2024-09-20T19:40:44.708528+0000","last_unstale":"2024-09-20T19:43:41.232909+0000","last_undegraded":"2024-09-20T19:43:41.232909+0000","last_fullsized":"2024-09-20T19:43:41.232909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'18","last_scrub_stamp":"2024-09-20T19:43:30.112938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.112938+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.648414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2189,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.510102+0000","last_change":"2024-09-20T19:43:46.510102+0000","last_active":"2024-09-20T19:43:46.510102+0000","last_peered":"2024-09-20T19:43:46.510102+0000","last_clean":"2024-09-20T19:43:46.510102+0000","last_became_active":"2024-09-20T19:40:44.704071+0000","last_became_peered":"2024-09-20T19:40:44.704071+0000","last_unstale":"2024-09-20T19:43:46.510102+0000","last_undegraded":"2024-09-20T19:43:46.510102+0000","last_fullsized":"2024-09-20T19:43:46.510102+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:46.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.509977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:54.182173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2141,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869163+0000","last_change":"2024-09-20T19:43:33.983755+0000","last_active":"2024-09-20T19:43:40.869163+0000","last_peered":"2024-09-20T19:43:40.869163+0000","last_clean":"2024-09-20T19:43:40.869163+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:43:40.869163+0000","last_undegraded":"2024-09-20T19:43:40.869163+0000","last_fullsized":"2024-09-20T19:43:40.869163+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:33.983608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.983608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.942731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"987'30","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.230893+0000","last_change":"2024-09-20T19:42:14.167422+0000","last_active":"2024-09-20T19:43:41.230893+0000","last_peered":"2024-09-20T19:43:41.230893+0000","last_clean":"2024-09-20T19:43:41.230893+0000","last_became_active":"2024-09-20T19:40:44.707276+0000","last_became_peered":"2024-09-20T19:40:44.707276+0000","last_unstale":"2024-09-20T19:43:41.230893+0000","last_undegraded":"2024-09-20T19:43:41.230893+0000","last_fullsized":"2024-09-20T19:43:41.230893+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:42:14.167284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:14.167284+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":2189,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306076+0000","last_change":"2024-09-20T19:42:51.740070+0000","last_active":"2024-09-20T19:43:41.306076+0000","last_peered":"2024-09-20T19:43:41.306076+0000","last_clean":"2024-09-20T19:43:41.306076+0000","last_became_active":"2024-09-20T19:40:44.704133+0000","last_became_peered":"2024-09-20T19:40:44.704133+0000","last_unstale":"2024-09-20T19:43:41.306076+0000","last_undegraded":"2024-09-20T19:43:41.306076+0000","last_fullsized":"2024-09-20T19:43:41.306076+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:51.739950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:51.739950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:19.122494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":2480,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306662+0000","last_change":"2024-09-20T19:43:12.620052+0000","last_active":"2024-09-20T19:43:41.306662+0000","last_peered":"2024-09-20T19:43:41.306662+0000","last_clean":"2024-09-20T19:43:41.306662+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:43:41.306662+0000","last_undegraded":"2024-09-20T19:43:41.306662+0000","last_fullsized":"2024-09-20T19:43:41.306662+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:43:12.619904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:12.619904+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.415120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1948,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929461+0000","last_change":"2024-09-20T19:43:30.154101+0000","last_active":"2024-09-20T19:43:40.929461+0000","last_peered":"2024-09-20T19:43:40.929461+0000","last_clean":"2024-09-20T19:43:40.929461+0000","last_became_active":"2024-09-20T19:40:44.723652+0000","last_became_peered":"2024-09-20T19:40:44.723652+0000","last_unstale":"2024-09-20T19:43:40.929461+0000","last_undegraded":"2024-09-20T19:43:40.929461+0000","last_fullsized":"2024-09-20T19:43:40.929461+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.154020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.154020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.442716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"987'21","reported_seq":2246,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306630+0000","last_change":"2024-09-20T19:42:49.725705+0000","last_active":"2024-09-20T19:43:41.306630+0000","last_peered":"2024-09-20T19:43:41.306630+0000","last_clean":"2024-09-20T19:43:41.306630+0000","last_became_active":"2024-09-20T19:40:44.706335+0000","last_became_peered":"2024-09-20T19:40:44.706335+0000","last_unstale":"2024-09-20T19:43:41.306630+0000","last_undegraded":"2024-09-20T19:43:41.306630+0000","last_fullsized":"2024-09-20T19:43:41.306630+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:49.725571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:49.725571+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:19.189521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1913,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923099+0000","last_change":"2024-09-20T19:43:20.275780+0000","last_active":"2024-09-20T19:43:40.923099+0000","last_peered":"2024-09-20T19:43:40.923099+0000","last_clean":"2024-09-20T19:43:40.923099+0000","last_became_active":"2024-09-20T19:40:44.697371+0000","last_became_peered":"2024-09-20T19:40:44.697371+0000","last_unstale":"2024-09-20T19:43:40.923099+0000","last_undegraded":"2024-09-20T19:43:40.923099+0000","last_fullsized":"2024-09-20T19:43:40.923099+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.275637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.275637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:46.317196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1066'18","reported_seq":138,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.231001+0000","last_change":"2024-09-20T19:42:58.421777+0000","last_active":"2024-09-20T19:43:41.231001+0000","last_peered":"2024-09-20T19:43:41.231001+0000","last_clean":"2024-09-20T19:43:41.231001+0000","last_became_active":"2024-09-20T19:40:44.719828+0000","last_became_peered":"2024-09-20T19:40:44.719828+0000","last_unstale":"2024-09-20T19:43:41.231001+0000","last_undegraded":"2024-09-20T19:43:41.231001+0000","last_fullsized":"2024-09-20T19:43:41.231001+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'18","last_scrub_stamp":"2024-09-20T19:42:58.421638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:58.421638+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.597390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1983,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701717+0000","last_change":"2024-09-20T19:42:32.812769+0000","last_active":"2024-09-20T19:43:40.701717+0000","last_peered":"2024-09-20T19:43:40.701717+0000","last_clean":"2024-09-20T19:43:40.701717+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:43:40.701717+0000","last_undegraded":"2024-09-20T19:43:40.701717+0000","last_fullsized":"2024-09-20T19:43:40.701717+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:32.812644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:32.812644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:49.192115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"987'21","reported_seq":2185,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701033+0000","last_change":"2024-09-20T19:42:21.918391+0000","last_active":"2024-09-20T19:43:40.701033+0000","last_peered":"2024-09-20T19:43:40.701033+0000","last_clean":"2024-09-20T19:43:40.701033+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:43:40.701033+0000","last_undegraded":"2024-09-20T19:43:40.701033+0000","last_fullsized":"2024-09-20T19:43:40.701033+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:21.918244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:21.918244+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:50.628965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928684+0000","last_change":"2024-09-20T19:42:23.303848+0000","last_active":"2024-09-20T19:43:40.928684+0000","last_peered":"2024-09-20T19:43:40.928684+0000","last_clean":"2024-09-20T19:43:40.928684+0000","last_became_active":"2024-09-20T19:40:44.698860+0000","last_became_peered":"2024-09-20T19:40:44.698860+0000","last_unstale":"2024-09-20T19:43:40.928684+0000","last_undegraded":"2024-09-20T19:43:40.928684+0000","last_fullsized":"2024-09-20T19:43:40.928684+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:23.303733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:23.303733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236651+0000","last_change":"2024-09-20T19:43:41.236651+0000","last_active":"2024-09-20T19:43:41.236651+0000","last_peered":"2024-09-20T19:43:41.236651+0000","last_clean":"2024-09-20T19:43:41.236651+0000","last_became_active":"2024-09-20T19:40:44.709755+0000","last_became_peered":"2024-09-20T19:40:44.709755+0000","last_unstale":"2024-09-20T19:43:41.236651+0000","last_undegraded":"2024-09-20T19:43:41.236651+0000","last_fullsized":"2024-09-20T19:43:41.236651+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:41.236526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.236526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.780221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1908,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924174+0000","last_change":"2024-09-20T19:43:04.233792+0000","last_active":"2024-09-20T19:43:40.924174+0000","last_peered":"2024-09-20T19:43:40.924174+0000","last_clean":"2024-09-20T19:43:40.924174+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:43:40.924174+0000","last_undegraded":"2024-09-20T19:43:40.924174+0000","last_fullsized":"2024-09-20T19:43:40.924174+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:04.233665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:04.233665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.915677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"987'27","reported_seq":2010,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700166+0000","last_change":"2024-09-20T19:43:16.756525+0000","last_active":"2024-09-20T19:43:40.700166+0000","last_peered":"2024-09-20T19:43:40.700166+0000","last_clean":"2024-09-20T19:43:40.700166+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:43:40.700166+0000","last_undegraded":"2024-09-20T19:43:40.700166+0000","last_fullsized":"2024-09-20T19:43:40.700166+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:43:16.756379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.756379+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.825382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":2182,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702508+0000","last_change":"2024-09-20T19:43:38.266008+0000","last_active":"2024-09-20T19:43:40.702508+0000","last_peered":"2024-09-20T19:43:40.702508+0000","last_clean":"2024-09-20T19:43:40.702508+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:43:40.702508+0000","last_undegraded":"2024-09-20T19:43:40.702508+0000","last_fullsized":"2024-09-20T19:43:40.702508+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:38.265791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.265791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:43.286102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":2183,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700723+0000","last_change":"2024-09-20T19:42:17.809206+0000","last_active":"2024-09-20T19:43:40.700723+0000","last_peered":"2024-09-20T19:43:40.700723+0000","last_clean":"2024-09-20T19:43:40.700723+0000","last_became_active":"2024-09-20T19:40:44.713438+0000","last_became_peered":"2024-09-20T19:40:44.713438+0000","last_unstale":"2024-09-20T19:43:40.700723+0000","last_undegraded":"2024-09-20T19:43:40.700723+0000","last_fullsized":"2024-09-20T19:43:40.700723+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:17.809077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:17.809077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:44.231836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.233829+0000","last_change":"2024-09-20T19:43:06.312356+0000","last_active":"2024-09-20T19:43:41.233829+0000","last_peered":"2024-09-20T19:43:41.233829+0000","last_clean":"2024-09-20T19:43:41.233829+0000","last_became_active":"2024-09-20T19:40:44.712828+0000","last_became_peered":"2024-09-20T19:40:44.712828+0000","last_unstale":"2024-09-20T19:43:41.233829+0000","last_undegraded":"2024-09-20T19:43:41.233829+0000","last_fullsized":"2024-09-20T19:43:41.233829+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.312231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.312231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:16.899539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"987'12","reported_seq":103,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.242601+0000","last_change":"2024-09-20T19:43:46.242601+0000","last_active":"2024-09-20T19:43:46.242601+0000","last_peered":"2024-09-20T19:43:46.242601+0000","last_clean":"2024-09-20T19:43:46.242601+0000","last_became_active":"2024-09-20T19:40:44.719536+0000","last_became_peered":"2024-09-20T19:40:44.719536+0000","last_unstale":"2024-09-20T19:43:46.242601+0000","last_undegraded":"2024-09-20T19:43:46.242601+0000","last_fullsized":"2024-09-20T19:43:46.242601+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12","last_scrub_stamp":"2024-09-20T19:43:46.242427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.242427+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:00.373429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":2162,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306972+0000","last_change":"2024-09-20T19:42:40.689145+0000","last_active":"2024-09-20T19:43:41.306972+0000","last_peered":"2024-09-20T19:43:41.306972+0000","last_clean":"2024-09-20T19:43:41.306972+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:43:41.306972+0000","last_undegraded":"2024-09-20T19:43:41.306972+0000","last_fullsized":"2024-09-20T19:43:41.306972+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:40.688957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:40.688957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:05.929470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2146,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.927892+0000","last_change":"2024-09-20T19:42:14.416139+0000","last_active":"2024-09-20T19:43:40.927892+0000","last_peered":"2024-09-20T19:43:40.927892+0000","last_clean":"2024-09-20T19:43:40.927892+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:43:40.927892+0000","last_undegraded":"2024-09-20T19:43:40.927892+0000","last_fullsized":"2024-09-20T19:43:40.927892+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:14.416012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:14.416012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2117,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868480+0000","last_change":"2024-09-20T19:42:26.140860+0000","last_active":"2024-09-20T19:43:40.868480+0000","last_peered":"2024-09-20T19:43:40.868480+0000","last_clean":"2024-09-20T19:43:40.868480+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:43:40.868480+0000","last_undegraded":"2024-09-20T19:43:40.868480+0000","last_fullsized":"2024-09-20T19:43:40.868480+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:26.140722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:26.140722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:42.826562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"987'27","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234145+0000","last_change":"2024-09-20T19:42:57.421725+0000","last_active":"2024-09-20T19:43:41.234145+0000","last_peered":"2024-09-20T19:43:41.234145+0000","last_clean":"2024-09-20T19:43:41.234145+0000","last_became_active":"2024-09-20T19:40:44.705478+0000","last_became_peered":"2024-09-20T19:40:44.705478+0000","last_unstale":"2024-09-20T19:43:41.234145+0000","last_undegraded":"2024-09-20T19:43:41.234145+0000","last_fullsized":"2024-09-20T19:43:41.234145+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:42:57.421643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:57.421643+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:17.252992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2151,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929992+0000","last_change":"2024-09-20T19:42:21.382437+0000","last_active":"2024-09-20T19:43:40.929992+0000","last_peered":"2024-09-20T19:43:40.929992+0000","last_clean":"2024-09-20T19:43:40.929992+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:43:40.929992+0000","last_undegraded":"2024-09-20T19:43:40.929992+0000","last_fullsized":"2024-09-20T19:43:40.929992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:21.382309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:21.382309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":2153,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868091+0000","last_change":"2024-09-20T19:43:13.264378+0000","last_active":"2024-09-20T19:43:40.868091+0000","last_peered":"2024-09-20T19:43:40.868091+0000","last_clean":"2024-09-20T19:43:40.868091+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:43:40.868091+0000","last_undegraded":"2024-09-20T19:43:40.868091+0000","last_fullsized":"2024-09-20T19:43:40.868091+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:13.264248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.264248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:28.002866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234515+0000","last_change":"2024-09-20T19:43:28.142134+0000","last_active":"2024-09-20T19:43:41.234515+0000","last_peered":"2024-09-20T19:43:41.234515+0000","last_clean":"2024-09-20T19:43:41.234515+0000","last_became_active":"2024-09-20T19:40:44.719130+0000","last_became_peered":"2024-09-20T19:40:44.719130+0000","last_unstale":"2024-09-20T19:43:41.234515+0000","last_undegraded":"2024-09-20T19:43:41.234515+0000","last_fullsized":"2024-09-20T19:43:41.234515+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:28.141976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:28.141976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.492524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":2124,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869251+0000","last_change":"2024-09-20T19:42:24.108812+0000","last_active":"2024-09-20T19:43:40.869251+0000","last_peered":"2024-09-20T19:43:40.869251+0000","last_clean":"2024-09-20T19:43:40.869251+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:43:40.869251+0000","last_undegraded":"2024-09-20T19:43:40.869251+0000","last_fullsized":"2024-09-20T19:43:40.869251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:42:24.108673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:24.108673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:51.077900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":2167,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.512212+0000","last_change":"2024-09-20T19:43:41.512212+0000","last_active":"2024-09-20T19:43:41.512212+0000","last_peered":"2024-09-20T19:43:41.512212+0000","last_clean":"2024-09-20T19:43:41.512212+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:43:41.512212+0000","last_undegraded":"2024-09-20T19:43:41.512212+0000","last_fullsized":"2024-09-20T19:43:41.512212+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:41.512085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.512085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.954600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"987'30","reported_seq":1995,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928969+0000","last_change":"2024-09-20T19:42:47.298115+0000","last_active":"2024-09-20T19:43:40.928969+0000","last_peered":"2024-09-20T19:43:40.928969+0000","last_clean":"2024-09-20T19:43:40.928969+0000","last_became_active":"2024-09-20T19:40:44.723592+0000","last_became_peered":"2024-09-20T19:40:44.723592+0000","last_unstale":"2024-09-20T19:43:40.928969+0000","last_undegraded":"2024-09-20T19:43:40.928969+0000","last_fullsized":"2024-09-20T19:43:40.928969+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:42:47.297946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:47.297946+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:14.910790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":2472,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929534+0000","last_change":"2024-09-20T19:42:27.414822+0000","last_active":"2024-09-20T19:43:40.929534+0000","last_peered":"2024-09-20T19:43:40.929534+0000","last_clean":"2024-09-20T19:43:40.929534+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:43:40.929534+0000","last_undegraded":"2024-09-20T19:43:40.929534+0000","last_fullsized":"2024-09-20T19:43:40.929534+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:42:27.414643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:27.414643+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:48.243741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2154,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701959+0000","last_change":"2024-09-20T19:43:30.172883+0000","last_active":"2024-09-20T19:43:40.701959+0000","last_peered":"2024-09-20T19:43:40.701959+0000","last_clean":"2024-09-20T19:43:40.701959+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:43:40.701959+0000","last_undegraded":"2024-09-20T19:43:40.701959+0000","last_fullsized":"2024-09-20T19:43:40.701959+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.172835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.172835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.998338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":101,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234003+0000","last_change":"2024-09-20T19:43:13.196425+0000","last_active":"2024-09-20T19:43:41.234003+0000","last_peered":"2024-09-20T19:43:41.234003+0000","last_clean":"2024-09-20T19:43:41.234003+0000","last_became_active":"2024-09-20T19:40:44.718639+0000","last_became_peered":"2024-09-20T19:40:44.718639+0000","last_unstale":"2024-09-20T19:43:41.234003+0000","last_undegraded":"2024-09-20T19:43:41.234003+0000","last_fullsized":"2024-09-20T19:43:41.234003+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:13.196296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.196296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.357746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"987'27","reported_seq":2214,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929971+0000","last_change":"2024-09-20T19:43:13.502075+0000","last_active":"2024-09-20T19:43:40.929971+0000","last_peered":"2024-09-20T19:43:40.929971+0000","last_clean":"2024-09-20T19:43:40.929971+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:43:40.929971+0000","last_undegraded":"2024-09-20T19:43:40.929971+0000","last_fullsized":"2024-09-20T19:43:40.929971+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:43:13.501926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.501926+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:15.519399+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1946,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929263+0000","last_change":"2024-09-20T19:42:17.393382+0000","last_active":"2024-09-20T19:43:40.929263+0000","last_peered":"2024-09-20T19:43:40.929263+0000","last_clean":"2024-09-20T19:43:40.929263+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:43:40.929263+0000","last_undegraded":"2024-09-20T19:43:40.929263+0000","last_fullsized":"2024-09-20T19:43:40.929263+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:17.393291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:17.393291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:47.244666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1973,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701327+0000","last_change":"2024-09-20T19:43:38.688743+0000","last_active":"2024-09-20T19:43:40.701327+0000","last_peered":"2024-09-20T19:43:40.701327+0000","last_clean":"2024-09-20T19:43:40.701327+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:43:40.701327+0000","last_undegraded":"2024-09-20T19:43:40.701327+0000","last_fullsized":"2024-09-20T19:43:40.701327+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.688527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.688527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.325502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2162,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928825+0000","last_change":"2024-09-20T19:42:09.402323+0000","last_active":"2024-09-20T19:43:40.928825+0000","last_peered":"2024-09-20T19:43:40.928825+0000","last_clean":"2024-09-20T19:43:40.928825+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:43:40.928825+0000","last_undegraded":"2024-09-20T19:43:40.928825+0000","last_fullsized":"2024-09-20T19:43:40.928825+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:09.402181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:09.402181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"987'39","reported_seq":797,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699700+0000","last_change":"2024-09-20T19:42:40.572544+0000","last_active":"2024-09-20T19:43:40.699700+0000","last_peered":"2024-09-20T19:43:40.699700+0000","last_clean":"2024-09-20T19:43:40.699700+0000","last_became_active":"2024-09-20T19:40:44.698792+0000","last_became_peered":"2024-09-20T19:40:44.698792+0000","last_unstale":"2024-09-20T19:43:40.699700+0000","last_undegraded":"2024-09-20T19:43:40.699700+0000","last_fullsized":"2024-09-20T19:43:40.699700+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:40.572392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:40.572392+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.670820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":2173,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702325+0000","last_change":"2024-09-20T19:42:59.068334+0000","last_active":"2024-09-20T19:43:40.702325+0000","last_peered":"2024-09-20T19:43:40.702325+0000","last_clean":"2024-09-20T19:43:40.702325+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:43:40.702325+0000","last_undegraded":"2024-09-20T19:43:40.702325+0000","last_fullsized":"2024-09-20T19:43:40.702325+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:59.068277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.068277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:27.700430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236126+0000","last_change":"2024-09-20T19:42:45.332595+0000","last_active":"2024-09-20T19:43:41.236126+0000","last_peered":"2024-09-20T19:43:41.236126+0000","last_clean":"2024-09-20T19:43:41.236126+0000","last_became_active":"2024-09-20T19:40:44.712077+0000","last_became_peered":"2024-09-20T19:40:44.712077+0000","last_unstale":"2024-09-20T19:43:41.236126+0000","last_undegraded":"2024-09-20T19:43:41.236126+0000","last_fullsized":"2024-09-20T19:43:41.236126+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:45.332470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:45.332470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:59.348624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234605+0000","last_change":"2024-09-20T19:42:41.240532+0000","last_active":"2024-09-20T19:43:41.234605+0000","last_peered":"2024-09-20T19:43:41.234605+0000","last_clean":"2024-09-20T19:43:41.234605+0000","last_became_active":"2024-09-20T19:40:44.718784+0000","last_became_peered":"2024-09-20T19:40:44.718784+0000","last_unstale":"2024-09-20T19:43:41.234605+0000","last_undegraded":"2024-09-20T19:43:41.234605+0000","last_fullsized":"2024-09-20T19:43:41.234605+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:41.240445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:41.240445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:49.394133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"987'24","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.231810+0000","last_change":"2024-09-20T19:43:04.358557+0000","last_active":"2024-09-20T19:43:41.231810+0000","last_peered":"2024-09-20T19:43:41.231810+0000","last_clean":"2024-09-20T19:43:41.231810+0000","last_became_active":"2024-09-20T19:40:44.706561+0000","last_became_peered":"2024-09-20T19:40:44.706561+0000","last_unstale":"2024-09-20T19:43:41.231810+0000","last_undegraded":"2024-09-20T19:43:41.231810+0000","last_fullsized":"2024-09-20T19:43:41.231810+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'24","last_scrub_stamp":"2024-09-20T19:43:04.358417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:04.358417+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:27.396306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1993,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700675+0000","last_change":"2024-09-20T19:43:23.682000+0000","last_active":"2024-09-20T19:43:40.700675+0000","last_peered":"2024-09-20T19:43:40.700675+0000","last_clean":"2024-09-20T19:43:40.700675+0000","last_became_active":"2024-09-20T19:40:44.709235+0000","last_became_peered":"2024-09-20T19:40:44.709235+0000","last_unstale":"2024-09-20T19:43:40.700675+0000","last_undegraded":"2024-09-20T19:43:40.700675+0000","last_fullsized":"2024-09-20T19:43:40.700675+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.681855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.681855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:26.131069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1528,"num_read_kb":1285,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":973,"ondisk_log_size":973,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1520,"num_read_kb":1515,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907262,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72532,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672620,"statfs":{"total":95995035648,"available":95920762880,"internally_reserved":0,"allocated":1118208,"data_stored":734393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234313,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76360,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668792,"statfs":{"total":95995035648,"available":95916843008,"internally_reserved":0,"allocated":778240,"data_stored":406049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561363,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76948,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668204,"statfs":{"total":95995035648,"available":95916240896,"internally_reserved":0,"allocated":1380352,"data_stored":997618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928674,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59156,"kb_used_data":772,"kb_used_omap":35,"kb_used_meta":58332,"kb_avail":93685996,"statfs":{"total":95995035648,"available":95934459904,"internally_reserved":0,"allocated":790528,"data_stored":406127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":59732913},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":550,"seq":2362232012956,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29932,"kb_used_data":1372,"kb_used_omap":51,"kb_used_meta":28492,"kb_avail":93715220,"statfs":{"total":95995035648,"available":95964385280,"internally_reserved":0,"allocated":1404928,"data_stored":997282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52777,"internal_metadata":29176279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378999,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80792,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664360,"statfs":{"total":95995035648,"available":95912304640,"internally_reserved":0,"allocated":1056768,"data_stored":669682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575235,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72216,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672936,"statfs":{"total":95995035648,"available":95921086464,"internally_reserved":0,"allocated":794624,"data_stored":407401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648166,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56092,"kb_used_data":780,"kb_used_omap":49,"kb_used_meta":55246,"kb_avail":93689060,"statfs":{"total":95995035648,"available":95937597440,"internally_reserved":0,"allocated":798720,"data_stored":407089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":56572457},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:43:51.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[32075]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:51.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:51.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:51 smithi042 bash[22706]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.145070+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.18 scrub starts 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.147215+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.18 scrub ok 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: audit 2024-09-20T19:43:50.244317+0000 mgr.y (mgr.24491) 963 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:52.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:51 smithi162 bash[24557]: cluster 2024-09-20T19:43:50.641028+0000 mgr.y (mgr.24491) 964 : cluster [DBG] pgmap v1760: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:52.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:52.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:52 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: audit 2024-09-20T19:43:50.951744+0000 mgr.y (mgr.24491) 965 : audit [DBG] from='client.16662 v1:172.21.15.42:0/421542068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:51.107074+0000 osd.0 (osd.0) 249 : cluster [DBG] 5.1e scrub starts 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:51.109309+0000 osd.0 (osd.0) 250 : cluster [DBG] 5.1e scrub ok 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.398299+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.1a scrub starts 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:52 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.400320+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.1a scrub ok 2024-09-20T19:43:53.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:53.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:53.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:53.934 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:53 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.298256+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.4 scrub starts 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.301606+0000 osd.3 (osd.3) 164 : cluster [DBG] 1.4 scrub ok 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.515696+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.17 scrub starts 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.518096+0000 osd.4 (osd.4) 328 : cluster [DBG] 6.17 scrub ok 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:54.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.642574+0000 mgr.y (mgr.24491) 966 : cluster [DBG] pgmap v1761: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.912662+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.12 scrub starts 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:52.914850+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.12 scrub ok 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.359589+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.7 scrub starts 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:54.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:53 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.361672+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.7 scrub ok 2024-09-20T19:43:54.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:54.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:54.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:54.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:54.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:54.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:54 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.533171+0000 osd.4 (osd.4) 329 : cluster [DBG] 2.f scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.536278+0000 osd.4 (osd.4) 330 : cluster [DBG] 2.f scrub ok 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.910762+0000 osd.7 (osd.7) 367 : cluster [DBG] 6.5 scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:53.912941+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub ok 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.376467+0000 osd.6 (osd.6) 344 : cluster [DBG] 2.14 scrub starts 2024-09-20T19:43:55.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:55.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:54 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.378652+0000 osd.6 (osd.6) 345 : cluster [DBG] 2.14 scrub ok 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:55.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:55.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:55.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:55.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:55 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:54.643600+0000 mgr.y (mgr.24491) 967 : cluster [DBG] pgmap v1762: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.335055+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.17 scrub starts 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.337267+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.17 scrub ok 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.483598+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.12 scrub starts 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:56.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:55 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.485914+0000 osd.1 (osd.1) 502 : cluster [DBG] 5.12 scrub ok 2024-09-20T19:43:56.559 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:56.642 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:43:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[22706]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:56.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:56 smithi042 bash[32075]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:56 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:57.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:56 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.263184+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1b scrub starts 2024-09-20T19:43:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:56 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:57.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:56 smithi162 bash[24557]: cluster 2024-09-20T19:43:55.265754+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1b scrub ok 2024-09-20T19:43:57.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:57.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:57.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:57.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:57.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:57.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:57.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:57.935 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:57.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:57.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:57.936 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:57 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.226940+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.229829+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.510309+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub starts 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.512667+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.2 scrub ok 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:58.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.644403+0000 mgr.y (mgr.24491) 968 : cluster [DBG] pgmap v1763: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.909684+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.1c scrub starts 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:56.912362+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.1c scrub ok 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.350290+0000 osd.6 (osd.6) 348 : cluster [DBG] 1.7 scrub starts 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:58.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:57 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.352656+0000 osd.6 (osd.6) 349 : cluster [DBG] 1.7 scrub ok 2024-09-20T19:43:58.347 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-20T19:43:58.349 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:58.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:58.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:58.933 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:58.934 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:58 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.528261+0000 osd.4 (osd.4) 333 : cluster [DBG] 4.1 scrub starts 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.533035+0000 osd.4 (osd.4) 334 : cluster [DBG] 4.1 scrub ok 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.576961+0000 osd.4 (osd.4) 335 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:59.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.579810+0000 osd.4 (osd.4) 336 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:57.660259+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e1102: 8 total, 8 up, 8 in 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.314532+0000 osd.6 (osd.6) 350 : cluster [DBG] 5.1f scrub starts 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.317389+0000 osd.6 (osd.6) 351 : cluster [DBG] 5.1f scrub ok 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.429086+0000 osd.1 (osd.1) 503 : cluster [DBG] 4.12 scrub starts 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:59.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:58 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.431240+0000 osd.1 (osd.1) 504 : cluster [DBG] 4.12 scrub ok 2024-09-20T19:43:59.110 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1764,"stamp":"2024-09-20T19:43:57.649808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3071,"num_read_kb":2823,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":858818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1971,"ondisk_log_size":1971,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":524028,"kb_used_data":7932,"kb_used_omap":143,"kb_used_meta":515824,"kb_avail":749437188,"statfs":{"total":767960285184,"available":767423680512,"internally_reserved":0,"allocated":8122368,"data_stored":5025641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147312,"internal_metadata":528203920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.011387"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234874+0000","last_change":"2024-09-20T19:43:32.115321+0000","last_active":"2024-09-20T19:43:41.234874+0000","last_peered":"2024-09-20T19:43:41.234874+0000","last_clean":"2024-09-20T19:43:41.234874+0000","last_became_active":"2024-09-20T19:40:44.720052+0000","last_became_peered":"2024-09-20T19:40:44.720052+0000","last_unstale":"2024-09-20T19:43:41.234874+0000","last_undegraded":"2024-09-20T19:43:41.234874+0000","last_fullsized":"2024-09-20T19:43:41.234874+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:32.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:32.115260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.334406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1952,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:42.057372+0000","last_change":"2024-09-20T19:43:42.057372+0000","last_active":"2024-09-20T19:43:42.057372+0000","last_peered":"2024-09-20T19:43:42.057372+0000","last_clean":"2024-09-20T19:43:42.057372+0000","last_became_active":"2024-09-20T19:34:58.530547+0000","last_became_peered":"2024-09-20T19:34:58.530547+0000","last_unstale":"2024-09-20T19:43:42.057372+0000","last_undegraded":"2024-09-20T19:43:42.057372+0000","last_fullsized":"2024-09-20T19:43:42.057372+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:42.057324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:42.057324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.036559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"987'39","reported_seq":113,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.231408+0000","last_change":"2024-09-20T19:43:09.288423+0000","last_active":"2024-09-20T19:43:41.231408+0000","last_peered":"2024-09-20T19:43:41.231408+0000","last_clean":"2024-09-20T19:43:41.231408+0000","last_became_active":"2024-09-20T19:40:44.709477+0000","last_became_peered":"2024-09-20T19:40:44.709477+0000","last_unstale":"2024-09-20T19:43:41.231408+0000","last_undegraded":"2024-09-20T19:43:41.231408+0000","last_fullsized":"2024-09-20T19:43:41.231408+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:43:09.288269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:09.288269+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:14.727998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1931,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699384+0000","last_change":"2024-09-20T19:43:06.736048+0000","last_active":"2024-09-20T19:43:40.699384+0000","last_peered":"2024-09-20T19:43:40.699384+0000","last_clean":"2024-09-20T19:43:40.699384+0000","last_became_active":"2024-09-20T19:30:31.363365+0000","last_became_peered":"2024-09-20T19:30:31.363365+0000","last_unstale":"2024-09-20T19:43:40.699384+0000","last_undegraded":"2024-09-20T19:43:40.699384+0000","last_fullsized":"2024-09-20T19:43:40.699384+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.735902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.735902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.049403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236163+0000","last_change":"2024-09-20T19:42:33.237917+0000","last_active":"2024-09-20T19:43:41.236163+0000","last_peered":"2024-09-20T19:43:41.236163+0000","last_clean":"2024-09-20T19:43:41.236163+0000","last_became_active":"2024-09-20T19:40:44.713295+0000","last_became_peered":"2024-09-20T19:40:44.713295+0000","last_unstale":"2024-09-20T19:43:41.236163+0000","last_undegraded":"2024-09-20T19:43:41.236163+0000","last_fullsized":"2024-09-20T19:43:41.236163+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:33.237754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:33.237754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.896189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"987'21","reported_seq":2044,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699793+0000","last_change":"2024-09-20T19:43:39.639518+0000","last_active":"2024-09-20T19:43:40.699793+0000","last_peered":"2024-09-20T19:43:40.699793+0000","last_clean":"2024-09-20T19:43:40.699793+0000","last_became_active":"2024-09-20T19:30:31.368347+0000","last_became_peered":"2024-09-20T19:30:31.368347+0000","last_unstale":"2024-09-20T19:43:40.699793+0000","last_undegraded":"2024-09-20T19:43:40.699793+0000","last_fullsized":"2024-09-20T19:43:40.699793+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:39.639367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:39.639367+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:09.093303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":2171,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700471+0000","last_change":"2024-09-20T19:43:16.071198+0000","last_active":"2024-09-20T19:43:40.700471+0000","last_peered":"2024-09-20T19:43:40.700471+0000","last_clean":"2024-09-20T19:43:40.700471+0000","last_became_active":"2024-09-20T19:18:11.327132+0000","last_became_peered":"2024-09-20T19:18:11.327132+0000","last_unstale":"2024-09-20T19:43:40.700471+0000","last_undegraded":"2024-09-20T19:43:40.700471+0000","last_fullsized":"2024-09-20T19:43:40.700471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:16.071077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.071077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:30.584638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1993,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700502+0000","last_change":"2024-09-20T19:42:43.656390+0000","last_active":"2024-09-20T19:43:40.700502+0000","last_peered":"2024-09-20T19:43:40.700502+0000","last_clean":"2024-09-20T19:43:40.700502+0000","last_became_active":"2024-09-20T19:30:31.346422+0000","last_became_peered":"2024-09-20T19:30:31.346422+0000","last_unstale":"2024-09-20T19:43:40.700502+0000","last_undegraded":"2024-09-20T19:43:40.700502+0000","last_fullsized":"2024-09-20T19:43:40.700502+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:43.656267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:43.656267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:59.187062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":2161,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.870080+0000","last_change":"2024-09-20T19:43:02.103353+0000","last_active":"2024-09-20T19:43:40.870080+0000","last_peered":"2024-09-20T19:43:40.870080+0000","last_clean":"2024-09-20T19:43:40.870080+0000","last_became_active":"2024-09-20T19:30:09.785404+0000","last_became_peered":"2024-09-20T19:30:09.785404+0000","last_unstale":"2024-09-20T19:43:40.870080+0000","last_undegraded":"2024-09-20T19:43:40.870080+0000","last_fullsized":"2024-09-20T19:43:40.870080+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:02.103206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:02.103206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.000717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"987'9","reported_seq":2054,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700485+0000","last_change":"2024-09-20T19:42:51.704440+0000","last_active":"2024-09-20T19:43:40.700485+0000","last_peered":"2024-09-20T19:43:40.700485+0000","last_clean":"2024-09-20T19:43:40.700485+0000","last_became_active":"2024-09-20T19:40:44.709660+0000","last_became_peered":"2024-09-20T19:40:44.709660+0000","last_unstale":"2024-09-20T19:43:40.700485+0000","last_undegraded":"2024-09-20T19:43:40.700485+0000","last_fullsized":"2024-09-20T19:43:40.700485+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'9","last_scrub_stamp":"2024-09-20T19:42:51.704299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:51.704299+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:20.045871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":2153,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868763+0000","last_change":"2024-09-20T19:43:21.144988+0000","last_active":"2024-09-20T19:43:40.868763+0000","last_peered":"2024-09-20T19:43:40.868763+0000","last_clean":"2024-09-20T19:43:40.868763+0000","last_became_active":"2024-09-20T19:30:31.352100+0000","last_became_peered":"2024-09-20T19:30:31.352100+0000","last_unstale":"2024-09-20T19:43:40.868763+0000","last_undegraded":"2024-09-20T19:43:40.868763+0000","last_fullsized":"2024-09-20T19:43:40.868763+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:21.144862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:21.144862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.664498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2169,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304395+0000","last_change":"2024-09-20T19:43:40.556085+0000","last_active":"2024-09-20T19:43:41.304395+0000","last_peered":"2024-09-20T19:43:41.304395+0000","last_clean":"2024-09-20T19:43:41.304395+0000","last_became_active":"2024-09-20T19:40:44.711101+0000","last_became_peered":"2024-09-20T19:40:44.711101+0000","last_unstale":"2024-09-20T19:43:41.304395+0000","last_undegraded":"2024-09-20T19:43:41.304395+0000","last_fullsized":"2024-09-20T19:43:41.304395+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:40.555955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:40.555955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:48.345498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":2150,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869690+0000","last_change":"2024-09-20T19:43:08.234634+0000","last_active":"2024-09-20T19:43:40.869690+0000","last_peered":"2024-09-20T19:43:40.869690+0000","last_clean":"2024-09-20T19:43:40.869690+0000","last_became_active":"2024-09-20T19:18:07.269890+0000","last_became_peered":"2024-09-20T19:18:07.269890+0000","last_unstale":"2024-09-20T19:43:40.869690+0000","last_undegraded":"2024-09-20T19:43:40.869690+0000","last_fullsized":"2024-09-20T19:43:40.869690+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:08.234509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:08.234509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.022931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"987'21","reported_seq":2097,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:45.552166+0000","last_change":"2024-09-20T19:43:45.552166+0000","last_active":"2024-09-20T19:43:45.552166+0000","last_peered":"2024-09-20T19:43:45.552166+0000","last_clean":"2024-09-20T19:43:45.552166+0000","last_became_active":"2024-09-20T19:40:44.707865+0000","last_became_peered":"2024-09-20T19:40:44.707865+0000","last_unstale":"2024-09-20T19:43:45.552166+0000","last_undegraded":"2024-09-20T19:43:45.552166+0000","last_fullsized":"2024-09-20T19:43:45.552166+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:45.551993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:45.551993+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:02.939959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2143,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305902+0000","last_change":"2024-09-20T19:43:18.636559+0000","last_active":"2024-09-20T19:43:41.305902+0000","last_peered":"2024-09-20T19:43:41.305902+0000","last_clean":"2024-09-20T19:43:41.305902+0000","last_became_active":"2024-09-20T19:30:09.787474+0000","last_became_peered":"2024-09-20T19:30:09.787474+0000","last_unstale":"2024-09-20T19:43:41.305902+0000","last_undegraded":"2024-09-20T19:43:41.305902+0000","last_fullsized":"2024-09-20T19:43:41.305902+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:18.636421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:18.636421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.222169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1129,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:50.147348+0000","last_change":"2024-09-20T19:43:50.147348+0000","last_active":"2024-09-20T19:43:50.147348+0000","last_peered":"2024-09-20T19:43:50.147348+0000","last_clean":"2024-09-20T19:43:50.147348+0000","last_became_active":"2024-09-20T19:31:16.267177+0000","last_became_peered":"2024-09-20T19:31:16.267177+0000","last_unstale":"2024-09-20T19:43:50.147348+0000","last_undegraded":"2024-09-20T19:43:50.147348+0000","last_fullsized":"2024-09-20T19:43:50.147348+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:50.147226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:50.147226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:07.961590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"92'79","reported_seq":1799,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:42.493667+0000","last_change":"2024-09-20T19:43:42.493667+0000","last_active":"2024-09-20T19:43:42.493667+0000","last_peered":"2024-09-20T19:43:42.493667+0000","last_clean":"2024-09-20T19:43:42.493667+0000","last_became_active":"2024-09-20T19:40:44.706211+0000","last_became_peered":"2024-09-20T19:40:44.706211+0000","last_unstale":"2024-09-20T19:43:42.493667+0000","last_undegraded":"2024-09-20T19:43:42.493667+0000","last_fullsized":"2024-09-20T19:43:42.493667+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:42.493535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:42.493535+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:58.459920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"987'45","reported_seq":2034,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700994+0000","last_change":"2024-09-20T19:43:21.673082+0000","last_active":"2024-09-20T19:43:40.700994+0000","last_peered":"2024-09-20T19:43:40.700994+0000","last_clean":"2024-09-20T19:43:40.700994+0000","last_became_active":"2024-09-20T19:30:31.364068+0000","last_became_peered":"2024-09-20T19:30:31.364068+0000","last_unstale":"2024-09-20T19:43:40.700994+0000","last_undegraded":"2024-09-20T19:43:40.700994+0000","last_fullsized":"2024-09-20T19:43:40.700994+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'45","last_scrub_stamp":"2024-09-20T19:43:21.672879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:21.672879+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.663091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":2154,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.870219+0000","last_change":"2024-09-20T19:43:16.175369+0000","last_active":"2024-09-20T19:43:40.870219+0000","last_peered":"2024-09-20T19:43:40.870219+0000","last_clean":"2024-09-20T19:43:40.870219+0000","last_became_active":"2024-09-20T19:30:31.374152+0000","last_became_peered":"2024-09-20T19:30:31.374152+0000","last_unstale":"2024-09-20T19:43:40.870219+0000","last_undegraded":"2024-09-20T19:43:40.870219+0000","last_fullsized":"2024-09-20T19:43:40.870219+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:16.175256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.175256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:17.100865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":2174,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701840+0000","last_change":"2024-09-20T19:43:31.174354+0000","last_active":"2024-09-20T19:43:40.701840+0000","last_peered":"2024-09-20T19:43:40.701840+0000","last_clean":"2024-09-20T19:43:40.701840+0000","last_became_active":"2024-09-20T19:30:31.367007+0000","last_became_peered":"2024-09-20T19:30:31.367007+0000","last_unstale":"2024-09-20T19:43:40.701840+0000","last_undegraded":"2024-09-20T19:43:40.701840+0000","last_fullsized":"2024-09-20T19:43:40.701840+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:31.174232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:31.174232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.759389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":748,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701689+0000","last_change":"2024-09-20T19:43:30.427709+0000","last_active":"2024-09-20T19:43:40.701689+0000","last_peered":"2024-09-20T19:43:40.701689+0000","last_clean":"2024-09-20T19:43:40.701689+0000","last_became_active":"2024-09-20T19:40:44.698643+0000","last_became_peered":"2024-09-20T19:40:44.698643+0000","last_unstale":"2024-09-20T19:43:40.701689+0000","last_undegraded":"2024-09-20T19:43:40.701689+0000","last_fullsized":"2024-09-20T19:43:40.701689+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.427581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.427581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:43.576650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"92'79","reported_seq":2247,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868011+0000","last_change":"2024-09-20T19:43:23.076869+0000","last_active":"2024-09-20T19:43:40.868011+0000","last_peered":"2024-09-20T19:43:40.868011+0000","last_clean":"2024-09-20T19:43:40.868011+0000","last_became_active":"2024-09-20T19:40:44.703846+0000","last_became_peered":"2024-09-20T19:40:44.703846+0000","last_unstale":"2024-09-20T19:43:40.868011+0000","last_undegraded":"2024-09-20T19:43:40.868011+0000","last_fullsized":"2024-09-20T19:43:40.868011+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:23.076715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.076715+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.917049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"987'30","reported_seq":2098,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700861+0000","last_change":"2024-09-20T19:42:37.674511+0000","last_active":"2024-09-20T19:43:40.700861+0000","last_peered":"2024-09-20T19:43:40.700861+0000","last_clean":"2024-09-20T19:43:40.700861+0000","last_became_active":"2024-09-20T19:40:44.707964+0000","last_became_peered":"2024-09-20T19:40:44.707964+0000","last_unstale":"2024-09-20T19:43:40.700861+0000","last_undegraded":"2024-09-20T19:43:40.700861+0000","last_fullsized":"2024-09-20T19:43:40.700861+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:42:37.674338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:37.674338+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:03.653539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":2197,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701242+0000","last_change":"2024-09-20T19:42:53.016414+0000","last_active":"2024-09-20T19:43:40.701242+0000","last_peered":"2024-09-20T19:43:40.701242+0000","last_clean":"2024-09-20T19:43:40.701242+0000","last_became_active":"2024-09-20T19:40:44.713805+0000","last_became_peered":"2024-09-20T19:40:44.713805+0000","last_unstale":"2024-09-20T19:43:40.701242+0000","last_undegraded":"2024-09-20T19:43:40.701242+0000","last_fullsized":"2024-09-20T19:43:40.701242+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:53.016283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:53.016283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:05.103857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"77'11","reported_seq":2492,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547080+0000","last_change":"2024-09-20T19:42:53.546824+0000","last_active":"2024-09-20T19:43:48.547080+0000","last_peered":"2024-09-20T19:43:48.547080+0000","last_clean":"2024-09-20T19:43:48.547080+0000","last_became_active":"2024-09-20T19:18:11.332533+0000","last_became_peered":"2024-09-20T19:18:11.332533+0000","last_unstale":"2024-09-20T19:43:48.547080+0000","last_undegraded":"2024-09-20T19:43:48.547080+0000","last_fullsized":"2024-09-20T19:43:48.547080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-20T19:42:53.546647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:53.546647+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:00.137973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":2003,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701080+0000","last_change":"2024-09-20T19:42:55.630806+0000","last_active":"2024-09-20T19:43:40.701080+0000","last_peered":"2024-09-20T19:43:40.701080+0000","last_clean":"2024-09-20T19:43:40.701080+0000","last_became_active":"2024-09-20T19:34:26.408471+0000","last_became_peered":"2024-09-20T19:34:26.408471+0000","last_unstale":"2024-09-20T19:43:40.701080+0000","last_undegraded":"2024-09-20T19:43:40.701080+0000","last_fullsized":"2024-09-20T19:43:40.701080+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:55.630688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:55.630688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:09.663889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"92'79","reported_seq":1751,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305604+0000","last_change":"2024-09-20T19:43:24.674112+0000","last_active":"2024-09-20T19:43:41.305604+0000","last_peered":"2024-09-20T19:43:41.305604+0000","last_clean":"2024-09-20T19:43:41.305604+0000","last_became_active":"2024-09-20T19:24:11.088849+0000","last_became_peered":"2024-09-20T19:24:11.088849+0000","last_unstale":"2024-09-20T19:43:41.305604+0000","last_undegraded":"2024-09-20T19:43:41.305604+0000","last_fullsized":"2024-09-20T19:43:41.305604+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:24.673991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:24.673991+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.741203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"987'39","reported_seq":2298,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702508+0000","last_change":"2024-09-20T19:42:35.126118+0000","last_active":"2024-09-20T19:43:40.702508+0000","last_peered":"2024-09-20T19:43:40.702508+0000","last_clean":"2024-09-20T19:43:40.702508+0000","last_became_active":"2024-09-20T19:40:44.712608+0000","last_became_peered":"2024-09-20T19:40:44.712608+0000","last_unstale":"2024-09-20T19:43:40.702508+0000","last_undegraded":"2024-09-20T19:43:40.702508+0000","last_fullsized":"2024-09-20T19:43:40.702508+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:35.125939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:35.125939+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.306367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":2205,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700621+0000","last_change":"2024-09-20T19:43:37.267471+0000","last_active":"2024-09-20T19:43:40.700621+0000","last_peered":"2024-09-20T19:43:40.700621+0000","last_clean":"2024-09-20T19:43:40.700621+0000","last_became_active":"2024-09-20T19:40:44.712329+0000","last_became_peered":"2024-09-20T19:40:44.712329+0000","last_unstale":"2024-09-20T19:43:40.700621+0000","last_undegraded":"2024-09-20T19:43:40.700621+0000","last_fullsized":"2024-09-20T19:43:40.700621+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.267320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.267320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.571815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"523'15","reported_seq":2278,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547174+0000","last_change":"2024-09-20T19:42:54.650140+0000","last_active":"2024-09-20T19:43:48.547174+0000","last_peered":"2024-09-20T19:43:48.547174+0000","last_clean":"2024-09-20T19:43:48.547174+0000","last_became_active":"2024-09-20T19:30:31.351353+0000","last_became_peered":"2024-09-20T19:30:31.351353+0000","last_unstale":"2024-09-20T19:43:48.547174+0000","last_undegraded":"2024-09-20T19:43:48.547174+0000","last_fullsized":"2024-09-20T19:43:48.547174+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'15","last_scrub_stamp":"2024-09-20T19:42:54.650016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:54.650016+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:18.726791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2172,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:45.495446+0000","last_change":"2024-09-20T19:43:45.495446+0000","last_active":"2024-09-20T19:43:45.495446+0000","last_peered":"2024-09-20T19:43:45.495446+0000","last_clean":"2024-09-20T19:43:45.495446+0000","last_became_active":"2024-09-20T19:30:09.781963+0000","last_became_peered":"2024-09-20T19:30:09.781963+0000","last_unstale":"2024-09-20T19:43:45.495446+0000","last_undegraded":"2024-09-20T19:43:45.495446+0000","last_fullsized":"2024-09-20T19:43:45.495446+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:45.495318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:45.495318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:06.383492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"987'27","reported_seq":2057,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701655+0000","last_change":"2024-09-20T19:42:45.695604+0000","last_active":"2024-09-20T19:43:40.701655+0000","last_peered":"2024-09-20T19:43:40.701655+0000","last_clean":"2024-09-20T19:43:40.701655+0000","last_became_active":"2024-09-20T19:30:31.362986+0000","last_became_peered":"2024-09-20T19:30:31.362986+0000","last_unstale":"2024-09-20T19:43:40.701655+0000","last_undegraded":"2024-09-20T19:43:40.701655+0000","last_fullsized":"2024-09-20T19:43:40.701655+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:42:45.695469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:45.695469+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:03.816337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.698947+0000","last_change":"2024-09-20T19:43:28.388629+0000","last_active":"2024-09-20T19:43:40.698947+0000","last_peered":"2024-09-20T19:43:40.698947+0000","last_clean":"2024-09-20T19:43:40.698947+0000","last_became_active":"2024-09-20T19:40:44.685956+0000","last_became_peered":"2024-09-20T19:40:44.685956+0000","last_unstale":"2024-09-20T19:43:40.698947+0000","last_undegraded":"2024-09-20T19:43:40.698947+0000","last_fullsized":"2024-09-20T19:43:40.698947+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:28.388542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:28.388542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.024679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306960+0000","last_change":"2024-09-20T19:43:06.757297+0000","last_active":"2024-09-20T19:43:41.306960+0000","last_peered":"2024-09-20T19:43:41.306960+0000","last_clean":"2024-09-20T19:43:41.306960+0000","last_became_active":"2024-09-20T19:30:31.367854+0000","last_became_peered":"2024-09-20T19:30:31.367854+0000","last_unstale":"2024-09-20T19:43:41.306960+0000","last_undegraded":"2024-09-20T19:43:41.306960+0000","last_fullsized":"2024-09-20T19:43:41.306960+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.757180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.757180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:16.939577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.235262+0000","last_change":"2024-09-20T19:43:10.269113+0000","last_active":"2024-09-20T19:43:41.235262+0000","last_peered":"2024-09-20T19:43:41.235262+0000","last_clean":"2024-09-20T19:43:41.235262+0000","last_became_active":"2024-09-20T19:40:44.720237+0000","last_became_peered":"2024-09-20T19:40:44.720237+0000","last_unstale":"2024-09-20T19:43:41.235262+0000","last_undegraded":"2024-09-20T19:43:41.235262+0000","last_fullsized":"2024-09-20T19:43:41.235262+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:10.268986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:10.268986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:20.106838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"92'79","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:52.301750+0000","last_change":"2024-09-20T19:43:52.301750+0000","last_active":"2024-09-20T19:43:52.301750+0000","last_peered":"2024-09-20T19:43:52.301750+0000","last_clean":"2024-09-20T19:43:52.301750+0000","last_became_active":"2024-09-20T19:40:44.706650+0000","last_became_peered":"2024-09-20T19:40:44.706650+0000","last_unstale":"2024-09-20T19:43:52.301750+0000","last_undegraded":"2024-09-20T19:43:52.301750+0000","last_fullsized":"2024-09-20T19:43:52.301750+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:52.301616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:52.301616+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:55.454774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"987'42","reported_seq":2036,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699999+0000","last_change":"2024-09-20T19:42:28.762499+0000","last_active":"2024-09-20T19:43:40.699999+0000","last_peered":"2024-09-20T19:43:40.699999+0000","last_clean":"2024-09-20T19:43:40.699999+0000","last_became_active":"2024-09-20T19:30:31.383091+0000","last_became_peered":"2024-09-20T19:30:31.383091+0000","last_unstale":"2024-09-20T19:43:40.699999+0000","last_undegraded":"2024-09-20T19:43:40.699999+0000","last_fullsized":"2024-09-20T19:43:40.699999+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:42:28.762361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:28.762361+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:54.586443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:53.361806+0000","last_change":"2024-09-20T19:43:53.361806+0000","last_active":"2024-09-20T19:43:53.361806+0000","last_peered":"2024-09-20T19:43:53.361806+0000","last_clean":"2024-09-20T19:43:53.361806+0000","last_became_active":"2024-09-20T19:18:07.266065+0000","last_became_peered":"2024-09-20T19:18:07.266065+0000","last_unstale":"2024-09-20T19:43:53.361806+0000","last_undegraded":"2024-09-20T19:43:53.361806+0000","last_fullsized":"2024-09-20T19:43:53.361806+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:53.361684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:53.361684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:17.296242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1066'17","reported_seq":148,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:56.229980+0000","last_change":"2024-09-20T19:43:56.229980+0000","last_active":"2024-09-20T19:43:56.229980+0000","last_peered":"2024-09-20T19:43:56.229980+0000","last_clean":"2024-09-20T19:43:56.229980+0000","last_became_active":"2024-09-20T19:40:44.707312+0000","last_became_peered":"2024-09-20T19:40:44.707312+0000","last_unstale":"2024-09-20T19:43:56.229980+0000","last_undegraded":"2024-09-20T19:43:56.229980+0000","last_fullsized":"2024-09-20T19:43:56.229980+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'17","last_scrub_stamp":"2024-09-20T19:43:56.229843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:56.229843+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:01.688997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":2140,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869362+0000","last_change":"2024-09-20T19:43:38.016361+0000","last_active":"2024-09-20T19:43:40.869362+0000","last_peered":"2024-09-20T19:43:40.869362+0000","last_clean":"2024-09-20T19:43:40.869362+0000","last_became_active":"2024-09-20T19:18:13.497317+0000","last_became_peered":"2024-09-20T19:18:13.497317+0000","last_unstale":"2024-09-20T19:43:40.869362+0000","last_undegraded":"2024-09-20T19:43:40.869362+0000","last_fullsized":"2024-09-20T19:43:40.869362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.016233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.016233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.872867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"92'79","reported_seq":1570,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:49.561629+0000","last_change":"2024-09-20T19:43:49.561629+0000","last_active":"2024-09-20T19:43:49.561629+0000","last_peered":"2024-09-20T19:43:49.561629+0000","last_clean":"2024-09-20T19:43:49.561629+0000","last_became_active":"2024-09-20T19:34:26.408399+0000","last_became_peered":"2024-09-20T19:34:26.408399+0000","last_unstale":"2024-09-20T19:43:49.561629+0000","last_undegraded":"2024-09-20T19:43:49.561629+0000","last_fullsized":"2024-09-20T19:43:49.561629+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:49.561501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:49.561501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:11.262968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"987'42","reported_seq":115,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.233243+0000","last_change":"2024-09-20T19:42:42.265268+0000","last_active":"2024-09-20T19:43:41.233243+0000","last_peered":"2024-09-20T19:43:41.233243+0000","last_clean":"2024-09-20T19:43:41.233243+0000","last_became_active":"2024-09-20T19:40:44.709362+0000","last_became_peered":"2024-09-20T19:40:44.709362+0000","last_unstale":"2024-09-20T19:43:41.233243+0000","last_undegraded":"2024-09-20T19:43:41.233243+0000","last_fullsized":"2024-09-20T19:43:41.233243+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:42:42.265093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:42.265093+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:04.684731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"70'1","reported_seq":2176,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:43.404096+0000","last_change":"2024-09-20T19:43:43.404096+0000","last_active":"2024-09-20T19:43:43.404096+0000","last_peered":"2024-09-20T19:43:43.404096+0000","last_clean":"2024-09-20T19:43:43.404096+0000","last_became_active":"2024-09-20T19:30:31.367017+0000","last_became_peered":"2024-09-20T19:30:31.367017+0000","last_unstale":"2024-09-20T19:43:43.404096+0000","last_undegraded":"2024-09-20T19:43:43.404096+0000","last_fullsized":"2024-09-20T19:43:43.404096+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:43.403961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:43.403961+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.066903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1989,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699384+0000","last_change":"2024-09-20T19:43:37.641806+0000","last_active":"2024-09-20T19:43:40.699384+0000","last_peered":"2024-09-20T19:43:40.699384+0000","last_clean":"2024-09-20T19:43:40.699384+0000","last_became_active":"2024-09-20T19:40:44.699997+0000","last_became_peered":"2024-09-20T19:40:44.699997+0000","last_unstale":"2024-09-20T19:43:40.699384+0000","last_undegraded":"2024-09-20T19:43:40.699384+0000","last_fullsized":"2024-09-20T19:43:40.699384+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.641686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.641686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.606200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1935,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699863+0000","last_change":"2024-09-20T19:42:48.764388+0000","last_active":"2024-09-20T19:43:40.699863+0000","last_peered":"2024-09-20T19:43:40.699863+0000","last_clean":"2024-09-20T19:43:40.699863+0000","last_became_active":"2024-09-20T19:40:44.708574+0000","last_became_peered":"2024-09-20T19:40:44.708574+0000","last_unstale":"2024-09-20T19:43:40.699863+0000","last_undegraded":"2024-09-20T19:43:40.699863+0000","last_fullsized":"2024-09-20T19:43:40.699863+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:48.764274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:48.764274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:52.833060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"92'79","reported_seq":1889,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929507+0000","last_change":"2024-09-20T19:43:33.429853+0000","last_active":"2024-09-20T19:43:40.929507+0000","last_peered":"2024-09-20T19:43:40.929507+0000","last_clean":"2024-09-20T19:43:40.929507+0000","last_became_active":"2024-09-20T19:40:44.713745+0000","last_became_peered":"2024-09-20T19:40:44.713745+0000","last_unstale":"2024-09-20T19:43:40.929507+0000","last_undegraded":"2024-09-20T19:43:40.929507+0000","last_fullsized":"2024-09-20T19:43:40.929507+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:33.429639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.429639+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.371654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"987'39","reported_seq":1985,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924355+0000","last_change":"2024-09-20T19:43:18.334308+0000","last_active":"2024-09-20T19:43:40.924355+0000","last_peered":"2024-09-20T19:43:40.924355+0000","last_clean":"2024-09-20T19:43:40.924355+0000","last_became_active":"2024-09-20T19:30:09.779890+0000","last_became_peered":"2024-09-20T19:30:09.779890+0000","last_unstale":"2024-09-20T19:43:40.924355+0000","last_undegraded":"2024-09-20T19:43:40.924355+0000","last_fullsized":"2024-09-20T19:43:40.924355+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:43:18.334155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:18.334155+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:22.767119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2117,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701209+0000","last_change":"2024-09-20T19:43:35.420958+0000","last_active":"2024-09-20T19:43:40.701209+0000","last_peered":"2024-09-20T19:43:40.701209+0000","last_clean":"2024-09-20T19:43:40.701209+0000","last_became_active":"2024-09-20T19:30:31.348089+0000","last_became_peered":"2024-09-20T19:30:31.348089+0000","last_unstale":"2024-09-20T19:43:40.701209+0000","last_undegraded":"2024-09-20T19:43:40.701209+0000","last_fullsized":"2024-09-20T19:43:40.701209+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:35.420883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:35.420883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.087239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2164,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.927771+0000","last_change":"2024-09-20T19:43:20.385183+0000","last_active":"2024-09-20T19:43:40.927771+0000","last_peered":"2024-09-20T19:43:40.927771+0000","last_clean":"2024-09-20T19:43:40.927771+0000","last_became_active":"2024-09-20T19:30:09.774851+0000","last_became_peered":"2024-09-20T19:30:09.774851+0000","last_unstale":"2024-09-20T19:43:40.927771+0000","last_undegraded":"2024-09-20T19:43:40.927771+0000","last_fullsized":"2024-09-20T19:43:40.927771+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.385063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.385063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.325790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":2159,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700317+0000","last_change":"2024-09-20T19:42:43.099657+0000","last_active":"2024-09-20T19:43:40.700317+0000","last_peered":"2024-09-20T19:43:40.700317+0000","last_clean":"2024-09-20T19:43:40.700317+0000","last_became_active":"2024-09-20T19:34:26.392336+0000","last_became_peered":"2024-09-20T19:34:26.392336+0000","last_unstale":"2024-09-20T19:43:40.700317+0000","last_undegraded":"2024-09-20T19:43:40.700317+0000","last_fullsized":"2024-09-20T19:43:40.700317+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:43.099599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:43.099599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:12.352694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"92'79","reported_seq":1874,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929299+0000","last_change":"2024-09-20T19:42:41.330947+0000","last_active":"2024-09-20T19:43:40.929299+0000","last_peered":"2024-09-20T19:43:40.929299+0000","last_clean":"2024-09-20T19:43:40.929299+0000","last_became_active":"2024-09-20T19:40:44.713893+0000","last_became_peered":"2024-09-20T19:40:44.713893+0000","last_unstale":"2024-09-20T19:43:40.929299+0000","last_undegraded":"2024-09-20T19:43:40.929299+0000","last_fullsized":"2024-09-20T19:43:40.929299+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:42:41.330790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:42:41.330790+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:43.745241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"987'30","reported_seq":2220,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701171+0000","last_change":"2024-09-20T19:43:34.246690+0000","last_active":"2024-09-20T19:43:40.701171+0000","last_peered":"2024-09-20T19:43:40.701171+0000","last_clean":"2024-09-20T19:43:40.701171+0000","last_became_active":"2024-09-20T19:30:31.345568+0000","last_became_peered":"2024-09-20T19:30:31.345568+0000","last_unstale":"2024-09-20T19:43:40.701171+0000","last_undegraded":"2024-09-20T19:43:40.701171+0000","last_fullsized":"2024-09-20T19:43:40.701171+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:43:34.246546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:34.246546+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:58.942888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2168,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.442004+0000","last_change":"2024-09-20T19:43:44.442004+0000","last_active":"2024-09-20T19:43:44.442004+0000","last_peered":"2024-09-20T19:43:44.442004+0000","last_clean":"2024-09-20T19:43:44.442004+0000","last_became_active":"2024-09-20T19:30:09.760349+0000","last_became_peered":"2024-09-20T19:30:09.760349+0000","last_unstale":"2024-09-20T19:43:44.442004+0000","last_undegraded":"2024-09-20T19:43:44.442004+0000","last_fullsized":"2024-09-20T19:43:44.442004+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.441901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.441901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.960477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"501'13","reported_seq":2204,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.096973+0000","last_change":"2024-09-20T19:43:46.096849+0000","last_active":"2024-09-20T19:43:46.096973+0000","last_peered":"2024-09-20T19:43:46.096973+0000","last_clean":"2024-09-20T19:43:46.096973+0000","last_became_active":"2024-09-20T19:30:09.782835+0000","last_became_peered":"2024-09-20T19:30:09.782835+0000","last_unstale":"2024-09-20T19:43:46.096973+0000","last_undegraded":"2024-09-20T19:43:46.096973+0000","last_fullsized":"2024-09-20T19:43:46.096973+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:43:46.096716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.096716+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:07.301177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1962,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923004+0000","last_change":"2024-09-20T19:43:06.185977+0000","last_active":"2024-09-20T19:43:40.923004+0000","last_peered":"2024-09-20T19:43:40.923004+0000","last_clean":"2024-09-20T19:43:40.923004+0000","last_became_active":"2024-09-20T19:40:44.696294+0000","last_became_peered":"2024-09-20T19:40:44.696294+0000","last_unstale":"2024-09-20T19:43:40.923004+0000","last_undegraded":"2024-09-20T19:43:40.923004+0000","last_fullsized":"2024-09-20T19:43:40.923004+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.185854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.185854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.466406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"92'79","reported_seq":1754,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304891+0000","last_change":"2024-09-20T19:43:01.736720+0000","last_active":"2024-09-20T19:43:41.304891+0000","last_peered":"2024-09-20T19:43:41.304891+0000","last_clean":"2024-09-20T19:43:41.304891+0000","last_became_active":"2024-09-20T19:30:09.791339+0000","last_became_peered":"2024-09-20T19:30:09.791339+0000","last_unstale":"2024-09-20T19:43:41.304891+0000","last_undegraded":"2024-09-20T19:43:41.304891+0000","last_fullsized":"2024-09-20T19:43:41.304891+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:01.736588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:01.736588+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.669744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"1066'58","reported_seq":2655,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547072+0000","last_change":"2024-09-20T19:43:33.197027+0000","last_active":"2024-09-20T19:43:48.547072+0000","last_peered":"2024-09-20T19:43:48.547072+0000","last_clean":"2024-09-20T19:43:48.547072+0000","last_became_active":"2024-09-20T19:40:44.714403+0000","last_became_peered":"2024-09-20T19:40:44.714403+0000","last_unstale":"2024-09-20T19:43:48.547072+0000","last_undegraded":"2024-09-20T19:43:48.547072+0000","last_fullsized":"2024-09-20T19:43:48.547072+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'58","last_scrub_stamp":"2024-09-20T19:43:33.196872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.196872+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.712005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2174,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306153+0000","last_change":"2024-09-20T19:43:27.740801+0000","last_active":"2024-09-20T19:43:41.306153+0000","last_peered":"2024-09-20T19:43:41.306153+0000","last_clean":"2024-09-20T19:43:41.306153+0000","last_became_active":"2024-09-20T19:18:07.266387+0000","last_became_peered":"2024-09-20T19:18:07.266387+0000","last_unstale":"2024-09-20T19:43:41.306153+0000","last_undegraded":"2024-09-20T19:43:41.306153+0000","last_fullsized":"2024-09-20T19:43:41.306153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:27.740678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:27.740678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.169334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1945,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928231+0000","last_change":"2024-09-20T19:42:36.425491+0000","last_active":"2024-09-20T19:43:40.928231+0000","last_peered":"2024-09-20T19:43:40.928231+0000","last_clean":"2024-09-20T19:43:40.928231+0000","last_became_active":"2024-09-20T19:30:31.353295+0000","last_became_peered":"2024-09-20T19:30:31.353295+0000","last_unstale":"2024-09-20T19:43:40.928231+0000","last_undegraded":"2024-09-20T19:43:40.928231+0000","last_fullsized":"2024-09-20T19:43:40.928231+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:36.425343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.425343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.749302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232172+0000","last_change":"2024-09-20T19:43:29.098576+0000","last_active":"2024-09-20T19:43:41.232172+0000","last_peered":"2024-09-20T19:43:41.232172+0000","last_clean":"2024-09-20T19:43:41.232172+0000","last_became_active":"2024-09-20T19:40:44.707825+0000","last_became_peered":"2024-09-20T19:40:44.707825+0000","last_unstale":"2024-09-20T19:43:41.232172+0000","last_undegraded":"2024-09-20T19:43:41.232172+0000","last_fullsized":"2024-09-20T19:43:41.232172+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:29.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:29.098515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.259043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"92'79","reported_seq":2226,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869652+0000","last_change":"2024-09-20T19:43:00.103265+0000","last_active":"2024-09-20T19:43:40.869652+0000","last_peered":"2024-09-20T19:43:40.869652+0000","last_clean":"2024-09-20T19:43:40.869652+0000","last_became_active":"2024-09-20T19:30:31.374444+0000","last_became_peered":"2024-09-20T19:30:31.374444+0000","last_unstale":"2024-09-20T19:43:40.869652+0000","last_undegraded":"2024-09-20T19:43:40.869652+0000","last_fullsized":"2024-09-20T19:43:40.869652+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":4,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:00.103109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:00.103109+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.133124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"987'21","reported_seq":2134,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702312+0000","last_change":"2024-09-20T19:43:37.431497+0000","last_active":"2024-09-20T19:43:40.702312+0000","last_peered":"2024-09-20T19:43:40.702312+0000","last_clean":"2024-09-20T19:43:40.702312+0000","last_became_active":"2024-09-20T19:30:09.767079+0000","last_became_peered":"2024-09-20T19:30:09.767079+0000","last_unstale":"2024-09-20T19:43:40.702312+0000","last_undegraded":"2024-09-20T19:43:40.702312+0000","last_fullsized":"2024-09-20T19:43:40.702312+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:37.431298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.431298+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:52.073391+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"84'5","reported_seq":3711,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:55.048250+0000","last_change":"2024-09-20T19:43:12.090271+0000","last_active":"2024-09-20T19:43:55.048250+0000","last_peered":"2024-09-20T19:43:55.048250+0000","last_clean":"2024-09-20T19:43:55.048250+0000","last_became_active":"2024-09-20T19:18:08.270531+0000","last_became_peered":"2024-09-20T19:18:08.270531+0000","last_unstale":"2024-09-20T19:43:55.048250+0000","last_undegraded":"2024-09-20T19:43:55.048250+0000","last_fullsized":"2024-09-20T19:43:55.048250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-09-20T19:43:12.090136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:12.090136+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.493942+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092567500000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1530,"num_read_kb":1525,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2202,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699335+0000","last_change":"2024-09-20T19:43:08.394716+0000","last_active":"2024-09-20T19:43:40.699335+0000","last_peered":"2024-09-20T19:43:40.699335+0000","last_clean":"2024-09-20T19:43:40.699335+0000","last_became_active":"2024-09-20T19:40:44.686713+0000","last_became_peered":"2024-09-20T19:40:44.686713+0000","last_unstale":"2024-09-20T19:43:40.699335+0000","last_undegraded":"2024-09-20T19:43:40.699335+0000","last_fullsized":"2024-09-20T19:43:40.699335+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:08.394592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:08.394592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:32.209356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":749,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.488692+0000","last_change":"2024-09-20T19:43:44.488692+0000","last_active":"2024-09-20T19:43:44.488692+0000","last_peered":"2024-09-20T19:43:44.488692+0000","last_clean":"2024-09-20T19:43:44.488692+0000","last_became_active":"2024-09-20T19:34:26.388772+0000","last_became_peered":"2024-09-20T19:34:26.388772+0000","last_unstale":"2024-09-20T19:43:44.488692+0000","last_undegraded":"2024-09-20T19:43:44.488692+0000","last_fullsized":"2024-09-20T19:43:44.488692+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.488563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:13.360055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":2165,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:53.913081+0000","last_change":"2024-09-20T19:43:53.913081+0000","last_active":"2024-09-20T19:43:53.913081+0000","last_peered":"2024-09-20T19:43:53.913081+0000","last_clean":"2024-09-20T19:43:53.913081+0000","last_became_active":"2024-09-20T19:40:44.702203+0000","last_became_peered":"2024-09-20T19:40:44.702203+0000","last_unstale":"2024-09-20T19:43:53.913081+0000","last_undegraded":"2024-09-20T19:43:53.913081+0000","last_fullsized":"2024-09-20T19:43:53.913081+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:53.912951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:53.912951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.437001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"92'79","reported_seq":1759,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.360224+0000","last_change":"2024-09-20T19:43:41.360224+0000","last_active":"2024-09-20T19:43:41.360224+0000","last_peered":"2024-09-20T19:43:41.360224+0000","last_clean":"2024-09-20T19:43:41.360224+0000","last_became_active":"2024-09-20T19:30:31.365427+0000","last_became_peered":"2024-09-20T19:30:31.365427+0000","last_unstale":"2024-09-20T19:43:41.360224+0000","last_undegraded":"2024-09-20T19:43:41.360224+0000","last_fullsized":"2024-09-20T19:43:41.360224+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:41.360089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.360089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:51.419942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"987'36","reported_seq":2003,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929064+0000","last_change":"2024-09-20T19:43:29.166809+0000","last_active":"2024-09-20T19:43:40.929064+0000","last_peered":"2024-09-20T19:43:40.929064+0000","last_clean":"2024-09-20T19:43:40.929064+0000","last_became_active":"2024-09-20T19:34:26.393782+0000","last_became_peered":"2024-09-20T19:34:26.393782+0000","last_unstale":"2024-09-20T19:43:40.929064+0000","last_undegraded":"2024-09-20T19:43:40.929064+0000","last_fullsized":"2024-09-20T19:43:40.929064+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'36","last_scrub_stamp":"2024-09-20T19:43:29.166694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:29.166694+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.598626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"77'1","reported_seq":1910,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923000+0000","last_change":"2024-09-20T19:43:07.192824+0000","last_active":"2024-09-20T19:43:40.923000+0000","last_peered":"2024-09-20T19:43:40.923000+0000","last_clean":"2024-09-20T19:43:40.923000+0000","last_became_active":"2024-09-20T19:40:44.715407+0000","last_became_peered":"2024-09-20T19:40:44.715407+0000","last_unstale":"2024-09-20T19:43:40.923000+0000","last_undegraded":"2024-09-20T19:43:40.923000+0000","last_fullsized":"2024-09-20T19:43:40.923000+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:43:07.192693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:07.192693+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:21.949938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2163,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868651+0000","last_change":"2024-09-20T19:42:42.956040+0000","last_active":"2024-09-20T19:43:40.868651+0000","last_peered":"2024-09-20T19:43:40.868651+0000","last_clean":"2024-09-20T19:43:40.868651+0000","last_became_active":"2024-09-20T19:30:09.771745+0000","last_became_peered":"2024-09-20T19:30:09.771745+0000","last_unstale":"2024-09-20T19:43:40.868651+0000","last_undegraded":"2024-09-20T19:43:40.868651+0000","last_fullsized":"2024-09-20T19:43:40.868651+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:42.955918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:42.955918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:00.713228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.307033+0000","last_change":"2024-09-20T19:43:37.565380+0000","last_active":"2024-09-20T19:43:41.307033+0000","last_peered":"2024-09-20T19:43:41.307033+0000","last_clean":"2024-09-20T19:43:41.307033+0000","last_became_active":"2024-09-20T19:30:09.765831+0000","last_became_peered":"2024-09-20T19:30:09.765831+0000","last_unstale":"2024-09-20T19:43:41.307033+0000","last_undegraded":"2024-09-20T19:43:41.307033+0000","last_fullsized":"2024-09-20T19:43:41.307033+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:37.565261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:37.565261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.417913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":2202,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:49.480362+0000","last_change":"2024-09-20T19:43:49.480362+0000","last_active":"2024-09-20T19:43:49.480362+0000","last_peered":"2024-09-20T19:43:49.480362+0000","last_clean":"2024-09-20T19:43:49.480362+0000","last_became_active":"2024-09-20T19:40:44.712370+0000","last_became_peered":"2024-09-20T19:40:44.712370+0000","last_unstale":"2024-09-20T19:43:49.480362+0000","last_undegraded":"2024-09-20T19:43:49.480362+0000","last_fullsized":"2024-09-20T19:43:49.480362+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:49.480231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:49.480231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.473277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"987'42","reported_seq":2529,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547732+0000","last_change":"2024-09-20T19:43:38.381338+0000","last_active":"2024-09-20T19:43:48.547732+0000","last_peered":"2024-09-20T19:43:48.547732+0000","last_clean":"2024-09-20T19:43:48.547732+0000","last_became_active":"2024-09-20T19:30:31.368442+0000","last_became_peered":"2024-09-20T19:30:31.368442+0000","last_unstale":"2024-09-20T19:43:48.547732+0000","last_undegraded":"2024-09-20T19:43:48.547732+0000","last_fullsized":"2024-09-20T19:43:48.547732+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'42","last_scrub_stamp":"2024-09-20T19:43:38.381167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.381167+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.532424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'2","reported_seq":101,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234541+0000","last_change":"2024-09-20T19:43:11.254899+0000","last_active":"2024-09-20T19:43:41.234541+0000","last_peered":"2024-09-20T19:43:41.234541+0000","last_clean":"2024-09-20T19:43:41.234541+0000","last_became_active":"2024-09-20T19:40:44.718297+0000","last_became_peered":"2024-09-20T19:40:44.718297+0000","last_unstale":"2024-09-20T19:43:41.234541+0000","last_undegraded":"2024-09-20T19:43:41.234541+0000","last_fullsized":"2024-09-20T19:43:41.234541+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-09-20T19:43:11.254763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:07.222546+0000","last_clean_scrub_stamp":"2024-09-20T19:43:11.254763+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:36.708223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2150,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.540062+0000","last_change":"2024-09-20T19:43:44.540062+0000","last_active":"2024-09-20T19:43:44.540062+0000","last_peered":"2024-09-20T19:43:44.540062+0000","last_clean":"2024-09-20T19:43:44.540062+0000","last_became_active":"2024-09-20T19:18:07.268953+0000","last_became_peered":"2024-09-20T19:18:07.268953+0000","last_unstale":"2024-09-20T19:43:44.540062+0000","last_undegraded":"2024-09-20T19:43:44.540062+0000","last_fullsized":"2024-09-20T19:43:44.540062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:44.539947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.539947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.862876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'79","reported_seq":2214,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.931692+0000","last_change":"2024-09-20T19:43:40.931558+0000","last_active":"2024-09-20T19:43:40.931692+0000","last_peered":"2024-09-20T19:43:40.931692+0000","last_clean":"2024-09-20T19:43:40.931692+0000","last_became_active":"2024-09-20T19:30:09.772795+0000","last_became_peered":"2024-09-20T19:30:09.772795+0000","last_unstale":"2024-09-20T19:43:40.931692+0000","last_undegraded":"2024-09-20T19:43:40.931692+0000","last_fullsized":"2024-09-20T19:43:40.931692+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'79","last_scrub_stamp":"2024-09-20T19:43:40.931434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:12:47.071633+0000","last_clean_scrub_stamp":"2024-09-20T19:43:40.931434+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.441953+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":2136,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869114+0000","last_change":"2024-09-20T19:43:14.236338+0000","last_active":"2024-09-20T19:43:40.869114+0000","last_peered":"2024-09-20T19:43:40.869114+0000","last_clean":"2024-09-20T19:43:40.869114+0000","last_became_active":"2024-09-20T19:18:11.331578+0000","last_became_peered":"2024-09-20T19:18:11.331578+0000","last_unstale":"2024-09-20T19:43:40.869114+0000","last_undegraded":"2024-09-20T19:43:40.869114+0000","last_fullsized":"2024-09-20T19:43:40.869114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:14.236216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:14.236216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.885413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2177,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304709+0000","last_change":"2024-09-20T19:42:55.692715+0000","last_active":"2024-09-20T19:43:41.304709+0000","last_peered":"2024-09-20T19:43:41.304709+0000","last_clean":"2024-09-20T19:43:41.304709+0000","last_became_active":"2024-09-20T19:40:44.711751+0000","last_became_peered":"2024-09-20T19:40:44.711751+0000","last_unstale":"2024-09-20T19:43:41.304709+0000","last_undegraded":"2024-09-20T19:43:41.304709+0000","last_fullsized":"2024-09-20T19:43:41.304709+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:55.692591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:55.692591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:21.189425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"987'51","reported_seq":2219,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928283+0000","last_change":"2024-09-20T19:42:36.344750+0000","last_active":"2024-09-20T19:43:40.928283+0000","last_peered":"2024-09-20T19:43:40.928283+0000","last_clean":"2024-09-20T19:43:40.928283+0000","last_became_active":"2024-09-20T19:18:10.114916+0000","last_became_peered":"2024-09-20T19:18:10.114916+0000","last_unstale":"2024-09-20T19:43:40.928283+0000","last_undegraded":"2024-09-20T19:43:40.928283+0000","last_fullsized":"2024-09-20T19:43:40.928283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'51","last_scrub_stamp":"2024-09-20T19:42:36.344612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.344612+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.398229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":2188,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.703221+0000","last_change":"2024-09-20T19:43:38.476998+0000","last_active":"2024-09-20T19:43:40.703221+0000","last_peered":"2024-09-20T19:43:40.703221+0000","last_clean":"2024-09-20T19:43:40.703221+0000","last_became_active":"2024-09-20T19:30:09.766181+0000","last_became_peered":"2024-09-20T19:30:09.766181+0000","last_unstale":"2024-09-20T19:43:40.703221+0000","last_undegraded":"2024-09-20T19:43:40.703221+0000","last_fullsized":"2024-09-20T19:43:40.703221+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.476943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.476943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.330919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":741,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.703657+0000","last_change":"2024-09-20T19:43:00.500715+0000","last_active":"2024-09-20T19:43:40.703657+0000","last_peered":"2024-09-20T19:43:40.703657+0000","last_clean":"2024-09-20T19:43:40.703657+0000","last_became_active":"2024-09-20T19:34:26.389855+0000","last_became_peered":"2024-09-20T19:34:26.389855+0000","last_unstale":"2024-09-20T19:43:40.703657+0000","last_undegraded":"2024-09-20T19:43:40.703657+0000","last_fullsized":"2024-09-20T19:43:40.703657+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:00.500632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:00.500632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:23.774619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":2148,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.867811+0000","last_change":"2024-09-20T19:42:59.082822+0000","last_active":"2024-09-20T19:43:40.867811+0000","last_peered":"2024-09-20T19:43:40.867811+0000","last_clean":"2024-09-20T19:43:40.867811+0000","last_became_active":"2024-09-20T19:40:44.704276+0000","last_became_peered":"2024-09-20T19:40:44.704276+0000","last_unstale":"2024-09-20T19:43:40.867811+0000","last_undegraded":"2024-09-20T19:43:40.867811+0000","last_fullsized":"2024-09-20T19:43:40.867811+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:59.082699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.082699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.506479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"987'21","reported_seq":1977,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:44.092427+0000","last_change":"2024-09-20T19:43:44.092427+0000","last_active":"2024-09-20T19:43:44.092427+0000","last_peered":"2024-09-20T19:43:44.092427+0000","last_clean":"2024-09-20T19:43:44.092427+0000","last_became_active":"2024-09-20T19:34:58.555087+0000","last_became_peered":"2024-09-20T19:34:58.555087+0000","last_unstale":"2024-09-20T19:43:44.092427+0000","last_undegraded":"2024-09-20T19:43:44.092427+0000","last_fullsized":"2024-09-20T19:43:44.092427+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:43:44.092289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:44.092289+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:50.539505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":2195,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702614+0000","last_change":"2024-09-20T19:43:23.105414+0000","last_active":"2024-09-20T19:43:40.702614+0000","last_peered":"2024-09-20T19:43:40.702614+0000","last_clean":"2024-09-20T19:43:40.702614+0000","last_became_active":"2024-09-20T19:40:44.713034+0000","last_became_peered":"2024-09-20T19:40:44.713034+0000","last_unstale":"2024-09-20T19:43:40.702614+0000","last_undegraded":"2024-09-20T19:43:40.702614+0000","last_fullsized":"2024-09-20T19:43:40.702614+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.105343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.105343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.627530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2201,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700469+0000","last_change":"2024-09-20T19:43:22.309399+0000","last_active":"2024-09-20T19:43:40.700469+0000","last_peered":"2024-09-20T19:43:40.700469+0000","last_clean":"2024-09-20T19:43:40.700469+0000","last_became_active":"2024-09-20T19:40:44.687389+0000","last_became_peered":"2024-09-20T19:40:44.687389+0000","last_unstale":"2024-09-20T19:43:40.700469+0000","last_undegraded":"2024-09-20T19:43:40.700469+0000","last_fullsized":"2024-09-20T19:43:40.700469+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:22.309270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:22.309270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:40.515709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"77'1","reported_seq":1969,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924077+0000","last_change":"2024-09-20T19:43:25.260488+0000","last_active":"2024-09-20T19:43:40.924077+0000","last_peered":"2024-09-20T19:43:40.924077+0000","last_clean":"2024-09-20T19:43:40.924077+0000","last_became_active":"2024-09-20T19:34:26.396313+0000","last_became_peered":"2024-09-20T19:34:26.396313+0000","last_unstale":"2024-09-20T19:43:40.924077+0000","last_undegraded":"2024-09-20T19:43:40.924077+0000","last_fullsized":"2024-09-20T19:43:40.924077+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:43:25.260361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:25.260361+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:37.041510+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"987'39","reported_seq":2222,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.304890+0000","last_change":"2024-09-20T19:42:56.682400+0000","last_active":"2024-09-20T19:43:41.304890+0000","last_peered":"2024-09-20T19:43:41.304890+0000","last_clean":"2024-09-20T19:43:41.304890+0000","last_became_active":"2024-09-20T19:18:10.119578+0000","last_became_peered":"2024-09-20T19:18:10.119578+0000","last_unstale":"2024-09-20T19:43:41.304890+0000","last_undegraded":"2024-09-20T19:43:41.304890+0000","last_fullsized":"2024-09-20T19:43:41.304890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:56.682198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:56.682198+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:09.143416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":756,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701312+0000","last_change":"2024-09-20T19:42:59.517014+0000","last_active":"2024-09-20T19:43:40.701312+0000","last_peered":"2024-09-20T19:43:40.701312+0000","last_clean":"2024-09-20T19:43:40.701312+0000","last_became_active":"2024-09-20T19:40:44.686132+0000","last_became_peered":"2024-09-20T19:40:44.686132+0000","last_unstale":"2024-09-20T19:43:40.701312+0000","last_undegraded":"2024-09-20T19:43:40.701312+0000","last_fullsized":"2024-09-20T19:43:40.701312+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:42:59.516859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.516859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:04.492595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"523'13","reported_seq":2469,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547222+0000","last_change":"2024-09-20T19:43:31.089047+0000","last_active":"2024-09-20T19:43:48.547222+0000","last_peered":"2024-09-20T19:43:48.547222+0000","last_clean":"2024-09-20T19:43:48.547222+0000","last_became_active":"2024-09-20T19:30:31.385236+0000","last_became_peered":"2024-09-20T19:30:31.385236+0000","last_unstale":"2024-09-20T19:43:48.547222+0000","last_undegraded":"2024-09-20T19:43:48.547222+0000","last_fullsized":"2024-09-20T19:43:48.547222+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:43:31.088914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:31.088914+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:38.119752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2156,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.305692+0000","last_change":"2024-09-20T19:43:20.685241+0000","last_active":"2024-09-20T19:43:41.305692+0000","last_peered":"2024-09-20T19:43:41.305692+0000","last_clean":"2024-09-20T19:43:41.305692+0000","last_became_active":"2024-09-20T19:30:09.772644+0000","last_became_peered":"2024-09-20T19:30:09.772644+0000","last_unstale":"2024-09-20T19:43:41.305692+0000","last_undegraded":"2024-09-20T19:43:41.305692+0000","last_fullsized":"2024-09-20T19:43:41.305692+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.685112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.685112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.777487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"987'24","reported_seq":2062,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699926+0000","last_change":"2024-09-20T19:43:26.719370+0000","last_active":"2024-09-20T19:43:40.699926+0000","last_peered":"2024-09-20T19:43:40.699926+0000","last_clean":"2024-09-20T19:43:40.699926+0000","last_became_active":"2024-09-20T19:40:44.723579+0000","last_became_peered":"2024-09-20T19:40:44.723579+0000","last_unstale":"2024-09-20T19:43:40.699926+0000","last_undegraded":"2024-09-20T19:43:40.699926+0000","last_fullsized":"2024-09-20T19:43:40.699926+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'24","last_scrub_stamp":"2024-09-20T19:43:26.719227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:26.719227+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:34.124353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":2037,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:53.536429+0000","last_change":"2024-09-20T19:43:53.536429+0000","last_active":"2024-09-20T19:43:53.536429+0000","last_peered":"2024-09-20T19:43:53.536429+0000","last_clean":"2024-09-20T19:43:53.536429+0000","last_became_active":"2024-09-20T19:30:31.374964+0000","last_became_peered":"2024-09-20T19:30:31.374964+0000","last_unstale":"2024-09-20T19:43:53.536429+0000","last_undegraded":"2024-09-20T19:43:53.536429+0000","last_fullsized":"2024-09-20T19:43:53.536429+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-20T19:43:53.536289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:53.536289+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:58.394392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2172,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699328+0000","last_change":"2024-09-20T19:43:23.354259+0000","last_active":"2024-09-20T19:43:40.699328+0000","last_peered":"2024-09-20T19:43:40.699328+0000","last_clean":"2024-09-20T19:43:40.699328+0000","last_became_active":"2024-09-20T19:30:09.766842+0000","last_became_peered":"2024-09-20T19:30:09.766842+0000","last_unstale":"2024-09-20T19:43:40.699328+0000","last_undegraded":"2024-09-20T19:43:40.699328+0000","last_fullsized":"2024-09-20T19:43:40.699328+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.354126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:45.164042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.230887+0000","last_change":"2024-09-20T19:42:30.185762+0000","last_active":"2024-09-20T19:43:41.230887+0000","last_peered":"2024-09-20T19:43:41.230887+0000","last_clean":"2024-09-20T19:43:41.230887+0000","last_became_active":"2024-09-20T19:40:44.706254+0000","last_became_peered":"2024-09-20T19:40:44.706254+0000","last_unstale":"2024-09-20T19:43:41.230887+0000","last_undegraded":"2024-09-20T19:43:41.230887+0000","last_fullsized":"2024-09-20T19:43:41.230887+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:30.185651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:30.185651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:53.980187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"987'21","reported_seq":1979,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928774+0000","last_change":"2024-09-20T19:42:28.260343+0000","last_active":"2024-09-20T19:43:40.928774+0000","last_peered":"2024-09-20T19:43:40.928774+0000","last_clean":"2024-09-20T19:43:40.928774+0000","last_became_active":"2024-09-20T19:40:44.716453+0000","last_became_peered":"2024-09-20T19:40:44.716453+0000","last_unstale":"2024-09-20T19:43:40.928774+0000","last_undegraded":"2024-09-20T19:43:40.928774+0000","last_fullsized":"2024-09-20T19:43:40.928774+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:28.260260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:28.260260+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:54.559454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":2204,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699741+0000","last_change":"2024-09-20T19:43:20.311874+0000","last_active":"2024-09-20T19:43:40.699741+0000","last_peered":"2024-09-20T19:43:40.699741+0000","last_clean":"2024-09-20T19:43:40.699741+0000","last_became_active":"2024-09-20T19:30:09.766314+0000","last_became_peered":"2024-09-20T19:30:09.766314+0000","last_unstale":"2024-09-20T19:43:40.699741+0000","last_undegraded":"2024-09-20T19:43:40.699741+0000","last_fullsized":"2024-09-20T19:43:40.699741+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.311486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.311486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.295045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232965+0000","last_change":"2024-09-20T19:42:36.262415+0000","last_active":"2024-09-20T19:43:41.232965+0000","last_peered":"2024-09-20T19:43:41.232965+0000","last_clean":"2024-09-20T19:43:41.232965+0000","last_became_active":"2024-09-20T19:40:44.708011+0000","last_became_peered":"2024-09-20T19:40:44.708011+0000","last_unstale":"2024-09-20T19:43:41.232965+0000","last_undegraded":"2024-09-20T19:43:41.232965+0000","last_fullsized":"2024-09-20T19:43:41.232965+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:36.262354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:36.262354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:57.124471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2160,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702120+0000","last_change":"2024-09-20T19:43:33.444532+0000","last_active":"2024-09-20T19:43:40.702120+0000","last_peered":"2024-09-20T19:43:40.702120+0000","last_clean":"2024-09-20T19:43:40.702120+0000","last_became_active":"2024-09-20T19:30:31.368145+0000","last_became_peered":"2024-09-20T19:30:31.368145+0000","last_unstale":"2024-09-20T19:43:40.702120+0000","last_undegraded":"2024-09-20T19:43:40.702120+0000","last_fullsized":"2024-09-20T19:43:40.702120+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:33.444414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.444414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.223120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":2147,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928393+0000","last_change":"2024-09-20T19:42:39.326616+0000","last_active":"2024-09-20T19:43:40.928393+0000","last_peered":"2024-09-20T19:43:40.928393+0000","last_clean":"2024-09-20T19:43:40.928393+0000","last_became_active":"2024-09-20T19:30:31.385443+0000","last_became_peered":"2024-09-20T19:30:31.385443+0000","last_unstale":"2024-09-20T19:43:40.928393+0000","last_undegraded":"2024-09-20T19:43:40.928393+0000","last_fullsized":"2024-09-20T19:43:40.928393+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:39.326492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:39.326492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:48.431680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"987'18","reported_seq":113,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.232909+0000","last_change":"2024-09-20T19:43:30.112997+0000","last_active":"2024-09-20T19:43:41.232909+0000","last_peered":"2024-09-20T19:43:41.232909+0000","last_clean":"2024-09-20T19:43:41.232909+0000","last_became_active":"2024-09-20T19:40:44.708528+0000","last_became_peered":"2024-09-20T19:40:44.708528+0000","last_unstale":"2024-09-20T19:43:41.232909+0000","last_undegraded":"2024-09-20T19:43:41.232909+0000","last_fullsized":"2024-09-20T19:43:41.232909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'18","last_scrub_stamp":"2024-09-20T19:43:30.112938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.112938+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:39.648414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2189,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.510102+0000","last_change":"2024-09-20T19:43:46.510102+0000","last_active":"2024-09-20T19:43:46.510102+0000","last_peered":"2024-09-20T19:43:46.510102+0000","last_clean":"2024-09-20T19:43:46.510102+0000","last_became_active":"2024-09-20T19:40:44.704071+0000","last_became_peered":"2024-09-20T19:40:44.704071+0000","last_unstale":"2024-09-20T19:43:46.510102+0000","last_undegraded":"2024-09-20T19:43:46.510102+0000","last_fullsized":"2024-09-20T19:43:46.510102+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:46.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.509977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:54.182173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":2141,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.869163+0000","last_change":"2024-09-20T19:43:33.983755+0000","last_active":"2024-09-20T19:43:40.869163+0000","last_peered":"2024-09-20T19:43:40.869163+0000","last_clean":"2024-09-20T19:43:40.869163+0000","last_became_active":"2024-09-20T19:30:31.375980+0000","last_became_peered":"2024-09-20T19:30:31.375980+0000","last_unstale":"2024-09-20T19:43:40.869163+0000","last_undegraded":"2024-09-20T19:43:40.869163+0000","last_fullsized":"2024-09-20T19:43:40.869163+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:33.983608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:33.983608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.942731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"987'30","reported_seq":117,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:49.263205+0000","last_change":"2024-09-20T19:43:49.263205+0000","last_active":"2024-09-20T19:43:49.263205+0000","last_peered":"2024-09-20T19:43:49.263205+0000","last_clean":"2024-09-20T19:43:49.263205+0000","last_became_active":"2024-09-20T19:40:44.707276+0000","last_became_peered":"2024-09-20T19:40:44.707276+0000","last_unstale":"2024-09-20T19:43:49.263205+0000","last_undegraded":"2024-09-20T19:43:49.263205+0000","last_fullsized":"2024-09-20T19:43:49.263205+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:43:49.263099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:49.263099+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.962796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":2190,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306188+0000","last_change":"2024-09-20T19:42:51.740070+0000","last_active":"2024-09-20T19:43:41.306188+0000","last_peered":"2024-09-20T19:43:41.306188+0000","last_clean":"2024-09-20T19:43:41.306188+0000","last_became_active":"2024-09-20T19:40:44.704133+0000","last_became_peered":"2024-09-20T19:40:44.704133+0000","last_unstale":"2024-09-20T19:43:41.306188+0000","last_undegraded":"2024-09-20T19:43:41.306188+0000","last_fullsized":"2024-09-20T19:43:41.306188+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:51.739950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:51.739950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:19.122494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"501'13","reported_seq":2482,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547218+0000","last_change":"2024-09-20T19:43:12.620052+0000","last_active":"2024-09-20T19:43:48.547218+0000","last_peered":"2024-09-20T19:43:48.547218+0000","last_clean":"2024-09-20T19:43:48.547218+0000","last_became_active":"2024-09-20T19:30:09.782393+0000","last_became_peered":"2024-09-20T19:30:09.782393+0000","last_unstale":"2024-09-20T19:43:48.547218+0000","last_undegraded":"2024-09-20T19:43:48.547218+0000","last_fullsized":"2024-09-20T19:43:48.547218+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-09-20T19:43:12.619904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:12.619904+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:35.415120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1948,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929461+0000","last_change":"2024-09-20T19:43:30.154101+0000","last_active":"2024-09-20T19:43:40.929461+0000","last_peered":"2024-09-20T19:43:40.929461+0000","last_clean":"2024-09-20T19:43:40.929461+0000","last_became_active":"2024-09-20T19:40:44.723652+0000","last_became_peered":"2024-09-20T19:40:44.723652+0000","last_unstale":"2024-09-20T19:43:40.929461+0000","last_undegraded":"2024-09-20T19:43:40.929461+0000","last_fullsized":"2024-09-20T19:43:40.929461+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.154020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.154020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.442716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"987'21","reported_seq":2246,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306630+0000","last_change":"2024-09-20T19:42:49.725705+0000","last_active":"2024-09-20T19:43:41.306630+0000","last_peered":"2024-09-20T19:43:41.306630+0000","last_clean":"2024-09-20T19:43:41.306630+0000","last_became_active":"2024-09-20T19:40:44.706335+0000","last_became_peered":"2024-09-20T19:40:44.706335+0000","last_unstale":"2024-09-20T19:43:41.306630+0000","last_undegraded":"2024-09-20T19:43:41.306630+0000","last_fullsized":"2024-09-20T19:43:41.306630+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:49.725571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:49.725571+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:19.189521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1913,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.923099+0000","last_change":"2024-09-20T19:43:20.275780+0000","last_active":"2024-09-20T19:43:40.923099+0000","last_peered":"2024-09-20T19:43:40.923099+0000","last_clean":"2024-09-20T19:43:40.923099+0000","last_became_active":"2024-09-20T19:40:44.697371+0000","last_became_peered":"2024-09-20T19:40:44.697371+0000","last_unstale":"2024-09-20T19:43:40.923099+0000","last_undegraded":"2024-09-20T19:43:40.923099+0000","last_fullsized":"2024-09-20T19:43:40.923099+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:20.275637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:20.275637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:46.317196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1066'18","reported_seq":140,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547336+0000","last_change":"2024-09-20T19:42:58.421777+0000","last_active":"2024-09-20T19:43:48.547336+0000","last_peered":"2024-09-20T19:43:48.547336+0000","last_clean":"2024-09-20T19:43:48.547336+0000","last_became_active":"2024-09-20T19:40:44.719828+0000","last_became_peered":"2024-09-20T19:40:44.719828+0000","last_unstale":"2024-09-20T19:43:48.547336+0000","last_undegraded":"2024-09-20T19:43:48.547336+0000","last_fullsized":"2024-09-20T19:43:48.547336+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'18","last_scrub_stamp":"2024-09-20T19:42:58.421638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:58.421638+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:11.597390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1991,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:52.518223+0000","last_change":"2024-09-20T19:43:52.518223+0000","last_active":"2024-09-20T19:43:52.518223+0000","last_peered":"2024-09-20T19:43:52.518223+0000","last_clean":"2024-09-20T19:43:52.518223+0000","last_became_active":"2024-09-20T19:30:31.377757+0000","last_became_peered":"2024-09-20T19:30:31.377757+0000","last_unstale":"2024-09-20T19:43:52.518223+0000","last_undegraded":"2024-09-20T19:43:52.518223+0000","last_fullsized":"2024-09-20T19:43:52.518223+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:52.518106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:52.518106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:20.024757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"987'21","reported_seq":2186,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701083+0000","last_change":"2024-09-20T19:42:21.918391+0000","last_active":"2024-09-20T19:43:40.701083+0000","last_peered":"2024-09-20T19:43:40.701083+0000","last_clean":"2024-09-20T19:43:40.701083+0000","last_became_active":"2024-09-20T19:18:09.292037+0000","last_became_peered":"2024-09-20T19:18:09.292037+0000","last_unstale":"2024-09-20T19:43:40.701083+0000","last_undegraded":"2024-09-20T19:43:40.701083+0000","last_fullsized":"2024-09-20T19:43:40.701083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'21","last_scrub_stamp":"2024-09-20T19:42:21.918244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:21.918244+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":2169,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:54.378799+0000","last_change":"2024-09-20T19:43:54.378799+0000","last_active":"2024-09-20T19:43:54.378799+0000","last_peered":"2024-09-20T19:43:54.378799+0000","last_clean":"2024-09-20T19:43:54.378799+0000","last_became_active":"2024-09-20T19:40:44.698860+0000","last_became_peered":"2024-09-20T19:40:44.698860+0000","last_unstale":"2024-09-20T19:43:54.378799+0000","last_undegraded":"2024-09-20T19:43:54.378799+0000","last_fullsized":"2024-09-20T19:43:54.378799+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:54.378671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:54.378671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:22.286281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236651+0000","last_change":"2024-09-20T19:43:41.236651+0000","last_active":"2024-09-20T19:43:41.236651+0000","last_peered":"2024-09-20T19:43:41.236651+0000","last_clean":"2024-09-20T19:43:41.236651+0000","last_became_active":"2024-09-20T19:40:44.709755+0000","last_became_peered":"2024-09-20T19:40:44.709755+0000","last_unstale":"2024-09-20T19:43:41.236651+0000","last_undegraded":"2024-09-20T19:43:41.236651+0000","last_fullsized":"2024-09-20T19:43:41.236651+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:41.236526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.236526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.780221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1908,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.924174+0000","last_change":"2024-09-20T19:43:04.233792+0000","last_active":"2024-09-20T19:43:40.924174+0000","last_peered":"2024-09-20T19:43:40.924174+0000","last_clean":"2024-09-20T19:43:40.924174+0000","last_became_active":"2024-09-20T19:30:09.763529+0000","last_became_peered":"2024-09-20T19:30:09.763529+0000","last_unstale":"2024-09-20T19:43:40.924174+0000","last_undegraded":"2024-09-20T19:43:40.924174+0000","last_fullsized":"2024-09-20T19:43:40.924174+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:04.233665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:04.233665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:25.915677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"987'27","reported_seq":2010,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700166+0000","last_change":"2024-09-20T19:43:16.756525+0000","last_active":"2024-09-20T19:43:40.700166+0000","last_peered":"2024-09-20T19:43:40.700166+0000","last_clean":"2024-09-20T19:43:40.700166+0000","last_became_active":"2024-09-20T19:30:31.363615+0000","last_became_peered":"2024-09-20T19:30:31.363615+0000","last_unstale":"2024-09-20T19:43:40.700166+0000","last_undegraded":"2024-09-20T19:43:40.700166+0000","last_fullsized":"2024-09-20T19:43:40.700166+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:43:16.756379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:16.756379+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.825382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"70'1","reported_seq":2182,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702508+0000","last_change":"2024-09-20T19:43:38.266008+0000","last_active":"2024-09-20T19:43:40.702508+0000","last_peered":"2024-09-20T19:43:40.702508+0000","last_clean":"2024-09-20T19:43:40.702508+0000","last_became_active":"2024-09-20T19:30:09.778317+0000","last_became_peered":"2024-09-20T19:30:09.778317+0000","last_unstale":"2024-09-20T19:43:40.702508+0000","last_undegraded":"2024-09-20T19:43:40.702508+0000","last_fullsized":"2024-09-20T19:43:40.702508+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:38.265791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.265791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:43.286102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":2192,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:55.486044+0000","last_change":"2024-09-20T19:43:55.486044+0000","last_active":"2024-09-20T19:43:55.486044+0000","last_peered":"2024-09-20T19:43:55.486044+0000","last_clean":"2024-09-20T19:43:55.486044+0000","last_became_active":"2024-09-20T19:40:44.713438+0000","last_became_peered":"2024-09-20T19:40:44.713438+0000","last_unstale":"2024-09-20T19:43:55.486044+0000","last_undegraded":"2024-09-20T19:43:55.486044+0000","last_fullsized":"2024-09-20T19:43:55.486044+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:55.485924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:55.485924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.001158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.233829+0000","last_change":"2024-09-20T19:43:06.312356+0000","last_active":"2024-09-20T19:43:41.233829+0000","last_peered":"2024-09-20T19:43:41.233829+0000","last_clean":"2024-09-20T19:43:41.233829+0000","last_became_active":"2024-09-20T19:40:44.712828+0000","last_became_peered":"2024-09-20T19:40:44.712828+0000","last_unstale":"2024-09-20T19:43:41.233829+0000","last_undegraded":"2024-09-20T19:43:41.233829+0000","last_fullsized":"2024-09-20T19:43:41.233829+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:06.312231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:06.312231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:16.899539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"987'12","reported_seq":103,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:46.242601+0000","last_change":"2024-09-20T19:43:46.242601+0000","last_active":"2024-09-20T19:43:46.242601+0000","last_peered":"2024-09-20T19:43:46.242601+0000","last_clean":"2024-09-20T19:43:46.242601+0000","last_became_active":"2024-09-20T19:40:44.719536+0000","last_became_peered":"2024-09-20T19:40:44.719536+0000","last_unstale":"2024-09-20T19:43:46.242601+0000","last_undegraded":"2024-09-20T19:43:46.242601+0000","last_fullsized":"2024-09-20T19:43:46.242601+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12","last_scrub_stamp":"2024-09-20T19:43:46.242427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:46.242427+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:00.373429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":2162,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.306972+0000","last_change":"2024-09-20T19:42:40.689145+0000","last_active":"2024-09-20T19:43:41.306972+0000","last_peered":"2024-09-20T19:43:41.306972+0000","last_clean":"2024-09-20T19:43:41.306972+0000","last_became_active":"2024-09-20T19:34:26.399211+0000","last_became_peered":"2024-09-20T19:34:26.399211+0000","last_unstale":"2024-09-20T19:43:41.306972+0000","last_undegraded":"2024-09-20T19:43:41.306972+0000","last_fullsized":"2024-09-20T19:43:41.306972+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:40.688957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:42:40.688957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:05.929470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2155,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:47.387887+0000","last_change":"2024-09-20T19:43:47.387769+0000","last_active":"2024-09-20T19:43:47.387887+0000","last_peered":"2024-09-20T19:43:47.387887+0000","last_clean":"2024-09-20T19:43:47.387887+0000","last_became_active":"2024-09-20T19:30:31.385934+0000","last_became_peered":"2024-09-20T19:30:31.385934+0000","last_unstale":"2024-09-20T19:43:47.387887+0000","last_undegraded":"2024-09-20T19:43:47.387887+0000","last_fullsized":"2024-09-20T19:43:47.387887+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:47.387647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:47.387647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:15.434317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":2126,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:52.915109+0000","last_change":"2024-09-20T19:43:52.914984+0000","last_active":"2024-09-20T19:43:52.915109+0000","last_peered":"2024-09-20T19:43:52.915109+0000","last_clean":"2024-09-20T19:43:52.915109+0000","last_became_active":"2024-09-20T19:30:31.352247+0000","last_became_peered":"2024-09-20T19:30:31.352247+0000","last_unstale":"2024-09-20T19:43:52.915109+0000","last_undegraded":"2024-09-20T19:43:52.915109+0000","last_fullsized":"2024-09-20T19:43:52.915109+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:52.914859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:52.914859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:02.466634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"987'27","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234145+0000","last_change":"2024-09-20T19:42:57.421725+0000","last_active":"2024-09-20T19:43:41.234145+0000","last_peered":"2024-09-20T19:43:41.234145+0000","last_clean":"2024-09-20T19:43:41.234145+0000","last_became_active":"2024-09-20T19:40:44.705478+0000","last_became_peered":"2024-09-20T19:40:44.705478+0000","last_unstale":"2024-09-20T19:43:41.234145+0000","last_undegraded":"2024-09-20T19:43:41.234145+0000","last_fullsized":"2024-09-20T19:43:41.234145+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:42:57.421643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:57.421643+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:17.252992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":2159,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:55.337403+0000","last_change":"2024-09-20T19:43:55.337403+0000","last_active":"2024-09-20T19:43:55.337403+0000","last_peered":"2024-09-20T19:43:55.337403+0000","last_clean":"2024-09-20T19:43:55.337403+0000","last_became_active":"2024-09-20T19:18:07.266782+0000","last_became_peered":"2024-09-20T19:18:07.266782+0000","last_unstale":"2024-09-20T19:43:55.337403+0000","last_undegraded":"2024-09-20T19:43:55.337403+0000","last_fullsized":"2024-09-20T19:43:55.337403+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:55.337280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:55.337280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:20.307816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":2154,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.868227+0000","last_change":"2024-09-20T19:43:13.264378+0000","last_active":"2024-09-20T19:43:40.868227+0000","last_peered":"2024-09-20T19:43:40.868227+0000","last_clean":"2024-09-20T19:43:40.868227+0000","last_became_active":"2024-09-20T19:30:31.357680+0000","last_became_peered":"2024-09-20T19:30:31.357680+0000","last_unstale":"2024-09-20T19:43:40.868227+0000","last_undegraded":"2024-09-20T19:43:40.868227+0000","last_fullsized":"2024-09-20T19:43:40.868227+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:13.264248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.264248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:28.002866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234515+0000","last_change":"2024-09-20T19:43:28.142134+0000","last_active":"2024-09-20T19:43:41.234515+0000","last_peered":"2024-09-20T19:43:41.234515+0000","last_clean":"2024-09-20T19:43:41.234515+0000","last_became_active":"2024-09-20T19:40:44.719130+0000","last_became_peered":"2024-09-20T19:40:44.719130+0000","last_unstale":"2024-09-20T19:43:41.234515+0000","last_undegraded":"2024-09-20T19:43:41.234515+0000","last_fullsized":"2024-09-20T19:43:41.234515+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:28.141976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:28.141976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:49.492524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":2132,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:56.912497+0000","last_change":"2024-09-20T19:43:56.912497+0000","last_active":"2024-09-20T19:43:56.912497+0000","last_peered":"2024-09-20T19:43:56.912497+0000","last_clean":"2024-09-20T19:43:56.912497+0000","last_became_active":"2024-09-20T19:18:13.369554+0000","last_became_peered":"2024-09-20T19:18:13.369554+0000","last_unstale":"2024-09-20T19:43:56.912497+0000","last_undegraded":"2024-09-20T19:43:56.912497+0000","last_fullsized":"2024-09-20T19:43:56.912497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-20T19:43:56.912371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:56.912371+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:59.005048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":2167,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.512212+0000","last_change":"2024-09-20T19:43:41.512212+0000","last_active":"2024-09-20T19:43:41.512212+0000","last_peered":"2024-09-20T19:43:41.512212+0000","last_clean":"2024-09-20T19:43:41.512212+0000","last_became_active":"2024-09-20T19:30:31.349313+0000","last_became_peered":"2024-09-20T19:30:31.349313+0000","last_unstale":"2024-09-20T19:43:41.512212+0000","last_undegraded":"2024-09-20T19:43:41.512212+0000","last_fullsized":"2024-09-20T19:43:41.512212+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:41.512085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:41.512085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:47.954600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"987'30","reported_seq":1995,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.928969+0000","last_change":"2024-09-20T19:42:47.298115+0000","last_active":"2024-09-20T19:43:40.928969+0000","last_peered":"2024-09-20T19:43:40.928969+0000","last_clean":"2024-09-20T19:43:40.928969+0000","last_became_active":"2024-09-20T19:40:44.723592+0000","last_became_peered":"2024-09-20T19:40:44.723592+0000","last_unstale":"2024-09-20T19:43:40.928969+0000","last_undegraded":"2024-09-20T19:43:40.928969+0000","last_fullsized":"2024-09-20T19:43:40.928969+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'30","last_scrub_stamp":"2024-09-20T19:42:47.297946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:47.297946+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:14.910790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"523'13","reported_seq":2474,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:48.547172+0000","last_change":"2024-09-20T19:42:27.414822+0000","last_active":"2024-09-20T19:43:48.547172+0000","last_peered":"2024-09-20T19:43:48.547172+0000","last_clean":"2024-09-20T19:43:48.547172+0000","last_became_active":"2024-09-20T19:30:31.368792+0000","last_became_peered":"2024-09-20T19:30:31.368792+0000","last_unstale":"2024-09-20T19:43:48.547172+0000","last_undegraded":"2024-09-20T19:43:48.547172+0000","last_fullsized":"2024-09-20T19:43:48.547172+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'13","last_scrub_stamp":"2024-09-20T19:42:27.414643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:27.414643+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":2154,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701959+0000","last_change":"2024-09-20T19:43:30.172883+0000","last_active":"2024-09-20T19:43:40.701959+0000","last_peered":"2024-09-20T19:43:40.701959+0000","last_clean":"2024-09-20T19:43:40.701959+0000","last_became_active":"2024-09-20T19:30:31.346238+0000","last_became_peered":"2024-09-20T19:30:31.346238+0000","last_unstale":"2024-09-20T19:43:40.701959+0000","last_undegraded":"2024-09-20T19:43:40.701959+0000","last_fullsized":"2024-09-20T19:43:40.701959+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:30.172835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:30.172835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:56.998338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"70'1","reported_seq":101,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.234003+0000","last_change":"2024-09-20T19:43:13.196425+0000","last_active":"2024-09-20T19:43:41.234003+0000","last_peered":"2024-09-20T19:43:41.234003+0000","last_clean":"2024-09-20T19:43:41.234003+0000","last_became_active":"2024-09-20T19:40:44.718639+0000","last_became_peered":"2024-09-20T19:40:44.718639+0000","last_unstale":"2024-09-20T19:43:41.234003+0000","last_undegraded":"2024-09-20T19:43:41.234003+0000","last_fullsized":"2024-09-20T19:43:41.234003+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-20T19:43:13.196296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.196296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:41.357746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"987'27","reported_seq":2214,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.929971+0000","last_change":"2024-09-20T19:43:13.502075+0000","last_active":"2024-09-20T19:43:40.929971+0000","last_peered":"2024-09-20T19:43:40.929971+0000","last_clean":"2024-09-20T19:43:40.929971+0000","last_became_active":"2024-09-20T19:19:57.010844+0000","last_became_peered":"2024-09-20T19:19:57.010844+0000","last_unstale":"2024-09-20T19:43:40.929971+0000","last_undegraded":"2024-09-20T19:43:40.929971+0000","last_fullsized":"2024-09-20T19:43:40.929971+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'27","last_scrub_stamp":"2024-09-20T19:43:13.501926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:13.501926+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:15.519399+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1954,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:51.109439+0000","last_change":"2024-09-20T19:43:51.109439+0000","last_active":"2024-09-20T19:43:51.109439+0000","last_peered":"2024-09-20T19:43:51.109439+0000","last_clean":"2024-09-20T19:43:51.109439+0000","last_became_active":"2024-09-20T19:34:26.392375+0000","last_became_peered":"2024-09-20T19:34:26.392375+0000","last_unstale":"2024-09-20T19:43:51.109439+0000","last_undegraded":"2024-09-20T19:43:51.109439+0000","last_fullsized":"2024-09-20T19:43:51.109439+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:51.109321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:51.109321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:07.418541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1973,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.701327+0000","last_change":"2024-09-20T19:43:38.688743+0000","last_active":"2024-09-20T19:43:40.701327+0000","last_peered":"2024-09-20T19:43:40.701327+0000","last_clean":"2024-09-20T19:43:40.701327+0000","last_became_active":"2024-09-20T19:30:31.366085+0000","last_became_peered":"2024-09-20T19:30:31.366085+0000","last_unstale":"2024-09-20T19:43:40.701327+0000","last_undegraded":"2024-09-20T19:43:40.701327+0000","last_fullsized":"2024-09-20T19:43:40.701327+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:38.688527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:43:38.688527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:44.325502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":2170,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:52.400480+0000","last_change":"2024-09-20T19:43:52.400480+0000","last_active":"2024-09-20T19:43:52.400480+0000","last_peered":"2024-09-20T19:43:52.400480+0000","last_clean":"2024-09-20T19:43:52.400480+0000","last_became_active":"2024-09-20T19:30:31.357260+0000","last_became_peered":"2024-09-20T19:30:31.357260+0000","last_unstale":"2024-09-20T19:43:52.400480+0000","last_undegraded":"2024-09-20T19:43:52.400480+0000","last_fullsized":"2024-09-20T19:43:52.400480+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:52.400331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:52.400331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:05.316735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"987'39","reported_seq":797,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.699700+0000","last_change":"2024-09-20T19:42:40.572544+0000","last_active":"2024-09-20T19:43:40.699700+0000","last_peered":"2024-09-20T19:43:40.699700+0000","last_clean":"2024-09-20T19:43:40.699700+0000","last_became_active":"2024-09-20T19:40:44.698792+0000","last_became_peered":"2024-09-20T19:40:44.698792+0000","last_unstale":"2024-09-20T19:43:40.699700+0000","last_undegraded":"2024-09-20T19:43:40.699700+0000","last_fullsized":"2024-09-20T19:43:40.699700+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'39","last_scrub_stamp":"2024-09-20T19:42:40.572392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:42:40.572392+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:08.670820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":2173,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.702325+0000","last_change":"2024-09-20T19:42:59.068334+0000","last_active":"2024-09-20T19:43:40.702325+0000","last_peered":"2024-09-20T19:43:40.702325+0000","last_clean":"2024-09-20T19:43:40.702325+0000","last_became_active":"2024-09-20T19:30:31.365801+0000","last_became_peered":"2024-09-20T19:30:31.365801+0000","last_unstale":"2024-09-20T19:43:40.702325+0000","last_undegraded":"2024-09-20T19:43:40.702325+0000","last_fullsized":"2024-09-20T19:43:40.702325+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:59.068277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:42:59.068277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:27.700430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":97,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.236126+0000","last_change":"2024-09-20T19:42:45.332595+0000","last_active":"2024-09-20T19:43:41.236126+0000","last_peered":"2024-09-20T19:43:41.236126+0000","last_clean":"2024-09-20T19:43:41.236126+0000","last_became_active":"2024-09-20T19:40:44.712077+0000","last_became_peered":"2024-09-20T19:40:44.712077+0000","last_unstale":"2024-09-20T19:43:41.236126+0000","last_undegraded":"2024-09-20T19:43:41.236126+0000","last_fullsized":"2024-09-20T19:43:41.236126+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:42:45.332470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:12.302527+0000","last_clean_scrub_stamp":"2024-09-20T19:42:45.332470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:43:59.348624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":105,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:55.265889+0000","last_change":"2024-09-20T19:43:55.265889+0000","last_active":"2024-09-20T19:43:55.265889+0000","last_peered":"2024-09-20T19:43:55.265889+0000","last_clean":"2024-09-20T19:43:55.265889+0000","last_became_active":"2024-09-20T19:40:44.718784+0000","last_became_peered":"2024-09-20T19:40:44.718784+0000","last_unstale":"2024-09-20T19:43:55.265889+0000","last_undegraded":"2024-09-20T19:43:55.265889+0000","last_fullsized":"2024-09-20T19:43:55.265889+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:55.265764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:06.213049+0000","last_clean_scrub_stamp":"2024-09-20T19:43:55.265764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:45:22.396176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"987'24","reported_seq":109,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:41.231810+0000","last_change":"2024-09-20T19:43:04.358557+0000","last_active":"2024-09-20T19:43:41.231810+0000","last_peered":"2024-09-20T19:43:41.231810+0000","last_clean":"2024-09-20T19:43:41.231810+0000","last_became_active":"2024-09-20T19:40:44.706561+0000","last_became_peered":"2024-09-20T19:40:44.706561+0000","last_unstale":"2024-09-20T19:43:41.231810+0000","last_undegraded":"2024-09-20T19:43:41.231810+0000","last_fullsized":"2024-09-20T19:43:41.231810+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'24","last_scrub_stamp":"2024-09-20T19:43:04.358417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:08.237223+0000","last_clean_scrub_stamp":"2024-09-20T19:43:04.358417+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:27.396306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1993,"reported_epoch":1101,"state":"active+clean","last_fresh":"2024-09-20T19:43:40.700675+0000","last_change":"2024-09-20T19:43:23.682000+0000","last_active":"2024-09-20T19:43:40.700675+0000","last_peered":"2024-09-20T19:43:40.700675+0000","last_clean":"2024-09-20T19:43:40.700675+0000","last_became_active":"2024-09-20T19:40:44.709235+0000","last_became_peered":"2024-09-20T19:40:44.709235+0000","last_unstale":"2024-09-20T19:43:40.700675+0000","last_undegraded":"2024-09-20T19:43:40.700675+0000","last_fullsized":"2024-09-20T19:43:40.700675+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:43:23.681855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:18:10.268062+0000","last_clean_scrub_stamp":"2024-09-20T19:43:23.681855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T19:44:26.131069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1528,"num_read_kb":1285,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":4786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":973,"ondisk_log_size":973,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1530,"num_read_kb":1525,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":64,"seq":274877907264,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72532,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672620,"statfs":{"total":95995035648,"available":95920762880,"internally_reserved":0,"allocated":1118208,"data_stored":734393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":54,"seq":231928234315,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76360,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668792,"statfs":{"total":95995035648,"available":95916843008,"internally_reserved":0,"allocated":778240,"data_stored":406049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561365,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76948,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668204,"statfs":{"total":95995035648,"available":95916240896,"internally_reserved":0,"allocated":1380352,"data_stored":997618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":522,"seq":2241972928675,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59156,"kb_used_data":772,"kb_used_omap":35,"kb_used_meta":58332,"kb_avail":93685996,"statfs":{"total":95995035648,"available":95934459904,"internally_reserved":0,"allocated":790528,"data_stored":406127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35919,"internal_metadata":59732913},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":550,"seq":2362232012958,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29932,"kb_used_data":1372,"kb_used_omap":51,"kb_used_meta":28492,"kb_avail":93715220,"statfs":{"total":95995035648,"available":95964385280,"internally_reserved":0,"allocated":1404928,"data_stored":997282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52777,"internal_metadata":29176279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379001,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80792,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664360,"statfs":{"total":95995035648,"available":95912304640,"internally_reserved":0,"allocated":1056768,"data_stored":669682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575237,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72216,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672936,"statfs":{"total":95995035648,"available":95921086464,"internally_reserved":0,"allocated":794624,"data_stored":407401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":500,"seq":2147483648167,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56092,"kb_used_data":780,"kb_used_omap":49,"kb_used_meta":55246,"kb_avail":93689060,"statfs":{"total":95995035648,"available":95937597440,"internally_reserved":0,"allocated":798720,"data_stored":407089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50647,"internal_metadata":56572457},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:43:59.116 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T19:43:59.116 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:88deadb8082f4a33832e228ccefc59cedba158a5 shell --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:43:59.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[32075]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:43:59.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:43:59.933 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:43:59 smithi042 bash[22706]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.155523+0000 osd.3 (osd.3) 169 : cluster [DBG] 2.1e scrub starts 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.157229+0000 osd.3 (osd.3) 170 : cluster [DBG] 2.1e scrub ok 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.249344+0000 osd.0 (osd.0) 251 : cluster [DBG] 4.16 scrub starts 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.253389+0000 osd.0 (osd.0) 252 : cluster [DBG] 4.16 scrub ok 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:44:00.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: audit 2024-09-20T19:43:58.324326+0000 mgr.y (mgr.24491) 969 : audit [DBG] from='client.16668 v1:172.21.15.42:0/1838797894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:44:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: cluster 2024-09-20T19:43:58.645311+0000 mgr.y (mgr.24491) 970 : cluster [DBG] pgmap v1765: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:44:00.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:43:59 smithi162 bash[24557]: audit 2024-09-20T19:43:59.242244+0000 mon.a (mon.0) 2876 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:44:00.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:44:00.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:00.962 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[22706]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:00.963 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:00.964 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:00.964 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:00.964 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:00 smithi042 bash[32075]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.148027+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.b scrub starts 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.150259+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.b scrub ok 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.553824+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.1a scrub starts 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:01.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:00 smithi162 bash[24557]: cluster 2024-09-20T19:43:59.556394+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.1a scrub ok 2024-09-20T19:44:01.930 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[22706]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:01.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:01.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:01 smithi042 bash[32075]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:02.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: audit 2024-09-20T19:44:00.246896+0000 mgr.y (mgr.24491) 971 : audit [DBG] from='client.24487 v1:172.21.15.162:0/1676211549' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.646355+0000 mgr.y (mgr.24491) 972 : cluster [DBG] pgmap v1766: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.916382+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.0 scrub starts 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:02.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:01 smithi162 bash[24557]: cluster 2024-09-20T19:44:00.919040+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.0 scrub ok 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:02.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:02.932 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:02 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:02 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:03.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:02 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.390922+0000 osd.1 (osd.1) 505 : cluster [DBG] 4.f scrub starts 2024-09-20T19:44:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:02 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:03.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:02 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.395922+0000 osd.1 (osd.1) 506 : cluster [DBG] 4.f scrub ok 2024-09-20T19:44:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:03.930 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[32075]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:03.931 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:03.932 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:03 smithi042 bash[22706]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.647775+0000 mgr.y (mgr.24491) 973 : cluster [DBG] pgmap v1767: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.681112+0000 osd.2 (osd.2) 441 : cluster [DBG] 5.d scrub starts 2024-09-20T19:44:04.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:04.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:02.684222+0000 osd.2 (osd.2) 442 : cluster [DBG] 5.d scrub ok 2024-09-20T19:44:04.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:04.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:03.220421+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-09-20T19:44:04.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:04.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:03 smithi162 bash[24557]: cluster 2024-09-20T19:44:03.222701+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-09-20T19:44:05.768 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE Bus STOPPING 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:05.769 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:05.770 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:05.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:05.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:05.771 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.331289+0000 osd.0 (osd.0) 253 : cluster [DBG] 5.5 scrub starts 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.333595+0000 osd.0 (osd.0) 254 : cluster [DBG] 5.5 scrub ok 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.602620+0000 osd.4 (osd.4) 339 : cluster [DBG] 4.c scrub starts 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.606441+0000 osd.4 (osd.4) 340 : cluster [DBG] 4.c scrub ok 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:04.648733+0000 mgr.y (mgr.24491) 974 : cluster [DBG] pgmap v1768: 142 pgs: 142 active+clean; 583 KiB data, 512 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:44:06.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:06.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.082829+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-20T19:44:06.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:06.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:05 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.087082+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-20T19:44:06.180 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:44:06.181 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE Bus STOPPED 2024-09-20T19:44:06.181 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE Bus STARTING 2024-09-20T19:44:06.181 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE Serving on http://:::9283 2024-09-20T19:44:06.181 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:05 smithi042 bash[22958]: [20/Sep/2024:19:44:05] ENGINE Bus STARTED 2024-09-20T19:44:06.527 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22958]: ::ffff:172.21.15.162 - - [20/Sep/2024:19:44:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-20T19:44:06.528 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T19:44:06.538 INFO:tasks.cephadm:Teardown begin 2024-09-20T19:44:06.538 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:44:06.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:44:06.567 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T19:44:06.567 DEBUG:teuthology.orchestra.run.smithi042:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:44:06.603 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:44:06.614 INFO:tasks.cephadm:Stopping all daemons... 2024-09-20T19:44:06.614 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-20T19:44:06.614 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:06.798 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:06.799 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:06.800 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:06.800 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:06.800 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[32075]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:06.800 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 systemd[1]: Stopping Ceph mon.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:06.800 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:06.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:06.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:06.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:06.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:06.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:06.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:06.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:06.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:06.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:06.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: debug 2024-09-20T19:44:06.790+0000 7fc2414ef640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-20T19:44:06.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22706]: debug 2024-09-20T19:44:06.790+0000 7fc2414ef640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T19:44:06.931 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:06 smithi042 bash[22958]: [20/Sep/2024:19:44:06] ENGINE Bus STOPPING 2024-09-20T19:44:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.226836+0000 osd.3 (osd.3) 173 : cluster [DBG] 5.17 scrub starts 2024-09-20T19:44:07.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.229298+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.17 scrub ok 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:05.688945+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e1103: 8 total, 8 up, 8 in 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.710859+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.42:0/624790637' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.711470+0000 mon.a (mon.0) 2878 : audit [INF] from='client.26519 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.717231+0000 mon.a (mon.0) 2879 : audit [INF] from='client.26519 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.734612+0000 mon.a (mon.0) 2880 : audit [DBG] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:07.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.768970+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.24491 v1:172.21.15.42:0/2440953934' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: audit 2024-09-20T19:44:05.769683+0000 mgr.y (mgr.24491) 975 : audit [DBG] from='mon.0 v1:172.21.15.42:0/2063231720' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:06.249070+0000 osd.6 (osd.6) 354 : cluster [DBG] 4.a scrub starts 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:07.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:06 smithi162 bash[24557]: cluster 2024-09-20T19:44:06.253612+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub ok 2024-09-20T19:44:07.364 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:07 smithi042 bash[22958]: [20/Sep/2024:19:44:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:44:07.364 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:07 smithi042 bash[22958]: [20/Sep/2024:19:44:07] ENGINE Bus STOPPED 2024-09-20T19:44:07.364 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:07 smithi042 bash[22958]: [20/Sep/2024:19:44:07] ENGINE Bus STARTING 2024-09-20T19:44:07.660 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:07 smithi042 bash[22958]: [20/Sep/2024:19:44:07] ENGINE Serving on http://:::9283 2024-09-20T19:44:07.661 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:07 smithi042 bash[22958]: [20/Sep/2024:19:44:07] ENGINE Bus STARTED 2024-09-20T19:44:07.877 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.a.service' 2024-09-20T19:44:07.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:07.904 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-20T19:44:07.904 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-20T19:44:07.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.c 2024-09-20T19:44:08.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:07 smithi042 systemd[1]: Stopping Ceph mon.c for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:08.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:08 smithi042 bash[32075]: debug 2024-09-20T19:44:08.074+0000 7fca2cb0c640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-20T19:44:08.181 INFO:journalctl@ceph.mon.c.smithi042.stdout:Sep 20 19:44:08 smithi042 bash[32075]: debug 2024-09-20T19:44:08.074+0000 7fca2cb0c640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:44:09.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.c.service' 2024-09-20T19:44:09.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:09.331 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-20T19:44:09.331 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-20T19:44:09.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.b 2024-09-20T19:44:09.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:09 smithi162 systemd[1]: Stopping Ceph mon.b for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:09.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:09 smithi162 bash[24557]: debug 2024-09-20T19:44:09.485+0000 7fa096f76640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-20T19:44:09.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 20 19:44:09 smithi162 bash[24557]: debug 2024-09-20T19:44:09.485+0000 7fa096f76640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:44:10.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mon.b.service' 2024-09-20T19:44:10.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:10.483 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-20T19:44:10.484 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-20T19:44:10.484 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y 2024-09-20T19:44:10.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:44:10 smithi162 bash[48936]: debug there is no tcmu-runner data available 2024-09-20T19:44:10.822 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:10 smithi042 systemd[1]: Stopping Ceph mgr.y for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:10.822 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:10 smithi042 bash[22958]: debug 2024-09-20T19:44:10.610+0000 7f1763322640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:10.822 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:10 smithi042 bash[22958]: debug 2024-09-20T19:44:10.610+0000 7f1763322640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-20T19:44:11.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-20T19:44:11.590 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:11 smithi042 bash[163746]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-mgr-y 2024-09-20T19:44:11.590 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:11 smithi042 bash[163808]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-mgr-y 2024-09-20T19:44:11.590 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:11 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.y.service: Deactivated successfully. 2024-09-20T19:44:11.590 INFO:journalctl@ceph.mgr.y.smithi042.stdout:Sep 20 19:44:11 smithi042 systemd[1]: Stopped Ceph mgr.y for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:44:11.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:11.606 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-20T19:44:11.606 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-20T19:44:11.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.x 2024-09-20T19:44:11.904 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 20 19:44:11 smithi162 systemd[1]: Stopping Ceph mgr.x for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:12.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-20T19:44:12.550 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:12.551 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-20T19:44:12.551 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T19:44:12.551 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0 2024-09-20T19:44:12.931 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:12 smithi042 systemd[1]: Stopping Ceph osd.0 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:12.931 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:12 smithi042 bash[109938]: debug 2024-09-20T19:44:12.778+0000 7fdceecb8640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:12.931 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:12 smithi042 bash[109938]: debug 2024-09-20T19:44:12.778+0000 7fdceecb8640 -1 osd.0 1103 *** Got signal Terminated *** 2024-09-20T19:44:12.931 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:12 smithi042 bash[109938]: debug 2024-09-20T19:44:12.778+0000 7fdceecb8640 -1 osd.0 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:19.036 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:18 smithi042 bash[163858]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:44:19.036 INFO:journalctl@ceph.osd.0.smithi042.stdout:Sep 20 19:44:18 smithi042 bash[166131]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:44:21.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T19:44:21.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:21.184 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T19:44:21.184 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-20T19:44:21.185 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.1 2024-09-20T19:44:21.431 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:21 smithi042 systemd[1]: Stopping Ceph osd.1 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:21.431 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:21 smithi042 bash[40690]: debug 2024-09-20T19:44:21.342+0000 7f207724f640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:21.431 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:21 smithi042 bash[40690]: debug 2024-09-20T19:44:21.342+0000 7f207724f640 -1 osd.1 1103 *** Got signal Terminated *** 2024-09-20T19:44:21.431 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:21 smithi042 bash[40690]: debug 2024-09-20T19:44:21.342+0000 7f207724f640 -1 osd.1 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:27.801 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:27 smithi042 bash[166260]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-1 2024-09-20T19:44:28.180 INFO:journalctl@ceph.osd.1.smithi042.stdout:Sep 20 19:44:27 smithi042 bash[166324]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-1 2024-09-20T19:44:29.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.1.service' 2024-09-20T19:44:30.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:30.026 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-20T19:44:30.026 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-20T19:44:30.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.2 2024-09-20T19:44:30.431 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:30 smithi042 systemd[1]: Stopping Ceph osd.2 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:30.431 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:30 smithi042 bash[46409]: debug 2024-09-20T19:44:30.186+0000 7f06ffa9f640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:30.431 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:30 smithi042 bash[46409]: debug 2024-09-20T19:44:30.186+0000 7f06ffa9f640 -1 osd.2 1103 *** Got signal Terminated *** 2024-09-20T19:44:30.431 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:30 smithi042 bash[46409]: debug 2024-09-20T19:44:30.186+0000 7f06ffa9f640 -1 osd.2 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:36.568 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:36 smithi042 bash[166458]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-2 2024-09-20T19:44:36.568 INFO:journalctl@ceph.osd.2.smithi042.stdout:Sep 20 19:44:36 smithi042 bash[166520]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-2 2024-09-20T19:44:38.574 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.2.service' 2024-09-20T19:44:38.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:38.603 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-20T19:44:38.603 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T19:44:38.603 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3 2024-09-20T19:44:38.931 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:38 smithi042 systemd[1]: Stopping Ceph osd.3 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:38.931 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:38 smithi042 bash[114337]: debug 2024-09-20T19:44:38.778+0000 7f678763a640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:38.931 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:38 smithi042 bash[114337]: debug 2024-09-20T19:44:38.778+0000 7f678763a640 -1 osd.3 1103 *** Got signal Terminated *** 2024-09-20T19:44:38.931 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:38 smithi042 bash[114337]: debug 2024-09-20T19:44:38.778+0000 7f678763a640 -1 osd.3 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:39.680 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:39 smithi042 bash[114337]: debug 2024-09-20T19:44:39.306+0000 7f6783c4f640 -1 osd.3 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.670975+0000 front 2024-09-20T19:44:13.671051+0000 (oldest deadline 2024-09-20T19:44:38.371188+0000) 2024-09-20T19:44:39.767 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:39 smithi162 bash[39432]: debug 2024-09-20T19:44:39.312+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:40.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:39 smithi162 bash[45241]: debug 2024-09-20T19:44:39.764+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:40.680 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:40 smithi042 bash[114337]: debug 2024-09-20T19:44:40.350+0000 7f6783c4f640 -1 osd.3 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.670975+0000 front 2024-09-20T19:44:13.671051+0000 (oldest deadline 2024-09-20T19:44:38.371188+0000) 2024-09-20T19:44:40.746 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:40 smithi162 bash[39432]: debug 2024-09-20T19:44:40.356+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:41.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:40 smithi162 bash[45241]: debug 2024-09-20T19:44:40.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:41.558 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:41 smithi162 bash[39432]: debug 2024-09-20T19:44:41.376+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:41.559 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:41 smithi162 bash[77847]: debug 2024-09-20T19:44:41.372+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:41.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:41 smithi162 bash[33802]: debug 2024-09-20T19:44:41.168+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:41.680 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:41 smithi042 bash[114337]: debug 2024-09-20T19:44:41.306+0000 7f6783c4f640 -1 osd.3 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.670975+0000 front 2024-09-20T19:44:13.671051+0000 (oldest deadline 2024-09-20T19:44:38.371188+0000) 2024-09-20T19:44:42.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:41 smithi162 bash[45241]: debug 2024-09-20T19:44:41.780+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:42.431 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:42 smithi162 bash[77847]: debug 2024-09-20T19:44:42.400+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:42.432 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:42 smithi162 bash[33802]: debug 2024-09-20T19:44:42.172+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:42.681 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:42 smithi042 bash[114337]: debug 2024-09-20T19:44:42.274+0000 7f6783c4f640 -1 osd.3 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.670975+0000 front 2024-09-20T19:44:13.671051+0000 (oldest deadline 2024-09-20T19:44:38.371188+0000) 2024-09-20T19:44:42.777 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:42 smithi162 bash[39432]: debug 2024-09-20T19:44:42.428+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:43.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:42 smithi162 bash[45241]: debug 2024-09-20T19:44:42.772+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:43.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:43 smithi162 bash[77847]: debug 2024-09-20T19:44:43.432+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:43.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:43 smithi162 bash[33802]: debug 2024-09-20T19:44:43.208+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:43.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:43 smithi162 bash[39432]: debug 2024-09-20T19:44:43.404+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:43.680 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:43 smithi042 bash[114337]: debug 2024-09-20T19:44:43.254+0000 7f6783c4f640 -1 osd.3 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.670975+0000 front 2024-09-20T19:44:13.671051+0000 (oldest deadline 2024-09-20T19:44:38.371188+0000) 2024-09-20T19:44:44.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:43 smithi162 bash[45241]: debug 2024-09-20T19:44:43.748+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:44.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:44 smithi162 bash[77847]: debug 2024-09-20T19:44:44.416+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:44.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:44 smithi162 bash[39432]: debug 2024-09-20T19:44:44.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:44.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:44 smithi162 bash[33802]: debug 2024-09-20T19:44:44.208+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:45.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:44 smithi162 bash[45241]: debug 2024-09-20T19:44:44.732+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:45.430 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:44 smithi042 bash[166649]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:44:45.431 INFO:journalctl@ceph.osd.3.smithi042.stdout:Sep 20 19:44:45 smithi042 bash[166712]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:44:45.558 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:45 smithi162 bash[33802]: debug 2024-09-20T19:44:45.256+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:45.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:45 smithi162 bash[39432]: debug 2024-09-20T19:44:45.488+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:45.559 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:45 smithi162 bash[77847]: debug 2024-09-20T19:44:45.368+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:46.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:45 smithi162 bash[45241]: debug 2024-09-20T19:44:45.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:46.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:46 smithi162 bash[77847]: debug 2024-09-20T19:44:46.368+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:46.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:46 smithi162 bash[33802]: debug 2024-09-20T19:44:46.220+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:46.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:46 smithi162 bash[39432]: debug 2024-09-20T19:44:46.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:47.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:46 smithi162 bash[45241]: debug 2024-09-20T19:44:46.692+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:47.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T19:44:47.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:47.110 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T19:44:47.110 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T19:44:47.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4 2024-09-20T19:44:47.401 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[33802]: debug 2024-09-20T19:44:47.248+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:47.401 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 systemd[1]: Stopping Ceph osd.4 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:47.401 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[77847]: debug 2024-09-20T19:44:47.316+0000 7feadac40640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:47.402 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[77847]: debug 2024-09-20T19:44:47.316+0000 7feadac40640 -1 osd.4 1103 *** Got signal Terminated *** 2024-09-20T19:44:47.402 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[77847]: debug 2024-09-20T19:44:47.316+0000 7feadac40640 -1 osd.4 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:47.402 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[77847]: debug 2024-09-20T19:44:47.396+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:47.666 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[77847]: debug 2024-09-20T19:44:47.396+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:47.666 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[39432]: debug 2024-09-20T19:44:47.460+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:48.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:47 smithi162 bash[45241]: debug 2024-09-20T19:44:47.660+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:48.507 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[77847]: debug 2024-09-20T19:44:48.368+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:48.507 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[77847]: debug 2024-09-20T19:44:48.368+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:48.507 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[33802]: debug 2024-09-20T19:44:48.248+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:48.507 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[33802]: debug 2024-09-20T19:44:48.248+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:48.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[39432]: debug 2024-09-20T19:44:48.504+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:48.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[45241]: debug 2024-09-20T19:44:48.696+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:48.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:48 smithi162 bash[45241]: debug 2024-09-20T19:44:48.696+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:49.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[77847]: debug 2024-09-20T19:44:49.320+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:49.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[77847]: debug 2024-09-20T19:44:49.320+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:49.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[39432]: debug 2024-09-20T19:44:49.468+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:49.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[33802]: debug 2024-09-20T19:44:49.252+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:49.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[33802]: debug 2024-09-20T19:44:49.252+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:50.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[45241]: debug 2024-09-20T19:44:49.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:50.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:49 smithi162 bash[45241]: debug 2024-09-20T19:44:49.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:50.471 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[77847]: debug 2024-09-20T19:44:50.308+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:50.471 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[77847]: debug 2024-09-20T19:44:50.308+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:50.472 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[33802]: debug 2024-09-20T19:44:50.208+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:50.472 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[33802]: debug 2024-09-20T19:44:50.208+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:50.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[39432]: debug 2024-09-20T19:44:50.468+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:50.808 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[45241]: debug 2024-09-20T19:44:50.696+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:50.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:50 smithi162 bash[45241]: debug 2024-09-20T19:44:50.696+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:51.462 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[77847]: debug 2024-09-20T19:44:51.296+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:51.462 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[77847]: debug 2024-09-20T19:44:51.296+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:51.462 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[33802]: debug 2024-09-20T19:44:51.196+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:51.462 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[33802]: debug 2024-09-20T19:44:51.196+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:51.748 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[39432]: debug 2024-09-20T19:44:51.460+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:51.748 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[39432]: debug 2024-09-20T19:44:51.460+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:52.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[45241]: debug 2024-09-20T19:44:51.744+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:52.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:51 smithi162 bash[45241]: debug 2024-09-20T19:44:51.744+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:52.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[77847]: debug 2024-09-20T19:44:52.296+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.209656+0000 front 2024-09-20T19:44:16.209731+0000 (oldest deadline 2024-09-20T19:44:40.909426+0000) 2024-09-20T19:44:52.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[77847]: debug 2024-09-20T19:44:52.296+0000 7fead324d640 -1 osd.4 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:23.210969+0000 front 2024-09-20T19:44:23.210722+0000 (oldest deadline 2024-09-20T19:44:47.310506+0000) 2024-09-20T19:44:52.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[39432]: debug 2024-09-20T19:44:52.436+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:52.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[39432]: debug 2024-09-20T19:44:52.436+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:52.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[33802]: debug 2024-09-20T19:44:52.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:52.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[33802]: debug 2024-09-20T19:44:52.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:53.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[45241]: debug 2024-09-20T19:44:52.752+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:53.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:52 smithi162 bash[45241]: debug 2024-09-20T19:44:52.752+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:53.558 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[106134]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-4 2024-09-20T19:44:53.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[33802]: debug 2024-09-20T19:44:53.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:53.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[33802]: debug 2024-09-20T19:44:53.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:53.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[39432]: debug 2024-09-20T19:44:53.404+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:53.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[39432]: debug 2024-09-20T19:44:53.404+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:54.058 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[106199]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-4 2024-09-20T19:44:54.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[45241]: debug 2024-09-20T19:44:53.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:54.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:53 smithi162 bash[45241]: debug 2024-09-20T19:44:53.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:54.558 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[33802]: debug 2024-09-20T19:44:54.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:54.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[33802]: debug 2024-09-20T19:44:54.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:54.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[39432]: debug 2024-09-20T19:44:54.428+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:54.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[39432]: debug 2024-09-20T19:44:54.428+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:54.969 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[45241]: debug 2024-09-20T19:44:54.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:54.969 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:54 smithi162 bash[45241]: debug 2024-09-20T19:44:54.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:55.558 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[39432]: debug 2024-09-20T19:44:55.432+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:55.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[39432]: debug 2024-09-20T19:44:55.432+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:55.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[33802]: debug 2024-09-20T19:44:55.272+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:55.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[33802]: debug 2024-09-20T19:44:55.272+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:55.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.4.service' 2024-09-20T19:44:55.650 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:44:55.650 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-20T19:44:55.650 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-20T19:44:55.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.5 2024-09-20T19:44:56.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[45241]: debug 2024-09-20T19:44:55.736+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:56.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[45241]: debug 2024-09-20T19:44:55.736+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:56.059 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 systemd[1]: Stopping Ceph osd.5 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:44:56.059 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[33802]: debug 2024-09-20T19:44:55.820+0000 7fedc75a2640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:44:56.059 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[33802]: debug 2024-09-20T19:44:55.820+0000 7fedc75a2640 -1 osd.5 1103 *** Got signal Terminated *** 2024-09-20T19:44:56.059 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:55 smithi162 bash[33802]: debug 2024-09-20T19:44:55.820+0000 7fedc75a2640 -1 osd.5 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:44:56.558 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[39432]: debug 2024-09-20T19:44:56.468+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:56.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[39432]: debug 2024-09-20T19:44:56.468+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:56.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[33802]: debug 2024-09-20T19:44:56.248+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:56.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[33802]: debug 2024-09-20T19:44:56.248+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:57.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[45241]: debug 2024-09-20T19:44:56.724+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:57.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:56 smithi162 bash[45241]: debug 2024-09-20T19:44:56.724+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:57.558 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[33802]: debug 2024-09-20T19:44:57.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:57.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[33802]: debug 2024-09-20T19:44:57.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:57.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[39432]: debug 2024-09-20T19:44:57.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:57.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[39432]: debug 2024-09-20T19:44:57.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:58.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[45241]: debug 2024-09-20T19:44:57.756+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:58.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[45241]: debug 2024-09-20T19:44:57.756+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:58.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:57 smithi162 bash[45241]: debug 2024-09-20T19:44:57.756+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:44:58.506 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[33802]: debug 2024-09-20T19:44:58.224+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:58.506 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[33802]: debug 2024-09-20T19:44:58.224+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:58.808 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[45241]: debug 2024-09-20T19:44:58.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:44:58.808 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[45241]: debug 2024-09-20T19:44:58.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:44:58.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[45241]: debug 2024-09-20T19:44:58.740+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:44:58.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[39432]: debug 2024-09-20T19:44:58.500+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:58.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[39432]: debug 2024-09-20T19:44:58.500+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:58.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:58 smithi162 bash[39432]: debug 2024-09-20T19:44:58.500+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:44:59.558 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[33802]: debug 2024-09-20T19:44:59.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:44:59.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[33802]: debug 2024-09-20T19:44:59.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:44:59.559 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[33802]: debug 2024-09-20T19:44:59.244+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:33.469269+0000 front 2024-09-20T19:44:33.469499+0000 (oldest deadline 2024-09-20T19:44:58.769198+0000) 2024-09-20T19:44:59.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[39432]: debug 2024-09-20T19:44:59.472+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:44:59.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[39432]: debug 2024-09-20T19:44:59.472+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:44:59.559 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[39432]: debug 2024-09-20T19:44:59.472+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:00.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[45241]: debug 2024-09-20T19:44:59.708+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:00.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[45241]: debug 2024-09-20T19:44:59.708+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:00.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:44:59 smithi162 bash[45241]: debug 2024-09-20T19:44:59.708+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:00.485 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[33802]: debug 2024-09-20T19:45:00.204+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:16.565975+0000 front 2024-09-20T19:44:16.566178+0000 (oldest deadline 2024-09-20T19:44:40.665928+0000) 2024-09-20T19:45:00.485 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[33802]: debug 2024-09-20T19:45:00.204+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.967467+0000 front 2024-09-20T19:44:25.967687+0000 (oldest deadline 2024-09-20T19:44:47.667231+0000) 2024-09-20T19:45:00.485 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[33802]: debug 2024-09-20T19:45:00.204+0000 7fedc3bb7640 -1 osd.5 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:33.469269+0000 front 2024-09-20T19:44:33.469499+0000 (oldest deadline 2024-09-20T19:44:58.769198+0000) 2024-09-20T19:45:00.736 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[39432]: debug 2024-09-20T19:45:00.480+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:00.736 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[39432]: debug 2024-09-20T19:45:00.480+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:00.736 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[39432]: debug 2024-09-20T19:45:00.480+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:01.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[45241]: debug 2024-09-20T19:45:00.732+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:01.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[45241]: debug 2024-09-20T19:45:00.732+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:01.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:00 smithi162 bash[45241]: debug 2024-09-20T19:45:00.732+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:01.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[39432]: debug 2024-09-20T19:45:01.532+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:01.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[39432]: debug 2024-09-20T19:45:01.532+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:01.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[39432]: debug 2024-09-20T19:45:01.532+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:01.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[45241]: debug 2024-09-20T19:45:01.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:01.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[45241]: debug 2024-09-20T19:45:01.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:01.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:01 smithi162 bash[45241]: debug 2024-09-20T19:45:01.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:02.472 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[106329]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-5 2024-09-20T19:45:02.472 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[106396]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-5 2024-09-20T19:45:02.781 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[39432]: debug 2024-09-20T19:45:02.496+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:02.781 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[39432]: debug 2024-09-20T19:45:02.496+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:02.781 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[39432]: debug 2024-09-20T19:45:02.496+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:03.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[45241]: debug 2024-09-20T19:45:02.776+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:03.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[45241]: debug 2024-09-20T19:45:02.776+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:03.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:02 smithi162 bash[45241]: debug 2024-09-20T19:45:02.776+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:03.771 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[39432]: debug 2024-09-20T19:45:03.476+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:03.771 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[39432]: debug 2024-09-20T19:45:03.476+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:03.771 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[39432]: debug 2024-09-20T19:45:03.476+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:04.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[45241]: debug 2024-09-20T19:45:03.768+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:04.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[45241]: debug 2024-09-20T19:45:03.768+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:04.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:03 smithi162 bash[45241]: debug 2024-09-20T19:45:03.768+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:04.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.5.service' 2024-09-20T19:45:04.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:45:04.538 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-20T19:45:04.538 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-20T19:45:04.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.6 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[45241]: debug 2024-09-20T19:45:04.724+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[45241]: debug 2024-09-20T19:45:04.724+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[45241]: debug 2024-09-20T19:45:04.724+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.516+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.516+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.516+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 systemd[1]: Stopping Ceph osd.6 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:04.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.700+0000 7faeaa5a7640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:45:04.810 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.700+0000 7faeaa5a7640 -1 osd.6 1103 *** Got signal Terminated *** 2024-09-20T19:45:04.810 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:04 smithi162 bash[39432]: debug 2024-09-20T19:45:04.700+0000 7faeaa5a7640 -1 osd.6 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:45:05.731 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[39432]: debug 2024-09-20T19:45:05.484+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:05.731 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[39432]: debug 2024-09-20T19:45:05.484+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:05.731 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[39432]: debug 2024-09-20T19:45:05.484+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.411Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.411Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.411Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.414Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.414Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.415Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:05.732 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[51898]: ts=2024-09-20T19:45:05.415Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.42:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.42:8765: connect: connection refused" 2024-09-20T19:45:06.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[45241]: debug 2024-09-20T19:45:05.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:06.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[45241]: debug 2024-09-20T19:45:05.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:06.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:05 smithi162 bash[45241]: debug 2024-09-20T19:45:05.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:06.808 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[45241]: debug 2024-09-20T19:45:06.704+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:06.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[45241]: debug 2024-09-20T19:45:06.704+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:06.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[45241]: debug 2024-09-20T19:45:06.704+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:06.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[39432]: debug 2024-09-20T19:45:06.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:06.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[39432]: debug 2024-09-20T19:45:06.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:06.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:06 smithi162 bash[39432]: debug 2024-09-20T19:45:06.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:07.734 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[39432]: debug 2024-09-20T19:45:07.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:07.734 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[39432]: debug 2024-09-20T19:45:07.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:07.734 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[39432]: debug 2024-09-20T19:45:07.452+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:08.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[45241]: debug 2024-09-20T19:45:07.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:08.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[45241]: debug 2024-09-20T19:45:07.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:08.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[45241]: debug 2024-09-20T19:45:07.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:08.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:07 smithi162 bash[45241]: debug 2024-09-20T19:45:07.728+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:08.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[39432]: debug 2024-09-20T19:45:08.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[39432]: debug 2024-09-20T19:45:08.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[39432]: debug 2024-09-20T19:45:08.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[39432]: debug 2024-09-20T19:45:08.492+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:42.961298+0000 front 2024-09-20T19:44:42.961475+0000 (oldest deadline 2024-09-20T19:45:07.661052+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[45241]: debug 2024-09-20T19:45:08.700+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[45241]: debug 2024-09-20T19:45:08.700+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[45241]: debug 2024-09-20T19:45:08.700+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:08.809 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:08 smithi162 bash[45241]: debug 2024-09-20T19:45:08.700+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:09.723 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[39432]: debug 2024-09-20T19:45:09.455+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:13.156579+0000 front 2024-09-20T19:44:13.156940+0000 (oldest deadline 2024-09-20T19:44:38.456283+0000) 2024-09-20T19:45:09.723 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[39432]: debug 2024-09-20T19:45:09.455+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:25.458412+0000 front 2024-09-20T19:44:25.458524+0000 (oldest deadline 2024-09-20T19:44:50.658581+0000) 2024-09-20T19:45:09.723 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[39432]: debug 2024-09-20T19:45:09.455+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:32.360338+0000 front 2024-09-20T19:44:32.360240+0000 (oldest deadline 2024-09-20T19:44:58.259918+0000) 2024-09-20T19:45:09.723 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[39432]: debug 2024-09-20T19:45:09.455+0000 7faea6bbc640 -1 osd.6 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:42.961298+0000 front 2024-09-20T19:44:42.961475+0000 (oldest deadline 2024-09-20T19:45:07.661052+0000) 2024-09-20T19:45:10.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[45241]: debug 2024-09-20T19:45:09.719+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:10.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[45241]: debug 2024-09-20T19:45:09.719+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:10.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[45241]: debug 2024-09-20T19:45:09.719+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:10.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:09 smithi162 bash[45241]: debug 2024-09-20T19:45:09.719+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:10.953 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:10 smithi162 bash[45241]: debug 2024-09-20T19:45:10.675+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:10.953 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:10 smithi162 bash[45241]: debug 2024-09-20T19:45:10.675+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:10.953 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:10 smithi162 bash[45241]: debug 2024-09-20T19:45:10.675+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:10.953 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:10 smithi162 bash[45241]: debug 2024-09-20T19:45:10.675+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:11.256 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:10 smithi162 bash[106524]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-6 2024-09-20T19:45:11.256 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 20 19:45:11 smithi162 bash[106590]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-6 2024-09-20T19:45:12.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:11 smithi162 bash[45241]: debug 2024-09-20T19:45:11.723+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:12.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:11 smithi162 bash[45241]: debug 2024-09-20T19:45:11.723+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:12.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:11 smithi162 bash[45241]: debug 2024-09-20T19:45:11.723+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:12.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:11 smithi162 bash[45241]: debug 2024-09-20T19:45:11.723+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:13.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:12 smithi162 bash[45241]: debug 2024-09-20T19:45:12.735+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:13.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:12 smithi162 bash[45241]: debug 2024-09-20T19:45:12.735+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:13.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:12 smithi162 bash[45241]: debug 2024-09-20T19:45:12.735+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:13.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:12 smithi162 bash[45241]: debug 2024-09-20T19:45:12.735+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:13.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.6.service' 2024-09-20T19:45:13.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:45:13.164 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-20T19:45:13.164 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-20T19:45:13.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.7 2024-09-20T19:45:13.559 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 systemd[1]: Stopping Ceph osd.7 for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:13.559 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.335+0000 7f85daeda640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:45:13.559 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.335+0000 7f85daeda640 -1 osd.7 1103 *** Got signal Terminated *** 2024-09-20T19:45:13.559 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.335+0000 7f85daeda640 -1 osd.7 1103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:45:14.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.767+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:14.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.767+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:14.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.767+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:14.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.767+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:14.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:13 smithi162 bash[45241]: debug 2024-09-20T19:45:13.767+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-20T19:44:50.981523+0000 front 2024-09-20T19:44:50.981634+0000 (oldest deadline 2024-09-20T19:45:13.281319+0000) 2024-09-20T19:45:15.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:14 smithi162 bash[45241]: debug 2024-09-20T19:45:14.783+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:15.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:14 smithi162 bash[45241]: debug 2024-09-20T19:45:14.783+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:15.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:14 smithi162 bash[45241]: debug 2024-09-20T19:45:14.783+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:15.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:14 smithi162 bash[45241]: debug 2024-09-20T19:45:14.783+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:15.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:14 smithi162 bash[45241]: debug 2024-09-20T19:45:14.783+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-20T19:44:50.981523+0000 front 2024-09-20T19:44:50.981634+0000 (oldest deadline 2024-09-20T19:45:13.281319+0000) 2024-09-20T19:45:16.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:15 smithi162 bash[45241]: debug 2024-09-20T19:45:15.763+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:16.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:15 smithi162 bash[45241]: debug 2024-09-20T19:45:15.763+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:16.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:15 smithi162 bash[45241]: debug 2024-09-20T19:45:15.763+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:16.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:15 smithi162 bash[45241]: debug 2024-09-20T19:45:15.763+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:16.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:15 smithi162 bash[45241]: debug 2024-09-20T19:45:15.763+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-20T19:44:50.981523+0000 front 2024-09-20T19:44:50.981634+0000 (oldest deadline 2024-09-20T19:45:13.281319+0000) 2024-09-20T19:45:17.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:16 smithi162 bash[45241]: debug 2024-09-20T19:45:16.751+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:17.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:16 smithi162 bash[45241]: debug 2024-09-20T19:45:16.751+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:17.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:16 smithi162 bash[45241]: debug 2024-09-20T19:45:16.751+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:17.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:16 smithi162 bash[45241]: debug 2024-09-20T19:45:16.751+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:17.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:16 smithi162 bash[45241]: debug 2024-09-20T19:45:16.751+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-20T19:44:50.981523+0000 front 2024-09-20T19:44:50.981634+0000 (oldest deadline 2024-09-20T19:45:13.281319+0000) 2024-09-20T19:45:18.058 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:17 smithi162 bash[45241]: debug 2024-09-20T19:45:17.731+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6803 osd.0 since back 2024-09-20T19:44:15.773135+0000 front 2024-09-20T19:44:15.773262+0000 (oldest deadline 2024-09-20T19:44:39.272627+0000) 2024-09-20T19:45:18.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:17 smithi162 bash[45241]: debug 2024-09-20T19:45:17.731+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6807 osd.1 since back 2024-09-20T19:44:24.975793+0000 front 2024-09-20T19:44:24.975768+0000 (oldest deadline 2024-09-20T19:44:47.776094+0000) 2024-09-20T19:45:18.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:17 smithi162 bash[45241]: debug 2024-09-20T19:45:17.731+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6811 osd.2 since back 2024-09-20T19:44:31.079352+0000 front 2024-09-20T19:44:31.079234+0000 (oldest deadline 2024-09-20T19:44:56.978730+0000) 2024-09-20T19:45:18.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:17 smithi162 bash[45241]: debug 2024-09-20T19:45:17.731+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.42:6815 osd.3 since back 2024-09-20T19:44:38.679811+0000 front 2024-09-20T19:44:38.679873+0000 (oldest deadline 2024-09-20T19:45:06.880302+0000) 2024-09-20T19:45:18.059 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:17 smithi162 bash[45241]: debug 2024-09-20T19:45:17.731+0000 7f85d74ef640 -1 osd.7 1103 heartbeat_check: no reply from 172.21.15.162:6802 osd.4 since back 2024-09-20T19:44:50.981523+0000 front 2024-09-20T19:44:50.981634+0000 (oldest deadline 2024-09-20T19:45:13.281319+0000) 2024-09-20T19:45:19.786 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:19 smithi162 bash[106723]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-7 2024-09-20T19:45:19.787 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 20 19:45:19 smithi162 bash[106788]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-osd-7 2024-09-20T19:45:21.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@osd.7.service' 2024-09-20T19:45:21.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:45:21.654 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-20T19:45:21.654 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-20T19:45:21.654 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@rgw.foo.a 2024-09-20T19:45:21.931 INFO:journalctl@ceph.rgw.foo.a.smithi042.stdout:Sep 20 19:45:21 smithi042 systemd[1]: Stopping Ceph rgw.foo.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:21.931 INFO:journalctl@ceph.rgw.foo.a.smithi042.stdout:Sep 20 19:45:21 smithi042 bash[55714]: debug 2024-09-20T19:45:21.794+0000 7fb0347f3640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-20T19:45:21.931 INFO:journalctl@ceph.rgw.foo.a.smithi042.stdout:Sep 20 19:45:21 smithi042 bash[55714]: debug 2024-09-20T19:45:21.794+0000 7fb03805f880 -1 shutting down 2024-09-20T19:45:32.621 DEBUG:teuthology.orchestra.run.smithi042:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@rgw.foo.a.service' 2024-09-20T19:45:32.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:45:32.645 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-20T19:45:32.645 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-20T19:45:32.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-49cb5aae-7783-11ef-baf5-efdc52797490@prometheus.a 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 systemd[1]: Stopping Ceph prometheus.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:45:33.027 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.795Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.798Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.798Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-20T19:45:33.028 INFO:journalctl@ceph.prometheus.a.smithi162.stdout:Sep 20 19:45:32 smithi162 bash[51898]: ts=2024-09-20T19:45:32.798Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-20T19:45:33.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-49cb5aae-7783-11ef-baf5-efdc52797490@prometheus.a.service' 2024-09-20T19:45:33.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T19:45:33.566 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-20T19:45:33.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T19:45:33.760 INFO:teuthology.orchestra.run.smithi042.stdout:Deleting cluster with fsid: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:45:38.609 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:38.609 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:38.863 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:38.863 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.180 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.181 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:38 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.504 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.504 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.504 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.504 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.797 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:39.797 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:40.143 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:40.143 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:39 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:40.415 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:40 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:40.415 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:40 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:40.415 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:40 smithi042 systemd[1]: Stopping Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:40.415 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:40 smithi042 bash[62791]: ts=2024-09-20T19:45:40.371Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:45:41.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:41 smithi042 bash[167272]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:45:41.430 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:41 smithi042 bash[167361]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:45:41.431 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:41 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T19:45:41.431 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:41 smithi042 systemd[1]: Stopped Ceph alertmanager.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:45:41.751 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:41 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:41.751 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:41 smithi042 systemd[1]: Stopping Ceph node-exporter.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:41.751 INFO:journalctl@ceph.alertmanager.a.smithi042.stdout:Sep 20 19:45:41 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:42.648 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 bash[167428]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T19:45:42.648 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:45:42.648 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 bash[167488]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T19:45:42.648 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-20T19:45:42.648 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 systemd[1]: Stopped Ceph node-exporter.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:45:42.914 INFO:journalctl@ceph.node-exporter.a.smithi042.stdout:Sep 20 19:45:42 smithi042 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:43.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T19:45:43.363 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:45:48.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.251 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.251 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.558 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.559 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.850 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.851 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.851 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:48.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.100 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:48 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.352 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.353 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.808 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.809 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.809 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:45:49.809 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:49 smithi162 systemd[1]: Stopping Ceph iscsi.iscsi.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:45:49.809 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:45:49 smithi162 bash[48936]: debug Shutdown received 2024-09-20T19:46:00.308 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 bash[107252]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-iscsi-iscsi-a 2024-09-20T19:46:00.622 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:00.622 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:00.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 bash[107318]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-iscsi-iscsi-a 2024-09-20T19:46:00.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-20T19:46:00.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-20T19:46:00.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: Stopped Ceph iscsi.iscsi.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:46:00.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi162.stdout:Sep 20 19:46:00 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:01.808 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:01 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:01.808 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:01 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:01.808 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:01 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:01.809 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:01 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:02.186 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:01 smithi162 systemd[1]: Stopping Ceph node-exporter.b for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:46:02.888 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 bash[107550]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T19:46:02.888 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:46:02.888 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 bash[107640]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T19:46:02.888 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-20T19:46:02.888 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: Stopped Ceph node-exporter.b for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:46:03.154 INFO:journalctl@ceph.node-exporter.b.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:03.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:03.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:02 smithi162 systemd[1]: Stopping Ceph grafana.a for 49cb5aae-7783-11ef-baf5-efdc52797490... 2024-09-20T19:46:03.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[51183]: logger=server t=2024-09-20T19:46:03.118893227Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-20T19:46:03.154 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[51183]: logger=ticker t=2024-09-20T19:46:03.119038527Z level=info msg=stopped last_tick=2024-09-20T19:46:00Z 2024-09-20T19:46:03.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[51183]: logger=tracing t=2024-09-20T19:46:03.119041013Z level=info msg="Closing tracing" 2024-09-20T19:46:03.155 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[51183]: logger=grafana-apiserver t=2024-09-20T19:46:03.119051072Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-20T19:46:04.035 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[107709]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-grafana-a 2024-09-20T19:46:04.036 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 bash[107771]: Error response from daemon: No such container: ceph-49cb5aae-7783-11ef-baf5-efdc52797490-grafana-a 2024-09-20T19:46:04.036 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 systemd[1]: ceph-49cb5aae-7783-11ef-baf5-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-20T19:46:04.036 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:03 smithi162 systemd[1]: Stopped Ceph grafana.a for 49cb5aae-7783-11ef-baf5-efdc52797490. 2024-09-20T19:46:04.310 INFO:journalctl@ceph.grafana.a.smithi162.stdout:Sep 20 19:46:04 smithi162 systemd[1]: /etc/systemd/system/ceph-49cb5aae-7783-11ef-baf5-efdc52797490@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-20T19:46:04.581 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:46:04.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:46:04.609 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-20T19:46:04.610 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi042/crash 2024-09-20T19:46:04.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash -- . 2024-09-20T19:46:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:tar: /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T19:46:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:tar: Error is not recoverable: exiting now 2024-09-20T19:46:04.653 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi162/crash 2024-09-20T19:46:04.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash -- . 2024-09-20T19:46:04.664 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T19:46:04.665 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-09-20T19:46:04.666 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-20T19:46:04.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-20T19:46:04.710 INFO:tasks.cephadm:Compressing logs... 2024-09-20T19:46:04.710 DEBUG:teuthology.orchestra.run.smithi042:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T19:46:04.756 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T19:46:04.765 INFO:teuthology.orchestra.run.smithi042.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-20T19:46:04.769 INFO:teuthology.orchestra.run.smithi162.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-20T19:46:04.775 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T19:46:04.776 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-20T19:46:04.776 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-20T19:46:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-20T19:46:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-20T19:46:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.6.log 2024-09-20T19:46:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T19:46:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.a.log: /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-20T19:46:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-20T19:46:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log: /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.3.log: 88.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T19:46:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log 2024-09-20T19:46:04.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.1.log: 94.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.6.log.gz 2024-09-20T19:46:04.783 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log 2024-09-20T19:46:04.784 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T19:46:04.784 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-20T19:46:04.784 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-20T19:46:04.785 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T19:46:04.785 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-20T19:46:04.786 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log 2024-09-20T19:46:04.786 INFO:teuthology.orchestra.run.smithi162.stderr: 79.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T19:46:04.786 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/tcmu-runner.log 2024-09-20T19:46:04.788 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-20T19:46:04.788 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log: /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/tcmu-runner.log: gzip -5 63.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/tcmu-runner.log.gz 2024-09-20T19:46:04.788 INFO:teuthology.orchestra.run.smithi162.stderr: --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log 2024-09-20T19:46:04.788 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T19:46:04.788 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T19:46:04.789 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log: 94.8% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log.gz 2024-09-20T19:46:04.789 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-20T19:46:04.792 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T19:46:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T19:46:04.795 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log: 90.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T19:46:04.796 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-20T19:46:04.800 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.b.log: 91.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-20T19:46:04.802 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.4.log: 86.4% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T19:46:04.805 INFO:teuthology.orchestra.run.smithi162.stderr: 91.5% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T19:46:04.806 INFO:teuthology.orchestra.run.smithi162.stderr: 91.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T19:46:04.807 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-20T19:46:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log: 95.3% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/objectstore_tool.7.log.gz 2024-09-20T19:46:04.819 INFO:teuthology.orchestra.run.smithi042.stderr: 92.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T19:46:04.820 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T19:46:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log 2024-09-20T19:46:04.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log: 94.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T19:46:04.839 INFO:teuthology.orchestra.run.smithi042.stderr: 91.7% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T19:46:04.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log: 59.0% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log.gz 2024-09-20T19:46:05.223 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-20T19:46:05.477 INFO:teuthology.orchestra.run.smithi162.stderr: 91.2% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-20T19:46:05.828 INFO:teuthology.orchestra.run.smithi042.stderr: 91.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-20T19:46:06.837 INFO:teuthology.orchestra.run.smithi042.stderr: 90.1% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-20T19:46:09.454 INFO:teuthology.orchestra.run.smithi042.stderr: 94.0% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-20T19:46:09.751 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-20T19:46:09.949 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-20T19:46:10.352 INFO:teuthology.orchestra.run.smithi042.stderr: 93.8% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-20T19:46:10.421 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-20T19:46:10.709 INFO:teuthology.orchestra.run.smithi162.stderr: 94.0% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-20T19:46:10.710 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-09-20T19:46:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m5.949s 2024-09-20T19:46:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m21.623s 2024-09-20T19:46:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.836s 2024-09-20T19:46:10.719 INFO:teuthology.orchestra.run.smithi042.stderr: 93.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-20T19:46:11.138 INFO:teuthology.orchestra.run.smithi042.stderr: 93.9% -- replaced with /var/log/ceph/49cb5aae-7783-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-20T19:46:11.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-20T19:46:11.139 INFO:teuthology.orchestra.run.smithi042.stderr:real 0m6.381s 2024-09-20T19:46:11.139 INFO:teuthology.orchestra.run.smithi042.stderr:user 0m25.239s 2024-09-20T19:46:11.139 INFO:teuthology.orchestra.run.smithi042.stderr:sys 0m0.951s 2024-09-20T19:46:11.140 INFO:tasks.cephadm:Archiving logs... 2024-09-20T19:46:11.140 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi042/log 2024-09-20T19:46:11.141 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T19:46:13.932 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi162/log 2024-09-20T19:46:13.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T19:46:16.429 INFO:tasks.cephadm:Removing cluster... 2024-09-20T19:46:16.429 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 --force 2024-09-20T19:46:16.621 INFO:teuthology.orchestra.run.smithi042.stdout:Deleting cluster with fsid: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:46:17.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 49cb5aae-7783-11ef-baf5-efdc52797490 --force 2024-09-20T19:46:18.182 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 49cb5aae-7783-11ef-baf5-efdc52797490 2024-09-20T19:46:19.544 INFO:tasks.cephadm:Removing cephadm ... 2024-09-20T19:46:19.545 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T19:46:19.551 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T19:46:19.557 INFO:tasks.cephadm:Teardown complete 2024-09-20T19:46:19.557 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-20T19:46:19.565 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-20T19:46:19.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-20T19:46:19.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-20T19:46:19.659 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-20T19:46:19.667 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-20T19:46:19.667 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:46:19.670 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:46:19.679 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-09-20T19:46:19.680 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-09-20T19:46:19.680 INFO:teuthology.orchestra.run.smithi042.stdout:*hv01.front.sepi 67.205.162.81 3 u 81 128 377 0.168 -0.203 0.117 2024-09-20T19:46:19.680 INFO:teuthology.orchestra.run.smithi042.stdout:+hv02.front.sepi 96.245.170.99 3 u 130 128 377 0.138 -0.495 0.200 2024-09-20T19:46:19.680 INFO:teuthology.orchestra.run.smithi042.stdout:+hv03.front.sepi 74.6.168.72 3 u 117 128 377 0.133 +2.138 0.078 2024-09-20T19:46:19.680 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout:*hv01.front.sepi 67.205.162.81 3 u 11 64 377 0.046 -0.279 1.016 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout:+hv02.front.sepi 96.245.170.99 3 u 7 64 377 0.096 -0.558 0.913 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 377 0.092 +2.141 1.122 2024-09-20T19:46:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-20T19:46:19.685 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-20T19:46:19.694 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-20T19:46:19.694 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-20T19:46:19.780 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-20T19:46:19.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-20T19:46:19.801 INFO:teuthology.task.internal:Duration was 3028.954534 seconds 2024-09-20T19:46:19.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-20T19:46:19.812 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-20T19:46:19.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T19:46:19.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T19:46:19.852 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-20T19:46:19.852 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-09-20T19:46:19.853 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T19:46:19.908 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-20T19:46:19.908 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T19:46:19.918 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-20T19:46:19.918 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T19:46:19.952 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T19:46:20.086 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-20T19:46:20.087 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T19:46:20.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T19:46:20.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-20T19:46:20.619 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-20T19:46:20.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T19:46:20.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T19:46:20.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-20T19:46:20.659 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:46:20.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:46:20.684 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-09-20T19:46:20.700 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-20T19:46:20.725 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:46:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:46:20.759 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T19:46:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:46:20.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-20T19:46:20.783 INFO:teuthology.task.internal:Transferring archived files... 2024-09-20T19:46:20.784 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi042 2024-09-20T19:46:20.784 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T19:46:20.841 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-20_12:24:58-orch:cephadm-wip-guits-main-2024-09-20-1028-distro-default-smithi/7912744/remote/smithi162 2024-09-20T19:46:20.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T19:46:20.872 INFO:teuthology.task.internal:Removing archive directory... 2024-09-20T19:46:20.872 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T19:46:20.876 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T19:46:20.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-20T19:46:20.930 INFO:teuthology.task.internal:Not uploading archives. 2024-09-20T19:46:20.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-20T19:46:20.950 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-20T19:46:20.950 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T19:46:20.953 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T19:46:20.956 INFO:teuthology.orchestra.run.smithi042.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 19:46 /home/ubuntu/cephtest 2024-09-20T19:46:20.962 INFO:teuthology.orchestra.run.smithi162.stdout: 658670 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 19:46 /home/ubuntu/cephtest 2024-09-20T19:46:20.964 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-20T19:46:20.979 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-20T19:46:21.048 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} duration: 3028.954534292221 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-20T19:46:21.048 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T19:46:21.137 INFO:teuthology.run:pass